scrapy-cluster:该Scrapy项目使用Redis和Kafka创建按需分布式抓取集群
需积分: 50 27 浏览量
2021-05-01
08:05:24
上传
评论
收藏 2.86MB ZIP 举报
scrapy-cluster:该Scrapy项目使用Redis和Kafka创建按需分布式抓取集群 （211个子文件） 
make.bat 7KB scrapy.cfg 68B scrapy-cluster-logstash.conf 432B scrapy-cluster-logstash-docker.conf 430B .coveragerc 77B .coveragerc 58B .coveragerc 43B .coveragerc 42B .coveragerc 26B .gitignore 695B
drudge.html 45KB
BreadthFirst.jpg 31KB export.json 36KB scraper_schema.json 2KB zookeeper_schema.json 1019B logs-template.json 822B action_schema.json 736B stats_schema.json 716B poll.json 301B LICENSE 1KB Makefile 7KB README.md 5KB README.md 509B README.md 490B README.md 277B
redismonitor_kibana.png 369KB crawler_kibana.png 340KB kafkamonitor_kibana.png 324KB overall_kibana.png 270KB ArchitectureOverview.png 256KB rest_kibana.png 255KB Throttle4.png 181KB RedisQueues.png 179KB Throttle3.png 152KB Throttle2.png 149KB Throttle1.png 127KB RedisPlugins.png 81KB KafkaPlugins.png 74KB scrapy-cluster-logo.png 71KB RestService.png 37KB rest_service.py 29KB test_rest_service.py 27KB distributed_scheduler.py 25KB kafka_monitor.py 23KB stats_collector.py 21KB redis_monitor.py 21KB test_distributed_scheduler.py 19KB test_plugins.py 18KB online.py 16KB zookeeper_watcher.py 14KB log_factory.py 11KB test_redis_monitor.py 11KB migrate.py 11KB test_log_factory.py 11KB conf.py 9KB stats_monitor.py 9KB test_kafka_monitor.py 9KB pipelines.py 8KB redis_throttled_queue.py 8KB kafkadump.py 8KB test_pipelines.py 8KB info_monitor.py 8KB settings.py 7KB log_retry_middleware.py 7KB redis_stats_middleware.py 6KB settings.py 6KB test_redis_stats_middleware.py 5KB online.py 5KB redis_queue.py 4KB test_plugins.py 4KB online.py 4KB zookeeper_monitor.py 4KB stop_monitor.py 4KB kafka_base_monitor.py 4KB settings_wrapper.py 3KB test_redis_queue.py 3KB link_spider.py 3KB wandering_spider.py 3KB test_zookeeper_watcher.py 3KB redis_global_page_per_domain_filter.py 3KB example_rtq.py 3KB lxmlhtml.py 3KB test_link_spider.py 3KB example_zw.py 3KB base_monitor.py 3KB online.py 3KB throttled_queue.py 3KB test_redis_throttled_queue.py 3KB redis_domain_max_page_filter.py 2KB example_sc.py 2KB test_stats_collector.py 2KB test_argparse_helper.py 2KB settings.py 2KB scraper_handler.py 2KB expire_monitor.py 2KB zookeeper_handler.py 2KB test_wandering_spider.py 2KB settings.py 2KB meta_passthrough_middleware.py 2KB test_settings_wrapper.py 2KB共 211 条
- 1
- 2
- 3
资源评论
