cocrawler:CoCrawler是使用现代工具和并发性构建的多功能Web搜寻器
需积分: 9 74 浏览量
2021-05-01
14:10:21
上传
评论
收藏 129KB ZIP 举报
cocrawler-master.zip （91个子文件） 
cocrawler-master 
.travis.yml 949B data 
top-1k.txt 12KB
html-parsing-test.html 8KB tests test-wide.yml 1000B .cocrawler-config.yml 146B test-scheduler.yml 1KB test-failures.yml 783B test-wide-save.yml 48B test.sh 4KB unit test_reppy.py 2KB test_facet.py 12KB test_accumulator.py 2KB test_dns.py 786B test_cocrawler.py 1KB test_timebin.py 967B test_parse.py 7KB test_useragent.py 2KB test_config.py 896B test_surt.py 9KB test_lxml.py 1KB test_seeds.py 309B test_robots.py 2KB test_stats.py 3KB test_datalayer.py 2KB test_urls.py 13KB test_burner.py 361B test_url_allowed.py 3KB test_topk.py 2KB test_00version.py 696B test-deep.yml 1KB test_burner.html 178B test-wide.hosts 587B TEST-TODO 231B warc CC-TEST-01-FOO-00002-hostname.warc.in 2KB CC-TEST-01-00001-hostname.warc.gz.in 1KB test.sh 1KB test-warc.py 2KB warc-diff.py 976B CC-TEST-01-FOO-00001-hostname.warc.in 2KB mock-webserver.py 5KB LICENSE 11KB azure-pipelines.yml 2KB cocrawler geoip.py 4KB webserver.py 2KB .coveragerc 92B dashboard.json 2KB timebin.py 2KB warc.py 10KB .pylintrc 395B datalayer.py 2KB robots.py 13KB content.py 5KB surt.py 8KB stats.py 11KB post_fetch.py 10KB timer.py 6KB fetcher.py 10KB dns.py 6KB pdeathsig.py 307B __init__.py 22KB url_allowed.py 4KB parse.py 14KB topk.py 3KB scheduler.py 11KB burner.py 4KB memory.py 5KB accumulator.py 2KB config.py 6KB seeds.py 6KB urls.py 15KB useragent.py 2KB facet.py 10KB requirements.txt 889B examples USECASES 1KB setup.py 2KB .gitignore 78B optional-requirements.txt 132B Makefile 2KB README.md 2KB scripts dump-soup.py 2KB run_burner_bench.py 2KB parse-html.py 422B bench_dns.py 4KB run_parsers.py 3KB bench_burner.py 3KB crawl.py 4KB run_burner.py 2KB aiohttp-fetch.py 2KB cocrawler-savefile-dump.py 294B .editorconfig 256B .flake8 29B资源评论
摔了个呆萌
- 粉丝: 28
- 资源: 4676
