robots.txt：robots.txt即服务。 抓取robots.txt文件，下载并解析它们以通过API检查规则
需积分: 18 85 浏览量
2021-02-04
03:19:01
上传
评论
收藏 2MB ZIP 举报
robots.txt：robots.txt即服务。 抓取robots.txt文件，下载并解析它们以通过API检查规则 （149个子文件） 
gradlew.bat 2KB Dockerfile 347B Dockerfile 343B Dockerfile 333B .dockerignore 53B .dockerignore 53B .dockerignore 53B secrets.dev.yml.encrypted 64B .env 16B CommonLexerRules.g4 2KB RobotsTxt.g4 724B .gitattributes 18B .gitignore 596B build.gradle 1KB build.gradle 1KB build.gradle 1KB build.gradle 684B build.gradle 406B settings.gradle 136B gradlew 5KB
antlr-4.7.2.jar 1.98MB gradle-wrapper.jar 55KB RuleMatcherTest.java 12KB RuleMatcher.java 6KB HttpRequestTest.java 4KB CrawlerApiService.java 4KB IncomingUrlService.java 4KB ApiController.java 4KB ApiService.java 4KB FileDownloaderService.java 3KB IncomingUrlServiceTest.java 3KB DefaultParserResultListener.java 3KB CrawlApiController.java 3KB IncomeStreamService.java 2KB UpdateStreamService.java 2KB ResponseErrorInterceptor.java 2KB AppConfiguration.java 2KB UrlValidator.java 1KB UpdateUrlService.java 1KB CacheConfiguration.java 1KB Rule.java 996B AsyncConfiguration.java 974B DomainUtilsTest.java 971B LoggingInterceptor.java 738B TestUtils.java 706B AllowData.java 702B AppConfiguration.java 603B UrlValidatorTest.java 596B CrawlerNameHeaderInterceptor.java 595B AllowDataTest.java 570B DomainUtils.java 557B UrlUtils.java 545B UrlUtilsTest.java 542B AppConfiguration.java 534B ParserResultListener.java 459B DownloaderApplicationTests.java 404B CrawlApiApplicationTests.java 400B ApiApplicationTests.java 390B DownloaderApplication.java 353B CrawlApiApplication.java 347B ApiApplication.java 332B AllowResponse.java 298B Group.java 268B Content.java 255B ResultType.java 137B comparison-violates-contract.json 6KB robots.txt.postman_collection.json 4KB custom-1.json 4KB group-multi-agent-and-rule.json 614B similar-agents-robots.json 576B group-multi-rule.json 479B dollar-pattern.json 436B wildcard-disallow-all.json 335B same-pattern-rules.json 274B package.json 257B same-default-rules.json 254B empty-default-agent.json 202B only-default-agent.json 195B group-one-disallow-rule.json 194B group-one-empty-rule.json 193B group-one-allow-rule.json 191B group-without-rules.json 114B empty-group.json 19B BaseParserTest.kt 11KB GroupListener.kt 3KB JsonSerializerTest.kt 2KB Entry.kt 2KB DefaultRobotParser.kt 2KB RedisUrlRepository.kt 1KB DataSourceConfiguration.kt 1KB RobotsRepository.kt 1KB Group.kt 892B ResultToEntryMapperTest.kt 837B ClusterService.kt 565B GenerateSerializerJson.kt 548B Result.kt 548B UrlRecord.kt 518B ResultToEntryMapper.kt 514B DisallowAllGroupTest.kt 514B AllowAllGroupTest.kt 502B共 149 条
- 1
- 2
资源评论
