nutch详细分析(包括配置文件等)资源-CSDN文库

5星 · 超过95%的资源需积分: 10 41 浏览量 2009-08-18 11:21:14 上传评论收藏 479KB DOC 举报

资源推荐

资源详情

资源评论

 分析
Nutch 简介......................................................................................................................................2
1  nutch 体系结构....................................................................................................................2
抓取部分.........................................................................................................................................3
1  爬虫的数据结构及含义......................................................................................................3
2  抓取目录分析......................................................................................................................4
3  抓取过程概述......................................................................................................................4
4  抓取过程分析......................................................................................................................5
4.1  inject 方法.................................................................................................................6
4.2  generate 方法..........................................................................................................12
4.3  fetch 方法................................................................................................................14
4.4  parse 方法................................................................................................................17
4.5  update 方法.............................................................................................................17
4.6  invert 方法...............................................................................................................20
4.7  index 方法...............................................................................................................24
4.8  dedup 方法..............................................................................................................28
4.9  merge 方法..............................................................................................................32
配置文件分析...............................................................................................................................32
1  nutch-default.xml 分析......................................................................................................33
1.1  <!-- file properties -->.............................................................................................33
1.2  <!-- HTTP properties -->.........................................................................................34
1.3  <!-- FTP properties -->............................................................................................37
1.4  <!-- web db properties -->.......................................................................................39
1.5  <!-- generate properties -->.....................................................................................43
1.6  <!-- fetcher properties -->.......................................................................................44
1.7  <!-- indexer properties -->......................................................................................45
1.8  <!-- indexingfilter plugin properties -->.................................................................47
1.9  <!-- analysis properties -->.....................................................................................47
1.10  <!-- searcher properties -->...................................................................................47
1.11  <!-- URL normalizer properties -->......................................................................50
1.12  <!-- mime properties -->.......................................................................................50
1.13  <!-- plugin properties -->......................................................................................51
1.14  <!-- parser properties -->.......................................................................................52
1.15  <!-- urlfilter plugin properties -->.........................................................................53
1.16  <!-- scoring filters properties -->..........................................................................54
1.17  <!-- clustering extension properties -->................................................................55
1.18  <!-- ontology extension properties -->..................................................................55
1.19  <!-- query-basic plugin properties -->..................................................................56
1.20  <!-- creative-commons plugin properties -->.......................................................57
1.21  <!-- query-more plugin properties -->..................................................................57
1.22  <!-- microformats-reltag plugin properties -->.....................................................58
1.23  <!-- language-identifier plugin properties -->......................................................58
1.24  <!-- Temporary Hadoop 0.17.x workaround. -->..................................................59