Nutch入门教程

Nutch入门教程 目 录 1. nutch简介.................................................................................................................................1 1.1什么是nutch.......................................................................................................................1 1.2研究nutch的原因...............................................................................................................1 1.3 nutch的目标.......................................................................................................................1 1.4 nutch VS lucene.................................................................................................................2 2. nutch的安装与配置.................................................................................................................3 2.1 JDK的安装与配置.............................................................................................................3 2.2 nutch的安装与配置...........................................................................................................5 2.3 tomcat的安装与配置.........................................................................................................5 3. nutch初体验..............................................................................................................................7 3.1 爬行企业内部网................................................................................................................7 3.1.1 配置nutch................................................................................................................7 3.1.2 配置tomcat..............................................................................................................8 3.1.3 执行抓取命令.........................................................................................................9 3.1.4 测试结果...............................................................................................................11 3.1.5 Intranet Recrawl.....................................................................................................13 3.2 爬行全网..........................................................................................................................18 3.2.1 nutch数据集的基本组成:......................................................................................18 3.2.2 爬行＂官方＂网址...............................................................................................18 3.2.3 爬行中文网址.......................................................................................................22 4. nutch基本原理分析...............................................................................................................23 4.1 nutch的基本组成.............................................................................................................23 4.2 nutch工作流程.................................................................................................................23 5. nutch工作流程分析...............................................................................................................25 5.1 爬虫.................................................................................................................................25 5.1.1 工作策略...............................................................................................................25 5.1.2 工作流程分析.......................................................................................................25 5.1.3 其它.......................................................................................................................27 5.2 索引.................................................................................................................................27 5.2.1 索引主要过程.......................................................................................................27 5.2.2 工作流程分析.......................................................................................................28 5.2.3 倒排索引(inverted index).....................................................................................29 5.2.4其它........................................................................................................................29 5.3 搜索.................................................................................................................................29 5.4 分析.................................................................................................................................30 5.5 nutch的其他一些特性.....................................................................................................31 6. nutch分析方法和工具...........................................................................................................33 6.1 Crawldb...........................................................................................................................33 6.2 Linkdb.............................................................................................................................35 6.3 Segments.........................................................................................................................35 6.4 Index................................................................................................................................39 7. nutch分布式文件系统...........................................................................................................41 2007-8-26 北京邮电大学－李阳 Nutch入门学习 7.1 概述.................................................................................................................................41 7.2 MapReduce...................................................................................................................41 7.3 文件系统语法..................................................................................................................42 7.4 文件系统设计..................................................................................................................42 7.5 系统的可用性..................................................................................................................43 7.6 Nutch文件系统工作架构.............................................................................................43 8. nutch应用...............................................................................................................................45 8.1 修改源码..........................................................................................................................45 8.2 插件机制---plugin...........................................................................................................45 8.2.1 什么是plugin.........................................................................................................45 8.2.2 使用plugin的好处.................................................................................................45 8.2.3 plugin工作原理......................................................................................................46 8.2.4 编写plugin............................................................................................................47 8.3 API接口............................................................................................................................53 8.3.1使用Nutch API.......................................................................................................53 8.3.2使用OpenSearch API.............................................................................................55 8.4 nutch的应用前景.............................................................................................................57 附录一: nutch的相关网站.........................................................................................................58 附录二: 参考文献.......................................................................................................................58