精选_基于Heritrix和Lucene实现的典故搜索引擎

共13441个文件

html：7654个

txt：3137个

class：742个

版权申诉

搜索引擎

lucene

全文检索

107 浏览量 2022-03-12 09:33:47 上传评论收藏 98.8MB ZIP 举报

资源推荐

资源详情

资源评论

收起资源包目录

精选_基于Heritrix和Lucene实现的典故搜索引擎_源码打包（13441个子文件）

crawl.log.00001 974KB

progress-statistics.log.00001 98KB

recover.gz.00001 95KB

uri-errors.log.00001 25KB

local-errors.log.00001 4KB

runtime-errors.log.00001 0B

ctl00_ContentPlaceHolder1_ID_Article_List.ascx_SearchTools1_txt_key 5KB

ctl00_ContentPlaceHolder1_ID_Article.ascx_SearchTools1_txt_key 5KB

ctl00_ContentPlaceHolder1_ID_index.ascx_SearchTools1_txt_key 5KB

defaultsource=liuchuanshao.asp 1B

unionadlocation=6.aspx 124B

avi.avi 81KB

heritrix.cacerts 21KB

_55.cfs 49KB

ckCode 1KB

Heritrix.class 54KB

CrawlJob.class 51KB

FetchHTTP.class 46KB

CrawlController.class 41KB

HttpMethodBase.class 34KB

WorkQueueFrontier.class 30KB

AdaptiveRevisitFrontier.class 30KB

StatisticsTracker.class 29KB

AbstractFrontier.class 29KB

CrawlJobHandler.class 27KB

CachedBdbMap.class 26KB

UURIFactoryTest.class 25KB

CrawlURI.class 23KB

ExtractorHTML.class 20KB

WriterPoolProcessor.class 20KB

ComplexType.class 19KB

WARCWriterProcessor.class 19KB

HttpConnection.class 19KB

DecideRuleSequenceTest.class 18KB

MirrorWriterProcessor.class 18KB

AdaptiveRevisitHostQueue.class 17KB

ARCReader.class 17KB

ARCWriterTest.class 17KB

StatisticsSummary.class 16KB

FetchFTP.class 16KB

JEMBeanHelper.class 15KB

XMLSettingsHandler.class 15KB

JerichoExtractorHTML.class 15KB

ExtractorHTMLTest.class 15KB

GenericObjectPool.class 15KB

ToeThread.class 14KB

JobConfigureUtils.class 14KB

BdbFrontier.class 14KB

UURIFactory.class 14KB

CrawlOrder.class 14KB

SettingsHandler.class 14KB

Kw3WriterProcessor.class 13KB

CrawlSettingsSAXHandler.class 13KB

WARCWriterTest.class 13KB

CookieSpecBase.class 13KB

HashtableAList.class 13KB

ObjectIdentityBdbCache.class 12KB

ArchiveUtils.class 12KB

BdbMultipleWorkQueues.class 12KB

PreconditionEnforcer.class 12KB

CrawlSettingsSAXSource.class 12KB

FileUtils.class 12KB

WARCWriter.class 12KB

CandidateURI.class 11KB

ArchiveReader.class 11KB

FetchDNS.class 11KB

LogReader.class 11KB

JmxUtils.class 11KB

PersistProcessor.class 10KB

ExtractorTool.class 10KB

RegexpHTMLLinkExtractor.class 10KB

AdaptiveRevisitQueueList.class 10KB

WriterPoolMember.class 10KB

SimpleHttpServer.class 10KB

WorkQueue.class 10KB

ARCWriter.class 10KB

ArchiveUtilsTest.class 10KB

RecordingOutputStream.class 10KB

ExtractorSWFTest.class 9KB

DomainSensitiveFrontier.class 9KB

HttpState.class 9KB

CrawlServer.class 9KB

ExtractorUniversal.class 9KB

LinksScoper.class 9KB

QuotaEnforcer.class 9KB

RecoveryJournal.class 9KB

XMLSettingsHandlerTest.class 9KB

ReplayCharSequenceTest.class 9KB

SelfTestCase.class 9KB

Arc2Warc.class 8KB

CrawlScope.class 8KB

JerichoExtractorHTMLTest.class 8KB

FPGenerator.class 8KB

CrawlMapper.class 8KB

AdaptiveRevisitHostQueueTest.class 8KB

MapTypeTest.class 8KB

HtmlFormCredential.class 8KB

Warc2Arc.class 8KB

WaitEvaluator.class 8KB

共 13441 条

Heritrix Release Notes

Internet Archive

Table of Contents

1. Release 1.14.4 - 2010-04-30 ......................................................................................2

2. Release 1.14.3 - 2009-03-02 ......................................................................................2

3. Release 1.14.2 - 2008-08-06 ......................................................................................2

4. Release 1.14.1 - 2008-08-06 ......................................................................................2

5. Release 1.14.0 - 2008-04-27 ......................................................................................2

6. Release 1.12.1 - 2007-05-06 ......................................................................................3

6.1. Contributors .................................................................................................3

6.2. All tracked changes .......................................................................................3

7. Release 1.12.0 - 2007-03-16 ......................................................................................3

7.1. Contributors .................................................................................................3

7.2. Notes .......................................................................................................... 3

7.3. Known Limitations/Issues ...............................................................................3

7.4. Changes ......................................................................................................4

8. Release 1.10.2 - 2007-01-15 ......................................................................................4

8.1. Contributors .................................................................................................5

8.2. Changes ......................................................................................................5

9. Release 1.10.1 - 2006-09-27 ......................................................................................8

9.1. Changes ......................................................................................................8

10. Release 1.10.0 - 2006-09-11 ..................................................................................... 8

10.1. Contributors ...............................................................................................8

10.2. Known Limitations/Issues .............................................................................9

10.3. Pre-1.10.0 checkpoints ..................................................................................9

10.4. Changes .....................................................................................................9

11. Release 1.8.0 - 2006-05-05 .................................................................................... 22

11.1. Known Limitations/Issues ........................................................................... 22

11.2. Changes ................................................................................................... 22

12. Release 1.6.0 - 2005-12-01 .................................................................................... 27

12.1. Known Limitations/Issues ........................................................................... 27

12.2. Changes ................................................................................................... 28

13. Release 1.4.0 - 2005-04-28 .................................................................................... 49

13.1. Known Limitations/Issues ........................................................................... 49

13.2. Changes ................................................................................................... 51

14. Release 1.2.0 - 2004-11-16 .................................................................................... 69

14.1. Known Limitations .................................................................................... 69

14.2. Changes ................................................................................................... 70

15. Release 1.0.4 - 2004-09-22 .................................................................................... 80

15.1. Changes ................................................................................................... 80

16. Release 1.0.2 - 2004-09-14 .................................................................................... 80

16.1. Changes ................................................................................................... 81

17. Release 1.0.0 - 2004-08-06 .................................................................................... 81

17.1. Known Limitations .................................................................................... 82

17.2. Changes ................................................................................................... 83

18. Release 0.10.0 - 2004-06-04 ................................................................................... 86

18.1. Changes ................................................................................................... 86

19. Release 0.10.0 - 2004-06-04 ................................................................................... 92

19.1. Changes ................................................................................................... 92

20. Release 0.8.0 - 2004-05-18 .................................................................................... 92

20.1. Synopsis .................................................................................................. 92

20.2. Changes ................................................................................................... 92

21. Release 0.6.0 - 2004-03-25 .................................................................................... 98

21.1. Changes ................................................................................................... 98

22. Release 0.4.1 - 2004-06-04 .................................................................................. 102

22.1. Changes ................................................................................................. 102

23. Release 0.4.0 - 2004-02-10 .................................................................................. 102

23.1. Changes ................................................................................................. 102

24. Release 0.2.0 - 2004-01-05 .................................................................................. 103

25. Release 0.1.0 - 2003-12-31 .................................................................................. 103

1. Release 1.14.4 - 2010-04-30

Release 1.14.4 is a bugfix release with 44 tracked fixes and small improvements.

Other details on this release, including a list of all tracked issues addressed, are available at: Release

Notes - Heritrix 1.14.4 at project wiki

[https://webarchive.jira.com/wiki/display/Heritrix/Release+Notes+-+1.14.4]

2. Release 1.14.3 - 2009-03-02

Release 1.14.3 is a bugfix release with 14 tracked fixes and small improvements.

Other details on this release, including a list of all tracked issues addressed, are available at: Release

Notes - Heritrix 1.14.3 at project wiki

[http://webteam.archive.org/confluence/display/Heritrix/Release+Notes+-+1.14.3]

3. Release 1.14.2 - 2008-08-06

Release 1.14.2 is a bugfix release with 6 tracked fixes.

Other details on this release, including a list of all tracked issues addressed, are available at: Release

Notes - Heritrix 1.14.2 at project wiki

[http://webteam.archive.org/confluence/display/Heritrix/Release+Notes+-+1.14.2]

4. Release 1.14.1 - 2008-08-06

Release 1.14.1 is a bugfix release with 25 tracked fixes and small requested/contributed features.

Other details on this release, including a list of all tracked issues addressed, are available at: Release

Notes - Heritrix 1.14.1 at project wiki

[http://webteam.archive.org/confluence/display/Heritrix/Release+Notes+-+1.14.1]

5. Release 1.14.0 - 2008-04-27

Release 1.14.0 adds a number of small features to the Heritrix 1.x line, most notably upgrading support

for the WARC archived-web-content format to version 0.17 (ISO Committee Draft). This release also

includes 41 bug fixes or other incremental improvements.

Other details on this release, including a list of all tracked issues addressed, are available at: Release

Notes - Heritrix 1.14.0 at project wiki

[http://webteam.archive.org/confluence/display/Heritrix/Release+Notes+-+1.14.0]

Heritrix Release Notes

6. Release 1.12.1 - 2007-05-06

Release 1.12.1 is primarily a bug-fix release. A total of 21 bugs have been addressed, including an ob-

ject-retention issue when running multiple jobs in sequence, a failure to enforce fetch timeouts on cer-

tain malformed server responses, and decide-rule overrides disappearing when basing jobs on earlier

jobs/profiles. There are also several enhancements, such as reporting the volume of data evaluated but

not transferred/stored when using duplicate-reduction features.

6.1. Contributors

Aside from the usual suspects [http://crawler.archive.org/team-list.html], the following people contrib-

uted to this release:

• Ahmed Ghouzia

6.2. All tracked changes

A dynamic list of all tracked changes marked as fixed in 1.12.1 is available at: Issues with 'Fix Version'

1.12.1.

[http://webteam.archive.org/confluence/display/Heritrix/Issues+with+%27Fix+Version%27+1.12.1]

7. Release 1.12.0 - 2007-03-16

Release 1.12.0 is the first of several planned releases enhancing Heritrix with "smart crawler" function-

ality. In this release, the theme has been offering new options to reduce the amount of duplicate content

crawled and stored when recrawling sites at regular intervals. A number of other enhancements and bug

fixes are also included.

7.1. Contributors

Aside from the usual suspects [http://crawler.archive.org/team-list.html], the following contributed to

this release:

• Oskar Grenholm

• Doug Judd

7.2. Notes

With this release, Heritrix project issue-tracking has moved from Sourceforge to a JIRA-based system at

http://webteam.archive.org/jira/browse/HER [http://webteam.archive.org/jira/browse/HER].

Those using Heritrix in a Hadoop environment may be interested in Doug Judd's HDFSWriterProcessor

[http://www.zvents.com/labs/hdfs_writer_processor], for storing crawled content directly into HDFS,

the Hadoop Distributed FileSystem [http://lucene.apache.org/hadoop/].

7.3. Known Limitations/Issues

7.3.1. java.io.IOException: No locks available

Heritrix Release Notes

See Section 11.1.1, “java.io.IOException: No locks available” in 1.8.0 Release Notes.

7.3.2. Older Checkpoints

Checkpoints from earlier versions are generally not supported for resume in later versions.

7.3.3. Older configurations (order.xml, etc.)

Crawler configuration files from jobs in previous versions may work in 1.12.0, though missing new set-

tings will be set to their default values, and obsolete old settings will generate log warnings. Re-creating

configurations from defaults or hand-editting to match newer files is recommended.

7.4. Changes

7.4.1. Duplication reduction features

A collection of Processors, including the FetchHistoryProcessor, PersistProcessor, and its subclasses,

may be used together with new options on the FetchHTTP and writer processors to carry information

forward between crawls and collect less duplicate content on later recrawls. The project wiki features

notes on using the new duplication-reduction functionality

[http://webteam.archive.org/confluence/display/Heritrix/Feature+Notes+-+1.12.0].

7.4.2. DecideRules have replaced Filters on Processors

All Processors which used internal Filters for differentially acting on URIs now use DecideRules in-

stead. In those cases where a DecideRule replacement for a Filter is not yet available, a legacy Filter can

be wrapped in a FilterDecideRule to preserve prior functionality. In a future release, all Filters will be

removed in favor of equivalent DecideRules.

7.4.3. WARC

ExperimentalWARCWriter has been updated to match proposed WARC version "WARC/0.12"

(revision H1.12-RC1)

[http://archive-access.svn.sourceforge.net/viewvc/*checkout*/archive-access/branches/gjm_warc_0_12/

warc/warc_file_format.html]. The implementation as of Heritrix 1.10.x remains for reference as

org.archive.io.warc.v10.ExperimentalV10WARCWriterProcessor. The WARC format remains under

discussion.

7.4.4. Kw3WriterProcessor

Oskar Grenholm of the Swedish National Library has contributed a module that writes the results of suc-

cessful fetches to files on disk. These files are MIME-files of the type used by the Swedish National

Library's Kulturarw3 web harvesting [http://www.kb.se/kw3/].

7.4.5. All tracked changes

A dynamic list of all tracked changes marked as fixed in 1.12.0 is available at: Issues with 'Fix Version'

1.12.0.

[http://webteam.archive.org/confluence/display/Heritrix/Issues+with+%27Fix+Version%27+1.12.0]

8. Release 1.10.2 - 2007-01-15

This is primarily a bug-fix release, with a couple of new features, provided before a number of signific-

ant changes to the Heritrix project that will require developer and crawl operator adjustments. Post-

Heritrix Release Notes

1.10.2, Heritrix source code control, issue tracking, and build process will migrate to new systems. Also,

updates to core classes, especially with regard to the settings architecture, will noticeably break back-

ward compatibility with 1.10.2 and prior crawler settings files and formats.

8.1. Contributors

• Olaf Freyer

• Max Schöfmann

8.2. Changes

8.2.1. Jericho HTML Extractor

Olaf Freyer has contributed an HTML Extractor named JerichoExtractorHTML based on the Jericho

HTML Parser. Following is a quote from the JerichoExtractorHTML class comment describing how the

new Extractor differs from ExtractorHTML, its advantages and downsides: “ This extractor extends Ex-

tractorHTML and mimics its workflow - but has some substantial differences when it comes to internal

implementation. Instead of heavily relying upon java regular expressions it uses a real html parser lib-

rary - namely Jericho HTML Parser (http://jerichohtml.sourceforge.net). Using this parser it can better

handle broken html (i.e. missing quotes) and also offer improved extraction of HTML form URLs (not

only extract the action of a form, but also its default values). Unfortunately this parser also has one ma-

jor drawback - it has to read the whole document into memory for parsing, thus has an inherent OOME

risk. This OOME risk can be reduced/eleminated by limiting the size of documents to be parsed (i.e. us-

ing NotExceedsDocumentLengthTresholdDecideRule). Also note that this extractor seems to have a

lower overall memory consumption compared to ExtractorHTML. (still to be confirmed on a larger

scale crawl) ”

Table 1. All Tracked Changes

ID Type Summary Open Date By Filer

913002

[https://sourcefo

rge.net/tracker/i

ndex.php?func=

de-

tail&aid=91300

2&group_id=73

833&atid=5391

02]

Add Make Extract-

orHTML ag-

gressiveness

configurable

2004-03-09 gojomo gojomo

1573708

[https://sourcefo

rge.net/tracker/i

ndex.php?func=

de-

tail&aid=15737

08&group_id=7

3833&atid=539

102]

Add [Contrib]

JerichoExtract-

orHTML

2006-10-09 nobody pandae

1633458 Add [arcreader] Sup- 2007-01-11 stack stack

Heritrix Release Notes

评论收藏

内容反馈

版权申诉

工具盒子

粉丝: 58
资源: 1313

精选_基于Heritrix和Lucene实现的典故搜索引擎_源码打包

基于Heritrix与Lucene的垂直搜索引擎研究

基于Heritrix与Lucene的垂直搜索引擎研究.pdf

bbs.rar_Lucene heritrix_bbs_heritrix_heritrix insta_lucene

Heritrix lucene开发自己的搜索引擎(源码)1

基于Lucene_Heritrix的垂直搜索引擎的研究与应用

lucene_heritrix 搜索引擎

基于Lucene和Heritrix的职位垂直搜索引擎的设计与实现

开发自己的搜索引擎 lucene + heritrix

基于Lucene_Heritrix的垂直搜索引擎的研究与应用.pdf

基于Lucene／Heritrix的垂直搜索引擎的研究与应用

lucene_book(1).rar_Lucene Heritrix_heritrix_project_search_sym

Heritrix lucene开发自己的搜索引擎(源码)3

开发自己的搜索引擎《lucene2.0+heritrix》一书对应的源码资料

基于LUCENE和HERITRIX技术搜索引擎的设计与实现

基于Lucene和Heritrix技术搜索引擎的设计与实现.pdf

heritrix与Lucene的垂直搜索引擎研究

基于Heritrix的内容搜索引擎系统.pdf

Lucene+Heritrix(搜索引擎开发)

ChatGPT教程（终极版）最全整理

博客中Kmeans以及FCM算法数据（免积分）

神经网络回归预测--气温数据集

hugging face的models-openai-clip-vit-large-patch14文件夹

XGBoost+LightGBM+LSTM-光伏发电量预测

Mathwork+Matlab+编程手册

Stable-Diffusion WEBUI 简体中文语言包（2023.05.30更新）

中文短信数据集-带标签

时间序列预测模型实战案例(Xgboost)(Python)(机器学习)包括时间序列预测和时间序列分类，点击即可运行！

亚博K210模型训练部署

Plecs电力电子仿真PLECS41.64 电力系统仿真软件免安装版本

最新资源