计算机研究 -基于增量聚类的新闻话题发现研究.pdf

ABSTRACT

With the rapid development of Internet technology, Internet has established itself

as the fourth largest media dominance, become the major carrier of releasing

information, getting information and transporting information to the public. Due to

timeliness and universality of news reports, news portals have become its vanguard of

news report. In addition, the point of view and quantity of the news reports also can

reflects current public opinion and the main social contradiction to a certain extent.

for real practical meaning.

This paper mainly includes three aspects. The first one is doing Chinese Word

Segmentation after getting reports on each portal news sites, aiming to make the

segmentation more suitable for next steps. This paper improves the traditional

segmentation methods, obtaining accurate and adaptive results. The second aspect is

improved traditional single-pass incremental clustering algorithm on accuracy and

recall rate of clustering, which is the focus of this article. The third one is improved

the objectivity and persuasive of news topic through combining news information

clustering results from present domestic news sites with machine translation

technology and overseas research on news report.

目 录

第一章 绪论.................................................................................................................. 1

1.1 课题研究的背景和意义.................................................................................. 1

1.2 国内外研究现状.............................................................................................. 2

1.3 现有的话题发现系统存在的问题.................................................................. 3

1.4 本文的研究内容和组织结构.......................................................................... 4

1.4.1 本文的研究内容.................................................................................... 4

1.4.2 本文的组织结构.................................................................................... 5

第二章 论文的相关工作.............................................................................................. 6

2.1 文本预处理...................................................................................................... 6

2.3.1 层次聚类算法........................................................................................ 8

2.3.2 划分式聚类算法.................................................................................... 9

2.3.3 基于密度的方法.................................................................................... 9

2.3.4 基于网格的方法.................................................................................... 9

2.4 机器翻译.......................................................................................................... 9

第三章 改进的增量聚类算法的实现........................................................................ 11

3.1 CRF 分词的改进实现 .................................................................................... 11

3.1.1 CRF 分词存在的问题 .......................................................................... 11

3.1.2 实验过程与结果.................................................................................. 13

3.1.3 改进分词的意义.................................................................................. 14

3.3.2 具有更新选择器的增量聚类算法设计与意义.................................. 19

3.3.3 引入待排队列的增量聚类算法设计与意义...................................... 20

3.3.4 ICCQ 算法设计 .................................................................................... 22

3.4 实验结果与分析............................................................................................. 23

3.4.1 数据来源.............................................................................................. 23

3.4.2 实验过程.............................................................................................. 23

3.4.3 评价指标.............................................................................................. 24

3.4.4 实验结果.............................................................................................. 24

3.4.5 结果分析.............................................................................................. 26

3.4.6 实验小结.............................................................................................. 26

4.3.3 实验的方法与流程.............................................................................. 33

4.4 实验结果........................................................................................................ 34

4.5 结果分析........................................................................................................ 37

第五章 总结与展望.................................................................................................... 39

5.1 本文所解决的问题........................................................................................ 39

5.2 不足之处........................................................................................................ 39

5.3 展望................................................................................................................ 40

参考文献...................................................................................................................... 41

发表文献和科研情况说明.......................................................................................... 47

致 谢...................................................................................................................... 48

闻信息进行处理，可以使网络新闻用户快速了解当前的新闻热点以及焦点。

发展和进步。

话题检测最早出现在话题检测与跟踪领域，两者采用的是相同的技术。TDT

是由美国国防部高级研究规划署（Advanced Research Project Agency，ARPA）和

国家标准技术局（NIST）于 1996 年发起研究的一项新技术，其目的是提取各类

信息进行分类整理以提取热点信息，从而帮助人们解决信息过载问题。NIST 为

系统能够针对具备时间顺序的新闻语料及时地检测出最新发生的事件；（2）报

道关系检测：判断语料中的两篇新闻报道是否属于同一新闻话题；（3）话题检

测：将关于同一话题的新闻报道聚集起来形成新话题；（4）话题跟踪：对于某

踪话题；（6）层次话题检测：层次话题检测是话题检测的扩展，在层次话题检

测中一篇新闻报道可以归属于多个新闻话题，一个新闻话题也可包含多个子话题。

等一流大学的众多知名学者参与其中，对 TDT 研究取得了初步成果，获取了宝

贵的经验。目前，包括 CMU、UMass、Cambridge 等在内的一流大学以及 IBM

等公司在 TDT 的评测中已经取得很好的成绩。TDT 研究任务主要集中在三个方

之后，随着科研人员对 TDT 研究

的不断深入，多种新方法新技术不断被应用到 TDT 中来，TDT 的研究领域不断

被拓宽。在话题检测方面，CMU 的科研人员最先探究了用凝聚层次聚类的方法

。Kumaran 和 Allan 将自然语言处理技术应用于