word(character)segmentation资源-CSDN文库

共40个文件

pdf：37个

biboliography：1个

gz：1个

word

(character)

segmentation

4星 · 超过85%的资源需积分: 13 145 浏览量 2010-01-15 04:25:50 上传评论收藏 10.46MB RAR 举报

资源推荐

资源详情

资源评论

收起资源包目录

segementation.rar （40个子文件）

dictionary-based

an empirical study of smoothing techniques for language modeling.pdf 738KB

chinese word segmentation using finite state transducer.pdf 48KB

A stochastic finite-state word-sgementation algorithm for Chinses.pdf 1.82MB

chinese word segmentation based on maximum matching and word binding force.pdf 315KB

comparition

Dinh_LREC2008.pdf 74KB

8A comparative study on thai word segementation approaches.pdf 167KB

Jin_SIGHAN2008.pdf 500KB

Wang_SIGHAN2008.pdf 345KB

vietnamese word segmentation with CRFs and SVMs An Investigation.pdf 140KB

statistical-based

A maximum Entropy Approach to Chinese Word Segmentation.pdf 86KB

Unigram Language Model for Chinese Word Segmentation.pdf 86KB

Chinese word segmentation as Character tagging.pdf 181KB

corpus-based methods in Chinese morphology and phonology.pdf 849KB

Chinese indexing using mutual information.pdf 1.2MB

an unsupervised learning and statistical approach for vietnamese word recognition and segmentation.pdf 228KB

A Fuzzy Synset-based Hidden Markov Model for Automatic Text Segmentation.pdf 331KB

SNLP2002-0051c.pdf 389KB

Conditional Random Fields - probabilistic models for segmenting and labeling sequence data.pdf 183KB

Thai Word Segmentation based-on GLR Parsing Technique .pdf 316KB

bayesian unsupervised word segmentation with nested pitman-yor language modeling.pdf 246KB

a stochastic morphological analysis for japanese employing character n-gram and k-nn method.pdf 191KB

non-dictionary-based thai word segmentation using decision trees.pdf 93KB

japanese word segmentation by adaboost using the decision list.gz 233KB

RIVF06_Word_Segmentation_for_Vietnamese_Text_Categorization_An_online_corpus_approach.pdf 372KB

feature-based Thai word segmentation.pdf 147KB

a Search-based chinese word segmentation method.pdf 314KB

a method for word segmentation in vietnamese.pdf 149KB

Effective Tag Set Selection in chinese Word Segmentation via Conditional Random File Modeling.pdf 102KB

chinese segmentation and new word detection using CRF.pdf 136KB

biboliography 4KB

hybrid

word segmentation in sentence analysis.pdf 128KB

Thai word segmentation a lexical semantic approach approach.pdf 162KB

Chinese word segmentatin by classification of characters.pdf 182KB

Gao_CL2005.pdf 1.58MB

chinese and japanese word segmentation using word-level and character-level information.ps 740KB

vietnamese segmentation.pdf 215KB

thai word segmentation using character-level information.pdf 222KB

Li_SIGHAN2008.pdf 323KB

a hybrid approach to chinese word segmentation.pdf 88KB

Le_LATA2008.pdf 137KB

A Stochastic Finite-State

Word-Segmentation Algorithm for

Chinese

Richard Sproat*

Bell Laboratories

William Gale~

Bell Laboratories

Chilin Shih~

Bell Laboratories

Nancy Chang ~

University of Cambridge

The initial stage of text analysis for any NLP task usually involves the tokenization of the

input into words. For languages like English one can assume, to a first approximation, that

word boundaries are given by whitespace or punctuation. In various Asian languages, including

Chinese, on the other hand, whitespace is never used to delimit words, so one must resort to lexical

information to "reconstruct" the word-boundary information. In this paper we present a stochastic

finite-state model wherein the basic workhorse is the weighted finite-state transducer. The

model segments Chinese text into dictionary entries and words derived by various productive

lexical processes, and--since the primary intended application of this model is to text-to-speech

synthesis--provides pronunciations for these words. We evaluate the system's performance by

comparing its segmentation 'Tudgments" with the judgments of a pool of human segmenters,

and the system is shown to perform quite well.

1. The Problem

Any NLP application that presumes as input unrestricted text requires an initial phase

of text analysis; such applications involve problems as diverse as machine translation,

information retrieval, and text-to-speech synthesis (TTS). An initial step of any text-

analysis task is the tokenization of the input into words. For a language like English,

this problem is generally regarded as trivial since words are delimited in English

text by whitespace or marks of punctuation. Thus in an English sentence such as

I'm going to show up at the ACL one would reasonably conjecture that there are eight

words separated by seven spaces. A moment's reflection will reveal that things are not

quite that simple. There are clearly eight orthographic words in the example given,

but if one were doing syntactic analysis one would probably want to consider I'm to

consist of two

syntactic

words, namely I and am. If one is interested in translation,

one would probably want to consider show up as a single dictionary word since its

semantic interpretation is not trivially derivable from the meanings of show and up.

And if one is interested in TTS, one would probably consider the single orthographic

word ACL to consist of three

phonological

words--/eJ s'i ~l/--corresponding to the

pronunciation of each of the letters in the acronym. Space- or punctuation-delimited

700 Mountain Avenue, 2d-451, Murray Hill, NJ 07974, USA. E-mail: rws@bell-labs,

corn

t 700 Mountain Avenue, 2d-451, Murray Hill, NJ 07974, USA. E-mail: ¢ls@bell-labs.

corn

~t 600 Mountain Avenue, 2c-278, Murray Hill, NJ 07974, USA. E-mail: gale@research, art. corn

§ Cambridge, UK. E-mail: nc201@eng.cam.ac.uk

Computational Linguistics Volume 22, Number 3

(a)

'How do you say octopus in Japanese?'

(b)

Plausible Segmentation

ri4-wen2 zhangl-yu2 zen3-meO shuol

'Japanese' 'octopus' 'how' 'say'

(c)

Implausible Segmentation

ri4 wen2-zhangl yu2 zen3-meO shuol

'Japan' 'essay' 'fish' 'how' 'say'

Figure 1

A Chinese sentence in (a) illustrating the lack of word boundaries. In (b) is a plausible

segmentation for this sentence; in (c) is an implausible segmentation.

orthographic words are thus only a starting point for further analysis and can only be

regarded as a useful hint at the desired division of the sentence into words.

Whether a language even has orthographic words is largely dependent on the

writing system used to represent the language (rather than the language itself); the

notion "orthographic word" is not universal. Most languages that use Roman, Greek,

Cyrillic, Armenian, or Semitic scripts, and many that use Indian-derived scripts, mark

orthographic word boundaries; however, languages written in a Chinese-derived writ-

ing system, including Chinese and Japanese, as well as Indian-derived writing systems

of languages like Thai, do not delimit orthographic words. 1

Put another way, written Chinese simply lacks orthographic words. In Chinese

text, individual characters of the script, to which we shall refer by their traditional

name of

hanzi, 2

are written one after another with no intervening spaces; a Chinese

sentence is shown in Figure

1. 3

Partly as a result of this, the notion "word" has never

played a role in Chinese philological tradition, and the idea that Chinese lacks any-

thing analogous to words in European languages has been prevalent among Western

sinologists; see DeFrancis (1984). Twentieth-century linguistic work on Chinese (Chao

1968; Li and Thompson 1981; Tang 1988, 1989, inter alia) has revealed the incorrectness

of this traditional view. All notions of word, with the exception of the orthographic

word, are as relevant in Chinese as they are in English, and just as is the case in other

languages, a word in Chinese may correspond to one or more symbols in the orthog-

1 For a related approach to the problem of word-segmention in Japanese, see Nagata (1994), inter alia.

2 Chinese ~--~

han4zi4

'Chinese character'; this is the same word as Japanese

kanji.

3 Throughout this paper we shall give Chinese examples in traditional orthography, followed

immediately by a Romanization into the pinyin transliteration scheme; numerals following each pinyin

syllable represent tones. Examples will usually be accompanied by a translation, plus a

morpheme-by-morpheme gloss given in parentheses whenever the translation does not adequately

serve this purpose. In the pinyin transliterations a dash (-) separates syllables that may be considered

part of the same phonological word; spaces are used to separate plausible phonological words; and a

plus sign (+) is used, where relevant, to indicate morpheme boundaries of interest.

378

Sproat, Shih, Gale and Chang Word-Segmentation for Chinese

raphy: ,K.

ren2

'person' is a fairly uncontroversial case of a monographemic word,

and ~]

zhongl-guo2

(middle country) 'China' a fairly uncontroversial case of a di-

graphemic word. The relevance of the distinction between, say, phonological words

and, say, dictionary words is shown by an example like ~J~,:t/4~l~I

zhongl-

hua2 ren2-min2 gong4-he2-guo2

(China people republic) 'People's Republic of China.'

Arguably this consists of about three phonological words. On the other hand, in a

translation system one probably wants to treat this string as a single dictionary word

since it has a conventional and somewhat unpredictable translation into English.

Thus, if one wants to

segment

words--for any purpose--from Chinese sentences,

one faces a more difficult task than one does in English since one cannot use spacing

as a guide. For example, suppose one is building a TTS system for Mandarin Chinese.

For that application, at a minimum, one would want to know the phonological word

boundaries. Now, for this application one might be tempted to simply bypass the

segmentation problem and pronounce the text character-by-character. However, there

are several reasons why this approach will not in general work:

1. Many hanzi have more than one pronunciation, where the correct

pronunciation depends upon word affiliation: ~ is pronounced

deO

when

it is a prenominal modification marker, but

di4 in

the word ~I ~

mu4-di4

'goal'; ~ is normally

ganl

'dry,' but

qian2

in a person's given name.

2. Some phonological rules depend upon correct word segmentation,

including Third Tone Sandhi (Shih 1986), which changes a 3 (low) tone

into a 2 (rising) tone before another 3 tone: tJ~

xiao3 [lao3 shu3]

'little

rat,' becomes

xiao3 [ lao2-shu3 ],

rather than

xiao2 [ lao2-shu3 ],

because the

rule first applies within the word

lao3-shu3

'rat,' blocking its phrasal

application.

3. In various dialects of Mandarin certain phonetic rules apply at the word

level. For example, in Northern dialects (such as Beijing), a full tone (1,

2, 3, or 4) is changed to a neutral tone (0) in the final syllable of many

words: ~,//~

dongl-gual

'winter melon' is often pronounced

dongl-guaO.

The high 1 tone of//~ would not normally neutralize in this fashion if it

were functioning as a word on its own.

4. TTS systems in general need to do more than simply compute the

pronunciations of individual words; they also need to compute

intonational phrase boundaries in long utterances and assign relative

prominence to words in those utterances. It has been shown for English

(Wang and Hirschberg 1992; Hirschberg 1993; Sproat 1994, inter alia) that

grammatical part of speech provides useful information for these tasks.

Given that part-of-speech labels are properties of words rather than

morphemes, it follows that one cannot do part-of-speech assignment

without having access to word-boundary information. Making the

reasonable assumption that similar information is relevant for solving

these problems in Chinese, it follows that a prerequisite for

intonation-boundary assignment and prominence assignment is word

segmentation.

The points enumerated above are particularly related to TTS, but analogous arguments

can easily be given for other applications; see for example Wu and Tseng's (1993)

discussion of the role of segmentation in information retrieval. There are thus some

very good reasons why segmentation into words is an important task.

379

Computational Linguistics Volume 22, Number 3

A minimal requirement for building a Chinese word segmenter is obviously a

dictionary; furthermore, as has been argued persuasively by Fung and Wu (1994), one

will perform much better at segmenting text by using a dictionary constructed with

text of the same genre as the text to be segmented. For novel texts, no lexicon that

consists simply of a list of word entries will ever be entirely satisfactory, since the

list will inevitably omit many constructions that should be considered words. Among

these are words derived by various productive processes, including:

Morphologically derived words such as r~r__~

xue2-shengl+menO

(student+plural) 'students,' which is derived by the affixation of the

plural affix ~

menO

to the noun _~

xue2-shengl.

Personal names such as J~,~,~]~

zhoul-enl-lai2

'Zhou Enlai.' Of course, we

can expect famous names like Zhou Enlai's to be in many dictionaries,

but names such as ;~t~

shi2-jil-lin2,

the name of the second author of

this paper, will not be found in any dictionary.

Transliterated foreign names such as ,,~. ~

ma3-1ai2-xil-ya3

'Malaysia.' Again, famous place names will most likely be found in the

dictionary, but less well-known names, such as ~l~..~

bu4-lang3-shi4-wei2-ke4

'Brunswick' (as in the New Jersey town name

'New Brunswick') will not generally be found.

In this paper we present a stochastic finite-state model for segmenting Chinese

text into words, both words found in a (static) lexicon as well as words derived via

the above-mentioned productive processes. The segmenter handles the grouping of

hanzi into words and outputs word pronunciations, with default pronunciations for

hanzi it cannot group; we focus here primarily on the system's ability to segment

text appropriately (rather than on its pronunciation abilities). The model incorporates

various recent techniques for incorporating and manipulating linguistic knowledge

using finite-state transducers. It also incorporates the Good-Turing method (Baayen

1989; Church and Gale 1991) in estimating the likelihoods of previously unseen con-

structions, including morphological derivatives and personal names. We will evaluate

various

specific

aspects of the segmentation, as well as the

overall

segmentation per-

formance. This latter evaluation compares the performance of the system with that of

several human judges since, as we shall show, even people do not agree on a single

correct way to segment a text.

Finally, this effort is part of a much larger program that we are undertaking to

develop stochastic finite-state methods for text analysis with applications to TTS and

other areas; in the final section of this paper we will briefly discuss this larger program

so as to situate the work discussed here in a broader context.

2. A Brief Introduction to the Chinese Writing System

Most readers will undoubtedly be at least somewhat familiar with the nature of the

Chinese writing system, but there are enough common misunderstandings that it is

as well to spend a few paragraphs on properties of the Chinese script that will be

relevant to topics discussed in this paper.

The first point we need to address is what type of linguistic object a hanzi repre-

sents. Much confusion has been sown about Chinese writing by the use of the term

ideograph,

suggesting that hanzi somehow directly represent ideas. The most accurate

characterization of Chinese writing is that it is morphosyllabic (DeFrancis 1984): each

380

Sproat, Shih, Gale and Chang Word-Segmentation for Chinese

hanzi represents one morpheme lexically and semantically, and one syllable phonologi-

cally. Thus in a two-hanzi word like ~]~ zhongl-guo2 (middle country) 'China' there

are two syllables, and at the same time two morphemes. Of course, since the number

of attested (phonemic) Mandarin syllables (roughly 1400, including tonal distinctions)

is far smaller than the number of morphemes, it follows that a given syllable could

in principle be written with any of several different hanzi, depending upon which

morpheme is intended: the syllable zhongl could be ~ 'middle,' ~ 'clock,' ,~ 'end,'

or ,~ 'loyal.' A morpheme, on the other hand, usually corresponds to a unique hanzi,

though there are a few cases where variant forms are found. Finally, quite a few hanzi

are homographs, meaning that they may be pronounced in several different ways, and

in extreme cases apparently represent different morphemes: The prenominal modifi-

cation marker ft~ deO is presumably a different morpheme from the second morpheme

of I~l~ mu4-di4, even though they are written the same way. 4

The second point, which will be relevant in the discussion of personal names in

Section 4.4, relates to the internal structure of hanzi. Following the system devised

under the Qing emperor Kang Xi, hanzi have traditionally been classified according

to a set of approximately 200 semantic radicals; members of a radical class share

a particular structural component, and often also share a common meaning (hence

the term 'semantic'). For example, hanzi containing the INSECT radical ~ tend to

denote insects and other crawling animals; examples include ~ wal 'frog,' ~ fengl

'wasp,' and ~ she2 'snake.' Similarly, hanzi sharing the GHOST radical ~ tend to

denote spirits and demons, such as ~ gui3 'ghost' itself, ~ mo2 'demon,' and

yan3 'nightmare.' While the semantic aspect of radicals is by no means completely

predictive, the semantic homogeneity of many classes is quite striking: for example

254 out of the 263 examples (97%) of the INSECT class listed by Wieger (1965, 773-76)

denote crawling or invertebrate animals; similarly 21 out of the 22 examples (95%) of

the GHOST class (page 808) denote ghosts or spirits. As we shall argue, the semantic

class affiliation of a hanzi constitutes useful information in predicting its properties.

3. Previous Work

There is a sizable literature on Chinese word segmentation: recent reviews include

Wang, Su, and Mo (1990) and Wu and Tseng (1993). Roughly speaking, previous work

can be divided into three categories, namely purely statistical approaches, purely lexi-

cal rule-based approaches, and approaches that combine lexical information with sta-

tistical information. The present proposal falls into the last group.

Purely statistical approaches have not been very popular, and so far as we are

aware earlier work by Sproat and Shih (1990) is the only published instance of such

an approach. In that work, mutual information was used to decide whether to group

adjacent hanzi into two-hanzi words. Mutual information was shown to be useful in

the segmentation task given that one does not have a dictionary. A related point is

that mutual information is helpful in augmenting existing electronic dictionaries, (cf.

4 To be sure, it is not always true that a hanzi represents a syllable or that it represents a morpheme. For

example, in Northern Mandarin dialects there is a morpheme -r that attaches mostly to nouns, and

which is phonologically incorporated into the syllable to which it attaches: thus men2+r (door+R)

'door" is realized as mer2. This is orthographically represented as ~ so that 'door' would be ~,

and in this case the hanzi ~t~ does not represent a syllable. Similarly, there is no compelling evidence

that either of the syllables of ~ binl-lang2 'betelnut' represents a morpheme, since neither can occur

in any context without the other: more likely ~]~1~ binl-lang2 is a disyllabic morpheme. (See Sproat

and Shih 1995.) However, the characterization given in the main body of the text is correct sufficiently

often to be useful.

381

评论收藏

内容反馈

taohu4

2012-12-25

可以用，虽然质量不是特别的高，不过已经很不错了~值得参考
Niwb_86

2017-10-11

刚才没下载下来啊

zhezhe7

粉丝: 0
资源: 20

word (character) segmentation

character-segmentation.rar_Line segmentation_图像字符分割

Chinese word segmentation 设计文档1

word-character-count:html5字符串中的单词和字符计数

is-word-character:检查一个字符是否是单词字符

character-and-word-count:字数统计

Improving Cross-Domain Chinese Word Segmentation with Word Embeddings

论文研究-AJWE: Jointly Learning Chinese Word Embeddings with Heterogeneous Attention.pdf

word-segment.zip_zip

自然语言处理 清华信息学院考博

thai-word-segmentation-sentencepiece:使用句子的泰语分词

前端开源库-jquery-word-and-character-counter-plugin

Approach of vehicle segmentation based on texture character

Integrating N-gram model and case-based learning for Chinese word segmentation

Thai word segmentation utility-开源

lingpipe4.1源码

tesseract3.02.02.7z

lingpipe-4.1.0.jar 自然语言处理工具包

sub-character-cws:子角色表示学习

自然语言处理.zip

Chinese-word-segmentation:通过Python完成中文分词

A language independent n-gram model for word segmentation

A novel unsupervised non-iterative approach to word segmentation

Microsoft Library MSDN4DOS.zip

哪种方法在新单词检测，基于字符或基于中文分词方面表现更好？

8-07-14_MegaCLI for linux_windows

计算机网络第六版答案

公式识别并转换为word格式工具

2019年数学建模大赛（美赛）论文模板（word和LaTeX）

简历模板大全

最新资源

自然语言处理清华信息学院考博