NicePlayer是一套免费好用的播放器

A composite approach to language/encoding detection

Shanjian Li (shanjian@netscape.com)

Katsuhiko Momoi (momoi@netscape.com)

Netscape Communications Corp.

helping transition browser users from frequent uses of a character encoding menu into a more desirable

state where an encoding menu is rarely, if ever, used. We envision that the transition to the Unicode would

have to be transparent to the users. Users need not know how characters are displayed as long as they are

displayed correctly -- whether it’s a native encoding or one of Unicode encodings. Good auto-detection

service could would help significantly in this effort as it takes most encoding issues out of the user’s

concerns.

2. Background:

Since the beginning of the computer age, many encoding schemes have been created to processing certain

language and languages for certain region. With the trend of global village, and especially the development

of Internet, Information exchange across regions become more andrepresent various writing

the default encoding, e.g. XML, XHTML, RDF, etc. Thus, today's global software applications haveare

required to handle multiple encodings in addition to supporting Unicode.

This work is done in the context to developing Internet browser. Today’s Internet is full of web pages in

various languages using various encoding. A lot of effort have been put into browser development to

handle web pages in various encoding, but inThe current work has been conducted in the context of

developing an Internet browser. To deal with a variety of languages using different encodings on the web

today, a lot of efforts have been expended. In order to get the correct display result, browser’s rely on http

server, html author or end user to provide the correctbrowsers should be able to utilize the encoding

information inprovided by http servers, web pages or end users via a character encoding menu. order to

interpret the text data correctly. Unfortunately, this piecetype of information is missing in many http server

As more Internet standard protocols designate Unicode as the default encoding, there will undoubtedly be a

significant shift toward the use of Unicode on web pages. Good universal auto-detection can make an

important contribution toward such a shift if it works seamlessly without the user ever having to use an

encoding menu. Under such a condition, gradual shift to Unicode could be painless and without noticeable

effects on web users since for users, pages simply display correctly without them doing anything or paying

attention to an encoding menu. in this situation thus becomes very important.Such a smooth transition

could be aided by making encodings issues less and less noticeable to the users. Auto-detection would play

an important role for such a scenario.

3.

original creator, pass it on to other programs, and so on.

The auto-detection methods discussed in this paper use an Internet Browser application as an example.

These auto-detection methods, however, can be easily adapted for other types of applications.

Browsers may use certain detection algorithms to auto-detect the encoding of web pages. A program can

potentially interpret a piece of text in any number of ways assuming different encodings, but except in

some extremely rare situations, only one interpretation is desired by the page'’s author. , Tand this is

normally thenormally the only reasonable way for the browser user to see that page correctly in the

intended language.

To list major factors in designing an auto-detection algorithm, we begin with might macke certain

3. The input text may contains certain noise which has no relation with its encoding, like some HTML tags,

English words,contain extraneous noises which have no relation to its encoding, e.g. HTML tags, non-

native words (e.g. English words in Chinese documents), space and other format/control characters found

in a typical web page. characters.

To enumerate and cover all languages and encoding is almost a mission impossible.cover all the known

languages and encodings for auto-detection is nearly an impossible task. In the current approaches, wWe

tried to cover all popular encodings used in East Asian languages, and provided a generic model to handle

single-byte encodings at the same time. The Russian language encodings was chosen as an implementation

example of the latter type and also our test bed for single-byte encodings.

4. Target multi-byte encodings include UTF8, Shift-JIS, EUC-JP, GB2312, Big5, EUC-TW, EUC-KR,

(KOI8-R, ISO8859-5, window1251, Mac-cyrillic, ibm866, ibm855) are covered in a test bed and as an

4. Three Methods of Auto-detectionBasic idea, Coding scheme,

Character Distribution and Precedence Distribution:

In this section, we discuss 3 different methods used infor detecting the encoding of text data. They are 1)

Coding scheme method, 2) Character Distribution, and 3) 2-Char Sequence Distribution. Each one has its

strengths and weaknesses used onby its ownself, but if we use all 3 in a complementary manner, the results

can be quite satisfying.

This method is probably the most obvious and the one most often tried first for multi-byte encodings. In

any of the multi-byte encoding coding schemes, not all possible code points are used. If an illegal byte or

byte sequence (i.e. unused code point) is meetencountered when verifying a certain encoding, we can

available, -- one byte at a time.. The state machine changes its state based on its previous state and the byte

it receive. Among all states a state machine might reach, there are 3 states are of detector’s interest receives.

There are 3 states in a state machine that are of interest to an auto-detector:

code point) for character has been identified.

charset it is designed for and that there is no other possible encoding which can contain this byte

sequence. This will to lead to an immediate positive answer for the detector.

encoding. This will lead to an immediate negative answer for this encoding. Detector will exclude this

legal character, we can query the state machine to see how many bytes this character has. This information

is used in 2 ways.

unlikely to beother encodinganything other than UTF-8. So we count the number of multi-byte

characters identified by the UTF-8 state machine. When it reaches a certain number (= the

threshold), conclusion is made.

(see below) so that , so that no more judgement is needed inside the Coding Scheanalyzer can deal

with character data rather than raw data.me analyzer.

SomeIn any given language, some characters are used more often than other characters. This fact can be

used to devise a data model for each language script. This is particularly useful for languages with a large

number of characters such as Chinese, Japanese and Korean. We often hear anecdotally about such

distributional statistics, but we have not found many published results. Thus for the following discussions,

A similar researchResearch by Taiwan’s Mandarin Promotion Council conducted annually(Year??) shows

thea similar result for traditional Chinese encoded in Big5. <http://www.edu.tw:81/mandr/

>