UTF-8 decoder capability and stress test
----------------------------------------
Markus Kuhn <http://www.cl.cam.ac.uk/~mgk25/> - 2003-02-19
This test file can help you examine, how your UTF-8 decoder handles
various types of correct, malformed, or otherwise interesting UTF-8
sequences. This file is not meant to be a conformance test. It does
not prescribes any particular outcome and therefore there is no way to
"pass" or "fail" this test file, even though the texts suggests a
preferable decoder behaviour at some places. The aim is instead to
help you think about and test the behaviour of your UTF-8 on a
systematic collection of unusual inputs. Experience so far suggests
that most first-time authors of UTF-8 decoders find at least one
serious problem in their decoder by using this file.
The test lines below cover boundary conditions, malformed UTF-8
sequences as well as correctly encoded UTF-8 sequences of Unicode code
points that should never occur in a correct UTF-8 file.
According to ISO 10646-1:2000, sections D.7 and 2.3c, a device
receiving UTF-8 shall interpret a "malformed sequence in the same way
that it interprets a character that is outside the adopted subset" and
"characters that are not within the adopted subset shall be indicated
to the user" by a receiving device. A quite commonly used approach in
UTF-8 decoders is to replace any malformed UTF-8 sequence by a
replacement character (U+FFFD), which looks a bit like an inverted
question mark, or a similar symbol. It might be a good idea to
visually distinguish a malformed UTF-8 sequence from a correctly
encoded Unicode character that is just not available in the current
font but otherwise fully legal, even though ISO 10646-1 doesn't
mandate this. In any case, just ignoring malformed sequences or
unavailable characters does not conform to ISO 10646, will make
debugging more difficult, and can lead to user confusion.
Please check, whether a malformed UTF-8 sequence is (1) represented at
all, (2) represented by exactly one single replacement character (or
equivalent signal), and (3) the following quotation mark after an
illegal UTF-8 sequence is correctly displayed, i.e. proper
resynchronization takes place immageately after any malformed
sequence. This file says "THE END" in the last line, so if you don't
see that, your decoder crashed somehow before, which should always be
cause for concern.
All lines in this file are exactly 79 characters long (plus the line
feed). In addition, all lines end with "|", except for the two test
lines 2.1.1 and 2.2.1, which contain non-printable ASCII controls
U+0000 and U+007F. If you display this file with a fixed-width font,
these "|" characters should all line up in column 79 (right margin).
This allows you to test quickly, whether your UTF-8 decoder finds the
correct number of characters in every line, that is whether each
malformed sequences is replaced by a single replacement character.
Note that as an alternative to the notion of malformed sequence used
here, it is also a perfectly acceptable (and in some situations even
preferable) solution to represent each individual byte of a malformed
sequence by a replacement character. If you follow this strategy in
your decoder, then please ignore the "|" column.
Here come the tests: |
|
1 Some correct UTF-8 text |
|
(The codepoints for this test are: |
U+03BA U+1F79 U+03C3 U+03BC U+03B5 --ryan.) |
|
You should see the Greek word 'kosme': "κόÏμε" |
|
|
2 Boundary condition test cases |
|
2.1 First possible sequence of a certain length |
|
(byte zero skipped...there's a null added at the end of the test. --ryan.) |
|
2.1.2 2 bytes (U-00000080): "Â" |
2.1.3 3 bytes (U-00000800): "à " |
2.1.4 4 bytes (U-00010000): "ð" |
|
(5 and 6 byte sequences were made illegal in rfc3629. --ryan.) |
2.1.5 5 bytes (U-00200000): "ø" |
2.1.6 6 bytes (U-04000000): "ü" |
|
2.2 Last possible sequence of a certain length |
|
2.2.1 1 byte (U-0000007F): "" |
2.2.2 2 bytes (U-000007FF): "ß¿" |
|
(Section 5.3.2 below calls this illegal. --ryan.) |
2.2.3 3 bytes (U-0000FFFF): "ï¿¿" |
|
(5 and 6 bytes sequences, and 4 bytes sequences > 0x10FFFF were made illegal |
in rfc3629, so these next three should be replaced with a invalid |
character codepoint. --ryan.) |
2.2.4 4 bytes (U-001FFFFF): "÷¿¿¿" |
2.2.5 5 bytes (U-03FFFFFF): "û¿¿¿¿" |
2.2.6 6 bytes (U-7FFFFFFF): "ý¿¿¿¿¿" |
|
2.3 Other boundary conditions |
|
2.3.1 U-0000D7FF = ed 9f bf = "í¿" |
2.3.2 U-0000E000 = ee 80 80 = "î" |
2.3.3 U-0000FFFD = ef bf bd = "�" |
2.3.4 U-0010FFFF = f4 8f bf bf = "ô¿¿" |
|
(This one is bogus in rfc3629. --ryan.) |
2.3.5 U-00110000 = f4 90 80 80 = "ô" |
|
3 Malformed sequences |
|
3.1 Unexpected continuation bytes |
|
Each unexpected continuation byte should be separately signalled as a |
malformed sequence of its own. |
|
3.1.1 First continuation byte 0x80: "" |
3.1.2 Last continuation byte 0xbf: "¿" |
没有合适的资源?快使用搜索试试~ 我知道了~
SDL-devel-1.2.13-mingw32.tar.gz
4星 · 超过85%的资源 需积分: 14 129 下载量 103 浏览量
2008-05-13
18:59:41
上传
评论
收藏 808KB GZ 举报
温馨提示
共480个文件
html:203个
3:174个
c:32个
经常看见有人寻找sdl.dll。这是Development Libraries <br>SDL-devel-1.2.13-mingw32.tar.gz (Mingw32)
资源推荐
资源详情
资源评论
收起资源包目录
SDL-devel-1.2.13-mingw32.tar.gz (480个子文件)
SDLKey.3 6KB
SDL_Event.3 5KB
SDL_OpenAudio.3 5KB
SDL_PixelFormat.3 4KB
SDL_SetAlpha.3 4KB
SDL_SetVideoMode.3 4KB
SDL_CreateRGBSurface.3 3KB
SDL_CreateCursor.3 3KB
SDL_ConvertAudio.3 3KB
SDL_SetPalette.3 3KB
SDL_AudioCVT.3 3KB
SDL_Surface.3 3KB
SDL_AudioSpec.3 3KB
SDL_BlitSurface.3 3KB
SDL_SetColors.3 2KB
SDL_AddTimer.3 2KB
SDL_LockSurface.3 2KB
SDL_keysym.3 2KB
SDL_Overlay.3 2KB
SDL_ListModes.3 2KB
SDL_SetTimer.3 2KB
SDL_SemWaitTimeout.3 2KB
SDL_SetEventFilter.3 2KB
SDL_VideoInfo.3 2KB
SDL_MouseMotionEvent.3 2KB
SDL_JoyHatEvent.3 2KB
SDL_SetColorKey.3 2KB
SDL_ActiveEvent.3 2KB
SDL_SemTryWait.3 1KB
SDL_CDPlayTracks.3 1KB
SDL_CDOpen.3 1KB
SDL_GLattr.3 1KB
SDL_LoadWAV.3 1KB
SDL_WasInit.3 1KB
SDL_CDStatus.3 1KB
SDL_CD.3 1KB
SDL_MouseButtonEvent.3 1KB
SDL_GL_SetAttribute.3 1KB
SDL_KeyboardEvent.3 1KB
SDL_VideoModeOK.3 1KB
SDL_JoyBallEvent.3 1KB
SDL_JoyButtonEvent.3 1KB
SDL_CreateSemaphore.3 1KB
SDL_FillRect.3 1KB
SDL_UserEvent.3 1KB
SDL_BuildAudioCVT.3 1KB
SDL_GL_GetProcAddress.3 1KB
SDL_DisplayFormatAlpha.3 1KB
SDL_SemWait.3 1KB
SDL_PeepEvents.3 1KB
SDL_EnableUNICODE.3 1KB
SDL_JoystickOpen.3 1KB
SDL_JoystickGetAxis.3 1KB
SDL_PollEvent.3 1KB
SDL_MapRGBA.3 1KB
SDL_UpdateRects.3 1KB
SDL_MixAudio.3 1KB
SDL_JoyAxisEvent.3 1KB
SDL_MapRGB.3 1KB
SDL_GetModState.3 1KB
SDL_SetGammaRamp.3 1KB
SDL_SysWMEvent.3 1KB
SDL_CreateRGBSurfaceFrom.3 1KB
SDL_Init.3 1KB
SDL_QuitEvent.3 1KB
SDL_InitSubSystem.3 1KB
SDL_ResizeEvent.3 1KB
SDL_PumpEvents.3 1KB
SDL_SemPost.3 1KB
SDL_JoystickEventState.3 1KB
SDL_DisplayFormat.3 1KB
SDL_Palette.3 1KB
SDL_GetKeyState.3 1KB
SDL_ConvertSurface.3 1KB
SDL_Flip.3 1016B
SDL_CreateYUVOverlay.3 986B
SDL_EventState.3 985B
SDL_CDtrack.3 978B
SDL_SetModState.3 967B
SDL_SetGamma.3 962B
SDL_JoystickGetBall.3 957B
SDL_PushEvent.3 954B
SDL_GetRGBA.3 947B
SDL_DestroySemaphore.3 944B
SDL_JoystickGetHat.3 930B
SDL_Quit.3 909B
SDL_SetClipRect.3 901B
SDL_UpdateRect.3 884B
SDL_GetRGB.3 862B
SDL_EnableKeyRepeat.3 844B
SDL_ShowCursor.3 843B
SDL_GetMouseState.3 827B
SDL_WM_SetIcon.3 824B
SDL_WM_GrabInput.3 819B
SDL_VideoDriverName.3 811B
SDL_DisplayYUVOverlay.3 805B
SDL_JoystickName.3 803B
SDL_SemValue.3 803B
SDL_CreateMutex.3 793B
SDL_GetEventFilter.3 765B
共 480 条
- 1
- 2
- 3
- 4
- 5
老虎太慢
- 粉丝: 5
- 资源: 26
上传资源 快速赚钱
- 我的内容管理 展开
- 我的资源 快来上传第一个资源
- 我的收益 登录查看自己的收益
- 我的积分 登录查看自己的积分
- 我的C币 登录后查看C币余额
- 我的收藏
- 我的下载
- 下载帮助
安全验证
文档复制为VIP权益,开通VIP直接复制
信息提交成功
- 1
- 2
前往页