DRAFT ISO/IEC 14496-10 : 2002 (E)
DRAFT ITU-T Rec. H.264 (2002 E) i
INTERNATIONAL ORGANISATION FOR STANDARDISATION
ORGANISATION INTERNATIONALE DE NORMALISATION
ISO/IEC JTC 1/SC 29/WG 11
CODING OF MOVING PICTURES AND AUDIO
ISO/IEC JTC 1/SC 29/WG 11
N5555
Pattaya, March 2003
Source JVT
Title
Draft Text of Final Draft International Standard for Advanced Video Coding
(ITU-T Rec. H.264 | ISO/IEC 14496-10 AVC)
Status Approved for final editing period with delivery by 31 March 2003
DRAFT ISO/IEC 14496-10 : 2002 (E)
ii DRAFT ITU-T Rec. H.264 (2002 E)
Title page to be provided by ITU-T | ISO/IEC
DRAFT INTERNATIONAL STANDARD
DRAFT ISO/IEC 14496-10 : 2002 (E)
DRAFT ITU-T Rec. H.264 (2002 E)
DRAFT ITU-T RECOMMENDATION
TABLE OF CONTENTS
Foreword..........................................................................................................................................................................xiii
0 Introduction .............................................................................................................................................................xiv
0.1 Prolog................................................................................................................................................................ xiv
0.2 Purpose.............................................................................................................................................................. xiv
0.3 Applications....................................................................................................................................................... xiv
0.4 Profiles and levels ............................................................................................................................................. xiv
0.5 Overview of the syntax........................................................................................................................................ xv
0.5.1 Predictive coding ........................................................................................................................................ xv
0.5.2 Coding interlaced video.............................................................................................................................. xv
0.5.3 Picture partitioning into macroblocks and smaller partitions......................................................................xv
0.5.4 Spatial redundancy reduction.....................................................................................................................xvi
0.6 How to read this specification ........................................................................................................................... xvi
1Scope............................................................................................................................................................................1
2 Normative references .................................................................................................................................................1
3 Definitions ...................................................................................................................................................................1
4 Abbreviations..............................................................................................................................................................7
5 Conventions.................................................................................................................................................................8
5.1 Arithmetic operators.............................................................................................................................................8
5.2 Logical operators .................................................................................................................................................8
5.3 Relational operators .............................................................................................................................................8
5.4 Bit-wise operators.................................................................................................................................................9
5.5 Assignment operators ...........................................................................................................................................9
5.6 Range notation......................................................................................................................................................9
5.7 Mathematical functions ........................................................................................................................................9
5.8 Variables, syntax elements and tables................................................................................................................10
5.9 Processes ............................................................................................................................................................11
6 Source, coded, decoded, output data formats, scanning processes, and neighbouring relationships................11
6.1 Bitstream formats ...............................................................................................................................................11
6.2 Source, decoded, and output picture formats .....................................................................................................11
6.3 Spatial subdivision of pictures and slices...........................................................................................................13
6.4 Inverse scanning processes and derivation processes for neighbours ...............................................................14
6.4.1 Inverse macroblock scanning process.........................................................................................................14
6.4.2 Inverse macroblock partition and sub-macroblock partition scanning process........................................... 14
6.4.2.1 Inverse macroblock partition scanning process ......................................................................................15
6.4.2.2 Inverse sub-macroblock partition scanning process ...............................................................................15
6.4.3 Inverse 4x4 luma block scanning process................................................................................................... 16
6.4.4 Derivation process of the availability for macroblock addresses................................................................ 16
6.4.5 Derivation process for neighbouring macroblock addresses and their availability..................................... 16
6.4.6 Derivation process for neighbouring macroblock addresses and their availability in MBAFF frames ......17
6.4.7 Derivation processes for neighbouring macroblocks, blocks, and partitions.............................................. 18
6.4.7.1 Derivation process for neighbouring macroblocks ................................................................................. 18
6.4.7.2 Derivation process for neighbouring 8x8 luma block.............................................................................19
6.4.7.3 Derivation process for neighbouring 4x4 luma blocks ........................................................................... 19
6.4.7.4 Derivation process for neighbouring 4x4 chroma blocks ....................................................................... 20
6.4.7.5 Derivation process for neighbouring partitions ...................................................................................... 20
6.4.8 Derivation process for neighbouring locations ........................................................................................... 21
6.4.8.1 Specification for neighbouring luma locations in fields and non-MBAFF frames ................................. 21
DRAFT ISO/IEC 14496-10 : 2002 (E)
DRAFT ITU-T Rec. H.264 (2002 E) iii
6.4.8.2 Specification for neighbouring luma locations in MBAFF frames
.........................................................22
7 Syntax and semantics ...............................................................................................................................................24
7.1 Method of describing syntax in tabular form......................................................................................................24
7.2 Specification of syntax functions, categories, and descriptors ...........................................................................25
7.3 Syntax in tabular form........................................................................................................................................27
7.3.1 NAL unit syntax .........................................................................................................................................27
7.3.2 Raw byte sequence payloads and RBSP trailing bits syntax ...................................................................... 28
7.3.2.1 Sequence parameter set RBSP syntax..................................................................................................... 28
7.3.2.2 Picture parameter set RBSP syntax.........................................................................................................29
7.3.2.3 Supplemental enhancement information RBSP syntax...........................................................................30
7.3.2.3.1 Supplemental enhancement information message syntax ................................................................ 30
7.3.2.4 Picture delimiter RBSP syntax ...............................................................................................................30
7.3.2.5 End of sequence RBSP syntax................................................................................................................30
7.3.2.6 End of stream RBSP syntax.................................................................................................................... 31
7.3.2.7 Filler data RBSP syntax.......................................................................................................................... 31
7.3.2.8 Slice layer without partitioning RBSP syntax......................................................................................... 31
7.3.2.9 Slice data partition RBSP syntax ............................................................................................................ 31
7.3.2.9.1 Slice data partition A RBSP syntax .................................................................................................31
7.3.2.9.2 Slice data partition B RBSP syntax.................................................................................................. 31
7.3.2.9.3 Slice data partition C RBSP syntax.................................................................................................. 32
7.3.2.10 RBSP slice trailing bits syntax.............................................................................................................. 32
7.3.2.11 RBSP trailing bits syntax ......................................................................................................................32
7.3.3 Slice header syntax ..................................................................................................................................... 33
7.3.3.1 Reference picture list reordering syntax .................................................................................................34
7.3.3.2 Prediction weight table syntax................................................................................................................ 35
7.3.3.3 Decoded reference picture marking syntax.............................................................................................36
7.3.4 Slice data syntax ......................................................................................................................................... 37
7.3.5 Macroblock layer syntax............................................................................................................................. 38
7.3.5.1 Macroblock prediction syntax ................................................................................................................39
7.3.5.2 Sub-macroblock prediction syntax ......................................................................................................... 40
7.3.5.3 Residual data syntax ............................................................................................................................... 41
7.3.5.3.1 Residual block CAVLC syntax........................................................................................................ 42
7.3.5.3.2 Residual block CABAC syntax........................................................................................................ 43
7.4 Semantics............................................................................................................................................................43
7.4.1 NAL unit semantics .................................................................................................................................... 43
7.4.1.1 Constraints on NAL unit order ............................................................................................................... 45
7.4.1.2 Association of NAL units to primary coded pictures..............................................................................47
7.4.1.3 Association of primary coded pictures to video sequences.....................................................................47
7.4.1.4 Encapsulation of an SODB within an RBSP (informative) .................................................................... 47
7.4.2 Raw byte sequence payloads and RBSP trailing bits semantics .................................................................48
7.4.2.1 Sequence parameter set RBSP semantics ...............................................................................................48
7.4.2.2 Picture parameter set RBSP semantics ...................................................................................................50
7.4.2.3 Supplemental enhancement information RBSP semantics .....................................................................52
7.4.2.3.1 Supplemental enhancement information message semantics...........................................................52
7.4.2.4 Picture delimiter RBSP semantics ..........................................................................................................52
7.4.2.5 End of sequence RBSP semantics...........................................................................................................52
7.4.2.6 End of stream RBSP semantics...............................................................................................................52
7.4.2.7 Filler data RBSP semantics..................................................................................................................... 52
7.4.2.8 Slice layer without partitioning RBSP semantics ................................................................................... 52
7.4.2.9 Slice data partition RBSP semantics.......................................................................................................52
7.4.2.9.1 Slice data partition A RBSP semantics ............................................................................................ 52
7.4.2.9.2 Slice data partition B RBSP semantics ............................................................................................53
7.4.2.9.3 Slice data partition C RBSP semantics ............................................................................................53
7.4.2.10 RBSP slice trailing bits semantics.........................................................................................................53
7.4.2.11 RBSP trailing bits semantics................................................................................................................. 53
7.4.3 Slice header semantics................................................................................................................................53
7.4.3.1 Reference picture list reordering semantics ............................................................................................57
7.4.3.2 Prediction weight table semantics...........................................................................................................58
7.4.3.3 Decoded reference picture marking semantics .......................................................................................59
7.4.4 Slice data semantics.................................................................................................................................... 61
7.4.5 Macroblock layer semantics .......................................................................................................................62
7.4.5.1 Macroblock prediction semantics ........................................................................................................... 67
7.4.5.2 Sub-macroblock prediction semantics ....................................................................................................67
DRAFT ISO/IEC 14496-10 : 2002 (E)
iv DRAFT ITU-T Rec. H.264 (2002 E)
7.4.5.3 Residual data semantics
..........................................................................................................................69
7.4.5.3.1 Residual block CAVLC semantics................................................................................................... 69
7.4.5.3.2 Residual block CABAC semantics ..................................................................................................70
8 Decoding process.......................................................................................................................................................70
8.1 NAL unit decoding process.................................................................................................................................70
8.2 Slice decoding process........................................................................................................................................71
8.2.1 Detection process of coded picture boundaries...........................................................................................71
8.2.2 Decoding process for picture order count...................................................................................................71
8.2.2.1 Decoding process for picture order count type 0 ....................................................................................72
8.2.2.2 Decoding process for picture order count type 1 ....................................................................................73
8.2.2.3 Decoding process for picture order count type 2 .................................................................................... 74
8.2.3 Decoding process for redundant slices .......................................................................................................74
8.2.4 Decoding process for macroblock to slice group map ................................................................................74
8.2.4.1 Specification for interleaved slice group map type................................................................................. 76
8.2.4.2 Specification for dispersed slice group map type ...................................................................................76
8.2.4.3 Specification for foreground with left-over slice group map type ..........................................................76
8.2.4.4 Specification for box-out slice group map types.....................................................................................76
8.2.4.5 Specification for raster scan slice group map types................................................................................ 77
8.2.4.6 Specification for wipe slice group map types ......................................................................................... 77
8.2.4.7 Specification for explicit slice group map type.......................................................................................77
8.2.4.8 Specification for conversion of map unit to slice group map to macroblock to slice group map ........... 77
8.2.5 Decoding process for slice data partitioning............................................................................................... 78
8.2.6 Decoding process for reference picture lists construction ..........................................................................78
8.2.6.1 Decoding process for picture numbers ...................................................................................................79
8.2.6.2 Initialisation process for reference picture lists ......................................................................................80
8.2.6.2.1 Initialisation process for the reference picture list for P and SP slices in frames............................. 80
8.2.6.2.2 Initialisation process for the reference picture list for P and SP slices in fields............................... 80
8.2.6.2.3 Initialisation process for reference picture lists for B slices in frames............................................. 81
8.2.6.2.4 Initialisation process for reference picture lists for B slices in fields............................................... 82
8.2.6.2.5 Initialisation process for reference picture lists in fields..................................................................82
8.2.6.3 Reordering process for reference picture lists......................................................................................... 83
8.2.6.3.1 Reordering process of reference picture lists for short-term pictures ..............................................83
8.2.6.3.2 Reordering process of reference picture lists for long-term pictures ...............................................84
8.2.7 Decoded reference picture marking process ...............................................................................................85
8.2.7.1 Sequence of operations for decoded reference picture marking process ................................................85
8.2.7.2 Decoding process for gaps in frame_num...............................................................................................85
8.2.7.3 Sliding window decoded reference picture marking process.................................................................. 86
8.2.7.4 Adaptive memory control decoded reference picture marking process..................................................86
8.2.7.4.1 Marking process of a short-term picture as “unused for reference”.................................................86
8.2.7.4.2 Marking process of a long-term picture as “unused for reference”.................................................. 87
8.2.7.4.3 Assignment process of a LongTermFrameIdx to a short-term reference picture............................. 87
8.2.7.4.4 Decoding process for MaxLongTermFrameIdx...............................................................................87
8.2.7.4.5 Marking process of all reference pictures as “unused for reference” and setting
MaxLongTermFrameIdx to “no long-term frame indices”.................................................................................87
8.2.7.4.6 Process for assigning a long-term frame index to the current picture ..............................................87
8.3 Intra prediction process......................................................................................................................................88
8.3.1 Intra_4x4 prediction process for luma samples ..........................................................................................88
8.3.1.1 Derivation process for the Intra4x4PredMode........................................................................................91
8.3.1.2 Intra_4x4 sample prediction ...................................................................................................................92
8.3.1.2.1 Specification of Intra_4x4_Vertical prediction mode......................................................................92
8.3.1.2.2 Specification of Intra_4x4_Horizontal prediction mode.................................................................. 93
8.3.1.2.3 Specification of Intra_4x4_DC prediction mode .............................................................................93
8.3.1.2.4 Specification of Intra_4x4_Diagonal_Down_Left prediction mode................................................ 93
8.3.1.2.5 Specification of Intra_4x4_Diagonal_Down_Right prediction mode.............................................. 93
8.3.1.2.6 Specification of Intra_4x4_Vertical_Right prediction mode ........................................................... 94
8.3.1.2.7 Specification of Intra_4x4_Horizontal_Down prediction mode ...................................................... 94
8.3.1.2.8 Specification of Intra_4x4_Vertical_Left prediction mode ............................................................. 94
8.3.1.2.9 Specification of Intra_4x4_Horizontal_Up prediction mode........................................................... 95
8.3.2 Intra_16x16 prediction process for luma samples ...................................................................................... 95
8.3.2.1 Specification of Intra_16x16_Vertical prediction mode.........................................................................96
8.3.2.2 Specification of Intra_16x16_Horizontal prediction mode..................................................................... 96
8.3.2.3 Specification of Intra_16x16_DC prediction mode ................................................................................96
8.3.2.4 Specification of Intra_16x16_Plane prediction mode.............................................................................96
DRAFT ISO/IEC 14496-10 : 2002 (E)
DRAFT ITU-T Rec. H.264 (2002 E) v
8.3.3 Intra prediction process for chroma samples
..............................................................................................97
8.3.3.1 Specification of Intra_Chroma_DC prediction mode .............................................................................97
8.3.3.2 Specification of Intra_Chroma_Horizontal prediction mode.................................................................. 98
8.3.3.3 Specification of Intra_Chroma_Vertical prediction mode......................................................................98
8.3.3.4 Specification of Intra_Chroma_Plane prediction mode..........................................................................98
8.4 Inter prediction process......................................................................................................................................99
8.4.1 Derivation process for motion vector components and reference indices.................................................100
8.4.1.1 Derivation process for luma motion vectors for skipped macroblocks in P and SP slices....................101
8.4.1.2 Derivation process for luma motion vectors for B_Skip, B_Direct_16x16 and B_Direct_8x8............101
8.4.1.2.1 Derivation process for the co-located 4x4 sub-macroblock partitions ........................................... 102
8.4.1.2.2 Derivation process for spatial direct luma motion vector and reference index prediction mode ... 104
8.4.1.2.3 Derivation process for temporal direct luma motion vector and reference index prediction mode106
8.4.1.3 Derivation process for luma motion vector prediction .........................................................................107
8.4.1.3.1 Derivation process for median luma motion vector prediction......................................................108
8.4.1.3.2 Derivation process for the neighbouring blocks for motion data of neighbouring partitions......... 109
8.4.1.4 Derivation process for chroma motion vectors..................................................................................... 109
8.4.2 Decoding process for Inter prediction samples......................................................................................... 110
8.4.2.1 Reference picture selection process...................................................................................................... 111
8.4.2.2 Fractional sample interpolation process................................................................................................111
8.4.2.2.1 Luma sample interpolation process................................................................................................ 112
8.4.2.2.2 Chroma sample interpolation process ............................................................................................ 114
8.4.2.3 Weighted sample prediction process ....................................................................................................115
8.4.2.3.1 Default weighted sample prediction process.................................................................................. 116
8.4.2.3.2 Weighted sample prediction process.............................................................................................. 116
8.5 Transform coefficient decoding process and picture construction process prior to deblocking filter process 118
8.5.1 Specification of transform decoding process for residual blocks ............................................................. 118
8.5.2 Specification of transform decoding process for luma samples of Intra_16x16 macroblock prediction
mode 119
8.5.3 Specification of transform decoding process for chroma samples............................................................ 119
8.5.4 Inverse scanning process for transform coefficients................................................................................. 120
8.5.5 Derivation process for the quantisation parameters and scaling function................................................. 121
8.5.6 Scaling and transformation process for luma DC coefficients for Intra_16x16 macroblock type ............122
8.5.7 Scaling and transformation process for chroma DC coefficients..............................................................122
8.5.8 Scaling and transformation process for residual 4x4 blocks.....................................................................123
8.5.9 Picture construction process prior to deblocking filter process ................................................................ 124
8.6 Decoding process for SP and SI slices .............................................................................................................125
8.6.1 SP decoding process for non-switching pictures ......................................................................................125
8.6.1.1 Luma transform coefficient decoding process ...................................................................................... 125
8.6.1.2 Chroma transform coefficient decoding process...................................................................................127
8.6.2 SP and SI slice decoding process for switching pictures ..........................................................................128
8.6.2.1 Luma transform coefficient decoding process ...................................................................................... 128
8.6.2.2 Chroma transform coefficient decoding process...................................................................................129
8.7 Deblocking filter process..................................................................................................................................130
8.7.1 Derivation process for the content dependent boundary filtering strength ............................................... 132
8.7.2 Derivation process for the thresholds for each block boundary................................................................ 133
8.7.3 Filtering process for edges with Bs smaller than 4 ................................................................................... 134
8.7.4 Filtering process for edges for Bs equal to 4 ............................................................................................135
9 Parsing process .......................................................................................................................................................136
9.1 Parsing process for Exp-Golomb codes ...........................................................................................................136
9.1.1 Mapping process for signed Exp-Golomb codes ...................................................................................... 137
9.1.2 Mapping process for coded block pattern................................................................................................. 138
9.2 CAVLC parsing process for transform coefficients..........................................................................................139
9.2.1 Parsing process for total number of coefficients and trailing ones ........................................................... 140
9.2.2 Parsing process for level information ....................................................................................................... 142
9.2.3 Parsing process for run information.......................................................................................................... 144
9.2.4 Combining level and run information....................................................................................................... 146
9.3 CABAC parsing process for slice data .............................................................................................................146
9.3.1 Initialisation process ................................................................................................................................. 147
9.3.1.1 Initialisation process for context variables............................................................................................ 148
9.3.1.2 Initialisation process for the arithmetic decoding engine......................................................................157
9.3.2 Binarization process.................................................................................................................................. 157
9.3.2.1 Unary (U) binarization process............................................................................................................. 159
9.3.2.2 Truncated unary (TU) binarization process .......................................................................................... 159
