電子書 H.264 and MPEG-4 Video Compression

3

Video Coding Concepts

3.1 INTRODUCTION

compress vb.: to squeeze together or compact into less space; condense

compress noun: the act of compression or the condition of being compressed

Compression is the process of compacting data into a smaller number of bits. Video compres-

sion (video coding) is the process of compacting or condensing a digital video sequence

into a smaller number of bits. ‘Raw’ or uncompressed digital video typically requires a

large bitrate (approximately 216 Mbits for 1 second of uncompressed TV-quality video, see

pair is often described as a CODEC (enCOder/ DECoder) (Figure 3.1).

Data compression is achieved by removing redundancy, i.e. components that are not nec-

essary for faithful reproduction of the data. Many types of data contain statistical redundancy

and can be effectively compressed using lossless compression, so that the reconstructed data

at the output of the decoder is a perfect copy of the original data. Unfortunately, lossless com-

pression of image and video information gives only a moderate amount of compression. The

best that can be achieved with current lossless image compression standards such as JPEG-LS

[1] is a compression ratio of around 3–4 times. Lossy compression is necessary to achieve

higher compression. In a lossy compression system, the decompressed data is not identical to

the source data and much higher compression ratios can be achieved at the expense of a loss



2003 John Wiley & Sons, Ltd. ISBN: 0-470-84837-5

compression. In the temporal domain, there is usually a high correlation (similarity) between

frames of video that were captured at around the same time. Temporally adjacent frames (suc-

cessive frames in time order) are often highly correlated, especially if the temporal sampling

rate (the frame rate) is high. In the spatial domain, there is usually a high correlation between

pixels (samples) that are close to each other, i.e. the values of neighbouring samples are often

very similar (Figure 3.2).

The H.264 and MPEG-4 Visual standards (described in detail in Chapters 5 and 6) share a

number of common features. Both standards assume a CODEC ‘model’ that uses block-based

motion compensation, transform, quantisation and entropy coding. In this chapter we examine

the main components of this model, starting with the temporal model (motion estimation and

decoded video sequence is identical to the original, then the coding process is lossless; if the

decoded sequence differs from the original, the process is lossy.

The CODEC represents the original video sequence by a model (an efficient coded

representation that can be used to reconstruct an approximation of the video data). Ideally, the

model should represent the sequence using as few bits as possible and with as high a fidelity

as possible. These two goals (compression efficiency and high quality) are usually conflicting,

because a lower compressed bit rate typically produces reduced image quality at the decoder.

This tradeoff between bit rate and quality (the rate-distortion trade off) is discussed further in

Chapter 7.

A video encoder (Figure 3.3) consists of three main functional units: a temporal model,

frame (created by subtracting the prediction from the actual current frame) and a set of model

parameters, typically a set of motion vectors describing how the motion was compensated.

The residual frame forms the input to the spatial model which makes use of similarities

between neighbouring samples in the residual frame to reduce spatial redundancy. In MPEG-4

Visual and H.264 this is achieved by applying a transform to the residual samples and quan-

tizing the results. The transform converts the samples into another domain in which they are

represented by transform coefficients. The coefficients are quantised to remove insignificant

values, leaving a small number of significant coefficients that provide a more compact repre-

sentation of the residual frame. The output of the spatial model is a set of quantised transform

coefficients.

The video decoder reconstructs a video frame from the compressed bit stream. The

coefficients and motion vectors are decoded by an entropy decoder after which the spatial

model is decoded to reconstruct a version of the residual frame. The decoder uses the motion

vector parameters, together with one or more previously decoded frames, to create a prediction

of the current frame and the frame itself is reconstructed by adding the residual frame to this

prediction.

3.3 TEMPORAL MODEL

The goal of the temporal model is to reduce redundancy between transmitted frames by forming

a predicted frame and subtracting this from the current frame. The output of this process is

a residual (difference) frame and the more accurate the prediction process, the less energy is

The simplest method of temporal prediction is to use the previous frame as the predictor

for the current frame. Two successive frames from a video sequence are shown in Figure

3.4 and Figure 3.5. Frame 1 is used as a predictor for frame 2 and the residual formed by

subtracting the predictor (frame 1) from the current frame (frame 2) is shown in Figure 3.6.

In this image, mid-grey represents a difference of zero and light or dark greys correspond

to positive and negative differences respectively. The obvious problem with this simple pre-

diction is that a lot of energy remains in the residual frame (indicated by the light and dark

areas) and this means that there is still a significant amount of information to compress after

temporal prediction. Much of the residual energy is due to object movements between the two

frames and a better prediction may be formed by compensating for motion between the two

uncovered regions and lighting changes, these differences correspond to pixel movements

between frames. It is possible to estimate the trajectory of each pixel between successive

video frames, producing a field of pixel trajectories known as the optical flow (optic flow) [2].

Figure 3.7 shows the optical flow field for the frames of Figure 3.4 and Figure 3.5. The complete

that only the vector for every 2nd pixel is shown.

If the optical flow field is accurately known, it should be possible to form an accu-

rate prediction of most of the pixels of the current frame by moving each pixel from the