2021 Enhanced Standard Compatible Image Compression Framework Ba

IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. XX, NO. X, X XXXX 1

Enhanced Standard Compatible Image Compression

Framework based on Auxiliary Codec Networks

Hanbin Son, Taeoh Kim, Hyeongmin Lee, and Sangyoun Lee, Member, IEEE

Abstract—Recent deep neural network-based research to en-

hance image compression performance can be divided into

three categories: learnable codecs, postprocessing networks, and

compact representation networks. The learnable codec has been

designed for end-to-end learning beyond the conventional com-

for increasing coding efficiency. Specifically, it is difficult to

achieve optimal learning in previous studies using a compact

representation network due to the inaccurate consideration of the

codecs. In this paper, we propose a novel standard compatible

image compression framework based on auxiliary codec net-

works (ACNs). In addition, ACNs are designed to imitate image

degradation operations of the existing codec, which delivers

more accurate gradients to the compact representation network.

Therefore, compact representation and postprocessing networks

can be learned effectively and optimally. We demonstrate that

W

ITH the development of media technology, the de-

mand for live streaming or communicating using high-

resolution visual data has increased, requiring better perfor-

mance of image and video compression algorithms. Standard

algorithms have been carefully developed and released for

compatibility between the encoder and decoder of compression

algorithms across users.

The JPEG standard [1], a traditional image compression

algorithm, has been the most widely used in still-image com-

development of human Re-identification and masked face recognition based

on CCTV camera) and the Institute of Information & communications Tech-

nology Planning & Evaluation (IITP) grant funded by the Korea government,

Ministry of Science and ICT (MSIT) (No.2016-0-00197, Development of the

high-precision natural 3D view generation technology using smart-car multi

sensors and deep learning).

H. Son, T. Kim, H. Lee and S. Lee are with the School of Elec-

trical and Electronic Engineering, Yonsei University, Seoul, South Ko-

rea (e-mail: [email protected]; [email protected]; [email protected];

[email protected])

compression standards, such as JPEG2000 [2], H.264/AVC [3],

end trainable deep neural networks [5]–[14] (learnable codecs

in this paper) have been proposed based on the rapid develop-

ment of deep learning. The approaches use trainable networks

to produce bitstreams and reconstruct the original image

(Fig. 1 (a)). Although these kinds of approaches structurally

consider the compression ratio and reconstruction quality,

their performance is still undesirable and incompatible with

standard codecs, which decreases the algorithm’s utility.

compression process to improve the compression performance

while being compatible with standard codecs. Following the

PPNets perform well in reconstruction and are compatible

with standard codecs, they only efficiently increase the visual

quality of the reconstructed image but do not consider the

compression ratio (Fig. 1 (b)).

gies have been applied to consider both image quality and

compression ratio. These kinds of approaches place a spatially

downsampled image into the codec and upsample [29]–[31],

or are processed by the PPNet [32], [33]. Generally, reducing

the spatial size of an image can increase the compression

proposed to overcome these problems. These algorithms train

tation in [38], abbreviated to CRNet in this paper) oper-

ation using the reconstruction loss after the PPNet. These

approaches can achieve both a high compression ratio and

better reconstruction quality with two networks. However,

IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. XX, NO. X, X XXXX 2

(b) Postprocessing networks (PPNet)

Fig. 1. Conceptual comparison between frameworks. Green and red arrows indicate forward (or inference) and backward pass (or gradients) to train the

CRNet, respectively. Gray modules indicate that it is not differentiable or a standard codec. Blue modules indicate a differentiable network.

do not consider the degradation process through the standard

codec (Fig. 1 (c)) because the standard codec, including the

quantization process, is a nondifferentiable module.

In this paper, we propose a novel standard compatible end-

to-end image compression framework based on auxiliary codec

networks (ACNs). The ACNs are designed to imitate the

forward image degradation process of existing codecs in dif-

a standard compatible manner. In addition, a bit estimation

network (BENet) is proposed for training as a regularization

function to prevent undesired bit-rate increments. As recent

CRNet-based [35]–[39] generate models at a single level, the

proposed framework is also trained and optimized per codec

and rate.

The contributions of this paper are summarized as follows:

based on the prior of the image compression process to

effectively and precisely train the CRNet.

between the ACN, PPNet, and CRNet. Furthermore, the

and any off-the-shelf image restoration networks. Based

on the highly accurate ACNs for two standard codecs:

JPEG and HEVC, our framework exhibits state-of-the-art

results compared to other image compression algorithms,

II. RELATED WORK

rate and is optimized for minimum distortion.

neural network is revealed, many follow-up studies have been

conducted. Theis et al. [7] proposed a compressive auto-

encoder based on a residual neural network [15] and used a

Laplace-smoothed histogram as the entropy model. Ball

´

e et al.

[8] jointly optimized the entire model for rate-distortion per-

formance using a generalized divisive normalization transform.

Further, Ball

e et al. [9] proposed a hyperprior to effectively

although many deep network-based approach algorithms have

compression schemes due to compatibility. Furthermore, al-

though the state-of-the-art approaches outperform even the

Better Portable Graphics (BPG) [40] codec, which is designed

based on the intra-mode of HEVC, a significant performance

IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. XX, NO. X, X XXXX 3

downsampling method and restoring images using the deep

et al. [38] proposed an end-to-end framework that consists of

three parts: a compact CNN (ComCNN), an image codec, and

a reconstruction CNN (RecCNN). The ComCNN produces a

compact representation of an input image, and the RecCNN

of the compressed bitstream from the codec. Then, we defined

the following objective function to train f and g to minimize

the rate-distortion cost: