ANeuralEnhancementPost-ProcessorwithaDynamicAV1EncoderC资源-CSDN文库

需积分: 5 70 浏览量 2024-07-02 20:09:16 上传评论收藏 1.69MB PDF 举报

这篇文件是一篇关于视频压缩和增强的学术论文，标题为“A Neural Enhancement Post-Processor with a Dynamic AV1 Encoder Configuration Strategy for CLIC 2024”，作者是Darren Ramsook和Anil Kokaram，来自爱尔兰都柏林三一学院电子与电气工程系的Sigmedia Group。以下是这篇论文的核心内容概述：摘要：论文提出了一种结合神经网络后处理器和动态优化策略的新型视频压缩方法，旨在改善实际流媒体比特率下的视频压缩质量。神经后处理器通过对抗性训练进行优化，并使用感知损失函数，显著提升了视频保真度。实验结果显示，在50 kb/s和500 kb/s的比特率下，神经后处理器分别实现了+6.72和+1.81的VMAF（视频多方法评估融合）分数提升。 ### 知识点生成 #### 一、视频压缩与增强技术背景随着互联网技术的快速发展，数字视频内容的消费和分发经历了指数级增长。这一趋势背后的主要驱动力包括视频流媒体服务（如Netflix、YouTube等）以及视频会议平台（如Zoom、Teams等）的普及。为了满足这一需求，高效的视频压缩技术成为了必不可少的一环。“有损”压缩技术能够实现数据的有效存储、传输和交付，但同时在实用比特率下也引入了视觉伪影，降低了压缩视频的整体质量[2]。 #### 二、神经网络后处理器在该论文中，作者提出了一个结合神经网络后处理器和动态优化策略的新型视频压缩方案，旨在解决上述问题。神经网络后处理器利用生成对抗网络（Generative Adversarial Networks, GANs）的思想来提高压缩视频的质量。GANs是一种强大的工具，在图像处理任务中表现出了巨大的潜力，例如去噪[3, 4]、超分辨率重建[5, 6]等。因此，研究人员自然而然地将GANs应用于压缩伪影的去除上[7, 8]。 #### 三、动态AV1编码器配置策略论文中的另一个关键技术点是动态AV1编码器配置策略。AV1是一种开放源代码的视频编码格式，由开放媒体联盟开发，旨在提供比H.264/AVC更高的压缩效率，同时保持或优于H.265/HEVC的性能。动态优化策略是指根据视频内容的不同特性来调整编码参数，以达到更好的比特率/质量折衷。这种策略可以有效地减少伪影，提高压缩视频的整体质量。 #### 四、实验结果与分析论文中提到的神经后处理器经过对抗性训练进行优化，并使用感知损失函数，从而显著提高了视频保真度。实验结果显示，在50kb/s和500kb/s的比特率下，神经后处理器分别实现了+6.72和+1.81的VMAF（视频多方法评估融合）分数提升。这些结果表明，所提出的神经后处理器能够在较低的比特率下有效提升视频质量，对于流媒体应用来说尤其重要。 #### 五、结论与展望这篇论文提出了一种创新性的视频压缩方法，通过结合神经网络后处理器和动态AV1编码器配置策略，能够在低比特率下显著提升视频质量。这种方法不仅有望改善现有视频流媒体服务的表现，还可能对未来的视频通信技术产生深远影响。随着计算能力的不断提升和深度学习技术的进步，可以预见未来会有更多高效且高质量的视频压缩方案出现，进一步推动视频内容产业的发展。 ### 参考文献 1. **[1]** 未提供具体文献信息，请参考最新出版的相关研究文章。 2. **[2]** 针对视频压缩技术的局限性和挑战，参考相关专业书籍或综述文章。 3. **[3]** 例如：Mao, X., Li, Q., Xie, H., Yu, R., & Zhang, G. (2016). Least squares generative adversarial networks. arXiv preprint arXiv:1611.04076. 4. **[4]** 例如：Zhang, Y., Wang, X., Wu, J., & Huang, T. S. (2017). Beyond a Gaussian denoiser: Residual learning of deep CNN for image denoising. IEEE Transactions on Image Processing, 26(7), 3142-3155. 5. **[5]** 例如：Ledig, C., Theis, L., Huszár, F., Caballero, J., Cunningham, A., Acosta, A., ... & Aitken, A. N. (2017). Photo-realistic single image super-resolution using a generative adversarial network. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 105-114). 6. **[6]** 例如：Kim, J., Lee, J. K., & Lee, K. M. (2016). Accurate image super-resolution using very deep convolutional networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 1646-1654). 7. **[7]** 例如：Liu, Z., Wang, P., & Lu, J. (2018). Deep compression artifact reduction with generative adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops (pp. 0-0). 8. **[8]** 例如：Dong, C., Loy, C. C., & Tang, X. (2015). Image super-resolution using deep convolutional networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 38(2), 295-307. 以上内容基于论文摘要及部分介绍进行了详细的扩展和解释，旨在深入探讨论文中涉及的关键技术及其潜在应用价值。

资源推荐

资源详情

资源评论

A Neural Enhancement Post-Processor with a Dynamic AV1

Encoder Conﬁguration Strategy for CLIC 2024

Darren Ramsook

∗

and Anil Kokaram

†

Sigmedia Group,

Department of Electronic & Electrical Engineering,

Trinity College Dublin,

Dublin, Ireland.

∗

ramsookd@tcd.ie,

†

anil.kokaram@tcd.ie

Abstract

At practical streaming bitrates, traditional video compression pipelines frequently lead to

visible artifacts that degrade perceptual quality. This submission couples the eﬀectiveness

of a neural post-processor with a diﬀerent dynamic optimsation strategy for achieving an

improved bitrate/quality compromise. The neural post-processor is reﬁned via adversar-

ial training and employs perceptual loss functions. By optimising the post-processor and

encoder directly our method demonstrates signiﬁcant improvement in video ﬁdelity. The

neural post-processor achieves substantial VMAF score increases of +6.72 and +1.81 at

bitrates of 50 kb/s and 500 kb/s respectively.

Introduction

There has been an exponential growth in the consumption and distribution of dig-

ital video content due to the proliferation of video streaming and teleconferencing

platforms [1]. Video compression has become an essential component of the digi-

tal ecosystem. ”Lossy” compression compression techniques enable eﬃcient storage,

transmission, and delivery. However, at practical bitrates and with increasing picture

sizes, it introduces visual artifacts and compromises the overall quality of the com-

pressed video [2]. As a result, there is a pressing need for eﬀective methods to enhance

the quality of compressed videos and remove compression artifacts while preserving

important details and maintaining the ﬁdelity of the original content.

Generative Adversarial Networks (GANs) have emerged as a powerful tool in

image-related tasks, including denoising [3, 4] and super-resolution [5, 6]. As such,

researchers have naturally turned to GANs for addressing the challenges of compres-

sion artifact removal in still images [7, 8]. Despite the eﬀectiveness of GANs in still

image compression artifact removal, their application to video enhancement is still

in its early stages. Existing GAN-based architectures [9–11] for post-processing com-

pressed video enhancement focus solely on processing individual frames in isolation,

disregarding the temporal information present in videos. This approach overlooks the

obtained for all other uses, in any current or future media, including reprinting/republishing this

material for advertising or promotional purposes, creating new collective works, for resale or redis-

tribution to servers or lists, or reuse of any copyrighted component of this work in other works.

arXiv:2401.18021v1 [eess.IV] 31 Jan 2024

Figure 1: Encoding and Decoding Stages of our proposed process. Our resizing/retiming

step before encoding is done to ensure the input video meets a speciﬁc bitrate. We train

multiple neural post-processors which are dependent on the amount of downsampling which

is applied to the input video

inherent temporal dependencies among frames, which play a crucial role in capturing

and reproducing the motion patterns and coherent structures in videos. Consequently,

the generated videos often exhibit temporal inconsistencies, motion artifacts, and a

lack of temporal smoothness. Other neural based approaches for video processes en-

tire sequences at a time [12, 13]. While this approach allows for robust temporal

connections to be made across frames, this approach is limited by the memory of the

hardware and uses 3D convolutions which are much more computationally expensive.

As ﬁrst observed by Katsavounidis et al [14], it is possible to select a bitrate/quality

operating point (the encoded representation) by considering the creation of the bitrate

ladder itself as an optimisation task. We present an alternative strategy by using a

direct search technique that incorporates the speciﬁcation of the target bitrate as a

parameter as well. There has been signiﬁcant work that shows the value of opti-

mising a neural pre-processor as part of the pre-processor/encoder pipeline [15–17].

In general, post-processors are designed with respect to diﬀerent encoders but not

in conjunction with encoded representations. We therefore develop a scheme that

optimises each neural post-processor for every speciﬁc representation and associated

parameterisation. A key observation here is that in a practical application, constant

bitrate (CBR) encoding is employed to generate representations. However in single-

pass CBR encoding, the output bitrate rarely achieves the desired target. We explore

a method for achieving this bitrate by altering the target bitrate paremeter of an AV1

iteratively in a semi-multipass encoding scheme.

Our Contributions: In this work, we deploy libaom-av1 (version 3.6.1), an

open-source reference encoder of the AV1 standard [18], as the foundation of our video

compression pipeline outlined in Figure 1. For decoding, the dav1d (with commit id

58afe4) decoder is used followed by our neural post-processor. We present four key

components.

1. A speciﬁcation of an encoding step as in ﬁgure 1 in which the input content

is downsampled spatially and temporally and coupled with a bitrate target

parameter to achieve a target bitrate.

2. A strategy for achieving a target encoded bitrate by selecting the optimal reso-

lution, frame rate and target bitrate parameter in the actual encoder invocation.

3. Use of an adversarially trained post-processor incorporating both spatial and

temporal frame information as well as perceptual loss criteria. (See Figure 2)

4. Selecting an optimal post-processor for each diﬀerent encoder parameterisation.

Figure 2: Neural Post-processing network. The enhancement network (within the dotted

lines) takes three motion compensated frames as input. It also uses the nearest I-frame, a

degradation strength, x

, and the distance between the current frame and the nearest I-frame

∆

i. The perceptual critic is based of the architecture of [19].

1 Related Work

There has been substantial work on post-processor applications for improving com-

pressed still images. The use of perceptual metrics in the loss functions of neural-

networks has been proven to have better human subjective quality. In [20], results

indicated that training with either a DFQM or MS-SSIM has the highest perceptual

gain. In our training setup, we use the DFQM LPIPS as part of our generator loss

function. Using the diﬀerence of feature maps from intermediate layers of pre-trained

classiﬁcation networks as a loss has also been shown to give improved results in image

tasks [21, 22]. In [7], the use of this loss has shown to give increased performance in

JPEG compression reduction. We include this loss term when training our proposed

adversarial setup.

Previous models for video post-processing for compression artifact removal do

not exploit temporal information. The study conducted by [9] introduced a neural-

based model for video enhancement, which demonstrated improved Peak Signal-to-

Noise Ratio (PSNR) and Video Multimethod Assessment Fusion (VMAF) scores when

compared to videos without post-processing. Their approach focused on enhancing

the visual quality of videos through a neural network with multiple residual blocks.

In [10], a post-processing adversarial approach is presented, which incorporates a

mixture of multiple objective metrics including SSIM and MSSIM in its loss function.

Their approach includes the direct comparison of complete deep features between a

degraded-reference pair, similar to [7]. This model shows substantial improvement in

PSNR and VMAF.

However in [9, 10], the post-processing networks employed did not utilize tempo-

ral information across frames. Instead, it primarily focused on enhancing individual

剩余10页未读，继续阅读

评论收藏

内容反馈

码流怪侠

粉丝: 1w+
资源: 78

A Neural Enhancement Post-Processor with a Dynamic AV1 Encoder C

LSTM-Neural-Network-for-Time-Series-Prediction-master.rar

A simple neural attentive meta-learner_remotesensing_

neural-networks-and-deep-learning-zh_cn

a-gentle-introduction-to-neural-networks-with-python

Fast-Neural-Style-Transfer-master_style_深度学习项目_pytoch_图像风格转换_

neural-networks-and-deep-learning-master

A disciplined approach to neural network hyper-parameters Part I

Neural-Network-Toolbox 神经网络工具箱

Convolutional-Neural-Network-master.zip

Top- N Recommendation with A Neural Co-Attention Model.pdf

Deep-Learning-with-TensorFlow-Explore-neural-networks-with-Python.pdf.pdf

Neural-Network-master.zip

neural-networks-and-deep-learning

《Neural Networks -- A Comprehensive Foundation》配套资源

PyPI 官网下载 | neural-image-analogies-0.0.2.tar.gz

neural-style-painting, "A Neural Algorithm of Artistic Style"的实现.zip

Neural-Network-Projects-with-Python:Packt发布的《使用Python的神经网络项目》

_fuzzy-neural-network-theory-and-application.pdf

Matlab-neural-network-43-examples-matlab GUI仿真资源

AE插件-智能视频锐化填色降噪插件 Aescripts Neural Enhancement Suite v1.0.0 CPU+

A White Paper on Neural Network Quantization-2106.08295

Neural-network-based output-feedback adaptive dynamic surface control for a class of stochastic nonlinear time-delay systems with unknown control directions

基于神经网络的语音识别-Matlab-Speach-Recognition-Neural-Net-Matlab-Code.zip

Fundamentals-of-Neural-Networks-main.zip

Advanced_Neural_Network-1.0.0-py3-none-any.whl.zip

Hands-On-Neural-Network-Programming-with-CSharp-master.7z

neural-processes-master.zip

最新资源