ASurveyofBatchNormalization（原理、背景、算法、演变、优缺点）资源-CSDN文库

需积分: 11 60 浏览量 2023-01-24 19:29:33 上传评论收藏 1.23MB PDF 举报

A Survey of BatchNormalization（原理、背景、算法、演变、优缺点）。Batch Normalization, 批标准化, 和普通的数据标准化类似, 是将分散的数据统一的一种做法, 也是优化神经网络的一种方法. Batch Normalization 将神经网络每层的输入都调整到均值为 0，方差为 1 的标准正态分布，其目的是解决神经网络中梯度消失的问题. BN操作使得原本偏移的特征数据，如5-11的第一个图，重新拉回到0均值，使进入激活函数的数据分布在激活函数线性区，使得输入数据的微小变化，更明显的体现到激活函数的输出，提升了激活函数对输入数据的区分力。《批标准化：深入理解与应用分析》深度学习在当今的计算机科学领域中扮演着至关重要的角色，尤其在图像识别、语音处理等多个领域取得了显著成就。然而，随着网络层次的增加，训练深度神经网络时面临的一大挑战就是梯度消失问题。随机梯度下降（SGD）作为主流的优化算法，虽然简洁高效，但对超参数的选择，如权重衰减系数、dropout比例以及学习率等，要求极其精确，否则可能导致训练过程复杂且收敛速度变慢。为了解决这些问题，批标准化（Batch Normalization, BN）应运而生。BN是一种旨在加速模型收敛速度并缓解深层网络中梯度消失问题的技术。它通过对每一层神经网络的输入进行规范化，使其均值接近于0，方差接近于1，从而使数据分布更为集中，便于优化。这一过程能够使得特征数据的微小变化在经过激活函数后有更明显的反映，增强了激活函数对输入数据的敏感性，提高了网络的区分能力。批标准化的核心思想是，对于每一层的输入，先计算当前批次数据的均值和方差，然后将每个样本的特征值按照这个均值和方差进行标准化，最后通过可学习的缩放和平移参数（γ和β）进行反规范化，以保持网络的表达能力。BN不仅改善了数据的分布，还通过稳定内部协变量转移（Internal Covariate Shift）来加速学习过程，使得模型在早期就能更快地捕获有用的信息。 BN的演变历程中，包括了对微型批次（mini-batch）的处理策略、动态调整批大小的适应性BN（Adaptive BatchNorm）以及在网络架构中的各种融合形式，如Inception模块中的BN融合等。这些发展进一步提升了BN在不同场景下的适用性和性能。尽管批标准化带来了诸多优势，但也存在一些潜在的问题。例如，BN在小批量或单样本的情况下表现可能不佳，因为计算的均值和方差可能不够稳定。此外，在某些任务，如生成对抗网络（GANs）中，BN可能会干扰生成样本的多样性。因此，研究人员提出了替代方案，如实例标准化（Instance Normalization）和组标准化（Group Normalization），以适应特定场景的需求。批标准化作为深度学习中的关键组件，极大地推动了神经网络的训练效率和性能。它不仅解决了深度网络中的梯度消失问题，还通过规范化输入数据，提升了模型的泛化能力和训练稳定性。然而，随着技术的不断发展，我们需要不断探索和改进，以应对批标准化所带来的挑战，以及寻找更适合不同任务的标准化方法。

资源推荐

资源详情

资源评论

A Survey of Batch Normalization

Dingkun Liu

Stuno:2200400616

Date:2022-12-05

Abstract

1 Basic introduction of Batch Normalization

1.1 Overview

In recent years, deep learning has been extremely popular[1], and has achieved great success in

vision, speech and many other ﬁelds. Stochastic gradient descent (SGD) has become the mainstream

method for training deep neural networks[2]. Although SGD is simple and efﬁcient for training deep

neural networks, it is criticized for the need to carefully select the hyperparameters of the model,

such as: weight decay coefﬁcient[3], dropout ratio[4], especially the learning used in optimization

Rate and model parameter initialization. Since the input of each layer is affected by the parameters of

all previous layers, when the network becomes deeper, small changes in the network parameters will

be gradually ampliﬁed, which makes the training more and more complicated and the convergence

slower and slower . This is a near-essential problem in the ﬁeld of deep learning. Many papers

have proposed solutions, such as: RELU activation function[5], Residual Network[6], etc. BN is

essentially one of the currently widely used solutions[7]. BN is a deep neural network training

technique, which can not only speed up the convergence speed of the model, but more importantly,

alleviate the problem of gradient disappearance in the deep network to a certain extent, thus making

the training of the deep network model easier and more stable.

The essence of deep learning is to describe the internal complex structural characteristics of the

problem and perform arbitrary nonlinear transformations. Simply put, it can be regarded as the

distribution of learning data. There is a very important assumption in the ﬁeld of machine learning:

the independent and identical distribution assumption, that is, it is assumed that the training data

and the test data satisfy the same distribution, which is a basic guarantee that the model obtained

through the training data can obtain good results in the test set. If the distribution of the training data

is different from the test data, then the generalization ability of the network will be greatly reduced.

On the other hand, if the distribution of each batch of training data is different, the network will have

to learn to adapt to different distributions in each iteration, then the training speed of the network

will be greatly reduced, which is why we need to do something to the input data. A normalization

process.

"Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift"

mentioned the motivation and innovation of Batch Normalization at the beginningTraining Deep

Neural Networks is complicated by the fact that the distribution of each layers inputs changes dur-

ing training, as the parameters of the previous layers change. This slows down the training by

requiring lower learning rates and careful parameter initialization, and makes it notoriously hard

to train models with saturating nonlinearities. We refer to this phenomenon as internal covariate

shift, and address the problem by normalizing layer inputs[8]. Our method draws its strength from

making normalization a part of the model architecture and performing the normalization for each

training mini-batch. Batch Normalization allows us to use much higher learning rates and be less

careful about initialization. It also acts as a regularizer, in some cases eliminating the need for

Preprint. Under review.

Dropout. Applied to a state-of-the-art image classiﬁcation model, batch Normalization achieves the

same accuracy with 14 times fewer training steps, and beats the original model by a signiﬁcant mar-

gin. Using an ensemble of batch normalized networks, they improve upon the best published result

on ImageNet classiﬁcation[9]: reaching 4.9% top-5 validation error (and 4.8% test error), exceeding

the accuracy of human raters.

1.2 Dataset introduction

The ImageNet project is a large visual database for the study of visual object recognition software.

Over 14 million image urls are manually annotated by ImageNet to indicate objects in the picture;

In at least a million images, a bounding box is also provided. ImageNet contains more than 20,000

categories; A typical category, such as "balloon" or "strawberry," contains hundreds of images. An

annotated database of third-party image urls is freely available directly from ImageNet; However,

the actual image does not belong to ImageNet. Since 2010, the ImageNet project has held an annual

software competition, the ImageNet Large-scale Visual Recognition Challenge (ILSVRC), in which

software programs compete to correctly classify objects and scenes. The ImageNet challenge uses

a "pruned" list of 1000 non-overlapping classes. 2012 was a huge breakthrough in solving the

ImageNet challenge and is widely regarded as the beginning of the deep learning revolution in 2010.

An example of a picture is shown in Figure 1.

Figure 1: The sample pictures of ImageNet dataset.

Compared with CIFAR-10[10], ImageNet dataset has more images, higher resolution, more cate-

gories (more than one image category), and more irrelevant noise and changes in the image, so the

difﬁculty of identiﬁcation is much higher than CIFAR-10. ILSVRC uses a subset of ImageNet, with

a total of about 1.2 million training images, 50,000 validation images, and 150,000 test images;

1000 category tag. The Large Scale Visual Recognition Challenge, ILSVRC(2012 2017), is a con-

test based on the dataset of 1000 categories. Training set 1.2 million images. Each year since 2010,

ImageNets project organization has held an Imagenetbased lmageNet Large Scale VisualRecogni-

tion Challenge (ILSVRC). ILSVRC competition has produced many successful image recognition

methods, many of which are deep learning methods, and they will be further developed and applied

after the competition. It can be said that the ImageNet dataset and ILSVRC competition have greatly

promoted the development of computer vision technology and even deep learning, playing a pivotal

role in the tide of deep learning.

1.3 Position of Batch Normalization

In the original paper of Batch Normalization, BN is placed in front of the nonlinear activation

layer[11]. We add the BN transform immediately before the nonlinearity. However, François Chollet

broke the news that Christian, one of the authors of the BN paper, put BN behind ReLU, it can guar-

antee that recent code written by Christian applies relu before BN[12]. In addition, Jeremy Howard

directly advocates putting BN behind nonlinear activation, You want the batchnorm after the non-

linearity, and before the dropout. "Should" come ﬁrst or later? At present, in practice, BN tends to

be placed in front of ReLU. There are also evaluations that BN is better after ReLU. Here I try to

make some explanations. BN is Batch Normalization. The purpose of this type of normalization

technology is to stabilize the distribution of each layer, that is, to normalize the input before passing

it to the neural network. This normalization operates on the input and is performed before the input

layer. Then, from this perspective, Batch Normalization can be regarded as the normalization of the

input to the hidden layer. Imagine that if we cut off all the network layers in front of a certain hidden

layer in the network, then this hidden layer becomes the input layer, and the input passed to it needs

normalization. In this respect, it is natural for the BN layer to be activated after nonlinear activation.

Then, let’s consider some speciﬁc activation functions shown in Figures 2, 3, and 4. We can see that

whether it is the change of tanh or sigmoid in x, the change of y is very small (tanh is the stretched

sigmoid). That is, it is prone to the problem of gradient decay. Then, if some normalization process-

ing is performed before tanh or sigmoid, the problem of gradient disappearance can be alleviated.

This may also be the reason why the original BN paper chose to place the BN layer before the

nonlinear activation.

Figure 2: Schematic diagram of tanh activation function.

Although the original Batch Normalization paper also experimented on Inception using ReLU, it

ﬁrst studied sigmoid activation. Therefore, when experimenting with ReLU, I guess the author may

have continued the previous conﬁguration of putting BN in front, instead of dealing with ReLU

separately. To sum up, the mechanism of the BN layer may be to help the stochastic gradient

descent by smoothing the distribution of the hidden layer input, and to alleviate the negative impact

of the stochastic gradient descent weight update on the subsequent layers. In fact, it may be able to

play this role whether it is before or after nonlinear activation. However, depending on the speciﬁc

activation function, the effect may be a little different (for example, for sigmoid and tanh, before

putting nonlinear activation, it may also alleviate the gradient decay problem of sigmoid/tanh by the

way, while for ReLU , this smoothing effect may be weakened after being "distorted" by ReLU).

2 Towards Reducing Internal Covariate Shift

2.1 Internal Covariate Shift

Why is training a deep neural network model difﬁcult? One of the important reasons is that the

deep neural network involves the superposition of many layers, and the parameter update of each

layer causes the input data distribution of the upper layer to change. Through the layer-by-layer

superposition[13], the input distribution of the upper layer will change very drastically, which makes

the upper layer Need to constantly re-adapt to the underlying parameter updates. In order to train

剩余15页未读，继续阅读

评论收藏

内容反馈

天使DiMaría

粉丝: 1w+
资源: 53

A Survey of BatchNormalization（原理、背景、算法、演变、优缺点）

A survey of computational location privacy

A Survey of Modern Algebra, Garrett Birkhoff

《A Survey of the Recent Architectures of Deep Convolutional Neural Networks》

A Survey of Monte Carlo Tree Search Methods

A Survey of Nonlinear Attitude Estimation Methods

A Survey of Fault Diagnosis and Fault-Tolerant Techniques 1

A Survey of the Recent Architectures of Deep Convolutional Neural Networks.pdf

Batch Normalization主要讲解

A survey of applications of second-order sliding .pdf

a survey of alternating permutations.pdf

Batch Normalization简介

A Survey of Graph Theoretical Approaches to

A Survey of Deep Learning-based Object Detection.pdf

A Survey of 5G Network Architecture and Emerging Technologies

2021 - A Survey of Binary Code Similarity.pdf

Imitation Learning A Survey of Learning Methods.pdf

A Survey of FPGA-Based LDPC Decoders

A Survey of Outlier Detection Methodologies

A survey of recent advances in visual feature detection

A survey of dimensionality reduction techniques

A survey of machine learning by Carl Burch 机器学习 综述

A survey of community detection methods in multilayernetworks.pdf

Survey of maneuvering target tracking: II. Ballistic target models

A survey of algorithms for dense subgraph discovery

About Face A Survey of Facial Recognition Evaluation.pdf

元学习综述，meta learning，A Survey of Zero-Shot Learning + poster

最新资源

A survey of machine learning by Carl Burch 机器学习综述