深度学习全面分析survey资源-CSDN文库

需积分: 48 65 浏览量 2018-10-29 22:39:03 上传评论 2 收藏 4.33MB PDF 举报

### 深度学习全面分析survey #### 一、引言与背景随着现代计算应用领域对高效解决方案的需求不断增长，深度神经网络（Deep Neural Networks, DNNs）已成为一个重要的工具。加速DNN训练是当前面临的一个重大挑战，解决这一问题的方法包括分布式算法到低级电路设计等多个方面。本文旨在从理论角度描述问题，并探讨DNN的并行化方法。此外，还讨论了DNN架构的发展趋势及其对并行策略的影响。 #### 二、并发分析在深度学习领域中，存在着多种类型的并发性，这些并发性可以从单个操作者层面到分布式深度学习系统层面进行探讨。 ##### 单一操作级别的并行单个操作级别是指在神经网络内部实现运算的并行化，比如矩阵乘法等基本算子可以通过GPU等硬件加速来实现。这种方法通常涉及利用现代硬件平台如GPU、TPU等的强大并行处理能力。 ##### 网络推理与训练中的并行网络推理与训练中的并行主要指的是在模型训练过程中的数据并行和模型并行两种方式： - **数据并行**：每个节点或设备拥有完整的模型副本，不同节点处理不同的数据集子集。这种方式适用于大数据集场景，可以显著加快训练速度。 - **模型并行**：对于大型模型而言，可以将模型分割成不同的部分部署在多个节点上，每个节点负责一部分模型的计算。这种方式在处理大规模复杂模型时尤其有效。 ##### 分布式深度学习分布式深度学习涉及到跨多个节点或设备之间的通信与协调，通常采用以下几种并行模式： - **同步模式**：所有节点完成一轮梯度计算后，更新模型参数。这种模式确保了全局模型的一致性，但可能受到“等待时间”较长的影响。 - **异步模式**：各节点根据自身进度更新模型参数，无需等待其他节点完成计算。这种方式可以提高计算效率，但可能会引入一致性问题。 #### 三、优化方法在分布式深度学习中，为了提高效率和效果，常常采用以下几种优化方法： ##### 异步随机优化异步随机优化是一种常见的方法，允许节点异步地更新全局模型参数。这种方法能够充分利用各个节点的计算资源，减少等待时间，但同时也需要解决因异步更新带来的不一致性问题。 ##### 分布式系统架构分布式系统架构的选择对于实现高效的深度学习训练至关重要。常见的架构包括： - **参数服务器架构**：通过中心化的参数服务器来协调多个工作节点的训练过程，负责参数的存储与分发。 - **AllReduce架构**：基于环形通信协议，每个节点都参与到参数的聚合过程中，从而避免了单一中心节点的瓶颈。 ##### 通信方案在分布式训练中，通信效率直接影响整体性能。常用的通信方案有： - **Gossip协议**：通过节点间随机的消息传递来达到参数同步的目的。 - **模型剪枝与参数压缩**：通过对模型进行剪枝或压缩，减少通信的数据量，从而加速训练过程。 #### 四、神经架构搜索神经架构搜索（Neural Architecture Search, NAS）是一种自动寻找最优模型结构的方法。随着计算资源的增加，NAS已经成为深度学习领域的一个研究热点。通过NAS，研究人员可以自动发现适合特定任务的最佳网络结构，进一步提高了模型的准确性和效率。 #### 五、未来方向基于现有的研究和发展趋势，未来的深度学习并行化可能朝着以下几个方向发展： 1. **更高效的通信机制**：开发新的通信协议和技术，进一步降低通信延迟和带宽消耗。 2. **自适应优化算法**：结合动态调整的学习率和其他超参数，实现更加灵活的优化过程。 3. **增强型神经架构搜索**：利用元学习等技术，提高NAS的效率和可扩展性。 4. **多模态融合**：探索如何将图像、文本等多种类型的数据有效地融合在一起，构建更加强大的多模态深度学习模型。随着硬件技术的进步和算法创新的不断涌现，深度学习领域的并行化技术将持续发展，为各种应用场景提供更加高效、智能的解决方案。

资源推荐

资源详情

资源评论

Demystifying Parallel and Distributed Deep Learning: An

In-Depth Concurrency Analysis

TAL BEN-NUN and TORSTEN HOEFLER, ETH Zurich, Switzerland

Deep Neural Networks (DNNs) are becoming an important tool in modern computing applications. Accelerating

their training is a major challenge and techniques range from distributed algorithms to low-level circuit

design. In this survey, we describe the problem from a theoretical perspective, followed by approaches for

its parallelization. We present trends in DNN architectures and the resulting implications on parallelization

strategies. We then review and model the dierent types of concurrency in DNNs: from the single operator,

through parallelism in network inference and training, to distributed deep learning. We discuss asynchronous

stochastic optimization, distributed system architectures, communication schemes, and neural architecture

search. Based on those approaches, we extrapolate potential directions for parallelism in deep learning.

CCS Concepts:

• General and reference →

Surveys and overviews ;

• Computing methodologies → Neural

networks; Parallel computing methodologies; Distributed computing methodologies;

Additional Key Words and Phrases: Deep Learning, Distributed Computing, Parallel Algorithms

ACM Reference Format:

Tal Ben-Nun and Torsten Hoeer. 2018. Demystifying Parallel and Distributed Deep Learning: An In-Depth

Concurrency Analysis. 47 pages.

1 INTRODUCTION

Machine Learning, and in particular Deep Learning

[143]

, is rapidly taking over a variety of aspects in

our daily lives. At the core of deep learning lies the Deep Neural Network (DNN), a construct inspired

by the interconnected nature of the human brain. Trained properly, the expressiveness of DNNs

provides accurate solutions for problems previously thought to be unsolvable, merely by observing

large amounts of data. Deep learning has been successfully implemented for a plethora of elds,

ranging from image classication

[108]

, through speech recognition

[6]

and medical diagnosis

[44]

, to

autonomous driving

[22]

and defeating human players in complex games

[215]

Since the 1980s, neural networks have attracted the attention of the machine learning commu-

nity

[144]

. However, DNNs’ rise into prominence was tightly coupled to the available computational

power, which allowed to exploit their inherent parallelism. Consequently, deep learning managed

to outperform all existing approaches in speech recognition

[147]

and image classication

[136]

, where

the latter (AlexNet) increased the accuracy by a factor of two, sparking interest outside of the

community and even academia.

As datasets increase in size and DNNs in complexity, the computational intensity and memory

demands of deep learning increase proportionally. Training a DNN to competitive accuracy today

Authors’ address: Tal Ben-Nun, talbn@inf.ethz.ch; Torsten Hoeer, htor@inf.ethz.ch, ETH Zurich, Department of Computer

Science, Zürich, 8006, Switzerland.

arXiv:1802.09941v2 [cs.LG] 15 Sep 2018

1:2 Tal Ben-Nun and Torsten Hoefler

Foundations (§2)

Supervised

Learning

Stochastic Gradient

Descent (SGD)

Work and Depth

Model

Message Passing

Interface (MPI)

The Efficiency Tradeoff: Generalization vs. Utilization (§3)

Operators (§4-5)

Concurrency

(§4-7)

Networks (§6) Training (§7)

Agent

AgentAgent

Agent

Param. Server

Fig. 1. Summary of Concurrency in Deep Learning

essentially requires a high-performance computing cluster. To harness such systems, dierent

aspects of training and inference (evaluation) of DNNs are modied to increase concurrency.

In this survey, we discuss the variety of topics in the context of parallelism and distribution

in deep learning, spanning from vectorization to ecient use of supercomputers. In particular,

we present parallelism strategies for DNN evaluation and implementations thereof, as well as

extensions to training algorithms and systems targeted at supporting distributed environments.

To provide comparative measures on the approaches, we analyze their concurrency and average

parallelism using the Work-Depth model

[21]

1.1 Related Surveys

Other surveys in the eld focus on applications of deep learning

[175]

, neural networks and their

history

[143,154,209,236]

, scaling up deep learning

[17]

, and hardware architectures for DNNs

[113,138,225]

In particular, three surveys

[143,209,236]

describe DNNs and the origins of deep learning method-

ologies from a historical perspective, as well as discuss the potential capabilities of DNNs w.r.t.

learnable functions and representational power. Two of the three surveys

[209,236]

also describe

optimization methods and applied regularization techniques in detail.

Bengio

[17]

discusses scaling deep learning from various perspectives, focusing on models, opti-

mization algorithms, and datasets. The paper also overviews some aspects of distributed computing,

including asynchronous and sparse communication.

Surveys of hardware architectures mostly focus on the computational side of training rather than

the optimization. This includes a recent survey

[225]

that reviews computation techniques for DNN

operators (layer types) and mapping computations to hardware, exploiting inherent parallelism.

The survey also includes discussion on data representation reduction (e.g., via quantization) to

reduce overall memory bandwidth within the hardware. Other surveys discuss accelerators for

traditional neural networks

[113]

and the use of FPGAs in deep learning

[138]

1.2 Scope

In this paper, we provide a comprehensive review and analysis of parallel and distributed deep

learning, summarized in Fig. 1 and organized as follows:

• Section 2 denes our terminology and algorithms.

• Section 3 discusses the tradeos between concurrency and accuracy in deep learning.

•

Section 4 describes DNN operators, how they are computed, and Section 5 shows how they

can be modied to expose concurrency.

•

Section 6 explores and analyzes the main approaches for parallelism in computations of full

networks for training and inference.

•

Section 7 provides an overview of distributed training, describing algorithm modications,

techniques to reduce communication, and system implementations.

• Section 8 gives concluding remarks and extrapolates potential directions in the eld.

Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis 1:3

Table 1. Yearly arXiv Papers in Computer Science (AI and Computer Vision)

Year 2012 2013 2014 2015 2016 2017

cs.AI 1,081 1,765 1,022 1,105 1,929 2,790

cs.CV 577 852 1,349 2,261 3,627 5,693

The paper surveys 240 other works, obtained by recursively tracking relevant bibliography from

seminal papers in the eld, dating back to the year 1984. We include additional papers resulting

from keyword searches on Google Scholar

and arXiv

. Due to the quadratic increase in deep

learning papers on the latter source (Table 1), some works may not have been included. The full

list of categorized papers in this survey can be found online

2 TERMINOLOGY AND ALGORITHMS

This section establishes theory and naming conventions for the material presented in the survey.

We rst discuss the class of supervised learning problems, followed by relevant foundations of

parallel programming.

2.1 Supervised Learning

In machine learning, Supervised Learning

[212]

is the process of optimizing a function from a set

of labeled samples (dataset) such that, given a sample, the function would return a value that

approximates the label. It is assumed that both the dataset and other, unobserved samples, are

sampled from the same probability distribution.

Throughout the survey, we refer to the operators

and

as the probability and expectation of

random variables;

z ∼ D

denotes that a random variable

is sampled from a probability distribution

; and

z∼D

[

f (z )

]

denotes the expected value of

f (z )

for a random variable

. The notations are

summarized in Table 2.

Formally, given a probability distribution of data

, random variable

z ∼ D

, a domain

where we

construct samples from, a label domain

, and a hypothesis class

containing functions

X → Y

we wish to minimize the generalization error, dened by the loss function

(f ) ≡ P

[

f (z ) , h(z)

]

where

h(z)

represents the true label of

. In practice, it is common to use a class

of functions

that are dened by a vector of parameters

(sometimes denoted as

), in order to dene a

continuous hypothesis space. For example,

may represent an N-dimensional hyperplane that

separates between samples of two classes, where

are its coecients. In deep neural networks,

we dene w in multiple layers, namely, w

l,i

is the parameter at layer l and index i.

We wish to nd w

∗

that minimizes the above loss function, as follows:

∗

= arg min

w ∈H

(

)

= arg min

w ∈H

z∼D

[

ℓ

(

w, z

)]

, (1)

where ℓ : H ×X → R

is the loss of an individual sample.

In this work, we consider two types of supervised learning problems, from which the sample loss

functions are derived: (multi-class) classication and regression. In the former, the goal is to identify

which class a sample most likely belongs to, e.g., inferring what type of animal appears in an image.

In regression, the goal is to nd a relation between the domains

and

, predicting values in

for

new samples in

. For instance, such a problem might predict the future temperature of a region,

given past observations.

https://scholar.google.com/

https://www.arxiv.org/

https://spcl.inf.ethz.ch/Research/Parallel_Programming/DistDL/

1:4 Tal Ben-Nun and Torsten Hoefler

Name Denition

D Data probability distribution

S Training dataset

w ∈ H Model parameters. w

(t )

denotes parameter i at

SGD iteration t

(z) Model function (learned predictor)

h(z) Ground-truth label (in Supervised Learning)

ℓ(w, z) Per-sample loss function

∇ℓ(w, z) Gradient of ℓ

u(д, w, t ) Parameter update rule. Function of loss gradient д,

parameters w, and iteration t

Table 2. Summary of Notations

0.54

0.28

0.02

0.07

0.03

0.04

0.02

Predicted Label True Label

Cat

Dog

Airplane

Truck

Strawberry

Horse

Bicycle





()

ℎ()

Sample



Cross-Entropy

ℓ(, )

Fig. 2. Multi-Class Classification Loss

For minimization purposes, a sample loss function

ℓ

should be continuous and dierentiable.

In regression problems, it is possible to use straightforward loss functions such as the squared

dierence

ℓ(w, z) =

(

(z) −h(z)

)

. On the other hand, in classication problems, a simple denition

of loss such as

ℓ(w, z) =

0 if

(z) = h(z)

or 1 otherwise (also known as binary or 0–1 loss), does

not match the continuity and dierentiability criteria.

To resolve this issue, prominent multi-class classication problems dene

as a probability

distribution of the inferred class types (see Fig. 2), instead of a single label. The model output is

typically normalized to a distribution using the softmax function

σ(z)

exp(z

)

exp(z

)

. The loss function

then computes the dierence of the prediction from the true label “distribution”, e.g., using cross-

entropy:

ℓ(w, z) = −

h(z)

log σ

(

(z)

)

. The cross-entropy loss can be seen as a generalization

of logistic regression, inducing a continuous loss function for multi-class classication.

Minimizing the loss function can be performed by using dierent approaches, such as iterative

methods (e.g., BFGS

[181]

) or meta-heuristics (e.g., evolutionary algorithms

[203]

). Optimization in

machine learning is prominently performed via Gradient Descent. Since the full

is, however,

never observed, it is necessary to obtain an unbiased estimator of the gradient. Observe that

∇L

(

)

= E

z∼D

[

∇ℓ

(

w, z

)]

(Eq. 1, linearity of the derivative). Thus, in expectation, we can descend

using randomly sampled data in each iteration, applying Stochastic Gradient Descent (SGD)

[207]

Algorithm 1 Stochastic Gradient Descent (SGD)

1: for t = 0 to T do ▷ SGD iterations

2: z ← Random element from S ▷ Sample dataset S

3: д ← ∇ℓ(w

(t)

, z) ▷ Compute gradient of ℓ

4: w

(t+1)

← w

(t)

+ u(д, w

(0, ...,t )

, t) ▷ Update weights with function u

5: end for

SGD (Algorithm 1) iteratively optimizes parameters dened by the sequence

(t)

}

t =0

, using

samples from a dataset

sampled from

with replacement. SGD is proven to converge at a rate

of O(1/

√

T ) for convex functions with Lipschitz-continuous and bounded gradient

[177]

Prior to running SGD, one must choose an initial estimate for the weights

(0)

. Due to the

ill-posed nature of some problems, the selection of

(0)

is important and may reect on the nal

quality of the result. The choice of initial weights can originate from random values, informed

decisions (e.g., Xavier initialization

[79]

), or from pre-trained weights in a methodology called

Transfer Learning

[191]

. In deep learning, recent works state that the optimization space is riddled

with saddle points

[143]

, and assume that the value of

(0)

does not aect the nal loss. In practice,

however, improper initialization may have an adverse eect on generalization as networks become

deeper

[92]

Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis 1:5

Table 3. Popular Weight Update Rules

Method Formula Denitions

Learning Rate w

(t +1)

= w

(t )

−η · ∇w

(t )

∇w

(t )

≡ ∇ℓ(w

(t )

, z)

Adaptive Learning Rate w

(t +1)

= w

(t )

−η

· ∇w

(t )

Momentum

[197]

(t +1)

= w

(t )

+ µ · (w

(t )

−w

(t −1)

) − η · ∇w

(t )

Nesterov Momentum

[178]

(t +1)

= w

(t )

+ v

t +1

= µ · v

−η · ∇ℓ(w

(t )

− µ ·v

, z)

AdaGrad

[68]

(t +1)

= w

(t )

−

η·∇w

(t )

√

i, t

+ε

i, t

τ =0



∇w

(t )



RMSProp

[95]

(t +1)

= w

(t )

−

η·∇w

(t )

′

i, t

+ε

′

i, t

= β ·A

′

t −1

+ (1 − β)



∇w

(t )



Adam

[130]

(t +1)

= w

(t )

−

η·M

(1)

i, t

(2)

i, t

+ε

(m)

i, t

·M

(m)

i, t −1

+(1−β

)



∇w

(t )



1−β

In line 1,

denotes the number of steps to run SGD for (known as the stopping condition or

computational budget). Typically, real-world instances of SGD run for a constant number of steps,

for a xed period of time, or until a desired accuracy is achieved. Line 2 then samples random

elements from the dataset, so as to provide the unbiased loss estimator. The gradient of the loss

function with respect to the weights

(t)

is subsequently computed (line 3). In deep neural networks,

the gradient is obtained with respect to each layer (

(t)

) using backpropagation (Section 4.2). This

gradient is then used for updating the weights, using a weight update rule (line 4).

2.1.1 Weight Update Rules. The weight update rule, denoted as

in Algorithm 1, can be dened

as a function of the gradient

, the previous weight values

(0)

, ··· , w

(t)

, and the current iteration

. Table 3 summarizes the popular

functions used in training. In the table, the basic SGD update

rule is

sдd

(д) = −η ·д

, where

represents the learning rate.

controls how much the gradient

values will overall aect the next estimate

(t+1)

, and in iterative nonlinear optimization methods

nding the correct

is a considerable part of the computation

[181]

. In machine learning problems,

it is customary to x

, or set an iteration-based weight update rule

alr

(д, t) = −η

·д

, where

decreases (decays) over time to bound the modication size and avoid local divergence.

Other popular weight update rules include Momentum, which uses the dierence between current

and past weights

(t)

−w

(t−1)

to avoid local minima and redundant steps with natural motion

[178,197]

More recent update rules, such as RMSProp

[95]

and Adam

[130]

, use the rst and second moments of

the gradient in order to adapt the learning rate per-weight, enhancing sparser updates over others.

Factors such as the learning rate and other symbols found in Table 3 are called hyper-parameters,

and are set before the optimization process begins. In the table,

µ, β, β

, and

represent the

momentum, RMS decay rate, and rst and second moment decay rate hyper-parameters, respectively.

To obtain the best results, hyper-parameters must be tuned, which can be performed by value

sweeps or by meta-optimization (Section 7.5.2). The multitude of hyper-parameters and the reliance

upon them is considered problematic by a part of the community

[199]

2.1.2 Minibatch SGD. When performing SGD, it is common to decrease the number of weight

updates by computing the sample loss in minibatches (Algorithm 2), averaging the gradient with

respect to subsets of the data

[141]

. Minibatches represent a tradeo between traditional SGD, which

is proven to converge when drawing one sample at a time, and batch methods

[181]

, which make

use of the entire dataset at each iteration.

In practice, minibatch sampling is implemented by shuing the dataset

, and processing

that permutation by obtaining contiguous segments of size

from it. An entire pass over the

dataset is called an epoch, and a full training procedure usually consists of tens to hundreds of such

剩余46页未读，继续阅读

评论收藏

内容反馈

L6084842

粉丝: 0
资源: 8

深度学习全面分析survey

Deep learning for sentiment analysis A survey 深度学习情感分析综述

深度强化学习中的泛化研究综述_A Survey of Generalisation in Deep Reinforcement

A General Survey on Attention Mechanisms in Deep Learning.pdf

深度元学习-A Survey of Deep Meta-Learning-翻译.docx

深度学习综述（英文）

survey-model:包提供用于调查系统的模型类

目标检测综述PPT-Object Detection in 20 Years: A Survey

Object Detection in 20 Years A Survey总结汇报.pptx

Object Detection in 20 Years A Survey

paper-survey::books:日语机器学习（尤其是深度学习）以往研究和相关工作的调查

Calculated.ellipsoid.arc.length.control.survey.rar_人工智能/神经网络/深度学习_C/C++_

语音情感分析survey

The Deep Learning Compiler A Comprehensive Survey.pdf

深度图相似学习综述（Deep Graph Similarity Learning：A Survey）.zip

使用深度学习进行海岸线监视的水下声学数据分类方法的综述（翻译版）

元学习综述，meta learning，A Survey of Zero-Shot Learning + poster

tidy-survey-aapor-2021:使用srvyr软件包进行R中的整理调查分析

A Survey of Deep Learning-based Object Detection.pdf

北航发布首篇「深度学习编译器」综述论文，36页pdf阐述DL编译器的全景.zip

A SURVEY ON DEEP LEARNING-BASED ARCHITECTUR.pdf

Deep Learning based Micro-expression Recognition A Survey.pdf

Deep Facial Expression Recognition A Survey.pdf

A survey on deep learning approaches for breast cancer diagnosi

Efficient Transformers： A Survey.pdf

A_Survey_on_Deep_Learning_in_Medical_Image_Analysis.pdf

Aesthetics, Personalization and Recommendation A survey on De

Deep Learning for Face Anti-Spoofing A Survey.zip

Deep Learning for Scene Classification A Survey.zip

Deep Learning based 3D Segmentation A Survey.pdf

最新资源