TimeTravelinDeepLearningSpace:AnIntroduction资源-CSDN文库

需积分: 5 42 浏览量 2016-02-26 16:48:52 上传评论收藏 1.07MB PDF 举报

标题“Time Travel in Deep Learning Space: An Introduction”引出了本文的主题，即深度学习领域的发展历程。作者Haohan Wang和Bhiksha Raj通过追溯深度学习模型的历史演变，从最初的神经网络概念开始，逐步深入到深度学习的各个子领域，包括深度判别模型、深度生成模型以及神经网络在时间序列数据建模中的应用。描述中提到的内容涵盖了深度学习模型的发展、优化技巧以及在不同领域中的应用效果和所遇到的问题，并对需要注意的问题进行了深入分析。根据这些信息，我们可以总结以下知识点： 1. 神经网络的起源和发展：神经网络的起源可以追溯到对人脑连接主义（connectionism）的模拟，这是人工智能领域早期的一个重要学派。该学派强调通过大量的简单单元之间的相互连接来模拟大脑功能。神经网络的发展历程包括了早期的人工神经网络研究，比如感知器（Perceptron）的出现，它是一种被用来作为主成分分析器（Principal Component Analyzer）的简单神经网络。 2. 深度学习的初始阶段和感知器：感知器是一种二分类线性分类器，它在20世纪60年代和70年代初期受到了广泛的关注。感知器模型的提出者是人工智能和计算机科学的先驱之一，这些先驱的工作为后来的神经网络发展奠定了基础。 3. 神经网络与反向传播算法：反向传播算法是深度学习中一种重要的训练方法，它通过网络误差的反向传播来调整网络权重和偏置。这种算法使得多层神经网络的训练成为可能，使得神经网络能够逼近任意复杂的函数，即所谓的通用逼近定理（Universal Approximation Power）。 4. 深度判别模型的发展：卷积神经网络（CNN）作为深度判别模型的代表，是从神经网络的基础上发展起来的。文章还讨论了深度学习模型优化过程中的许多实用技术，例如层间初始化（Layerwise Initialization）或称作预训练（Pretraining），这些技术有助于神经网络的训练过程。 5. 深度生成模型的演进：研究者们在深度生成模型的发展上，必须平衡表示能力和计算复杂性之间的关系。受限玻尔兹曼机（Restricted Boltzmann Machine, RBM）和深度信念网络（Deep Belief Nets, DBN）是这一领域的关键进展。 6. 时间序列数据建模：文章将回顾使用神经网络对时间序列数据进行建模的发展历程，从时延神经网络（Time Delay Neural Networks）开始，逐步介绍到当前流行的循环神经网络（Recurrent Neural Networks, RNN）及其扩展长短期记忆网络（Long Short-Term Memory, LSTM）。同时，也会简要探讨深度循环神经网络的构建方法。 7. 深度神经网络的未来问题：报告最后提出了关于深度神经网络一些引人入胜的开放性问题，这些问题暗示了深度学习领域未来的研究方向。深度学习从其简单的神经网络起源，发展至今天的复杂模型，经历了从基础理论研究到应用实践的种种探索。目前，深度学习在图像识别、自然语言处理、时间序列预测等多个领域展现出了惊人的效果，但同时也面临着模型复杂度、训练难度、可解释性等挑战。未来的深度学习研究将需要在这些方面作出更深入的探索。

资源推荐

资源详情

资源评论

A Survey: Time Travel in Deep Learning Space:

An Introduction to Deep Learning Models

and How Deep Learning Models Evolved from

the Initial Ideas

Haohan Wang and Bhiksha Raj

Language Technologies Institute

School of Computer Science

Carnegie Mellon University

Pittsburgh, PA 15213

{haohanw, bhiksha}@cs.cmu.edu

Abstract

This report will show the history of deep learning evolves. It will trace back

as far as the initial belief of connectionism modelling of brain, and come back

to look at its early stage realization: neural networks. With the background of

neural network, we will gradually introduce how convolutional neural networks,

as a representative of deep discriminative models, is developed from neural net-

works, together with many practical techniques that can help in optimization of

neural networks. On the other hand, we will also trace back to see the evolution

history of deep generative models, to see how researchers balance the representa-

tion power and computation complexity to reach Restricted Boltzmann Machine

and eventually reach Deep Belief Nets. Further, we will also look into the devel-

opment history of modelling time series data with neural networks. We start with

Time Delay Neural Networks and move further to currently famous model named

Recurrent Neural Network and its extension Lone Time Short Memory. We will

also brieﬂy look into how to construct deep recurrent neural networks. Finally,

we will conclude this report with some interesting open-ended questions of deep

neural networks.

arXiv:1510.04781v1 [cs.LG] 16 Oct 2015

Contents

1 Introduction 4

2 From the Very Beginning to Neural Network 4

2.1 Connectionism and Early Stage of Artiﬁcial Neural Network . . . . . 4

2.1.1 Connectionism . . . . . . . . . . . . . . . . . . . . . . . . . 5

2.1.2 Early Stage Pioneers . . . . . . . . . . . . . . . . . . . . . . 5

2.1.3 Perceptron in Neural Network . . . . . . . . . . . . . . . . . 6

2.2 Perceptron as a Principal Component Analyzer . . . . . . . . . . . . 6

2.2.1 Perceptron . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

2.2.2 Perceptron as a Principal Component Analyzer . . . . . . . . 7

2.3 Neural Network, Backpropagation, Global Optimum and Universal Ap-

proximation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

2.3.1 Neural Network . . . . . . . . . . . . . . . . . . . . . . . . . 9

2.3.2 Backpropagation . . . . . . . . . . . . . . . . . . . . . . . . 9

2.3.3 Universal Approximation Power . . . . . . . . . . . . . . . . 10

3 Neural Networks Extended to Deep Discriminative Models 12

3.1 The Necessity of Depth . . . . . . . . . . . . . . . . . . . . . . . . . 12

3.1.1 Natural Motivation . . . . . . . . . . . . . . . . . . . . . . . 12

3.1.2 Computation Efﬁciency . . . . . . . . . . . . . . . . . . . . 12

3.1.3 Computation vs. Statistics . . . . . . . . . . . . . . . . . . . 13

3.2 Stacked Autoencoders . . . . . . . . . . . . . . . . . . . . . . . . . 14

3.2.1 Autoencoder . . . . . . . . . . . . . . . . . . . . . . . . . . 14

3.2.2 Stacked Autoencoders . . . . . . . . . . . . . . . . . . . . . 15

3.2.3 Layerwise Initialization (Pretraining) . . . . . . . . . . . . . 15

3.2.4 Stacked Sparse Autoencoders . . . . . . . . . . . . . . . . . 16

3.2.5 Stacked Denoising Autoencoders . . . . . . . . . . . . . . . 16

3.3 Convolutional Neural Network . . . . . . . . . . . . . . . . . . . . . 16

3.3.1 CNN Architecture . . . . . . . . . . . . . . . . . . . . . . . 17

3.3.2 Convolutional Layer . . . . . . . . . . . . . . . . . . . . . . 18

3.3.3 Subsampling Layer . . . . . . . . . . . . . . . . . . . . . . . 18

3.3.4 Relevant Techniques . . . . . . . . . . . . . . . . . . . . . . 19

3.4 Dropout & Maxout . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

3.4.1 Dropout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

3.4.2 Maxout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

4 Deep Generative Models 21

4.1 Self Organizing Map . . . . . . . . . . . . . . . . . . . . . . . . . . 21

4.2 Generative (Deep) Network Family . . . . . . . . . . . . . . . . . . . 23

4.3 Hopﬁeld Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

4.3.1 Energy of Hopﬁeld Network . . . . . . . . . . . . . . . . . . 25

4.3.2 Application of Hopﬁeld Network . . . . . . . . . . . . . . . 25

4.3.3 Learning of Hopﬁeld Network . . . . . . . . . . . . . . . . . 26

4.4 Boltzmann Machine . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

1 Introduction

This report is an introduction of deep learning models. Different from other deep learn-

ing tutorial, this report aims to draw a very comprehensive and yet brief picture about

how these deep learning models are developed. It is comprehensive because is covers

a lot of initial ideas about network models, like basic Neural Network, Self Oraganiz-

ing Map, Time Delay Neural Network. Yet it is brief because we cannot introduce

everything about deep learning models due to the fast development of this ﬁeld.

In this report, we will ﬁrst trace back to the very beginning of how researchers start

to model brain. With a belief of connectionism, a premier notion of neural networks

is formed. Further, artiﬁcial neural networks are introduced with the successful design

of perceptrons and backpropagation algorithm and start to play a dominating role in

machine learning area.

With the background of early stage work, the Section 3 is devoted to some famous

discriminative deep learning models. From the necessity of depth of models, we ﬁrst

move to stacked autoencoders and then introduce convolutional neural networks and

some of its inﬁnitely many optimizing techniques. Most of these techniques could be

generalized to other deep learning models.

Section 4 is devoted to generative models. Again, we travel through a long journey

from the early stage work like self organizing map and Hopﬁeld network, to the middle

age Boltzmann Machine and Restricted Boltzmann Machine, and to nowadays Deep

Belief Nets, which is a natural extension of earlier work.

Then, Section 5 is a collection of neural network models for time series data, or

generally sequential data. As a pioneer, Time Delay Neural Network is introduced

ﬁrst, followed by Recurrent Neural Network and the famous model called Long Short

Term Memory (LSTM). Then, an intuition of how these models could be beneﬁted

from deep representation learning is introduced at the end of that section.

Last but not least, Section 6 is about some other interesting topics of deep learning

and some topics that are yet to be fully explored when this report is prepared. Here, we

will actually show some validation for the claim that deep learning is the best model

so far. However, we will also see some ﬂaws of the deep learning, that may possibility

hinder its further development.

2 From the Very Beginning to Neural Network

2.1 Connectionism and Early Stage of Artiﬁcial Neural Network

This very ﬁrst section is devoted to the very initial stage of Artiﬁcial Neural Network.

It is built on people’s one point of view of understanding human mental phenomena:

connectionism. The remaining is about the pioneer work of neural network modeling,

how the information is passed and stored in the network and the early mathematics

model. These works may not be directly useful for understanding neural network.

However, we believe this whole process will be helpful for the researchers who are

interested in pushing the deep learning work forward.

2.1.1 Connectionism

It is a long way for researchers and philosophers to go in understanding how human

brain and neural system function. Finally, they end up in two different, yet both inter-

esting tracks: connectionism [10] and computationalism[8]. Needless to mention that

which path is more promising or which path is the subset of the other, here we only fo-

cus on the connectionism points of view, where people believe that human intellectual

abilities can be explained with a network of very simple units (neuron).

2.1.2 Early Stage Pioneers

Figure 1: An example of Bain’s neural groupings structure. (Figure from [1])

Relevant work started in 19th century. To make it easier to understand, it starts out

as a philosophy question, rather than a computer science problem or a biology problem.

Alexander Bain ﬁrst mentioned the connectionism ideas in [1]. Even earlier, he

offered an interpretation of mental phenomena within an association framework. Then,

in his book, he argues a brand new point of view stating memory is a model capable

of ”putting required stuff together” rather than ”storing everything beforehand”. He

named this memory structure as ”neural groupings”, as a predecessor of ”neural net-

work” in our world, and further explained that under his view, connected nerve ﬁbers

are channeled to different parts of the network under different simulations.

As a very simple example, showed in Figure 1. X, Y , Z is triggered by different

combinations of simulations from a, b or c.

Bain’s work is recognized as the earliest work of discovering neural networks of

human brain, however, Bain was not convinced by himself and argued that this structure

shows no practical value.

Donald O. Hebb was inﬂuenced by Bain. In 1949, he stated the famous rule: ”Cells

that ﬁre together, wire together” [18], which emphasized on the activation behavior of

co-ﬁred cells. This theory attempts to explain neural groupings theory and provides a

biological basis for learning methods in the rehabilitation of memory. In a more formal

way, it says that when an axon of cell A is near enough to excite cell B and repeatedly

or persistently takes part in ﬁring it, some growth process or metabolic change takes

place in one or both cells such that As efﬁciency as one of the cells ﬁring B is increased.

剩余42页未读，继续阅读

评论收藏

内容反馈

nxzlj

粉丝: 6
资源: 12

Time Travel in Deep Learning Space: An Introduction

timeTravel书籍

characteristics:不同编码下的字符信息

acm贪心 gone fishing

Female Characteristics: How Gender Differences Influence Best Answers in Online Travel Q&A Community

Differences

Introduction to Deep Learning.pdf

An Introduction to Deep Learning for the Physical Layer

Introduction to Deep Learning Using R

DFT的matlab源代码-Travel_Time_prediction_with-deep-learning:在英国行驶的车辆的行驶时间预测

2022最新版：TRAVEL TOUR V4.3.2主题：旅游预订，旅游预订主题.rar

CRACKING CODES WITH PYTHON An Introduction to Building and Breaking Ciphers

OTC_Learning_Guides-Introduction

React-Bacon-Timetravel-Example:带有Timetravel示例的React.js + Bacon.js

Deep Learning: An Introduction from the NLP Perspective

Deep Learning with Python: A Hands-on Introduction

Deep Learning with Python A Hands-on Introduction

Oriented prestack time migration

timetravel:一个用 JavaScript 模拟时间的小模块

multi-agent systems for traffic and transportation engineering

Introduction to Apache Flink

Expectations about recipients' prosociality and mental time travel relate to。。。。

An Introduction to Deep Reinforcement Learning.pdf

Introduction to Deep Learning with Python

Introduction to Deep Learning with Python

最新资源