ATourofTensorFlow资源-CSDN文库

TensorFlow

需积分: 10 23 浏览量 2017-09-27 13:36:22 上传评论收藏 480KB PDF 举报

资源详情

资源评论

A Tour of TensorFlow

Proseminar Data Mining

Peter Goldsborough

Fakultät für Informatik

Technische Universität München

Email: peter.goldsborough@in.tum.de

Abstract— Deep learning is a branch of artiﬁcial intelligence

employing deep neural network architectures that has signiﬁ-

cantly advanced the state-of-the-art in computer vision, speech

recognition, natural language processing and other domains. In

November 2015, Google released TensorFlow, an open source deep

learning software library for deﬁning, training and deploying

machine learning models. In this paper, we review TensorFlow

and put it in context of modern deep learning concepts and

software. We discuss its basic computational paradigms and

distributed execution model, its programming interface as well

as accompanying visualization toolkits. We then compare Ten-

sorFlow to alternative libraries such as Theano, Torch or Caffe

on a qualitative as well as quantitative basis and ﬁnally comment

on observed use-cases of TensorFlow in academia and industry.

Index Terms— Artiﬁcial Intelligence, Machine Learning, Neu-

ral Networks, Distributed Computing, Open source software,

Software packages

I. INTRODUCTION

Modern artiﬁcial intelligence systems and machine learn-

ing algorithms have revolutionized approaches to scientiﬁc

and technological challenges in a variety of ﬁelds. We can

observe remarkable improvements in the quality of state-of-

the-art computer vision, natural language processing, speech

recognition and other techniques. Moreover, the beneﬁts of

recent breakthroughs have trickled down to the individual,

improving everyday life in numerous ways. Personalized dig-

ital assistants, recommendations on e-commerce platforms,

ﬁnancial fraud detection, customized web search results and

social network feeds as well as novel discoveries in genomics

have all been improved, if not enabled, by current machine

learning methods.

A particular branch of machine learning, deep learning,

has proven especially effective in recent years. Deep learning

is a family of representation learning algorithms employing

complex neural network architectures with a high number

of hidden layers, each composed of simple but non-linear

transformations to the input data. Given enough such trans-

formation modules, very complex functions may be modeled

to solve classiﬁcation, regression, transcription and numerous

other learning tasks [1].

It is noteworthy that the rise in popularity of deep learning

can be traced back to only the last few years, enabled primarily

by the greater availability of large data sets, containing more

training examples; the efﬁcient use of graphical processing

units (GPUs) and massively parallel commodity hardware to

train deep learning models on these equally massive data

sets as well as the discovery of new methods such as the

rectiﬁed linear unit (ReLU) activation function or dropout as

a regularization technique [1]–[4].

While deep learning algorithms and individual architectural

components such as representation transformations, activa-

tion functions or regularization methods may initially be

expressed in mathematical notation, they must eventually be

transcribed into a computer program for real world usage.

For this purpose, there exist a number of open source as

well as commercial machine learning software libraries and

frameworks. Among these are Theano [5], Torch [6], scikit-

learn [7] and many more, which we review in further detail

in Section II of this paper. In November 2015, this list was

extended by TensorFlow, a novel machine learning software

library released by Google [8]. As per the initial publication,

TensorFlow aims to be “an interface for expressing machine

learning algorithms” in “large-scale [. . . ] on heterogeneous

distributed systems” [8].

The remainder of this paper aims to give a thorough review

of TensorFlow and put it in context of the current state of

machine learning. In detail, the paper is further structured

as follows. Section II will provide a brief overview and

history of machine learning software libraries, listing but

not comparing projects similar to TensorFlow. Subsequently,

Section III discusses in depth the computational paradigms

underlying TensorFlow. In Section IV we explain the current

programming interface in the various supported languages. To

inspect and debug models expressed in TensorFlow, there exist

powerful visualization tools, which we examine in Section

V. Section VI then gives a comparison of TensorFlow and

alternative deep learning libraries on a qualitative as well as

quantitative basis. Before concluding our review in Section

VIII, Section VII studies current real world use cases of

TensorFlow in literature and industry.

II. HISTORY OF MACHINE LEARNING LIBRARIES

In this section, we aim to give a brief overview and key

milestones in the history of machine learning software li-

braries. We begin with a review of libraries suitable for a wide

range of machine learning and data analysis purposes, reaching

back more than 20 years. We then perform a more focused

study of recent programming frameworks suited especially to

the task of deep learning. Figure 1 visualizes this section in

a timeline. We wish to emphasize that this section does in no

arXiv:1610.01178v1 [cs.LG] 1 Oct 2016

way compare TensorFlow, as we have dedicated Section VI to

this speciﬁc purpose.

A. General Machine Learning

In the following paragraphs we list and brieﬂy review a

small set of general machine learning libraries in chronolog-

ical order. With general, we mean to describe any particular

library whose common use cases in the machine learning and

data science community include but are not limited to deep

learning. As such, these libraries may be used for statisti-

cal analysis, clustering, dimensionality reduction, structured

prediction, anomaly detection, shallow (as opposed to deep)

neural networks and other tasks.

We begin our review with a library published 21 years

before TensorFlow: MLC++ [9]. MLC++ is a software library

developed in the C++ programming language providing algo-

rithms alongside a comparison framework for a number of

data mining, statistical analysis as well as pattern recognition

techniques. It was originally developed at Stanford University

in 1994 and is now owned and maintained by Silicon Graphics,

Inc (SGI

). To the best of our knowledge, MLC++ is the oldest

machine learning library still available today.

Following MLC++ in the chronological order, OpenCV

(Open Computer Vision) was released in the year 2000 by

Bradski et al. [10]. It is aimed primarily at solving learning

tasks in the ﬁeld of computer vision and image recognition,

including a collection of algorithms for face recognition,

object identiﬁcation, 3D-model extraction and other purposes.

It is released under a BSD license and provides interfaces in

multiple programming languages such as C++, Python and

MATLAB.

Another machine learning library we wish to mention is

scikit-learn

[7]. The scikit-learn project was originally devel-

oped by David Cournapeu as part of the Google Summer of

Code program

in 2008. It is an open source machine learning

library written in Python, on top of the NumPy, SciPy and

matplotlib frameworks. It is useful for a large class of both

supervised and unsupervised learning problems.

The Accord.NET

library stands apart from the aforemen-

tioned examples in that it is written in the C# (“C Sharp”)

programming language. Released in 2008, it is composed not

only of a variety of machine learning algorithms, but also

signal processing modules for speech and image recognition

[11].

Massive Online Analysis

(MOA) is an open source frame-

work for online and ofﬂine analysis of massive, potentially

inﬁnite, data streams. MOA includes a variety of tools for

classiﬁcation, regression, recommender systems and other

disciplines. It is written in the Java programming language

https://www.sgi.com/tech/mlc/

http://opencv.org

http://scikit-learn.org/stable/

https://summerofcode.withgoogle.com

http://accord-framework.net/index.html

http://moa.cms.waikato.ac.nz

and maintained by staff of the University of Waikato, New

Zealand. It was conceived in 2010 [12].

The Mahout

project, part of Apache Software Foundation

is a Java programming environment for scalable machine

learning applications, built on top of the Apache Hadoop

plat-

form. It allows for analysis of large datasets distributed in the

Hadoop Distributed File System (HDFS) using the MapReduce

programming paradigm. Mahout provides machine learning

algorithms for classiﬁcation, clustering and ﬁltering.

Pattern

is a Python machine learning module we include

in our list due to its rich set of web mining facilities. It com-

prises not only general machine learning algorithms (e.g. clus-

tering, classiﬁcation or nearest neighbor search) and natural

language processing methods (e.g. n-gram search or sentiment

analysis), but also a web crawler that can, for example, fetch

Tweets or Wikipedia entries, facilitating quick data analysis on

these sources. It was published by the University of Antwerp

in 2012 and is open source.

Lastly, Spark MLlib

is an open source machine learning

and data analysis platform released in 2015 and built on top

of the Apache Spark

project [13], a fast cluster computing

system. Similar to Apache Mahout, it supports processing

of large scale distributed datasets and training of machine

learning models across a cluster of commodity hardware. For

this, it includes classiﬁcation, regression, clustering and other

machine learning algorithms [14].

B. Deep Learning

While the software libraries mentioned in the previous

section are useful for a great variety of different machine

learning and statistical analysis tasks, the following paragraphs

list software frameworks especially effective in training deep

learning models.

The ﬁrst and oldest framework in our list suited to the

development and training of deep neural networks is Torch

released already in 2002 [6]. Torch consisted originally of

a pure C++ implementation and interface. Today, its core

is implemented in C/CUDA while it exposes an interface

in the Lua

scripting language. For this, Torch makes use

of a LuaJIT (just-in-time) compiler to connect Lua routines

to the underlying C implementations. It includes, inter alia,

numerical optimization routines, neural network models as

well as general purpose n-dimensional array (tensor) objects.

Theano

, released in 2008 [5], is another noteworthy deep

learning library. We note that while Theano enjoys greatest

popularity among the machine learning community, it is, in

essence, not a machine learning library at all. Rather, it is a

http://mahout.apache.org

http://www.apache.org

http://hadoop.apache.org

http://www.clips.ua.ac.be/pages/pattern

http://spark.apache.org/mllib

http://spark.apache.org/

http://torch.ch

https://www.lua.org

http://deeplearning.net/software/theano/

1992

1994

1996

1998

2000

2002

2004

2006

2008

2010

2012

2014

2016

MLC++

OpenCV

Torch

scikit

Accord

Theano

MOA

Mahout

Pattern

DL4J

Caffe

cuDNN

TensorFlow

MLlib

Fig. 1: A timeline showing the release of machine-learning libraries discussed in section I in the last 25 years.

programming framework that allows users to declare math-

ematical expressions symbolically, as computational graphs.

These are then optimized, eventually compiled and ﬁnally

executed on either CPU or GPU devices. As such, [5] labels

Theano a “mathematical compiler”.

Caffe

is an open source deep learning library maintained

by the Berkeley Vision and Learning Center (BVLC). It

was released in 2014 under a BSD-License [15]. Caffe is

implemented in C++ and uses neural network layers as its

basic computational building blocks (as opposed to Theano

and others, where the user must deﬁne individual mathematical

operations making up layers). A deep learning model, con-

sisting of many such layers, is stored in the Google Protocol

Buffer format. While models can be deﬁned manually in this

Protocol Buffer “language”, there exist bindings to Python

and MATLAB to generate them programmatically. Caffe is

especially well suited to the development and training of

convolutional neural networks (CNNs or ConvNets), used

extensively in the domain of image recognition.

While the aforementioned machine learning frameworks

allowed for the deﬁnition of deep learning models in Python,

MATLAB and Lua, the Deeplearning4J

(DL4J) library

enables also the Java programmer to create deep neural

networks. DL4J includes functionality to create Restricted

Boltzmann machines, convolutional and recurrent neural net-

works, deep belief networks and other types of deep learning

models. Moreover, DL4J enables horizontal scalability using

distributed computing platforms such as Apache Hadoop or

Spark. It was released in 2014 by Adam Gibson under an

Apache 2.0 open source license.

Lastly, we add the NVIDIA Deep Learning SDK

to this list. Its main goal is to maximize the performance

of deep learning algorithms on (NVIDIA) GPUs. The SDK

consists of three core modules. The ﬁrst, cuDNN, provides

high performance GPU implementations for deep learning

algorithms such as convolutions, activation functions and

tensor transformations. The second is a linear algebra library,

cuBLAS, enabling GPU-accelerated mathematical operations

on n-dimensional arrays. Lastly, cuSPARSE includes a set of

routines for sparse matrices tuned for high efﬁciency on GPUs.

While it is possible to program in these libraries directly, there

http://caffe.berkeleyvision.org

http://deeplearning4j.org

https://developer.nvidia.com/deep-learning-software

exist also bindings to other deep learning libraries, such as

Torch

III. THE TENSORFLOW PROGRAMMING MODEL

In this section we provide an in-depth discussion of the

abstract computational principles underlying the TensorFlow

software library. We begin with a thorough examination of the

basic structural and architectural decisions made by the Ten-

sorFlow development team and explain how machine learning

algorithms may be expressed in its dataﬂow graph language.

Subsequently, we study TensorFlow’s execution model and

provide insight into the way TensorFlow graphs are assigned

to available hardware units in a local as well as distributed

environment. Then, we investigate the various optimizations

incorporated into TensorFlow, targeted at improving both

software and hardware efﬁciency. Lastly, we list extensions to

the basic programming model that aid the user in both com-

putational as well as logistical aspects of training a machine

learning model with TensorFlow.

A. Computational Graph Architecture

In TensorFlow, machine learning algorithms are represented

as computational graphs. A computational or dataﬂow graph

is a form of directed graph where vertices or nodes describe

operations, while edges represent data ﬂowing between these

operations. If an output variable z is the result of applying a

binary operation to two inputs x and y, then we draw directed

edges from x and y to an output node representing z and

annotate the vertex with a label describing the performed

computation. Examples for computational graphs are given

in Figure 2. The following paragraphs discuss the principle

elements of such a dataﬂow graph, namely operations, tensors,

variables and sessions.

1) Operations: The major beneﬁt of representing an algo-

rithm in form of a graph is not only the intuitive (visual)

expression of dependencies between units of a computational

model, but also the fact that the deﬁnition of a node within

the graph can be kept very general. In TensorFlow, nodes

represent operations, which in turn express the combination or

transformation of data ﬂowing through the graph [8]. An oper-

ation can have zero or more inputs and produce zero or more

outputs. As such, an operation may represent a mathematical

https://github.com/soumith/cudnn.torch

z = x + y

x w

dot

ˆy

ˆy = σ(x

w + b)

Fig. 2: Examples of computational graphs. The left graph

displays a very simple computation, consisting of just an

addition of the two input variables x and y. In this case, z

is the result of the operation +, as the annotation suggests.

The right graph gives a more complex example of computing

a logistic regression variable ˆy in for some example vector x,

weight vector w as well as a scalar bias b. As shown in the

graph, ˆy is the result of the sigmoid or logistic function σ.

equation, a variable or constant, a control ﬂow directive, a

ﬁle I/O operation or even a network communication port.

It may seem unintuitive that an operation, which the reader

may associate with a function in the mathematical sense, can

represent a constant or variable. However, a constant may be

thought of as an operation that takes no inputs and always

produces the same output corresponding to the constant it

represents. Analogously, a variable is really just an operation

taking no input and producing the current state or value of

that variable. Table ?? gives an overview of different kinds of

operations that may be declared in a TensorFlow graph.

Any operation must be backed by an associated implemen-

tation. In [8] such an implementation is referred to as the

operation’s kernel. A particular kernel is always speciﬁcally

built for execution on a certain kind of device, such as a CPU,

GPU or other hardware unit.

2) Tensors: In TensorFlow, edges represent data ﬂowing

from one operation to another and are referred to as tensors.

A tensor is a multi-dimensional collection of homogeneous

values with a ﬁxed, static type. The number of dimensions

of a tensor is termed its rank. A tensor’s shape is the

tuple describing its size, i.e. the number of components, in

each dimension. In the mathematical sense, a tensor is the

generalization of two-dimensional matrices, one-dimensional

Category Examples

Element-wise operations Add, Mul, Exp

Matrix operations MatMul, MatrixInverse

Value-producing operations Constant, Variable

Neural network units SoftMax, ReLU, Conv2D

Checkpoint operations Save, Restore

TABLE I: Examples for TensorFlow operations [8].

vectors and also scalars, which are simply tensors of rank zero.

In terms of the computational graph, a tensor can be seen

as a symbolic handle to one of the outputs of an operation.

A tensor itself does not hold or store values in memory, but

provides only an interface for retrieving the value referenced

by the tensor. When creating an operation in the TensorFlow

programming environment, such as for the expression x + y,

a tensor object is returned. This tensor may then be supplied

as input to other computations, thereby connecting the source

and destination operations with an edge. By these means, data

ﬂows through a TensorFlow graph.

Next to regular tensors, TensorFlow also provides a

SparseTensor data structure, allowing for a more space-

efﬁcient dictionary-like representation of sparse tensors with

only few non-zeros entries.

3) Variables: In a typical situation, such as when per-

forming stochastic gradient descent (SGD), the graph of a

machine learning model is executed from start to end multiple

times for a single experiment. Between two such invocations,

the majority of tensors in the graph are destroyed and do

not persist. However, it is often necessary to maintain state

across evaluations of the graph, such as for the weights and

parameters of a neural network. For this purpose, there exist

variables in TensorFlow, which are simply special operations

that can be added to the computational graph.

In detail, variables can be described as persistent, mutable

handles to in-memory buffers storing tensors. As such, vari-

ables are characterized by a certain shape and a ﬁxed type.

To manipulate and update variables, TensorFlow provides the

assign family of graph operations.

When creating a variable node for a TensorFlow graph, it

is necessary to supply a tensor with which the variable is

initialized upon graph execution. The shape and data type of

the variable is then deduced from this initializer. Interestingly,

the variable itself does not store this initial tensor. Rather,

constructing a variable results in the addition of three distinct

nodes to the graph:

1) The actual variable node, holding the mutable state.

2) An operation producing the initial value, often a con-

stant.

3) An initializer operation, that assigns the initial value

to the variable tensor upon evaluation of the graph.

An example for this is given in Figure 3.

4) Sessions: In TensorFlow, the execution of operations

and evaluation of tensors may only be performed in a special

environment referred to as session. One of the responsibilities

of a session is to encapsulate the allocation and management

of resources such as variable buffers. Moreover, the Session

interface of the TensorFlow library provides a run routine,

which is the primary entry point for executing parts or the

entirety of a computational graph. This method takes as input

the nodes in the graph whose tensors should be computed and

returned. Moreover, an optional mapping from arbitrary nodes

in the graph to respective replacement values — referred to as

feed nodes — may be supplied to run as well [8].

剩余15页未读，继续阅读

评论收藏

内容反馈

A Tour of TensorFlow

评论0

最新资源

A Tour of TensorFlow

评论0

最新资源

相关推荐

TensorFlow

A Tour of C++, 2nd Edition

A tour of C++ 2nd

A Tour of C++ 2nd

TensorFlow:TensorFlow项目

C++语言导学 A Tour of C++（C++之父写的入门书）

A Tour of C++

Bjarne Stroustrup A Tour of C++ (2013).pdf

A wavelet tour of signal processing 3rd edition

Whirlwind Tour Of Python

Tour-Setup

Tour-Of-Heroes

Tour-of-Heroes1

Tour-of-heroes

Tour-Planner

A Guided Tour Of Mathematical Physics (By Roel Snieder, Department Of Geophysics, Utrecht Univers.pdf

A Wavelet Tour of Signal Processing 3rd Edition

新版图书 A.Tour.of.C++.2nd.Edition.2018.7

A tour of c++ 2014

A tour of git_the basics