Tensorflow Tutorial (英文)

TensorFlow:

Large-Scale Machine Learning on Heterogeneous Distributed Systems

(Preliminary White Paper, November 9, 2015)

Mart

´

ın Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro,

Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow,

Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser,

Manjunath Kudlur, Josh Levenberg, Dan Man

´

Abstract

TensorFlow [1] is an interface for expressing machine learn-

ing algorithms, and an implementation for executing such al-

gorithms. A computation expressed using TensorFlow can be

executed with little or no change on a wide variety of hetero-

geneous systems, ranging from mobile devices such as phones

and tablets up to large-scale distributed systems of hundreds

of machines and thousands of computational devices such as

GPU cards. The system is flexible and can be used to express

a wide variety of algorithms, including training and inference

computational drug discovery. This paper describes the Ten-

sorFlow interface and an implementation of that interface that

we have built at Google. The TensorFlow API and a reference

implementation were released as an open-source package under

the Apache 2.0 license in November, 2015 and are available at

www.tensorflow.org.

1 Introduction

The Google Brain project started in 2011 to explore the

use of very-large-scale deep neural networks, both for

research and for use in Google’s products. As part of

for image classification and object detection [16, 48],

video classification [27], speech recognition [56, 21, 20],

∗

{jeff,rajatmonga}@google.com

sequence prediction [47], move selection for Go [34],

pedestrian detection [2], reinforcement learning [38],

and other areas [17, 5]. In addition, often in close collab-

oration with the Google Brain team, more than 50 teams

at Google and other Alphabet companies have deployed

deep neural networks using DistBelief in a wide variety

ties and requirements for training and using neural net-

works, we have built TensorFlow, our second-generation

system for the implementation and deployment of large-

scale machine learning models. TensorFlow takes com-

putations described using a dataflow-like model and

maps them onto a wide variety of different hardware

platforms, ranging from running inference on mobile

device platforms such as Android and iOS to modest-

sized training and inference systems using single ma-

chines containing one or many GPU cards to large-scale

to significant maintenance burdens and leaky abstrac-

tions. TensorFlow computations are expressed as stateful

dataflow graphs (described in more detail in Section 2),

and we have focused on making the system both flexible

enough for quickly experimenting with new models for

research purposes and sufficiently high performance and

robust for production training and deployment of ma-

chine learning models. For scaling neural network train-

ing to larger deployments, TensorFlow allows clients to

easily express various kinds of parallelism through repli-

cation and parallel execution of a core model dataflow

[14, 29, 42]. Some TensorFlow uses allow some flexibil-

ity in terms of the consistency of parameter updates, and

we can easily express and take advantage of these relaxed

synchronization requirements in some of our larger de-

ployments. Compared to DistBelief, TensorFlow’s pro-

gramming model is more flexible, its performance is sig-

nificantly better, and it supports training and using a

broader range of models on a wider variety of hetero-

geneous hardware platforms.

Dozens of our internal clients of DistBelief have al-

many hundreds of machines [11, 47, 48, 18, 53, 41].

Although these applications have concentrated on ma-

chine learning and deep neural networks in particular,

we expect that TensorFlow’s abstractions will be useful

in a variety of other domains, including other kinds of

machine learning algorithms, and possibly other kinds

of numerical computations. We have open-sourced the

TensorFlow API and a reference implementation under

the Apache 2.0 license in November, 2015, available at

www.tensorflow.org.

several optimizations to the basic implementations. Sec-

tion 6 describes some of our experiences in using Ten-

sorFlow, Section 7 describes several programming id-

ioms we have found helpful when using TensorFlow, and

Section 9 describes several auxiliary tools we have built

around the core TensorFlow system. Sections 10 and 11

discuss future and related work, respectively, and Sec-

tion 12 offers concluding thoughts.

2 Programming Model and Basic Concepts

A TensorFlow computation is described by a directed

graph, which is composed of a set of nodes. The graph

represents a dataflow computation, with extensions for

allowing some kinds of nodes to maintain and update

persistent state and for branching and looping control

ecute a TensorFlow graph using the Python front end is

shown in Figure 1, and the resulting computation graph

in Figure 2.

time. Special edges, called control dependencies, can

also exist in the graph: no data flows along such edges,

but they indicate that the source node for the control de-

pendence must finish executing before the destination

node for the control dependence starts executing. Since

our model includes mutable state, control dependencies

can be used directly by clients to enforce happens before

relationships. Our implementation also sometimes in-

serts control dependencies to enforce orderings between

otherwise independent operations as a way of, for exam-

creating a Session. To create a computation graph, the

Session interface supports an Extend method to augment

the current graph managed by the session with additional

nodes and edges (the initial graph when a session is cre-

Devices are the computational heart of TensorFlow. Each

worker is responsible for one or more devices, and

each device has a device type, and a name. Device

names are composed of pieces that identify the de-

vice’s type, the device’s index within the worker, and,

in our distributed setting, an identification of the job

and task of the worker (or localhost for the case where

the devices are local to the process). Example device

names are "/job:localhost/device:cpu:0" or

"/job:worker/task:17/device:gpu:3". We

tion and deallocation of device memory, and for arrang-

ing for the execution of any kernels that are requested by

higher levels in the TensorFlow implementation.

ment types, including signed and unsigned integers rang-

ing in size from 8 bits to 64 bits, IEEE float and double

types, a complex number type, and a string type (an ar-

bitrary byte array). Backing store of the appropriate size

is managed by an allocator that is specific to the device

on which the tensor resides. Tensor backing store buffers

are reference counted and are deallocated when no refer-

ences remain.

3.1 Single-Device Execution

Let’s first consider the simplest execution scenario: a sin-

added to a ready queue. The ready queue is processed in

some unspecified order, delegating execution of the ker-

nel for a node to the device object. When a node has

finished executing, the counts of all nodes that depend

on the completed node are decremented.

3.2 Multi-Device Execution

Once a system has multiple devices, there are two main

complications: deciding which device to place the com-

putation for each node in the graph, and then managing

the required communication of data across device bound-

plified version of this algorithm is presented here. See

Section 4.3 for extensions supported by this algorithm.

One input to the placement algorithm is a cost model,

which contains estimates of the sizes (in bytes) of the

when presented with its input tensors. This cost model is

either statically estimated based on heuristics associated

with different operation types, or is measured based on

an actual set of placement decisions for earlier execu-

tions of the graph.

The placement algorithm first runs a simulated execu-

tion of the graph. The simulation is described below and

ends up picking a device for each node in the graph using

greedy heuristics. The node to device placement gener-

ated by this simulation is also used as the placement for

The placement algorithm starts with the sources of the

computation graph, and simulates the activity on each

device in the system as it progresses. For each node that

is reached in this traversal, the set of feasible devices is

considered (a device may not be feasible if the device

does not provide a kernel that implements the particular

operation). For nodes with multiple feasible devices, the

placement algorithm uses a greedy heuristic that exam-

to this node from other devices to the considered device.

The device where the node’s operation would finish the

soonest is selected as the device for that operation, and

the placement process then continues onwards to make

placement decisions for other nodes in the graph, includ-

ing downstream nodes that are now ready for their own

simulated execution. Section 4.3 describes some exten-

sions that allow users to provide hints and partial con-

straints to guide the placement algorithm. The placement

algorithm is an area of ongoing development within the

system.

transformation.

ceive nodes coordinate to transfer data across devices.

This allows us to isolate all communication inside Send

and Receive implementations, which simplifies the rest

of the runtime.

When we insert Send and Receive nodes, we canoni-

calize all users of a particular tensor on a particular de-

vice to use a single Receive node, rather than one Re-

ceive node per downstream user on a particular device.

This ensures that the data for the needed tensor is only

transmitted once between a source device → destination

By handling communication in this manner, we also

allow the scheduling of individual nodes of the graph

on different devices to be decentralized into the work-

ers: the Send and Receive nodes impart the necessary