1 INTRODUCTION
Reinforcement learning (RL) is about an agent interacting with the environment, learning an optimal
policy, by trail and error, for sequential decision making problems in a wide range of fields in both
natural and social sciences, and engineering (Sutton and Barto, 1998; 2017; Bertsekas and Tsitsiklis,
1996; Bertsekas, 2012; Szepesv
´
ari, 2010; Powell, 2011).
The integration of reinforcement learning and neural networks has a long history (Sutton and Barto,
2017; Bertsekas and Tsitsiklis, 1996; Schmidhuber, 2015). With recent exciting achievements of
deep learning (LeCun et al., 2015; Goodfellow et al., 2016), benefiting from big data, powerful
computation, new algorithmic techniques, mature software packages and architectures, and strong
financial support, we have been witnessing the renaissance of reinforcement learning (Krakovsky,
2016), especially, the combination of deep neural networks and reinforcement learning, i.e., deep
reinforcement learning (deep RL).
Deep learning, or deep neural networks, has been prevailing in reinforcement learning in the last
several years, in games, robotics, natural language processing, etc. We have been witnessing break-
throughs, like deep Q-network (Mnih et al., 2015) and AlphaGo (Silver et al., 2016a); and novel ar-
chitectures and applications, like differentiable neural computer (Graves et al., 2016), asynchronous
methods (Mnih et al., 2016), dueling network architectures (Wang et al., 2016b), value iteration
networks (Tamar et al., 2016), unsupervised reinforcement and auxiliary learning (Jaderberg et al.,
2017; Mirowski et al., 2017), neural architecture design (Zoph and Le, 2017), dual learning for
machine translation (He et al., 2016a), spoken dialogue systems (Su et al., 2016b), information
extraction (Narasimhan et al., 2016), guided policy search (Levine et al., 2016a), and generative
adversarial imitation learning (Ho and Ermon, 2016), etc.
Why has deep learning been helping reinforcement learning make so many and so enormous achieve-
ments? Representation learning with deep learning enables automatic feature engineering and end-
to-end learning through gradient descent, so that reliance on domain knowledge is significantly
reduced or even removed. Feature engineering used to be done manually and is usually time-
consuming, over-specified, and incomplete. Deep, distributed representations exploit the hierar-
chical composition of factors in data to combat the exponential challenges of the curse of dimen-
sionality. Generality, expressiveness and flexibility of deep neural networks make some tasks easier
or possible, e.g., in the breakthroughs and novel architectures and applications discussed above.
Deep learning and reinforcement learning, being selected as one of the MIT Technology Review 10
Breakthrough Technologies in 2013 and 2017 respectively, will play their crucial role in achieving
artificial general intelligence. David Silver, the major contributor of AlphaGo (Silver et al., 2016a),
even made a formula: artificial intelligence = reinforcement learning + deep learning (Silver, 2016).
We invent a sentence: Deep reinforcement learning is artificial intelligence.
The outline of this overview follows. First we discuss background of machine learning, deep learn-
ing and reinforcement learning in Section 2. Next we discuss core RL elements, including value
function in Section 3.1, policy in Section 3.2, reward in Section 3.3, model in Section 3.4, plan-
ning in Section 3.5, and exploration in Section 3.6. Then we discuss important mechanisms for
RL, including attention and memory in Section 4.1, unsupervised learning in Section 4.2, transfer
learning in Section 4.3, multi-agent RL in Section 4.4, hierarchical RL in Section 4.5, and, learning
to learn in Section 4.6. After that, we discuss various RL applications, including games in Sec-
tion 5.1, robotics in Section 5.2, natural language processing in Section 5.3, computer vision in
Section 5.4, neural architecture design in Section 5.5, business management in Section 5.6, finance
in Section 5.7, healthcare in Section 5.8, Industry 4.0 in Section 5.9, smart grid in Section 5.10, in-
telligent transportation systems in Section 5.11, and computer systems in Section 5.12. We present
a list of topics not reviewed yet in Section 6, give a brief summary in Section 8, and close with
discussions in Section 9.
In Section 7, we list a collection of RL resources including books, surveys, reports, online courses,
tutorials, conferences, journals and workshops, blogs, and open sources. If picking a single RL
resource, it is Sutton and Barto’s RL book (Sutton and Barto, 2017), 2nd edition in preparation. It
covers RL fundamentals and reflects new progress, e.g., in deep Q-network, AlphaGo, policy gra-
dient methods, as well as in psychology and neuroscience. Deng and Dong (2014) and Goodfellow
et al. (2016) are recent deep learning books. Bishop (2011), Hastie et al. (2009), and Murphy (2012)
5
