Reinforcement Learning 14
class_link: https://icml.cc/Conferences/2018/Schedule?showParentSession=3462
Title: RLlib: Abstractions for Distributed Reinforcement Learning
Author: Eric Liang · Richard Liaw · Robert Nishihara · Philipp Moritz · Roy Fox · Ken Goldberg ·
Joseph Gonzalez · Michael Jordan · Ion Stoica
Abstract: Reinforcement learning (RL) algorithms involve the deep nesting of highly irregular
computation patterns, each of which typically exhibits opportunities for distributed computation.
We argue for distributing RL components in a composable way by adapting algorithms for top-
down hierarchical control, thereby encapsulating parallelism and resource requirements within
short-running compute tasks. We demonstrate the benefits of this principle through RLlib: a
library that provides scalable software primitives for RL. These primitives enable a broad range of
algorithms to be implemented with high performance, scalability, and substantial code reuse.
RLlib is available as part of the open source Ray project at http://rllib.io/.
Link: http://proceedings.mlr.press/v80/liang18b/liang18b.pdf
Title: IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner
Architectures
Author: Lasse Espeholt · Hubert Soyer · Remi Munos · Karen Simonyan · Vlad Mnih · Tom Ward ·
Yotam Doron · Vlad Firoiu · Tim Harley · Iain Dunning · Shane Legg · koray kavukcuoglu
Abstract: In this work we aim to solve a large collection of tasks using a single reinforcement
learning agent with a single set of parameters. A key challenge is to handle the increased amount
of data and extended training time. We have developed a new distributed agent IMPALA
(Importance Weighted Actor-Learner Architecture) that not only uses resources more efficiently in
single-machine training but also scales to thousands of machines without sacrificing data
efficiency or resource utilisation. We achieve stable learning at high throughput by combining
decoupled acting and learning with a novel off-policy correction method called V-trace. We
demonstrate the effectiveness of IMPALA for multi-task reinforcement learning on DMLab-30 (a
set of 30 tasks from the DeepMind Lab environment (Beattie et al., 2016)) and Atari57 (all
available Atari games in Arcade Learning Environment (Bellemare et al., 2013a)). Our results show
that IMPALA is able to achieve better performance than previous agents with less data, and
crucially exhibits positive transfer between tasks as a result of its multi-task approach.
Link: http://proceedings.mlr.press/v80/espeholt18a/espeholt18a.pdf
Title: Mix & Match - Agent Curricula for Reinforcement Learning
Author: Wojciech Czarnecki · Siddhant Jayakumar · Max Jaderberg · Leonard Hasenclever · Yee
Teh · Nicolas Heess · Simon Osindero · Razvan Pascanu
Abstract: We introduce Mix and match (M&M) – a training framework designed to facilitate
rapid and effective learning in RL agents that would be too slow or too challenging to train
otherwise.The key innovation is a procedure that allows us to automatically form a curriculum
