模仿学习Imitation Learning最新论文2018

1 Introduction

We are interested in robotic systems that are able to perform a variety of complex useful tasks, e.g.

tidying up a home or preparing a meal. The robot should be able to learn new tasks without long

perform the task. We can compare this with an alternative form of communication, namely natural

language. Although language is highly versatile, effective, and efficient, natural language processing

31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA.

when it is conditioned on a single demonstration for task F, it should behave like a good policy for

task F. (c) We can phrase this as a supervised learning problem, where we train this network on a set

of training tasks, and with enough examples it should generalize to unseen, but related tasks. To train

this network, in each iteration we sample a demonstration from one of the training tasks, and feed it

to the network. Then, we sample another pair of observation and action from a second demonstration

of the same task. When conditioned on both the first demonstration and this observation, the network

is trained to output the corresponding action.

systems are not yet at a level where we could easily use language to precisely describe a complex

task to a robot. Compared to language, using demonstrations has two fundamental advantages: first,

these applications have either required careful feature engineering, or a significant amount of system

interaction time. This is far from what what we desire: ideally, we hope to demonstrate a certain task

only once or a few times to the robot, and have it instantly generalize to new situations of the same

task, without long system interaction time or domain knowledge about individual tasks.

In this paper we explore the one-shot imitation learning setting illustrated in Fig. 1, where the

objective is to maximize the expected performance of the learned policy when faced with a new,

previously unseen, task, and having received as input only one demonstration of that task. For the

interaction, once it has received the demonstration.

of soft attention over both types of inputs made strong generalization possible. In particular, on a

configurations which were not present in any training data. Videos of our experiments are available

at http://bit.ly/nips2017-oneshot.

2 Related Work

Imitation learning considers the problem of acquiring skills from observing demonstrations. Survey

indicating success) for each task, although this is not required during training. The objective is to

and demonstrations d 2 D(t).

In our experiments, we use

During test time, we can sample multiple downsampled trajectories, use each of them to compute

this consistently improves the performance of the policy.

Neighborhood Attention:

tions, composed of dilated temporal convolution [

] and neighborhood attention. We now describe

this second operation in more detail.

Since our neural network needs to handle demonstrations with variable numbers of blocks, it must

a query

is given by

tanh(q + c

weighted combination of the memory content, where the weights are given by a softmax operation

heads, in which case there will be as many output vectors as there are queries.

Now we turn to neighborhood attention. We assume there are

The input to neighborhood attention is a list of embeddings

which can be the result of a projection operation over a list of block positions, or the output of a

previous neighborhood attention operation. Given this list of embeddings, we use two separate linear