Generating Videos with Scene Dynamics
Carl Vondrick
MIT
vondrick@mit.edu
Hamed Pirsiavash
UMBC
hpirsiav@umbc.edu
Antonio Torralba
MIT
torralba@mit.edu
Abstract
We capitalize on large amounts of unlabeled video in order to learn a model of
scene dynamics for both video recognition tasks (e.g. action classification) and
video generation tasks (e.g. future prediction). We propose a generative adversarial
network for video with a spatio-temporal convolutional architecture that untangles
the scene’s foreground from the background. Experiments suggest this model can
generate tiny videos up to a second at full frame rate better than simple baselines,
and we show its utility at predicting plausible futures of static images. Moreover,
experiments and visualizations show the model internally learns useful features for
recognizing actions with minimal supervision, suggesting scene dynamics are a
promising signal for representation learning. We believe generative video models
can impact many applications in video understanding and simulation.
1 Introduction
Understanding object motions and scene dynamics is a core problem in computer vision. For both
video recognition tasks (e.g., action classification) and video generation tasks (e.g., future prediction),
a model of how scenes transform is needed. However, creating a model of dynamics is challenging
because there is a vast number of ways that objects and scenes can change.
In this work, we are interested in the fundamental problem of learning how scenes transform with
time. We believe investigating this question may yield insight into the design of predictive models for
computer vision. However, since annotating this knowledge is both expensive and ambiguous, we
instead seek to learn it directly from large amounts of in-the-wild, unlabeled video. Unlabeled video
has the advantage that it can be economically acquired at massive scales yet contains rich temporal
signals “for free” because frames are temporally coherent.
With the goal of capturing some of the temporal knowledge contained in large amounts of unlabeled
video, we present an approach that learns to generate tiny videos which have fairly realistic dynamics
and motions. To do this, we capitalize on recent advances in generative adversarial networks [
9
,
31
,
4
],
which we extend to video. We introduce a two-stream generative model that explicitly models the
foreground separately from the background, which allows us to enforce that the background is
stationary, helping the network to learn which objects move and which do not.
Our experiments suggest that our model has started to learn about dynamics. In our generation
experiments, we show that our model can generate scenes with plausible motions.
1
We conducted a
psychophysical study where we asked over a hundred people to compare generated videos, and people
preferred videos from our full model more often. Furthermore, by making the model conditional
on an input image, our model can sometimes predict a plausible (but “incorrect”) future. In our
recognition experiments, we show how our model has learned, without supervision, useful features
for human action classification. Moreover, visualizations of the learned representation suggest future
generation may be a promising supervisory signal for learning to recognize objects of motion.
1
See http://mit.edu/vondrick/tinyvideo for the animated videos.
29th Conference on Neural Information Processing Systems (NIPS 2016), Barcelona, Spain.
arXiv:1609.02612v3 [cs.CV] 26 Oct 2016
资源评论
