VideoGPT: Video Generation using VQ-VAE and Transformers
Wilson Yan
* 1
Yunzhi Zhang
* 1
Pieter Abbeel
1
Aravind Srinivas
1
Abstract
We present VideoGPT: a conceptually simple ar-
chitecture for scaling likelihood based generative
modeling to natural videos. VideoGPT uses VQ-
VAE that learns downsampled discrete latent rep-
resentations of a raw video by employing 3D con-
volutions and axial self-attention. A simple GPT-
like architecture is then used to autoregressively
model the discrete latents using spatio-temporal
position encodings. Despite the simplicity in for-
mulation and ease of training, our architecture is
able to generate samples competitive with state-
of-the-art GAN models for video generation on
the BAIR Robot dataset, and generate high fi-
delity natural videos from UCF-101 and Tum-
bler GIF Dataset (TGIF). We hope our proposed
architecture serves as a reproducible reference
for a minimalistic implementation of transformer
based video generation models. Samples and
code are available at
https://wilson1yan.
github.io/videogpt/index.html.
1. Introduction
Deep generative models of multiple types (Kingma &
Welling, 2013; Goodfellow et al., 2014; van den Oord et al.,
2016b; Dinh et al., 2016) have seen incredible progress in
the last few years on multiple modalities including natural
images (van den Oord et al., 2016c; Zhang et al., 2019;
Brock et al., 2018; Kingma & Dhariwal, 2018; Ho et al.,
2019a; Karras et al., 2017; 2019; Van Den Oord et al.,
2017; Razavi et al., 2019; Vahdat & Kautz, 2020; Ho et al.,
2020; Chen et al., 2020; Ramesh et al., 2021), audio wave-
forms conditioned on language features (van den Oord et al.,
2016a; Oord et al., 2017; Prenger et al., 2019; Bi
´
nkowski
et al., 2019), natural language in the form of text (Rad-
ford et al., 2019; Brown et al., 2020), and music genera-
tion (Dhariwal et al., 2020). These results have been made
possible thanks to fundamental advances in deep learning
*
Equal contribution
1
University of California, Berkeley.
Correspondence to: Wilson Yan, Aravind Srinivas
<
wil-
son1.yan@berkeley.edu, aravind srinivas@berkeley.edu>.
architectures (He et al., 2015; van den Oord et al., 2016b;c;
Vaswani et al., 2017; Zhang et al., 2019; Menick & Kalch-
brenner, 2018) as well as the availability of compute re-
sources (Jouppi et al., 2017; Amodei & Hernandez, 2018)
that are more powerful and plentiful than a few years ago.
While there have certainly been impressive efforts to model
videos (Vondrick et al., 2016; Kalchbrenner et al., 2016;
Tulyakov et al., 2018; Clark et al., 2019), high-fidelity nat-
ural videos is one notable modality that has not seen the
same level of progress in generative modeling as compared
to images, audio, and text. This is reasonable since the
complexity of natural videos requires modeling correlations
across both space and time with much higher input dimen-
sions. Video modeling is therefore a natural next challenge
for current deep generative models. The complexity of the
problem also demands more compute resources which can
also be deemed as one important reason for the relatively
slow progress in generative modeling of videos.
Why is it useful to build generative models of videos? Con-
ditional and unconditional video generation implicitly ad-
dresses the problem of video prediction and forecasting.
Video prediction (Srivastava et al., 2015; Finn et al., 2016;
Kalchbrenner et al., 2017; Sønderby et al., 2020) can be
seen as learning a generative model of future frames con-
ditioned on the past frames. Architectures developed for
video generation can be useful in forecasting applications
for weather prediction (Sønderby et al., 2020), autonomous
driving (for e.g., such as predicting the future in more se-
mantic and dense abstractions like segmentation masks (Luc
et al., 2017)). Finally, building generative models of the
world around us is considered as one way to measure our
understanding of physical common sense and predictive
intelligence (Lake et al., 2015).
Multiple classes of generative models have been shown
to produce strikingly good samples such as autoregres-
sive models (van den Oord et al., 2016b;c; Parmar et al.,
2018; Menick & Kalchbrenner, 2018; Radford et al.,
2019; Chen et al., 2020), generative adversarial networks
(GANs) (Goodfellow et al., 2014; Radford et al., 2015),
variational autoencoders (VAEs) (Kingma & Welling, 2013;
Kingma et al., 2016; Mittal et al., 2017; Marwah et al., 2017;
Vahdat & Kautz, 2020; Child, 2020), Flows (Dinh et al.,
2014; 2016; Kingma & Dhariwal, 2018; Ho et al., 2019a),
arXiv:2104.10157v2 [cs.CV] 14 Sep 2021
剩余13页未读，继续阅读
资源评论
