transitions. The key components are: (i) a short-term mem-
ory block called conditional attention module (CAM), which
conditions the current generation on the features extracted
from the previous chunk via an attentional mechanism, lead-
ing to consistent chunk transitions, (ii) a long-term mem-
ory block called appearance preservation module, which
extracts high-level scene and object features from the first
video chunk to prevent the model from forgetting the ini-
tial scene, and (iii) a randomized blending approach that
enables to apply a video enhancer autoregressively for in-
finitely long videos without inconsistencies between chunks.
Experiments show that StreamingT2V generates high mo-
tion amount. In contrast, all competing image-to-video
methods are prone to video stagnation when applied naively
in an autoregressive manner. Thus, we propose with Stream-
ingT2V a high-quality seamless text-to-long video genera-
tor that outperforms competitors with consistency and mo-
tion.
1. Introduction
In recent years, with the raise of Diffusion Models [15, 26,
28, 34], the task of text-guided image synthesis and manipu-
lation gained enormous attention from the community. The
huge success in image generation led to the further exten-
sion of diffusion models to generate videos conditioned by
textual prompts [4, 5, 7, 11–13, 17, 18, 20, 32, 37, 39, 45].
Despite the impressive generation quality and text align-
ment, the majority of existing approaches such as [4, 5,
17, 39, 45] are mostly focused on generating short frame
sequences (typically of 16 or 24 frame-length). However,
short videos are limited in real-world use-cases such as ad
making, storytelling, etc.
The na
¨
ıve approach of simply training existing methods
on long videos (e.g. ≥ 64 frames) is normally unfeasi-
ble. Even for generating short sequences, a very expensive
training (e.g. using more that 260K steps and 4500 batch
size [39]) is typically required. Without training on longer
videos, video quality commonly degrades when short video
generators are made to output long videos (see appendix).
Existing approaches, such as [5, 17, 23], thus extend the
baselines to autoregressively generate short video chunks
conditioned on the last frame(s) of the previous chunk.
However, the straightforward long-video generation ap-
proach of simply concatenating the noisy latents of a video
chunk with the last frame(s) of the previous chunk leads to
poor conditioning with inconsistent scene transitions (see
Sec. 5.3). Some works [4, 8, 40, 43, 48] incorporate also
CLIP [25] image embeddings of the last frame of the previ-
ous chunk reaching to a slightly better consistency, but are
still prone to inconsistent global motion across chunks (see
Fig. 5) as the CLIP image encoder loses information impor-
tant for perfectly reconstructing the conditional frames. The
concurrent work SparseCtrl [12] utilizes a more sophisti-
cated conditioning mechanism by sparse encoder. Its archi-
tecture requires to concatenate additional zero-filled frames
to the conditioning frames before being plugged into sparse
encoder. However, this inconsistency in the input leads to
inconsistencies in the output (see Sec.5.4). Moreover, we
observed that all image-to-video methods that we evaluated
in our experiments (see Sec.5.4) lead eventually to video
stagnation, when applied autoregressively by conditioning
on the last frame of the previous chunk.
To overcome the weaknesses and limitations of current
works, we propose StreamingT2V, an autoregressive text-
to-video method equipped with long/short-term memory
blocks that generates long videos without temporal incon-
sistencies.
To this end, we propose the Conditional Attention Mod-
ule (CAM) which, due to its attentional nature, effectively
borrows the content information from the previous frames
to generate new ones, while not restricting their motion by
the previous structures/shapes. Thanks to CAM, our results
are smooth and with artifact-free video chunk transitions.
Existing approaches are not only prone to temporal in-
consistencies and video stagnation, but they suffer from
object appearance/characteristic changes and video quality
degradation over time (see e.g., SVD [4] in Fig. 7). The
reason is that, due to conditioning only on the last frame(s)
of the previous chunk, they overlook the long-term depen-
dencies of the autoregressive process. To address this is-
sue we design an Appearance Preservation Module (APM)
that extracts object or global scene appearance information
from an initial image (anchor frame), and conditions the
video generation process of all chunks with that informa-
tion, which helps to keep object and scene features across
the autoregressive process.
To further improve the quality and resolution of our long
video generation, we adapt a video enhancement model for
autoregressive generation. For this purpose, we choose a
high-resolution text-to-video model and utilize the SDEdit
[22] approach for enhancing consecutive 24-frame chunks
(overlapping with 8 frames) of our video. To make the
chunk enhancement transitions smooth, we design a ran-
domized blending approach for seamless blending of over-
lapping enhanced chunks.
Experiments show that StreamingT2V successfully gen-
erates long and temporal consistent videos from text with-
out video stagnation. To summarize, our contributions are
three-fold:
• We introduce StreamingT2V, an autoregressive approach
for seamless synthesis of extended video content using
short and long-term dependencies.
• Our Conditional Attention Module (CAM) and Appear-
ance Preservation Module (APM) ensure the natural
continuity of the global scene and object characteristics
2
