稀疏注意力：时间序列预测的局部性和Transformer的存储瓶颈

Enhancing the Locality and Breaking the Memory

Bottleneck of Transformer on Time Series Forecasting

Shiyang Li

shiyangli@ucsb.edu

Xiaoyong Jin

x_jin@ucsb.edu

Yao Xuan

yxuan@ucsb.edu

Xiyou Zhou

xiyou@ucsb.edu

University of California, Santa Barbara

Abstract

Time series forecasting is an important problem across many domains, including

bottleneck: space complexity of canonical Transformer grows quadratically with

sequence length

L

, making directly modeling long time series infeasible. In

order to solve these two issues, we first propose convolutional self-attention by

producing queries and keys with causal convolution so that local context can

be better incorporated into attention mechanism. Then, we propose LogSparse

Transformer with only

O(L(log L)

2

)

for time series with fine granularity and strong long-term dependencies under

constrained memory budget. Our experiments on both synthetic data and real-

world datasets show that it compares favorably to the state-of-the-art.

also require practitioners’ expertise in manually selecting trend, seasonality and other components.

To sum up, these two major weaknesses have greatly hindered their applications in the modern

large-scale time series forecasting tasks.

To tackle the aforementioned challenges, deep neural networks [

,

5

,

6

] have been proposed as

an alternative solution, where Recurrent Neural Network (RNN) [

,

8

,

9

] has been employed to

10

because of gradient vanishing and exploding problem. Despite the emergence of various variants,

including LSTM [

11

] and GRU [

that language models using LSTM have an effective context size of about 200 tokens on average

but are only able to sharply distinguish 50 tokens nearby, indicating that even LSTM struggles to

model to access any part of the history regardless of distance, making it potentially more suitable

for grasping the recurring patterns with long-term dependencies. However, canonical dot-product

L

on directly modeling long time series with fine granularity. We specifically delve into these two

issues and investigate the applications of Transformer to time series forecasting. Our contributions

are three fold:

We successfully apply Transformer architecture to time series forecasting and perform extensive

2

memory bottleneck, not only making fine-grained long time series modeling feasible but also

producing comparable or even better results with much less memory usage, compared to canonical

Transformer.

2 Related Work

Due to the wide applications of forecasting, various methods have been proposed to solve the problem.

One of the most prominent models is

known Box-Jenkins methodology [

practitioners. However, its linear assumption and limited scalability make it unsuitable for large-scale

forecasting tasks. Further, information across similar time series cannot be shared since each time

Perceptrons (MLPs) as a decoder to solve the so-called error accumulation issue and conduct multi-

ahead forecasting in parallel. [

6

] uses a global RNN to directly output the parameters of a linear

SSM at each step for each time series, aiming to approximate nonlinear dynamics with locally linear

segments. In contrast, [

20

and SSMs, and maintain a complex latent process to conduct multi-step forecasting in parallel.

The well-known self-attention based Transformer [

1

] has recently been proposed for sequence

modeling and has achieved great success. Several recent works apply it to translation, speech, music

and image generation [

1

,

computationally prohibitive since the space complexity of self-attention grows quadratically with

where

, z

]

and

denotes the value of time series

i

at time

t

.We

are going to predict the next

+⌧

. Besides, let

the following conditional distribution

, x

; )=

+⌧

Y

+1

p(z

, x

1

Transformer

f

by taking advantage of the multi-head self-

dependencies, and different attention heads learn to focus on different aspects of temporal patterns.

respectively, with

, W