# Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting (AAAI'21 Best Paper)
![Python 3.6](https://img.shields.io/badge/python-3.6-green.svg?style=plastic)
![PyTorch 1.2](https://img.shields.io/badge/PyTorch%20-%23EE4C2C.svg?style=plastic)
![cuDNN 7.3.1](https://img.shields.io/badge/cudnn-7.3.1-green.svg?style=plastic)
![License CC BY-NC-SA](https://img.shields.io/badge/license-CC_BY--NC--SA--green.svg?style=plastic)
This is the origin Pytorch implementation of Informer in the following paper:
[Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting](https://arxiv.org/abs/2012.07436). Special thanks to `Jieqi Peng`@[cookieminions](https://github.com/cookieminions) for building this repo.
:triangular_flag_on_post:**News**(Mar 27, 2023): We will release Informer V2 soon.
:triangular_flag_on_post:**News**(Feb 28, 2023): The Informer's [extension paper](https://www.sciencedirect.com/science/article/pii/S0004370223000322) is online on AIJ.
:triangular_flag_on_post:**News**(Mar 25, 2021): We update all experiment [results](#resultslink) with hyperparameter settings.
:triangular_flag_on_post:**News**(Feb 22, 2021): We provide [Colab Examples](#colablink) for friendly usage.
:triangular_flag_on_post:**News**(Feb 8, 2021): Our Informer paper has been awarded [AAAI'21 Best Paper \[Official\]](https://aaai.org/Conferences/AAAI-21/aaai-outstanding-and-distinguished-papers/)[\[Beihang\]](http://scse.buaa.edu.cn/info/1097/7443.htm)[\[Rutgers\]](https://www.business.rutgers.edu/news/hui-xiong-and-research-colleagues-receive-aaai-best-paper-award)! We will continue this line of research and update on this repo. Please star this repo and [cite](#citelink) our paper if you find our work is helpful for you.
<p align="center">
<img src=".\img\informer.png" height = "360" alt="" align=center />
<br><br>
<b>Figure 1.</b> The architecture of Informer.
</p>
## ProbSparse Attention
The self-attention scores form a long-tail distribution, where the "active" queries lie in the "head" scores and "lazy" queries lie in the "tail" area. We designed the ProbSparse Attention to select the "active" queries rather than the "lazy" queries. The ProbSparse Attention with Top-u queries forms a sparse Transformer by the probability distribution.
`Why not use Top-u keys?` The self-attention layer's output is the re-represent of input. It is formulated as a weighted combination of values w.r.t. the score of dot-product pairs. The top queries with full keys encourage a complete re-represent of leading components in the input, and it is equivalent to selecting the "head" scores among all the dot-product pairs. If we choose Top-u keys, the full keys just preserve the trivial sum of values within the "long tail" scores but wreck the leading components' re-represent.
<p align="center">
<img src=".\img\probsparse_intro.png" height = "320" alt="" align=center />
<br><br>
<b>Figure 2.</b> The illustration of ProbSparse Attention.
</p>
## Requirements
- Python 3.6
- matplotlib == 3.1.1
- numpy == 1.19.4
- pandas == 0.25.1
- scikit_learn == 0.21.3
- torch == 1.8.0
Dependencies can be installed using the following command:
```bash
pip install -r requirements.txt
```
## Data
The ETT dataset used in the paper can be downloaded in the repo [ETDataset](https://github.com/zhouhaoyi/ETDataset).
The required data files should be put into `data/ETT/` folder. A demo slice of the ETT data is illustrated in the following figure. Note that the input of each dataset is zero-mean normalized in this implementation.
<p align="center">
<img src="./img/data.png" height = "168" alt="" align=center />
<br><br>
<b>Figure 3.</b> An example of the ETT data.
</p>
The ECL data and Weather data can be downloaded here.
- [Google Drive](https://drive.google.com/drive/folders/1ohGYWWohJlOlb2gsGTeEq3Wii2egnEPR?usp=sharing)
- [BaiduPan](https://pan.baidu.com/s/1wyaGUisUICYHnfkZzWCwyA), password: 6gan
## Reproducibility
To easily reproduce the results you can follow the next steps:
1. Initialize the docker image using: `make init`.
2. Download the datasets using: `make dataset`.
3. Run each script in `scripts/` using `make run_module module="bash ETTh1.sh"` for each script.
4. Alternatively, run all the scripts at once:
```
for file in `ls scripts`; do make run_module module="bash scripts/$script"; done
```
## Usage
<span id="colablink">Colab Examples:</span> We provide google colabs to help reproduce and customize our repo, which includes `experiments(train and test)`, `prediction`, `visualization` and `custom data`.
[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1_X7O2BkFLvqyCdZzDZvV2MB0aAvYALLC)
Commands for training and testing the model with *ProbSparse* self-attention on Dataset ETTh1, ETTh2 and ETTm1 respectively:
```bash
# ETTh1
python -u main_informer.py --model informer --data ETTh1 --attn prob --freq h
# ETTh2
python -u main_informer.py --model informer --data ETTh2 --attn prob --freq h
# ETTm1
python -u main_informer.py --model informer --data ETTm1 --attn prob --freq t
```
More parameter information please refer to `main_informer.py`.
We provide a more detailed and complete command description for training and testing the model:
```python
python -u main_informer.py --model <model> --data <data>
--root_path <root_path> --data_path <data_path> --features <features>
--target <target> --freq <freq> --checkpoints <checkpoints>
--seq_len <seq_len> --label_len <label_len> --pred_len <pred_len>
--enc_in <enc_in> --dec_in <dec_in> --c_out <c_out> --d_model <d_model>
--n_heads <n_heads> --e_layers <e_layers> --d_layers <d_layers>
--s_layers <s_layers> --d_ff <d_ff> --factor <factor> --padding <padding>
--distil --dropout <dropout> --attn <attn> --embed <embed> --activation <activation>
--output_attention --do_predict --mix --cols <cols> --itr <itr>
--num_workers <num_workers> --train_epochs <train_epochs>
--batch_size <batch_size> --patience <patience> --des <des>
--learning_rate <learning_rate> --loss <loss> --lradj <lradj>
--use_amp --inverse --use_gpu <use_gpu> --gpu <gpu> --use_multi_gpu --devices <devices>
```
The detailed descriptions about the arguments are as following:
| Parameter name | Description of parameter |
| --- | --- |
| model | The model of experiment. This can be set to `informer`, `informerstack`, `informerlight(TBD)` |
| data | The dataset name |
| root_path | The root path of the data file (defaults to `./data/ETT/`) |
| data_path | The data file name (defaults to `ETTh1.csv`) |
| features | The forecasting task (defaults to `M`). This can be set to `M`,`S`,`MS` (M : multivariate predict multivariate, S : univariate predict univariate, MS : multivariate predict univariate) |
| target | Target feature in S or MS task (defaults to `OT`) |
| freq | Freq for time features encoding (defaults to `h`). This can be set to `s`,`t`,`h`,`d`,`b`,`w`,`m` (s:secondly, t:minutely, h:hourly, d:daily, b:business days, w:weekly, m:monthly).You can also use more detailed freq like 15min or 3h |
| checkpoints | Location of model checkpoints (defaults to `./checkpoints/`) |
| seq_len | Input sequence length of Informer encoder (defaults to 96) |
| label_len | Start token length of Informer decoder (defaults to 48) |
| pred_len | Prediction sequence length (defaults to 24) |
| enc_in | Encoder input size (defaults to 7) |
| dec_in | Decoder input size (defaults to 7) |
| c_out | Output size (defaults to 7) |
| d_model | Dimension of model (defaults to 512) |
| n_heads | Num of heads (defaults to 8) |
| e_layers | Num of encoder layers (defaults to 2) |
| d_layers | Num of decoder layers (defaults to 1) |
| s_layers | Num of stack encoder layers (defaults to `3,2,1`) |
| d_ff | Dimension of fcn (defaults to 2048) |
| factor | Probsparse attn factor (defaults t
没有合适的资源?快使用搜索试试~ 我知道了~
Informer-hp-高质量精讲.rarInformer-hp-高质量精讲.rar
共106个文件
pyc:32个
py:19个
xml:15个
需积分: 4 41 下载量 16 浏览量
2023-07-05
22:45:55
上传
评论
收藏 156.24MB RAR 举报
温馨提示
Informer-hp-高质量精讲.rarInformer-hp-高质量精讲.rar
资源推荐
资源详情
资源评论
收起资源包目录
Informer-hp-高质量精讲.rarInformer-hp-高质量精讲.rar (106个子文件)
HP.csv 25KB
Dockerfile 206B
.gitignore 50B
.gitignore 50B
.gitignore 50B
Informer-hp.iml 491B
Informer2020-main.iml 486B
results.iml 291B
LICENSE 11KB
Makefile 1KB
README.md 13KB
true.npy 36KB
pred.npy 36KB
pred.npy 36KB
true.npy 36KB
pred.npy 3KB
pred.npy 3KB
pred.npy 3KB
true.npy 3KB
true.npy 3KB
true.npy 3KB
metrics.npy 148B
metrics.npy 148B
metrics.npy 148B
metrics.npy 148B
metrics.npy 148B
result_univariate.png 381KB
probsparse_intro.png 339KB
result_multivariate.png 296KB
informer.png 74KB
data.png 10KB
checkpoint.pth 62.83MB
checkpoint.pth 62.83MB
data_loader.py 13KB
exp_informer.py 11KB
main_informer.py 7KB
model.py 7KB
attn.py 6KB
timefeatures.py 5KB
embed.py 4KB
encoder.py 3KB
tools.py 3KB
decoder.py 2KB
exp_basic.py 875B
masking.py 851B
metrics.py 826B
huatu.py 395B
huatu.py 395B
__init__.py 1B
__init__.py 0B
__init__.py 0B
__init__.py 0B
exp_informer.cpython-311.pyc 17KB
data_loader.cpython-311.pyc 17KB
timefeatures.cpython-311.pyc 11KB
attn.cpython-311.pyc 11KB
embed.cpython-311.pyc 10KB
data_loader.cpython-37.pyc 9KB
model.cpython-311.pyc 9KB
timefeatures.cpython-37.pyc 8KB
exp_informer.cpython-37.pyc 7KB
encoder.cpython-311.pyc 7KB
tools.cpython-311.pyc 6KB
embed.cpython-37.pyc 5KB
attn.cpython-37.pyc 5KB
model.cpython-37.pyc 5KB
decoder.cpython-311.pyc 4KB
encoder.cpython-37.pyc 4KB
tools.cpython-37.pyc 3KB
metrics.cpython-311.pyc 3KB
masking.cpython-311.pyc 3KB
exp_basic.cpython-311.pyc 2KB
decoder.cpython-37.pyc 2KB
exp_basic.cpython-37.pyc 1KB
metrics.cpython-37.pyc 1KB
masking.cpython-37.pyc 1KB
__init__.cpython-311.pyc 161B
__init__.cpython-311.pyc 160B
__init__.cpython-311.pyc 159B
__init__.cpython-311.pyc 158B
__init__.cpython-37.pyc 142B
__init__.cpython-37.pyc 141B
__init__.cpython-37.pyc 140B
__init__.cpython-37.pyc 139B
Informer-hp.rar 49.84MB
ETTh1.sh 2KB
ETTh2.sh 2KB
WTH.sh 2KB
ETTm1.sh 2KB
requirements.txt 91B
Project_Default.xml 9KB
workspace.xml 5KB
workspace.xml 4KB
workspace.xml 1KB
Project_Default.xml 811B
modules.xml 293B
modules.xml 281B
modules.xml 273B
misc.xml 194B
encodings.xml 189B
共 106 条
- 1
- 2
资源评论
程序员奇奇
- 粉丝: 3w+
- 资源: 294
上传资源 快速赚钱
- 我的内容管理 展开
- 我的资源 快来上传第一个资源
- 我的收益 登录查看自己的收益
- 我的积分 登录查看自己的积分
- 我的C币 登录后查看C币余额
- 我的收藏
- 我的下载
- 下载帮助
安全验证
文档复制为VIP权益,开通VIP直接复制
信息提交成功