# Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting (AAAI'21 Best Paper)
![Python 3.6](https://img.shields.io/badge/python-3.6-green.svg?style=plastic)
![PyTorch 1.2](https://img.shields.io/badge/PyTorch%20-%23EE4C2C.svg?style=plastic)
![cuDNN 7.3.1](https://img.shields.io/badge/cudnn-7.3.1-green.svg?style=plastic)
![License CC BY-NC-SA](https://img.shields.io/badge/license-CC_BY--NC--SA--green.svg?style=plastic)
This is the origin Pytorch implementation of Informer in the following paper:
[Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting](https://arxiv.org/abs/2012.07436). Special thanks to `Jieqi Peng`@[cookieminions](https://github.com/cookieminions) for building this repo.
:triangular_flag_on_post:**News**(Mar 25, 2021): We update all experiment [results](#resultslink) with hyperparameter settings.
:triangular_flag_on_post:**News**(Feb 22, 2021): We provide [Colab Examples](#colablink) for friendly usage.
:triangular_flag_on_post:**News**(Feb 8, 2021): Our Informer paper has been awarded [AAAI'21 Best Paper \[Official\]](https://aaai.org/Conferences/AAAI-21/aaai-outstanding-and-distinguished-papers/)[\[Beihang\]](http://scse.buaa.edu.cn/info/1097/7443.htm)[\[Rutgers\]](https://www.business.rutgers.edu/news/hui-xiong-and-research-colleagues-receive-aaai-best-paper-award)! We will continue this line of research and update on this repo. Please star this repo and [cite](#citelink) our paper if you find our work is helpful for you.
<p align="center">
<img src=".\img\informer.png" height = "360" alt="" align=center />
<br><br>
<b>Figure 1.</b> The architecture of Informer.
</p>
## ProbSparse Attention
The self-attention scores form a long-tail distribution, where the "active" queries lie in the "head" scores and "lazy" queries lie in the "tail" area. We designed the ProbSparse Attention to select the "active" queries rather than the "lazy" queries. The ProbSparse Attention with Top-u queries forms a sparse Transformer by the probability distribution.
`Why not use Top-u keys?` The self-attention layer's output is the re-represent of input. It is formulated as a weighted combination of values w.r.t. the score of dot-product pairs. The top queries with full keys encourage a complete re-represent of leading components in the input, and it is equivalent to selecting the "head" scores among all the dot-product pairs. If we choose Top-u keys, the full keys just preserve the trivial sum of values within the "long tail" scores but wreck the leading components' re-represent.
<p align="center">
<img src=".\img\probsparse_intro.png" height = "320" alt="" align=center />
<br><br>
<b>Figure 2.</b> The illustration of ProbSparse Attention.
</p>
## Requirements
- Python 3.6
- matplotlib == 3.1.1
- numpy == 1.19.4
- pandas == 0.25.1
- scikit_learn == 0.21.3
- torch == 1.8.0
Dependencies can be installed using the following command:
```bash
pip install -r requirements.txt
```
## Data
The ETT dataset used in the paper can be downloaded in the repo [ETDataset](https://github.com/zhouhaoyi/ETDataset).
The required data files should be put into `data/ETT/` folder. A demo slice of the ETT data is illustrated in the following figure. Note that the input of each dataset is zero-mean normalized in this implementation.
<p align="center">
<img src="./img/data.png" height = "168" alt="" align=center />
<br><br>
<b>Figure 3.</b> An example of the ETT data.
</p>
The ECL data and Weather data can be downloaded here.
- [Google Drive](https://drive.google.com/drive/folders/1ohGYWWohJlOlb2gsGTeEq3Wii2egnEPR?usp=sharing)
- [BaiduPan](https://pan.baidu.com/s/1wyaGUisUICYHnfkZzWCwyA), password: 6gan
## Reproducibility
To easily reproduce the results you can follow the next steps:
1. Initialize the docker image using: `make init`.
2. Download the datasets using: `make dataset`.
3. Run each script in `scripts/` using `make run_module module="bash ETTh1.sh"` for each script.
4. Alternatively, run all the scripts at once:
```
for file in `ls scripts`; do make run_module module="bash scripts/$script"; done
```
## Usage
<span id="colablink">Colab Examples:</span> We provide google colabs to help reproduce and customize our repo, which includes `experiments(train and test)`, `prediction`, `visualization` and `custom data`.
[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1_X7O2BkFLvqyCdZzDZvV2MB0aAvYALLC)
Commands for training and testing the model with *ProbSparse* self-attention on Dataset ETTh1, ETTh2 and ETTm1 respectively:
```bash
# ETTh1
python -u main_informer.py --model informer --data ETTh1 --attn prob --freq h
# ETTh2
python -u main_informer.py --model informer --data ETTh2 --attn prob --freq h
# ETTm1
python -u main_informer.py --model informer --data ETTm1 --attn prob --freq t
```
More parameter information please refer to `main_informer.py`.
We provide a more detailed and complete command description for training and testing the model:
```python
python -u main_informer.py --model <model> --data <data>
--root_path <root_path> --data_path <data_path> --features <features>
--target <target> --freq <freq> --checkpoints <checkpoints>
--seq_len <seq_len> --label_len <label_len> --pred_len <pred_len>
--enc_in <enc_in> --dec_in <dec_in> --c_out <c_out> --d_model <d_model>
--n_heads <n_heads> --e_layers <e_layers> --d_layers <d_layers>
--s_layers <s_layers> --d_ff <d_ff> --factor <factor> --padding <padding>
--distil --dropout <dropout> --attn <attn> --embed <embed> --activation <activation>
--output_attention --do_predict --mix --cols <cols> --itr <itr>
--num_workers <num_workers> --train_epochs <train_epochs>
--batch_size <batch_size> --patience <patience> --des <des>
--learning_rate <learning_rate> --loss <loss> --lradj <lradj>
--use_amp --inverse --use_gpu <use_gpu> --gpu <gpu> --use_multi_gpu --devices <devices>
```
The detailed descriptions about the arguments are as following:
| Parameter name | Description of parameter |
| --- | --- |
| model | The model of experiment. This can be set to `informer`, `informerstack`, `informerlight(TBD)` |
| data | The dataset name |
| root_path | The root path of the data file (defaults to `./data/ETT/`) |
| data_path | The data file name (defaults to `ETTh1.csv`) |
| features | The forecasting task (defaults to `M`). This can be set to `M`,`S`,`MS` (M : multivariate predict multivariate, S : univariate predict univariate, MS : multivariate predict univariate) |
| target | Target feature in S or MS task (defaults to `OT`) |
| freq | Freq for time features encoding (defaults to `h`). This can be set to `s`,`t`,`h`,`d`,`b`,`w`,`m` (s:secondly, t:minutely, h:hourly, d:daily, b:business days, w:weekly, m:monthly).You can also use more detailed freq like 15min or 3h |
| checkpoints | Location of model checkpoints (defaults to `./checkpoints/`) |
| seq_len | Input sequence length of Informer encoder (defaults to 96) |
| label_len | Start token length of Informer decoder (defaults to 48) |
| pred_len | Prediction sequence length (defaults to 24) |
| enc_in | Encoder input size (defaults to 7) |
| dec_in | Decoder input size (defaults to 7) |
| c_out | Output size (defaults to 7) |
| d_model | Dimension of model (defaults to 512) |
| n_heads | Num of heads (defaults to 8) |
| e_layers | Num of encoder layers (defaults to 2) |
| d_layers | Num of decoder layers (defaults to 1) |
| s_layers | Num of stack encoder layers (defaults to `3,2,1`) |
| d_ff | Dimension of fcn (defaults to 2048) |
| factor | Probsparse attn factor (defaults to 5) |
| padding | Padding type(defaults to 0). |
| distil | Whether to use distilling in encoder, using this argument means not using distilling (defaults to `True`) |
| dropout | The probability of dropout (defaults to 0.05) |
| attn | Attention used i
没有合适的资源?快使用搜索试试~ 我知道了~
Informer代码详细注释版
共63个文件
py:17个
pyc:16个
xml:7个
1.该资源内容由用户上传,如若侵权请联系客服进行举报
2.虚拟产品一经售出概不退款(资源遇到问题,请及时私信上传者)
2.虚拟产品一经售出概不退款(资源遇到问题,请及时私信上传者)
版权申诉
5星 · 超过95%的资源 3 下载量 68 浏览量
2023-10-09
11:40:25
上传
评论 3
收藏 62.33MB ZIP 举报
温馨提示
逐行注释,十分详细! ## <span id="citelink">Citation</span> If you find this repository useful in your research, please consider citing the following paper: ``` @inproceedings{haoyietal-informer-2021, author = {Haoyi Zhou and Shanghang Zhang and Jieqi Peng and Shuai Zhang and Jianxin Li and Hui Xiong and Wancai Zhang}, title = {Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecas
资源推荐
资源详情
资源评论
收起资源包目录
Informer2020-main.zip (63个子文件)
Informer2020-main
Makefile 1KB
data
__init__.py 1B
ETT
ETTh1.csv 2.47MB
ETTh2.csv 2.31MB
ETTm2.csv 9.23MB
ETTm1.csv 9.88MB
data_loader.py 13KB
__pycache__
data_loader.cpython-37.pyc 9KB
__init__.cpython-37.pyc 143B
LICENSE 11KB
utils
__init__.py 0B
metrics.py 826B
masking.py 851B
timefeatures.py 5KB
tools.py 3KB
__pycache__
timefeatures.cpython-37.pyc 8KB
metrics.cpython-37.pyc 1KB
tools.cpython-37.pyc 3KB
__init__.cpython-37.pyc 144B
masking.cpython-37.pyc 1KB
.idea
Informer2020-main.iml 342B
workspace.xml 2KB
misc.xml 206B
inspectionProfiles
Project_Default.xml 2KB
profiles_settings.xml 174B
markdown.xml 302B
modules.xml 293B
deployment.xml 1KB
.gitignore 190B
img
data.png 10KB
probsparse_intro.png 339KB
informer.png 74KB
result_multivariate.png 296KB
result_univariate.png 381KB
exp
__init__.py 0B
exp_informer.py 11KB
exp_basic.py 885B
__pycache__
exp_informer.cpython-37.pyc 7KB
__init__.cpython-37.pyc 142B
exp_basic.cpython-37.pyc 1KB
Dockerfile 206B
main_informer.py 7KB
test anything.ipynb 12KB
environment.yml 198B
requirements.txt 91B
models
__init__.py 0B
attn.py 6KB
model.py 9KB
decoder.py 2KB
embed.py 5KB
encoder.py 4KB
__pycache__
decoder.cpython-37.pyc 2KB
attn.cpython-37.pyc 5KB
embed.cpython-37.pyc 5KB
encoder.cpython-37.pyc 4KB
__init__.cpython-37.pyc 145B
model.cpython-37.pyc 6KB
checkpoints
informer_ETTh1_ftM_sl96_ll48_pl24_dm512_nh8_el2_dl1_df2048_atprob_fc5_ebtimeF_dtTrue_mxTrue_test_0
checkpoint.pth 62.78MB
README.md 12KB
scripts
ETTm1.sh 2KB
WTH.sh 2KB
ETTh1.sh 2KB
ETTh2.sh 2KB
共 63 条
- 1
资源评论
- lsz20212024-08-22非常有用的资源,可以直接使用,对我很有用,果断支持!
- i_am_a_frog2024-05-24资源中能够借鉴的内容很多,值得学习的地方也很多,大家一起进步!
- a2301_766582024-01-19资源是宝藏资源,实用也是真的实用,感谢大佬分享~
小风飞子
- 粉丝: 374
- 资源: 1961
上传资源 快速赚钱
- 我的内容管理 展开
- 我的资源 快来上传第一个资源
- 我的收益 登录查看自己的收益
- 我的积分 登录查看自己的积分
- 我的C币 登录后查看C币余额
- 我的收藏
- 我的下载
- 下载帮助
最新资源
- 校园失物招领网站:设计与实现的全流程解析
- 基于java的公司固定资产管理系统.doc
- 基于java+springboot+vue+mysql的学科竞赛管理系统 源码+数据库+论文(高分毕业设计).zip
- 人工智能领域计算断层成像技术研究最新进展综述
- 微藻检测10-YOLO(v5至v11)、COCO、CreateML、Paligemma、TFRecord、VOC数据集合集.rar
- 快速排序算法在Rust语言的实现及其优化
- 2024年超融合网络架构研究与实践报告.pdf
- 埃森哲:2024年360°价值报告(英文版).pdf
- ISACA中国社区2024女性职业现状调查报告.pdf
- 如何看待“适度宽松”的货币政策.pdf
- 双目立体匹配三维重建点云C++ 本工程基于网上开源代码进行修改,内容如下: 1.修改为 VS2015 Debug win32 版本,支持利用特征点和 OpenCV 立体匹配算法进行进行三维重建及显示
- 华为云AI数字人生态赋能千行百业高效发展.pdf
- 金融业数据安全发展与实践报告.pdf
- 候鸟生命线—共筑候鸟迁徙保护网络.pdf
- 2024年全国统一电力市场建设情况及展望报告.pdf
- 2018-2023年粤港澳、京津冀、长三角三大区域高校本科专业调整趋势.pdf
资源上传下载、课程学习等过程中有任何疑问或建议,欢迎提出宝贵意见哦~我们会及时处理!
点击此处反馈
安全验证
文档复制为VIP权益,开通VIP直接复制
信息提交成功