Python-ESRGANEnhancedSRGAN增强的超分辨率生成对抗网络

## ESRGAN (Enhanced SRGAN) [[BasicSR]](https://github.com/xinntao/BasicSR) [[EDVR]](https://github.com/xinntao/EDVR) [[DNI]](https://xinntao.github.io/projects/DNI) ### :smiley: Training codes are in [BasicSR](https://github.com/xinntao/BasicSR) repo. We have simplified the network structure file.<br/> You can convert the previously save models (`*.pth`) with the script `transer_RRDB_models.py`;<br/> If you want to use the old arch, you can find it [here](https://github.com/xinntao/ESRGAN/releases/tag/old-arch). --- Check out our new work on:<br/> 1. **Video Super-Resolution**: [`EDVR: Video Restoration with Enhanced Deformable Convolutional Networks`](https://xinntao.github.io/projects/EDVR), which has won all four tracks in NTIRE 2019 Challenges on Video Restoration and Enhancement (CVPR19 Workshops). 2. **DNI (CVPR19)**: [`Deep Network Interpolation for Continuous Imagery Effect Transition`](https://xinntao.github.io/projects/DNI) --- ### Enhanced Super-Resolution Generative Adversarial Networks By Xintao Wang, [Ke Yu](https://yuke93.github.io/), Shixiang Wu, [Jinjin Gu](http://www.jasongt.com/), Yihao Liu, [Chao Dong](https://scholar.google.com.hk/citations?user=OSDCB0UAAAAJ&hl=en), [Yu Qiao](http://mmlab.siat.ac.cn/yuqiao/), [Chen Change Loy](http://personal.ie.cuhk.edu.hk/~ccloy/) This repo only provides simple testing codes, pretrained models and the network strategy demo. For full training and testing codes, please refer to [BasicSR](https://github.com/xinntao/BasicSR). We won the first place in [PIRM2018-SR competition](https://www.pirm2018.org/PIRM-SR.html) (region 3) and got the best perceptual index. The paper is accepted to [ECCV2018 PIRM Workshop](https://pirm2018.org/). :triangular_flag_on_post: Add [Frequently Asked Questions](https://github.com/xinntao/ESRGAN/blob/master/QA.md). > For instance, > 1. How to reproduce your results in the PIRM18-SR Challenge (with low perceptual index)? > 2. How do you get the perceptual index in your ESRGAN paper? #### BibTeX <!-- @article{wang2018esrgan, author={Wang, Xintao and Yu, Ke and Wu, Shixiang and Gu, Jinjin and Liu, Yihao and Dong, Chao and Loy, Chen Change and Qiao, Yu and Tang, Xiaoou}, title={ESRGAN: Enhanced super-resolution generative adversarial networks}, journal={arXiv preprint arXiv:1809.00219}, year={2018} } --> @InProceedings{wang2018esrgan, author = {Wang, Xintao and Yu, Ke and Wu, Shixiang and Gu, Jinjin and Liu, Yihao and Dong, Chao and Qiao, Yu and Loy, Chen Change}, title = {ESRGAN: Enhanced super-resolution generative adversarial networks}, booktitle = {The European Conference on Computer Vision Workshops (ECCVW)}, month = {September}, year = {2018} } <p align="center"> <img src="figures/baboon.jpg"> </p> The **RRDB_PSNR** PSNR_oriented model trained with DF2K dataset (a merged dataset with [DIV2K](https://data.vision.ee.ethz.ch/cvl/DIV2K/) and [Flickr2K](http://cv.snu.ac.kr/research/EDSR/Flickr2K.tar) (proposed in [EDSR](https://github.com/LimBee/NTIRE2017))) is also able to achive high PSNR performance. | _Method | _{Training dataset} | _Set5 | _Set14 | _BSD100 | _Urban100 | _Manga109 | |:---:|:---:|:---:|:---:|:---:|:---:|:---:| | _{[SRCNN](http://mmlab.ie.cuhk.edu.hk/projects/SRCNN.html)}| ₂₉₁| _30.48/0.8628 |_27.50/0.7513|_26.90/0.7101|_24.52/0.7221|_27.58/0.8555| | _{[EDSR](https://github.com/thstkdgus35/EDSR-PyTorch)} | _DIV2K | _32.46/0.8968 | _28.80/0.7876 | _27.71/0.7420 | _26.64/0.8033 | _31.02/0.9148 | | _{[RCAN](https://github.com/yulunzhang/RCAN)} | _DIV2K | _32.63/0.9002 | _28.87/0.7889 | _27.77/0.7436 | _{26.82/ 0.8087}| _{31.22/ 0.9173}| |_RRDB(ours)| _DF2K| _{**32.73/0.9011**} |_{**28.99/0.7917**} |_{**27.85/0.7455**} |_{**27.03/0.8153**} |_{**31.66/0.9196**}| ## Quick Test #### Dependencies - Python 3 - [PyTorch >= 1.0](https://pytorch.org/) (CUDA version >= 7.5 if installing with CUDA. [More details](https://pytorch.org/get-started/previous-versions/)) - Python packages: `pip install numpy opencv-python` ### Test models 1. Clone this github repo. ``` git clone https://github.com/xinntao/ESRGAN cd ESRGAN ``` 2. Place your own **low-resolution images** in `./LR` folder. (There are two sample images - baboon and comic). 3. Download pretrained models from [Google Drive](https://drive.google.com/drive/u/0/folders/17VYV_SoZZesU6mbxz2dMAIccSSlqLecY) or [Baidu Drive](https://pan.baidu.com/s/1-Lh6ma-wXzfH8NqeBtPaFQ). Place the models in `./models`. We provide two models with high perceptual quality and high PSNR performance (see [model list](https://github.com/xinntao/ESRGAN/tree/master/models)). 4. Run test. We provide ESRGAN model and RRDB_PSNR model and you can config in the `test.py`. ``` python test.py ``` 5. The results are in `./results` folder. ### Network interpolation demo You can interpolate the RRDB_ESRGAN and RRDB_PSNR models with alpha in [0, 1]. 1. Run `python net_interp.py 0.8`, where *0.8* is the interpolation parameter and you can change it to any value in [0,1]. 2. Run `python test.py models/interp_08.pth`, where *models/interp_08.pth* is the model path. <p align="center"> <img height="400" src="figures/43074.gif"> </p> ## Perceptual-driven SR Results You can download all the resutls from [Google Drive](https://drive.google.com/drive/folders/1iaM-c6EgT1FNoJAOKmDrK7YhEhtlKcLx?usp=sharing). (:heavy_check_mark: included; :heavy_minus_sign: not included; :o: TODO) HR images can be downloaed from [BasicSR-Datasets](https://github.com/xinntao/BasicSR#datasets). | Datasets |LR | [*ESRGAN*](https://arxiv.org/abs/1809.00219) | [SRGAN](https://arxiv.org/abs/1609.04802) | [EnhanceNet](http://openaccess.thecvf.com/content_ICCV_2017/papers/Sajjadi_EnhanceNet_Single_Image_ICCV_2017_paper.pdf) | [CX](https://arxiv.org/abs/1803.04626) | |:---:|:---:|:---:|:---:|:---:|:---:| | Set5 |:heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark:| :o: | | Set14 | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark:| :o: | | BSDS100 | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark: | :heavy_check_mark:| :o: | | [PIRM](https://pirm.github.io/) <br>^{(val, test)} | :heavy_check_mark: | :heavy_check_mark: | :heavy_minus_sign: | :heavy_check_mark:| :heavy_check_mark: | | [OST300](https://arxiv.org/pdf/1804.02815.pdf) |:heavy_check_mark: | :heavy_check_mark: | :heavy_minus_sign: | :heavy_check_mark:| :o: | | urban100 | :heavy_check_mark: | :heavy_check_mark: | :heavy_minus_sign: | :heavy_check_mark:| :o: | | [DIV2K](https://data.vision.ee.ethz.ch/cvl/DIV2K/) <br>^{(val, test)} | :heavy_check_mark: | :heavy_check_mark: | :heavy_minus_sign: | :heavy_check_mark:| :o: | ## ESRGAN We improve the [SRGAN](https://arxiv.org/abs/1609.04802) from three aspects: 1. adopt a deeper model using Residual-in-Residual Dense Block (RRDB) without batch normalization layers. 2. employ [Relativistic average GAN](https://ajolicoeur.wordpress.com/relativisticgan/) instead of the vanilla GAN. 3. improve the perceptual loss by using the features before activation. In contrast to SRGAN, which claimed that **deeper models are increasingly difficult to train**, our deeper ESRGAN model shows its superior performance with easy training. <p align="center"> <img height="120" src="figures/architecture.jpg"> </p> <p align="center"> <img height="180" src="figures/RRDB.png"> </p> ## Network Interpolation We propose the **network interpolation strategy** to balance the visual quality and PSNR. <p align="center"> <img height="500" src="figures/net_interp.jpg"> </p> We sho