Bringing-Old-Photos-Back-to-Life（打包版）

# Old Photo Restoration (Official PyTorch Implementation) <img src='imgs/0001.jpg'/> ### [Project Page](http://raywzy.com/Old_Photo/) | [Paper (CVPR version)](https://arxiv.org/abs/2004.09484) | [Paper (Journal version)](https://arxiv.org/pdf/2009.07047v1.pdf) | [Pretrained Model](https://hkustconnect-my.sharepoint.com/:f:/g/personal/bzhangai_connect_ust_hk/Em0KnYOeSSxFtp4g_dhWdf0BdeT3tY12jIYJ6qvSf300cA?e=nXkJH2) | [Colab Demo](https://colab.research.google.com/drive/1NEm6AsybIiC5TwTU_4DqDkQO0nFRB-uA?usp=sharing) | [Replicate Demo & Docker Image](https://replicate.ai/zhangmozhe/bringing-old-photos-back-to-life) :fire: **Bringing Old Photos Back to Life, CVPR2020 (Oral)** **Old Photo Restoration via Deep Latent Space Translation, TPAMI 2022** [Ziyu Wan](http://raywzy.com/)¹, [Bo Zhang](https://www.microsoft.com/en-us/research/people/zhanbo/)², [Dongdong Chen](http://www.dongdongchen.bid/)³, [Pan Zhang](https://panzhang0212.github.io/)⁴, [Dong Chen](https://www.microsoft.com/en-us/research/people/doch/)², [Jing Liao](https://liaojing.github.io/html/)¹, [Fang Wen](https://www.microsoft.com/en-us/research/people/fangwen/)² <br> ¹City University of Hong Kong, ²Microsoft Research Asia, ³Microsoft Cloud AI, ⁴USTC <!-- ## Notes of this project The code originates from our research project and the aim is to demonstrate the research idea, so we have not optimized it from a product perspective. And we will spend time to address some common issues, such as out of memory issue, limited resolution, but will not involve too much in engineering problems, such as speedup of the inference, fastapi deployment and so on. **We welcome volunteers to contribute to this project to make it more usable for practical application.** --> ## :sparkles: News **2022.3.31**: Our new work regarding old film restoration will be published in CVPR 2022. For more details, please refer to the [project website](http://raywzy.com/Old_Film/) and [github repo](https://github.com/raywzy/Bringing-Old-Films-Back-to-Life). The framework now supports the restoration of high-resolution input. <img src='imgs/HR_result.png'> Training code is available and welcome to have a try and learn the training details. You can now play with our [Colab](https://colab.research.google.com/drive/1NEm6AsybIiC5TwTU_4DqDkQO0nFRB-uA?usp=sharing) and try it on your photos. ## Requirement The code is tested on Ubuntu with Nvidia GPUs and CUDA installed. Python>=3.6 is required to run the code. ## Installation Clone the Synchronized-BatchNorm-PyTorch repository for ``` cd Face_Enhancement/models/networks/ git clone https://github.com/vacancy/Synchronized-BatchNorm-PyTorch cp -rf Synchronized-BatchNorm-PyTorch/sync_batchnorm . cd ../../../ ``` ``` cd Global/detection_models git clone https://github.com/vacancy/Synchronized-BatchNorm-PyTorch cp -rf Synchronized-BatchNorm-PyTorch/sync_batchnorm . cd ../../ ``` Download the landmark detection pretrained model ``` cd Face_Detection/ wget http://dlib.net/files/shape_predictor_68_face_landmarks.dat.bz2 bzip2 -d shape_predictor_68_face_landmarks.dat.bz2 cd ../ ``` Download the pretrained model, put the file `Face_Enhancement/checkpoints.zip` under `./Face_Enhancement`, and put the file `Global/checkpoints.zip` under `./Global`. Then unzip them respectively. ``` cd Face_Enhancement/ wget https://github.com/microsoft/Bringing-Old-Photos-Back-to-Life/releases/download/v1.0/face_checkpoints.zip unzip face_checkpoints.zip cd ../ cd Global/ wget https://github.com/microsoft/Bringing-Old-Photos-Back-to-Life/releases/download/v1.0/global_checkpoints.zip unzip global_checkpoints.zip cd ../ ``` Install dependencies: ```bash pip install -r requirements.txt ``` ## :rocket: How to use? **Note**: GPU can be set 0 or 0,1,2 or 0,2; use -1 for CPU ### 1) Full Pipeline You could easily restore the old photos with one simple command after installation and downloading the pretrained model. For images without scratches: ``` python run.py --input_folder [test_image_folder_path] \ --output_folder [output_path] \ --GPU 0 ``` For scratched images: ``` python run.py --input_folder [test_image_folder_path] \ --output_folder [output_path] \ --GPU 0 \ --with_scratch ``` **For high-resolution images with scratches**: ``` python run.py --input_folder [test_image_folder_path] \ --output_folder [output_path] \ --GPU 0 \ --with_scratch \ --HR ``` Note: Please try to use the absolute path. The final results will be saved in `./output_path/final_output/`. You could also check the produced results of different steps in `output_path`. ### 2) Scratch Detection Currently we don't plan to release the scratched old photos dataset with labels directly. If you want to get the paired data, you could use our pretrained model to test the collected images to obtain the labels. ``` cd Global/ python detection.py --test_path [test_image_folder_path] \ --output_dir [output_path] \ --input_size [resize_256|full_size|scale_256] ``` <img src='imgs/scratch_detection.png'> ### 3) Global Restoration A triplet domain translation network is proposed to solve both structured degradation and unstructured degradation of old photos. <p align="center"> <img src='imgs/pipeline.PNG' width="50%" height="50%"/> </p> ``` cd Global/ python test.py --Scratch_and_Quality_restore \ --test_input [test_image_folder_path] \ --test_mask [corresponding mask] \ --outputs_dir [output_path] python test.py --Quality_restore \ --test_input [test_image_folder_path] \ --outputs_dir [output_path] ``` <img src='imgs/global.png'> ### 4) Face Enhancement We use a progressive generator to refine the face regions of old photos. More details could be found in our journal submission and `./Face_Enhancement` folder. <p align="center"> <img src='imgs/face_pipeline.jpg' width="60%" height="60%"/> </p> <img src='imgs/face.png'> > *NOTE*: > This repo is mainly for research purpose and we have not yet optimized the running performance. > > Since the model is pretrained with 256*256 images, the model may not work ideally for arbitrary resolution. ### 5) GUI A user-friendly GUI which takes input of image by user and shows result in respective window. #### How it works: 1. Run GUI.py file. 2. Click browse and select your image from test_images/old_w_scratch folder to remove scratches. 3. Click Modify Photo button. 4. Wait for a while and see results on GUI window. 5. Exit window by clicking Exit Window and get your result image in output folder. <img src='imgs/gui.PNG'> ## How to train? ### 1) Create Training File Put the folders of VOC dataset, collected old photos (e.g., Real_L_old and Real_RGB_old) into one shared folder. Then ``` cd Global/data/ python Create_Bigfile.py ``` Note: Remember to modify the code based on your own environment. ### 2) Train the VAEs of domain A and domain B respectively ``` cd .. python train_domain_A.py --use_v2_degradation --continue_train --training_dataset domain_A --name domainA_SR_old_photos --label_nc 0 --loadSize 256 --fineSize 256 --dataroot [your_data_folder] --no_instance --resize_or_crop crop_only --batchSize 100 --no_html --gpu_ids 0,1,2,3 --self_gen --nThreads 4 --n_downsample_global 3 --k_size 4 --use_v2 --mc 64 --start_r 1 --kl 1 --no_cgan --outputs_dir [your_output_folder] --checkpoints_dir [your_ckpt_folder] python train_domain_B.py --continue_train --training_dataset domain_B --name domainB_old_photos --label_nc 0 --loadSize 256 --fineSize 256 --dataroot [your_data_folder] --no_instance --resize_or_crop crop_only --batchSize 120 --no_html --gpu_ids 0,1,2,3 --self_gen --nThreads 4 --n_downsample_global 3 --k_