# VQGAN-CLIP Overview
A repo for running VQGAN+CLIP locally. This started out as a Katherine Crowson VQGAN+CLIP derived Google colab notebook.
<a href="https://replicate.ai/nerdyrodent/vqgan-clip"><img src="https://img.shields.io/static/v1?label=Replicate&message=Demo and Docker Image&color=blue"></a>
Original notebook: [![Open In Colab][colab-badge]][colab-notebook]
[colab-notebook]: <https://colab.research.google.com/drive/1ZAus_gn2RhTZWzOWUpPERNC0Q8OhZRTZ>
[colab-badge]: <https://colab.research.google.com/assets/colab-badge.svg>
Some example images:
<img src="./samples/Cartoon3.png" width="256px"></img><img src="./samples/Cartoon.png" width="256px"></img><img src="./samples/Cartoon2.png" width="256px"></img>
<img src="./samples/Bedroom.png" width="256px"></img><img src="./samples/DemonBiscuits.png" width="256px"></img><img src="./samples/Football.png" width="256px"></img>
<img src="./samples/Fractal_Landscape3.png" width="256px"></img><img src="./samples/Games_5.png" width="256px"></img>
Environment:
* Tested on Ubuntu 20.04
* GPU: Nvidia RTX 3090
* Typical VRAM requirements:
* 24 GB for a 900x900 image
* 10 GB for a 512x512 image
* 8 GB for a 380x380 image
You may also be interested in [CLIP Guided Diffusion](https://github.com/nerdyrodent/CLIP-Guided-Diffusion)
## Set up
This example uses [Anaconda](https://www.anaconda.com/products/individual#Downloads) to manage virtual Python environments.
Create a new virtual Python environment for VQGAN-CLIP:
```sh
conda create --name vqgan python=3.9
conda activate vqgan
```
Install Pytorch in the new enviroment:
Note: This installs the CUDA version of Pytorch, if you want to use an AMD graphics card, read the [AMD section below](#using-an-amd-graphics-card).
```sh
pip install torch==1.9.0+cu111 torchvision==0.10.0+cu111 torchaudio==0.9.0 -f https://download.pytorch.org/whl/torch_stable.html
```
Install other required Python packages:
```sh
pip install ftfy regex tqdm omegaconf pytorch-lightning IPython kornia imageio imageio-ffmpeg einops torch_optimizer setuptools==59.5.0
```
Or use the ```requirements.txt``` file, which includes version numbers.
Clone required repositories:
```sh
git clone 'https://github.com/nerdyrodent/VQGAN-CLIP'
cd VQGAN-CLIP
git clone 'https://github.com/openai/CLIP'
git clone 'https://github.com/CompVis/taming-transformers'
```
Note: In my development environment both CLIP and taming-transformers are present in the local directory, and so aren't present in the `requirements.txt` or `vqgan.yml` files.
As an alternative, you can also pip install taming-transformers and CLIP.
You will also need at least 1 VQGAN pretrained model. E.g.
```sh
mkdir checkpoints
curl -L -o checkpoints/vqgan_imagenet_f16_16384.yaml -C - 'https://heibox.uni-heidelberg.de/d/a7530b09fed84f80a887/files/?p=%2Fconfigs%2Fmodel.yaml&dl=1' #ImageNet 16384
curl -L -o checkpoints/vqgan_imagenet_f16_16384.ckpt -C - 'https://heibox.uni-heidelberg.de/d/a7530b09fed84f80a887/files/?p=%2Fckpts%2Flast.ckpt&dl=1' #ImageNet 16384
```
Note that users of ```curl``` on Microsoft Windows should use double quotes.
The `download_models.sh` script is an optional way to download a number of models. By default, it will download just 1 model.
See <https://github.com/CompVis/taming-transformers#overview-of-pretrained-models> for more information about VQGAN pre-trained models, including download links.
By default, the model .yaml and .ckpt files are expected in the `checkpoints` directory.
See <https://github.com/CompVis/taming-transformers> for more information on datasets and models.
Video guides are also available:
* Linux - https://www.youtube.com/watch?v=1Esb-ZjO7tw
* Windows - https://www.youtube.com/watch?v=XH7ZP0__FXs
### Using an AMD graphics card
Note: This hasn't been tested yet.
ROCm can be used for AMD graphics cards instead of CUDA. You can check if your card is supported here:
<https://github.com/RadeonOpenCompute/ROCm#supported-gpus>
Install ROCm accordng to the instructions and don't forget to add the user to the video group:
<https://rocmdocs.amd.com/en/latest/Installation_Guide/Installation-Guide.html>
The usage and set up instructions above are the same, except for the line where you install Pytorch.
Instead of `pip install torch==1.9.0+cu111 ...`, use the one or two lines which are displayed here (select Pip -> Python-> ROCm):
<https://pytorch.org/get-started/locally/>
### Using the CPU
If no graphics card can be found, the CPU is automatically used and a warning displayed.
Regardless of an available graphics card, the CPU can also be used by adding this command line argument: `-cd cpu`
This works with the CUDA version of Pytorch, even without CUDA drivers installed, but doesn't seem to work with ROCm as of now.
### Uninstalling
Remove the Python enviroment:
```sh
conda remove --name vqgan --all
```
and delete the `VQGAN-CLIP` directory.
## Run
To generate images from text, specify your text prompt as shown in the example below:
```sh
python generate.py -p "A painting of an apple in a fruit bowl"
```
<img src="./samples/A_painting_of_an_apple_in_a_fruitbowl.png" width="256px"></img>
## Multiple prompts
Text and image prompts can be split using the pipe symbol in order to allow multiple prompts.
You can also use a colon followed by a number to set a weight for that prompt. For example:
```sh
python generate.py -p "A painting of an apple in a fruit bowl | psychedelic | surreal:0.5 | weird:0.25"
```
<img src="./samples/Apple_weird.png" width="256px"></img>
Image prompts can be split in the same way. For example:
```sh
python generate.py -p "A picture of a bedroom with a portrait of Van Gogh" -ip "samples/VanGogh.jpg | samples/Bedroom.png"
```
### Story mode
Sets of text prompts can be created using the caret symbol, in order to generate a sort of story mode. For example:
```sh
python generate.py -p "A painting of a sunflower|photo:-1 ^ a painting of a rose ^ a painting of a tulip ^ a painting of a daisy flower ^ a photograph of daffodil" -cpe 1500 -zvid -i 6000 -zse 10 -vl 20 -zsc 1.005 -opt Adagrad -lr 0.15 -se 6000
```
## "Style Transfer"
An input image with style text and a low number of iterations can be used create a sort of "style transfer" effect. For example:
```sh
python generate.py -p "A painting in the style of Picasso" -ii samples/VanGogh.jpg -i 80 -se 10 -opt AdamW -lr 0.25
```
| Output | Style |
| ------------------------------------------------------------- | ----------- |
| <img src="./samples/vvg_picasso.png" width="256px"></img> | Picasso |
| <img src="./samples/vvg_sketch.png" width="256px"></img> | Sketch |
| <img src="./samples/vvg_psychedelic.png" width="256px"></img> | Psychedelic |
A video style transfer effect can be achived by specifying a directory of video frames in `video_style_dir`. Output will be saved in the steps directory, using the original video frame filenames. You can also use this as a sort of "batch mode" if you have a directory of images you want to apply a style to. This can also be combined with Story Mode if you don't wish to apply the same style to every images, but instead roll through a list of styles.
## Feedback example
By feeding back the generated images and making slight changes, some interesting effects can be created.
The example `zoom.sh` shows this by applying a zoom and rotate to generated images, before feeding them back in again.
To use `zoom.sh`, specifying a text prompt, output filename and number of frames. E.g.
```sh
./zoom.sh "A painting of a red telephone box spinning through a time vortex" Telephone.png 150
```
If you don't have ImageMagick installed, you can install it with ```sudo apt install imagemagick```
<img src="./s
没有合适的资源?快使用搜索试试~ 我知道了~
深度学习图像生成开源项目VQGAN-CLIP项目源码
共56个文件
png:14个
sample:13个
sh:5个
需积分: 0 3 下载量 20 浏览量
2023-08-23
20:51:46
上传
评论
收藏 63.09MB ZIP 举报
温馨提示
github仓库地址: https://github.com/nerdyrodent/VQGAN-CLIP 基于CLIP算法实现的图像文本编码空间共享为相关图像文本多模态提供了有力指导,VQGAN-CLIP基于CLIP算法,通过计算所生成图像与文本信息的损失进行训练指导。
资源推荐
资源详情
资源评论
收起资源包目录
VQGAN-CLIP.zip (56个子文件)
zoom.sh 1KB
download_models.sh 5KB
LICENSE 2KB
samples
vvg_picasso.png 484KB
zoom.gif 23.77MB
Cartoon.png 414KB
Apple_weird.png 511KB
DemonBiscuits.png 404KB
Bedroom.png 515KB
Fractal_Landscape3.png 492KB
samples.txt 33B
Football.png 462KB
Games_5.png 405KB
A_painting_of_an_apple_in_a_fruitbowl.png 458KB
vvg_sketch.png 529KB
Cartoon2.png 394KB
Cartoon3.png 414KB
pencil_sketch_2.png 531KB
VanGogh.jpg 416KB
vvg_psychedelic.png 578KB
random.sh 1KB
predict.py 22KB
.git
index 3KB
HEAD 21B
refs
heads
main 41B
tags
remotes
origin
HEAD 30B
objects
pack
pack-0be68c251ed037c3671b6a03d36d1516c445eeb3.pack 32.47MB
pack-0be68c251ed037c3671b6a03d36d1516c445eeb3.idx 14KB
info
description 73B
packed-refs 112B
info
exclude 240B
logs
HEAD 189B
refs
heads
main 189B
remotes
origin
HEAD 189B
hooks
post-update.sample 189B
prepare-commit-msg.sample 1KB
commit-msg.sample 896B
pre-receive.sample 544B
update.sample 4KB
pre-commit.sample 2KB
pre-rebase.sample 5KB
applypatch-msg.sample 478B
fsmonitor-watchman.sample 5KB
push-to-checkout.sample 3KB
pre-applypatch.sample 424B
pre-push.sample 1KB
pre-merge-commit.sample 416B
config 300B
opt_tester.sh 2KB
cog.yaml 594B
vqgan.yml 2KB
generate.py 41KB
requirements.txt 1KB
.gitignore 190B
README.md 13KB
video_styler.sh 2KB
共 56 条
- 1
资源评论
西山点子王
- 粉丝: 3
- 资源: 6
上传资源 快速赚钱
- 我的内容管理 展开
- 我的资源 快来上传第一个资源
- 我的收益 登录查看自己的收益
- 我的积分 登录查看自己的积分
- 我的C币 登录后查看C币余额
- 我的收藏
- 我的下载
- 下载帮助
安全验证
文档复制为VIP权益,开通VIP直接复制
信息提交成功