多模态大模型应用-本地化部署VQGAN+CLIP-不使用colab-附流程教程-优质大模型应用实战.zip

# VQGAN-CLIP Overview A repo for running VQGAN+CLIP locally. This started out as a Katherine Crowson VQGAN+CLIP derived Google colab notebook. Original notebook: [![Open In Colab][colab-badge]][colab-notebook] [colab-notebook]: <https://colab.research.google.com/drive/1ZAus_gn2RhTZWzOWUpPERNC0Q8OhZRTZ> [colab-badge]: <https://colab.research.google.com/assets/colab-badge.svg> Some example images: <img src="./samples/Cartoon3.png" width="256px"></img><img src="./samples/Cartoon.png" width="256px"></img><img src="./samples/Cartoon2.png" width="256px"></img> <img src="./samples/Bedroom.png" width="256px"></img><img src="./samples/DemonBiscuits.png" width="256px"></img><img src="./samples/Football.png" width="256px"></img> <img src="./samples/Fractal_Landscape3.png" width="256px"></img><img src="./samples/Games_5.png" width="256px"></img> Environment: * Tested on Ubuntu 20.04 * GPU: Nvidia RTX 3090 * Typical VRAM requirements: * 24 GB for a 900x900 image * 10 GB for a 512x512 image * 8 GB for a 380x380 image You may also be interested in [CLIP Guided Diffusion](https://github.com/nerdyrodent/CLIP-Guided-Diffusion) ## Set up This example uses [Anaconda](https://www.anaconda.com/products/individual#Downloads) to manage virtual Python environments. Create a new virtual Python environment for VQGAN-CLIP: ```sh conda create --name vqgan python=3.9 conda activate vqgan ``` Install Pytorch in the new enviroment: Note: This installs the CUDA version of Pytorch, if you want to use an AMD graphics card, read the [AMD section below](#using-an-amd-graphics-card). ```sh pip install torch==1.9.0+cu111 torchvision==0.10.0+cu111 torchaudio==0.9.0 -f https://download.pytorch.org/whl/torch_stable.html ``` Install other required Python packages: ```sh pip install ftfy regex tqdm omegaconf pytorch-lightning IPython kornia imageio imageio-ffmpeg einops torch_optimizer setuptools==59.5.0 ``` Or use the ```requirements.txt``` file, which includes version numbers. Clone required repositories: ```sh cd project git clone 'https://github.com/openai/CLIP' git clone 'https://github.com/CompVis/taming-transformers' ``` Note: In my development environment both CLIP and taming-transformers are present in the local directory, and so aren't present in the `requirements.txt` or `vqgan.yml` files. As an alternative, you can also pip install taming-transformers and CLIP. You will also need at least 1 VQGAN pretrained model. E.g. ```sh mkdir checkpoints curl -L -o checkpoints/vqgan_imagenet_f16_16384.yaml -C - 'https://heibox.uni-heidelberg.de/d/a7530b09fed84f80a887/files/?p=%2Fconfigs%2Fmodel.yaml&dl=1' #ImageNet 16384 curl -L -o checkpoints/vqgan_imagenet_f16_16384.ckpt -C - 'https://heibox.uni-heidelberg.de/d/a7530b09fed84f80a887/files/?p=%2Fckpts%2Flast.ckpt&dl=1' #ImageNet 16384 ``` Note that users of ```curl``` on Microsoft Windows should use double quotes. The `download_models.sh` script is an optional way to download a number of models. By default, it will download just 1 model. See <https://github.com/CompVis/taming-transformers#overview-of-pretrained-models> for more information about VQGAN pre-trained models, including download links. By default, the model .yaml and .ckpt files are expected in the `checkpoints` directory. See <https://github.com/CompVis/taming-transformers> for more information on datasets and models. Video guides are also available: * Linux - https://www.youtube.com/watch?v=1Esb-ZjO7tw * Windows - https://www.youtube.com/watch?v=XH7ZP0__FXs ### Using an AMD graphics card Note: This hasn't been tested yet. ROCm can be used for AMD graphics cards instead of CUDA. You can check if your card is supported here: <https://github.com/RadeonOpenCompute/ROCm#supported-gpus> Install ROCm accordng to the instructions and don't forget to add the user to the video group: <https://rocmdocs.amd.com/en/latest/Installation_Guide/Installation-Guide.html> The usage and set up instructions above are the same, except for the line where you install Pytorch. Instead of `pip install torch==1.9.0+cu111 ...`, use the one or two lines which are displayed here (select Pip -> Python-> ROCm): <https://pytorch.org/get-started/locally/> ### Using the CPU If no graphics card can be found, the CPU is automatically used and a warning displayed. Regardless of an available graphics card, the CPU can also be used by adding this command line argument: `-cd cpu` This works with the CUDA version of Pytorch, even without CUDA drivers installed, but doesn't seem to work with ROCm as of now. ### Uninstalling Remove the Python enviroment: ```sh conda remove --name vqgan --all ``` and delete the `VQGAN-CLIP` directory. ## Run To generate images from text, specify your text prompt as shown in the example below: ```sh python generate.py -p "A painting of an apple in a fruit bowl" ``` <img src="./samples/A_painting_of_an_apple_in_a_fruitbowl.png" width="256px"></img> ## Multiple prompts Text and image prompts can be split using the pipe symbol in order to allow multiple prompts. You can also use a colon followed by a number to set a weight for that prompt. For example: ```sh python generate.py -p "A painting of an apple in a fruit bowl | psychedelic | surreal:0.5 | weird:0.25" ``` <img src="./samples/Apple_weird.png" width="256px"></img> Image prompts can be split in the same way. For example: ```sh python generate.py -p "A picture of a bedroom with a portrait of Van Gogh" -ip "samples/VanGogh.jpg | samples/Bedroom.png" ``` ### Story mode Sets of text prompts can be created using the caret symbol, in order to generate a sort of story mode. For example: ```sh python generate.py -p "A painting of a sunflower|photo:-1 ^ a painting of a rose ^ a painting of a tulip ^ a painting of a daisy flower ^ a photograph of daffodil" -cpe 1500 -zvid -i 6000 -zse 10 -vl 20 -zsc 1.005 -opt Adagrad -lr 0.15 -se 6000 ``` ## "Style Transfer" An input image with style text and a low number of iterations can be used create a sort of "style transfer" effect. For example: ```sh python generate.py -p "A painting in the style of Picasso" -ii samples/VanGogh.jpg -i 80 -se 10 -opt AdamW -lr 0.25 ``` | Output | Style | | ------------------------------------------------------------- | ----------- | | <img src="./samples/vvg_picasso.png" width="256px"></img> | Picasso | | <img src="./samples/vvg_sketch.png" width="256px"></img> | Sketch | | <img src="./samples/vvg_psychedelic.png" width="256px"></img> | Psychedelic | A video style transfer effect can be achived by specifying a directory of video frames in `video_style_dir`. Output will be saved in the steps directory, using the original video frame filenames. You can also use this as a sort of "batch mode" if you have a directory of images you want to apply a style to. This can also be combined with Story Mode if you don't wish to apply the same style to every images, but instead roll through a list of styles. ## Feedback example By feeding back the generated images and making slight changes, some interesting effects can be created. The example `zoom.sh` shows this by applying a zoom and rotate to generated images, before feeding them back in again. To use `zoom.sh`, specifying a text prompt, output filename and number of frames. E.g. ```sh ./zoom.sh "A painting of a red telephone box spinning through a time vortex" Telephone.png 150 ``` If you don't have ImageMagick installed, you can install it with ```sudo apt install imagemagick``` <img src="./samples/zoom.gif" width="256px"></img> There is also a simple zoom video creation option available. For example: ```sh python generate.py -p "The inside of a sphere" -zvid -i 4500 -zse 20 -vl 10 -zsc 0.97 -opt Adagrad -lr 0.15 -se 4500 ``` ## Random text example Use `random.sh` to make a batch of images from random text. Edit the text and number of generated images to your taste! ```sh ./random.sh