text-generation-webui-main

# Text generation web UI A gradio web UI for running Large Language Models like LLaMA, llama.cpp, GPT-J, OPT, and GALACTICA. Its goal is to become the [AUTOMATIC1111/stable-diffusion-webui](https://github.com/AUTOMATIC1111/stable-diffusion-webui) of text generation. | |  | |:---:|:---:| | |  | ## Features * 3 interface modes: default, notebook, and chat * Multiple model backends: transformers, llama.cpp, ExLlama, AutoGPTQ, GPTQ-for-LLaMa * Dropdown menu for quickly switching between different models * LoRA: load and unload LoRAs on the fly, train a new LoRA * Precise instruction templates for chat mode, including Llama 2, Alpaca, Vicuna, WizardLM, StableLM, and many others * [Multimodal pipelines, including LLaVA and MiniGPT-4](https://github.com/oobabooga/text-generation-webui/tree/main/extensions/multimodal) * 8-bit and 4-bit inference through bitsandbytes * CPU mode for transformers models * [DeepSpeed ZeRO-3 inference](docs/DeepSpeed.md) * [Extensions](docs/Extensions.md) * [Custom chat characters](docs/Chat-mode.md) * Very efficient text streaming * Markdown output with LaTeX rendering, to use for instance with [GALACTICA](https://github.com/paperswithcode/galai) * Nice HTML output for GPT-4chan * API, including endpoints for websocket streaming ([see the examples](https://github.com/oobabooga/text-generation-webui/blob/main/api-examples)) To learn how to use the various features, check out the Documentation: https://github.com/oobabooga/text-generation-webui/tree/main/docs ## Installation ### One-click installers | Windows | Linux | macOS | WSL | |--------|--------|--------|--------| | [oobabooga-windows.zip](https://github.com/oobabooga/text-generation-webui/releases/download/installers/oobabooga_windows.zip) | [oobabooga-linux.zip](https://github.com/oobabooga/text-generation-webui/releases/download/installers/oobabooga_linux.zip) |[oobabooga-macos.zip](https://github.com/oobabooga/text-generation-webui/releases/download/installers/oobabooga_macos.zip) | [oobabooga-wsl.zip](https://github.com/oobabooga/text-generation-webui/releases/download/installers/oobabooga_wsl.zip) | Just download the zip above, extract it, and double-click on "start". The web UI and all its dependencies will be installed in the same folder. * The source codes are here: https://github.com/oobabooga/one-click-installers * There is no need to run the installers as admin. * AMD doesn't work on Windows. * Huge thanks to [@jllllll](https://github.com/jllllll), [@ClayShoaf](https://github.com/ClayShoaf), and [@xNul](https://github.com/xNul) for their contributions to these installers. ### Manual installation using Conda Recommended if you have some experience with the command line. #### 0. Install Conda https://docs.conda.io/en/latest/miniconda.html On Linux or WSL, it can be automatically installed with these two commands: ``` curl -sL "https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh" > "Miniconda3.sh" bash Miniconda3.sh ``` Source: https://educe-ubc.github.io/conda.html #### 1. Create a new conda environment ``` conda create -n textgen python=3.10.9 conda activate textgen ``` #### 2. Install Pytorch | System | GPU | Command | |--------|---------|---------| | Linux/WSL | NVIDIA | `pip3 install torch torchvision torchaudio` | | Linux/WSL | CPU only | `pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cpu` | | Linux | AMD | `pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/rocm5.4.2` | | MacOS + MPS | Any | `pip3 install torch torchvision torchaudio` | | Windows | NVIDIA | `pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu117` | | Windows | CPU only | `pip3 install torch torchvision torchaudio` | The up-to-date commands can be found here: https://pytorch.org/get-started/locally/. #### 2.1 Special instructions * MacOS users: https://github.com/oobabooga/text-generation-webui/pull/393 * AMD users: https://rentry.org/eq3hg #### 3. Install the web UI ``` git clone https://github.com/oobabooga/text-generation-webui cd text-generation-webui pip install -r requirements.txt ``` #### bitsandbytes bitsandbytes >= 0.39 may not work on older NVIDIA GPUs. In that case, to use `--load-in-8bit`, you may have to downgrade like this: * Linux: `pip install bitsandbytes==0.38.1` * Windows: `pip install https://github.com/jllllll/bitsandbytes-windows-webui/raw/main/bitsandbytes-0.38.1-py3-none-any.whl` ### Alternative: Docker ``` ln -s docker/{Dockerfile,docker-compose.yml,.dockerignore} . cp docker/.env.example .env # Edit .env and set TORCH_CUDA_ARCH_LIST based on your GPU model docker compose up --build ``` * You need to have docker compose v2.17 or higher installed. See [this guide](https://github.com/oobabooga/text-generation-webui/blob/main/docs/Docker.md) for instructions. * For additional docker files, check out [this repository](https://github.com/Atinoda/text-generation-webui-docker). ### Updating the requirements From time to time, the `requirements.txt` changes. To update, use this command: ``` conda activate textgen cd text-generation-webui pip install -r requirements.txt --upgrade ``` ## Downloading models Models should be placed inside the `models/` folder. [Hugging Face](https://huggingface.co/models?pipeline_tag=text-generation&sort=downloads) is the main place to download models. These are some examples: * [Pythia](https://huggingface.co/models?sort=downloads&search=eleutherai%2Fpythia+deduped) * [OPT](https://huggingface.co/models?search=facebook/opt) * [GALACTICA](https://huggingface.co/models?search=facebook/galactica) * [GPT-J 6B](https://huggingface.co/EleutherAI/gpt-j-6B/tree/main) You can automatically download a model from HF using the script `download-model.py`: python download-model.py organization/model For example: python download-model.py facebook/opt-1.3b To download a protected model, set env vars `HF_USER` and `HF_PASS` to your Hugging Face username and password (or [User Access Token](https://huggingface.co/settings/tokens)). The model's terms must first be accepted on the HF website. #### GGML models You can drop these directly into the `models/` folder, making sure that the file name contains `ggml` somewhere and ends in `.bin`. #### GPT-4chan <details> <summary> Instructions </summary> [GPT-4chan](https://huggingface.co/ykilcher/gpt-4chan) has been shut down from Hugging Face, so you need to download it elsewhere. You have two options: * Torrent: [16-bit](https://archive.org/details/gpt4chan_model_float16) / [32-bit](https://archive.org/details/gpt4chan_model) * Direct download: [16-bit](https://theswissbay.ch/pdf/_notpdf_/gpt4chan_model_float16/) / [32-bit](https://theswissbay.ch/pdf/_notpdf_/gpt4chan_model/) The 32-bit version is only relevant if you intend to run the model in CPU mode. Otherwise, you should use the 16-bit version. After downloading the model, follow these steps: 1. Place the files under `models/gpt4chan_model_float16` or `models/gpt4chan_model`. 2. Place GPT-J 6B's config.json file in that same folder: [config.json](https://huggingface.co/EleutherAI/gpt-j-6B/raw/main/config.json). 3. Download GPT-J 6B's tokenizer files (they will be automatically detected when you attempt to load GPT-4chan): ``` python download-model.py EleutherAI/gpt-j-6B --text-only ``` When you load this model in default or notebook modes, the "HTML" tab will show the generated text in 4chan format. </details> ## Starting the web UI conda activate textgen cd text-generation-webui python server.py Then browse to `http://localhost:7860/?__theme=dark` Optionally, you can use the f