训练模型、源码-Lora

# LoRA_Easy_Training_Scripts A set of training scripts written in python for use in Kohya's [SD-Scripts](https://github.com/kohya-ss/sd-scripts). The main two scripts, titled `lora_train_command_line.py` and `lora_train_popup.py` are the main training scripts. They both effectively do the same thing, just in different ways. `lora_train_command_line.py` is designed for those who want to directly edit the script, if you need some help on what the various commands are, you can just run the script like so to get help with for every arg, at the bottom of this readme will be a big list of all of the commands as well ``` venv\Scripts\accelerate.exe launch lora_train_command_line.py -h or venv\Scripts\accelerate.exe launch lora_train_popup.py -h ``` `lora_train_popup.py` is designed for those who are unable or unwilling to directly edit the script, you give up some control this way, but the defaults are generally good, so you won't need to worry too much about the hidden elements. It attempts to keep things simple by asking the user questions through popups, it's slower than `lora_train_command_line.py` but it is a guided experience `lora_resize.py` is a third script that is to be used for _resizing_ lora, as SD-Scripts includes a way to do that now. This is a great way to reduce your dim size after training. It should be plenty easy to use as well because it uses popups much like `lora_train_popup.py` does. ## Installation ### Windows If you have a windows device I have created batch files [here](https://github.com/derrian-distro/LoRA_Easy_Training_Scripts/releases/latest) that auto install for you. if you want to specifically use _my_ scripts, then grab the one titled `install_sd_scripts.bat` which will install SD-Scripts as well as my scripts. **Make sure you don't have spaces in your path when you install them** ### Linux Unfortunatly, I haven't actually gotten around to creating installers for Linux devices, mainly because I don't use a Linux device as my daily driver. that being said, you can still install SD-Scripts! Follow the install instructions on SD-Scripts then just move these files over to the SD-Scripts directory. ## Usage ### Windows If you are using windows, then you can just run any of the scripts by using their respective `run_*.bat` files, so `lora_train_popup.py` would have you run `run_popup.py`. Just make sure that they are in the same folder as the `.py` files, which should be the root of SD-Scripts. ### Linux If you are using Linux, you can run it by running accelerate like so. ``` venv/Scripts/accelerate launch lora_train_command_line.py or venv/Scripts/accelerate launch lora_train_popup.py ``` I believe it also works without accelerate if you just want to run it from the venv and python directly. ## JSON Saving And Loading One of the big features of this set of scripts is that they can create and use JSON files. how they load json files are slightly different between the two training scripts, so lets explain why. `lora_train_command_line.py` will load everything unconditionally by default. There is a way to exclude things though. you must modify the variable called `json_load_skip_list` which can be added to to exclude things, more explanation of it will be below `lora_train_popup.py` will load everything except for the following items, the path to save a json, the path to load regularization images, the path to a lora model to resume training, the name you can set to change the output name, your training comment, and the skip list itself Additionally, you can load or save JSON files from the command line with their respective commands, `--save_json_path "path\to\folder"` for saving, and `--load_json_path "path\to\json.json"` for loading. You can also just set them, in `lora_train_command_line.py` and it is asked in `lora_train_popup.py` Finally, I also set up the JSON loading so that it supports the JSON files Kohya_ss generates ## Queuing Training I have implemented queues to both the `lora_train_command_line.py` and `lora_train_popup.py` like the JSON saving, they work slightly differently from one another. so I'll explain the differences. `lora_train_command_line.py` has a variable called `multi_run_folder` that can take a path to a folder that has a bunch of JSON files in it. it will run through all of them one by one, and train every model in that folder. Unlike when loading JSON files normally, this will ignore the exclude list because it cannot wait for the user to change the variables during run time. Since it loads everything through JSON files, I have opted to have it create a "completed" folder of the JSON files that have already been trained, doing this means that if you quit before all are finished, you know what hasn't been done. If you would rather set it through the command line, you can call `--multi_run_path "path\to\folder"` `lora_train_popup.py` just loops through the popups until you say you want to stop, and then queues them up to run after you are done entering them. I don't have a way to track what has been done for this version because of the way it's implemented. ## Tag Occurrence Printout new with this update is a way to generate a txt file that outputs all of the tags that was used to train with in an easy to read way that has both the number of times it appeared in all caption files, as well as the tag itself. I have now changed it's implementation so that if you wanted to output alphabetically, you can. ## LoRA Resize Script `lora_resize.py` is a script I wrote to run the resize script that is within SD-Scripts, much like the other two, it has a batch file that can be used to run it. It does things in the popup style, and supports queueing. This script should simplify reducing the dim size of LoRA. ## LoRA Merging and Image Resizing scripts I added two new scripts to handle the scripts that sd-scripts added since I created the resizing script. The first one, `lora_merge.py` and it's accompanying bat file `lora_merge.bat` can take in any number of loras to merge together, it walks you through the process, so you shouldn't need to know how it works under the hood! The second one, `image_resize.py` is a script for downscaling your images. This is great if you are training at a high resolution as you can disable bucket upscaling and have a set of images that are "different" to the model, it should improve gens at lower resolutions, and might even help smaller datasets ## Lion Optimizer I added support for the new Lion optimizer in both scripts. I don't know what the best lr is for them, and still have 8bit adam as the default for now. if you want to know more, take a look at the original github [here](https://github.com/lucidrains/lion-pytorch). They seem to have some insights, namely, use less steps than usual and a smaller lr. stating that lr should be anywhere from 3-10x _smaller_ than normal. ## Changelog - Feb 20, 2023 - Updated the scripts to support the new parameters added - Lion support added on both scripts, under the param `use_lion` - added the `lowram` option even though I don't think it will be needed for anybody who actually uses my scripts, as that seems to be geared towards colab users. - Added a script for merging LoRA, called `lora_merge.py` and it's bat file `lora_merge.bat` - It uses the same popup style for the other sub scripts, and will walk you through the process. This script seems to be able to merge more than one LoRA at a time - Added a script for downscaling images, called `image_resize.py` and it's bat file `image_resize.bat`. This will be a very useful tool for replacing repeats with a better alternative if you use the no_upscale feature. - Feb 14, 2023 - Updated the scripts to support the new parameter Kohya added, noise_offset - I haven't tested it myself yet, but supposedly having this set will improve generation