# fine-tune-mistral
Code used to fine-tune this model: [abacaj/mistral-7b-sft](https://huggingface.co/abacaj/mistral-7b-sft). Add your data in the data folder as `train.jsonl` and `validation.jsonl`.
**Note** this repo is intended for full fine-tuning of mistral not qlora or other methods.
# How to run
Install dependencies:
```
python -m venv env \
&& source env/bin/activate \
&& pip install -r requirements.txt
```
[Get a Hugging Face token](https://huggingface.co/settings/tokens) and set the variable:
```
export HF_TOKEN="[insert token here]"
```
Run training code:
```
torchrun --nnodes=1 --nproc-per-node=<REPLACE_WITH_NUMBER_OF_GPUS> train.py
```
# Tips
- If running with a small batch size, lower the learning rate
- I did not have to adjust grad clip or weight_decay but YMMV
- Use enough data, I recommend > 1k samples
- I ran this for 3 epochs on 40k samples, will need to experiment more on epochs because the model was still improving.
- The better way to tell if your model is improving or just overfitting or even getting worse, you should add evaluation on your task. This is data that is not part of training. For example, on code completion you can evaluate your model on the mbpp validation set or a custom set you have.
- Use FSDP option: `backward_prefetch=BackwardPrefetch.BACKWARD_PRE` if you have the GPU memory, or `backward_prefetch=BackwardPrefetch.BACKWARD_POST`. This can cause OOM so it was set to None
没有合适的资源?快使用搜索试试~ 我知道了~
资源推荐
资源详情
资源评论
收起资源包目录
大模型微调_在3090+A100+H100上微调Mistra-7B大模型_附流程教程+项目源码_优质大模型项目分享.zip (8个子文件)
大模型微调_在3090+A100+H100上微调Mistra-7B大模型_附流程教程+项目源码_优质大模型项目分享
data
validation.jsonl 868KB
train.jsonl 4.55MB
core
supervised_dataset.py 5KB
multipack_sampler.py 5KB
requirements.txt 1KB
train.py 14KB
README.md 1KB
scripts
split_validation.py 869B
共 8 条
- 1
资源评论
极智视界
- 粉丝: 2w+
- 资源: 1510
上传资源 快速赚钱
- 我的内容管理 展开
- 我的资源 快来上传第一个资源
- 我的收益 登录查看自己的收益
- 我的积分 登录查看自己的积分
- 我的C币 登录后查看C币余额
- 我的收藏
- 我的下载
- 下载帮助
最新资源
- update9-20240601.5.205.slice.img.7z.002
- 微信小程序优惠券页面前端模板源码
- 林子雨编著《Spark编程基础(Python版)》 实验7 Spark机器学习库MLib编程实践数据
- 一种支持微信文本限制长度限制使用的,在合适的位置截取限制字节长度的字符串子串的java算法
- MoonPdfLib中解析PDF的库
- 学习Spring-使用Typora进行整理小白可以自学,有什么问题dd我
- update9-20240601.5.205.slice.img.7z.001.pd
- 2030.5 CSIP通信报文参考
- CSIP test程序解读
- POElight流放之路特效优化
资源上传下载、课程学习等过程中有任何疑问或建议,欢迎提出宝贵意见哦~我们会及时处理!
点击此处反馈
安全验证
文档复制为VIP权益,开通VIP直接复制
信息提交成功