轩辕：度小满中文金融对话大模型.zip

<p align="center"> <img src="resources/title_logo.jpg" style="width: 40%;" id="title-icon"> </p> <p align="center" style="display: flex; flex-direction: row; justify-content: center; align-items: center"> 🤗 <a href="https://huggingface.co/Duxiaoman-DI" target="_blank" style="margin-left: 10px">HuggingFace</a> • 🤖 <a href="https://modelscope.cn/organization/Duxiaoman-DI" target="_blank" style="margin-left: 10px">ModelScope</a> • 💬 <a href="https://github.com/Duxiaoman-DI/XuanYuan/blob/main/resources/wechat.jpg" target="_blank" style="margin-left: 10px">WeChat</a> </p> # News [3/11/2024] 🔥开源**XuanYuan-6B、XuanYuan-13B、XuanYuan2-70B系列模型** [1/19/2023] 🔥开源**XuanYuan-13B-Chat模型** [11/1/2023] 🔥开源**XuanYuan-70B-Chat模型及8-bit和4bit量化模型** [9/22/2023] 🔥开源**XuanYuan-70B Base**模型 [9/22/2023] 🔥开源60G高质量中文金融数据。[Hugging Face](https://huggingface.co/datasets/Duxiaoman-DI/FinCorpus) [9/22/2023] 🔥开源中文金融领域知识评估数据集 **FinanceIQ**。[GitHub](https://github.com/Duxiaoman-DI/XuanYuan/tree/main/FinanceIQ) | [HuggingFace](https://huggingface.co/datasets/Duxiaoman-DI/FinanceIQ) [5/21/2023] 开源度小满轩辕-176B大模型，在BLOOM-176B的基础上针对中文通用领域和金融领域进行了针对性的预训练与微调。是国内首个开源的千亿级中文对话大模型 **目前发布的模型和下载链接**如下： | | 基座模型 | Chat模型 | 8-bit量化Chat模型 | 4-bit量化Chat模型 | | --------------- | ------------------------------------------------------------ | ------------------------------------------------------------ | ------------------------------------------------------------ | ------------------------------------------------------------ | | XuanYuan-6B | [🤗](https://huggingface.co/Duxiaoman-DI/XuanYuan-6B) [🤖](https://modelscope.cn/models/Duxiaoman-DI/XuanYuan-6B/summary)| [🤗](https://huggingface.co/Duxiaoman-DI/XuanYuan-6B-Chat) [🤖](https://modelscope.cn/models/Duxiaoman-DI/XuanYuan-6B-Chat/summary)| [🤗](https://huggingface.co/Duxiaoman-DI/XuanYuan-6B-Chat-8bit) [🤖](https://modelscope.cn/models/Duxiaoman-DI/XuanYuan-6B-Chat-8bit/summary) | [🤗](https://huggingface.co/Duxiaoman-DI/XuanYuan-6B-Chat-4bit) [🤖](https://modelscope.cn/models/Duxiaoman-DI/XuanYuan-6B-Chat-4bit/summary) | | XuanYuan-13B | [🤗](https://huggingface.co/Duxiaoman-DI/XuanYuan-13B) [🤖](https://modelscope.cn/models/Duxiaoman-DI/XuanYuan-13B/summary)| [🤗](https://huggingface.co/Duxiaoman-DI/XuanYuan-13B-Chat) [🤖](https://modelscope.cn/models/Duxiaoman-DI/XuanYuan-13B-Chat/summary)| [🤗](https://huggingface.co/Duxiaoman-DI/XuanYuan-13B-Chat-8bit) [🤖](https://modelscope.cn/models/Duxiaoman-DI/XuanYuan-13B-Chat-8bit/summary) | [🤗](https://huggingface.co/Duxiaoman-DI/XuanYuan-13B-Chat-4bit) [🤖](https://modelscope.cn/models/Duxiaoman-DI/XuanYuan-13B-Chat-4bit/summary) | | XuanYuan2-70B|[🤗](https://huggingface.co/Duxiaoman-DI/XuanYuan2-70B) [🤖](https://modelscope.cn/models/Duxiaoman-DI/XuanYuan2-70B/summary)| [🤗](https://huggingface.co/Duxiaoman-DI/XuanYuan2-70B-Chat) [🤖](https://modelscope.cn/models/Duxiaoman-DI/XuanYuan2-70B-Chat/summary)| [🤗](https://huggingface.co/Duxiaoman-DI/XuanYuan2-70B-Chat-8bit) [🤖](https://modelscope.cn/models/Duxiaoman-DI/XuanYuan2-70B-Chat-8bit/summary) | [🤗](https://huggingface.co/Duxiaoman-DI/XuanYuan2-70B-Chat-4bit) [🤖](https://modelscope.cn/models/Duxiaoman-DI/XuanYuan2-70B-Chat-4bit/summary) | | XuanYuan-70B|[🤗](https://huggingface.co/Duxiaoman-DI/XuanYuan-70B) [🤖](https://modelscope.cn/models/Duxiaoman-DI/XuanYuan-70B/summary)| [🤗](https://huggingface.co/Duxiaoman-DI/XuanYuan-70B-Chat) [🤖](https://modelscope.cn/models/Duxiaoman-DI/XuanYuan-70B-Chat/summary)| [🤗](https://huggingface.co/Duxiaoman-DI/XuanYuan-70B-Chat-8bit) [🤖](https://modelscope.cn/models/Duxiaoman-DI/XuanYuan-70B-Chat-8bit/summary) | [🤗](https://huggingface.co/Duxiaoman-DI/XuanYuan-70B-Chat-4bit) [🤖](https://modelscope.cn/models/Duxiaoman-DI/XuanYuan-70B-Chat-4bit/summary) | | XuanYuan-176B | | [🤗](https://huggingface.co/xyz-nlp/XuanYuan2.0) | | | 技术文档： [XuanYuan-6B](xuanyuan_6b_report.md) [XuanYuan-13B](https://github.com/Duxiaoman-DI/XuanYuan/blob/main/xuanyuan_13b_report.md) [XuanYuan2-70B](xuanyuan2_70b_report.md) [XuanYuan-70B](xuanyuan_70b_report.md) # 目录 - [XuanYuan-6系列模型](#xuanyuan-6b) - [XuanYuan-13系列模型](https://github.com/Duxiaoman-DI/XuanYuan#xuanyuan-13b) - [XuanYuan2-70系列模型](#xuanyuan2-70b) - [XuanYuan-70系列模型](https://github.com/Duxiaoman-DI/XuanYuan#xuanyuan-70b) - [介绍](https://github.com/Duxiaoman-DI/XuanYuan#%E4%BB%8B%E7%BB%8D) - [通用评测](https://github.com/Duxiaoman-DI/XuanYuan#%E9%80%9A%E7%94%A8%E8%AF%84%E6%B5%8B) - [金融评测](https://github.com/Duxiaoman-DI/XuanYuan#%E9%87%91%E8%9E%8D%E8%AF%84%E6%B5%8B) - [推理部署](https://github.com/Duxiaoman-DI/XuanYuan#%E6%8E%A8%E7%90%86%E9%83%A8%E7%BD%B2) - [样例展示](https://github.com/Duxiaoman-DI/XuanYuan#%E6%A0%B7%E4%BE%8B%E5%B1%95%E7%A4%BA) - [开源金融数据集](https://github.com/Duxiaoman-DI/XuanYuan#%E5%BC%80%E6%BA%90%E9%87%91%E8%9E%8D%E6%95%B0%E6%8D%AE%E9%9B%86) - [XuanYuan-176B](https://github.com/Duxiaoman-DI/XuanYuan#%E8%BD%A9%E8%BE%95-176b%E9%A6%96%E4%B8%AA%E5%8D%83%E4%BA%BF%E7%BA%A7%E4%B8%AD%E6%96%87%E9%87%91%E8%9E%8D%E5%AF%B9%E8%AF%9D%E6%A8%A1%E5%9E%8B) # XuanYuan-6B ## 介绍 在轩辕系列大模型研发过程中，我们积累了大量的高质量数据和模型训练经验，构建了完善的训练平台，搭建了合理的评估流水线。在此基础上，为丰富轩辕系列模型矩阵，降低轩辕大模型使用门槛，我们进一步推出了XuanYuan-6B系列大模型。不同于XuanYuan-13B和XuanYuan-70B系列模型在LLaMA2上继续预训练的范式，XuanYuan-6B是我们从零开始进行预训练的大模型。当然，XuanYuan-6B仍采用类LLaMA的模型架构。在预训练基础上，我们构建了丰富、高质量的问答数据和人类偏好数据，并通过指令微调和强化学习进一步对齐了模型表现和人类偏好，显著提升了模型在对话场景中的表现。XuanYuan6B系列模型在多个评测榜单和人工评估中均获得了亮眼的结果。模型训练细节请参考我们的技术报告：[Report](xuanyuan_6b_report.md) 本次开源的XuanYuan-6B系列模型包含基座模型XuanYuan-6B，经指令微调和强化对齐的chat模型XuanYuan-6B-Chat，以及chat模型的量化版本XuanYuan-6B-Chat-4bit和XuanYuan-6B-Chat-8bit。 主要特点： * 收集多个领域大量的训练语料，进行了多维度数据清洗和去重，保证数据的量级和质量 * 从零开始预训练，预训练中动态调整数据配比，模型基座能力较强 * 结合Self-QA方法构建高质量问答数据，采用混合训练方式进行监督微调 * 构建高质量人类偏好数据训练奖励模型并进行强化训练，对齐模型表现和人类偏好 * 模型尺寸小并包含量化版本，硬件要求低，适用性更强 * 在多个榜单和人工评估中均展现出良好的性能，具备领先的金融能力 ## 性能评测 ### 基础评测 金融一直是轩辕大模型重点关注的领域和主要应用目标，因此我们首先在金融场景评测了XuanYuan-6B模型。我们使用自己构建并开源的FinanceIQ数据集，该数据集是一份专业的大模型金融能力评估数据集，涵盖了10个金融大类，36个金融小类，总计7173题。评估结果如下表所示。从表中可以看出，XuanYuan-6B模型在该评估数据中的性能甚至超越了GPT4，显示出了其