❗️❗️❗️NOTICE: For optimal performance, we refrain from fine-tuning the model's identity. Thus, inquiries such as "Who are you" or "Who developed you" may yield random responses that are not necessarily accurate.
# Updates:
- 🚀🚀🚀 [May 9, 2024] We're excited to introduce Llama3-70B-Chinese-Chat! Fine-tuned on a mixed Chinese-English dataset of **~100K preference pairs**, its Chinese proficiency **surpasses ChatGPT and matches GPT-4**, as shown by C-Eval and CMMLU results.
- 🔥 We provide the official Ollama model for the q4_0 GGUF version of Llama3-70B-Chinese-Chat at [wangshenzhi/llama3-70b-chinese-chat-ollama-q4](https://ollama.com/wangshenzhi/llama3-70b-chinese-chat-ollama-q4)! Run the following command for quick use of this model: `ollama run wangshenzhi/llama3-70b-chinese-chat-ollama-q4:latest`.
- 🔥 We provide the official Ollama model for the q8_0 GGUF version of Llama3-70B-Chinese-Chat at [wangshenzhi/llama3-70b-chinese-chat-ollama-q8](https://ollama.com/wangshenzhi/llama3-70b-chinese-chat-ollama-q8)! Run the following command for quick use of this model: `ollama run wangshenzhi/llama3-70b-chinese-chat-ollama-q8:latest`.
# Model Summary
Llama3-70B-Chinese-Chat is **one of the first instruction-tuned LLMs for Chinese & English users with various abilities** such as roleplaying, tool-using, and math, built upon the [meta-llama/Meta-Llama-3-70B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3-70B-Instruct) model.
🎉**According to the results from C-Eval and CMMLU, the performance of Llama3-70B-Chinese-Chat in Chinese significantly exceeds that of ChatGPT and is comparable to GPT-4!**
Developed by: [Shenzhi Wang](https://shenzhi-wang.netlify.app) (王慎执) and [Yaowei Zheng](https://github.com/hiyouga) (郑耀威)
- License: [Llama-3 License](https://llama.meta.com/llama3/license/)
- Base Model: Meta-Llama-3-70B-Instruct
- Model Size: 70.6B
- Context length: 8K
# 1. Introduction
This is **one of the first LLM fine-tuned specifically for Chinese and English users**, based on the [Meta-Llama-3-70B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3-70B-Instruct) model. The fine-tuning algorithm used is **ORPO** [1].
Our Llama3-70B-Chinese-Chat model was trained on a dataset containing **over 100K preference pairs**, with a roughly equal ratio of Chinese and English data. This dataset will be available soon.
**Compared to the original [Meta-Llama-3-70B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3-70B-Instruct) model, the Llama3-70B-Chinese-Chat model greatly reduces the issues of "Chinese questions with English answers" and the mixing of Chinese and English in responses. Additionally, Llama3-70B-Chinese-Chat excels at roleplaying, function calling, and mathematics.**
With much more parameters than our [Llama3-8B-Chinese-Chat](https://huggingface.co/shenzhi-wang/Llama3-8B-Chinese-Chat) model, our Llama3-70B-Chinese-Chat offers significant performance enhancements. If you enjoyed our [Llama3-8B-Chinese-Chat](https://huggingface.co/shenzhi-wang/Llama3-8B-Chinese-Chat), the Llama3-70B-Chinese-Chat is a must-try!
[1] Hong, Jiwoo, Noah Lee, and James Thorne. "Reference-free Monolithic Preference Optimization with Odds Ratio." arXiv preprint arXiv:2403.07691 (2024).
Training framework: [LLaMA-Factory](https://github.com/hiyouga/LLaMA-Factory).
Training details:
- epochs: 3 (we also provide a 2-epoch model version at the [`epoch_2` branch](https://ai.gitee.com/wang-shenzhi/llama3-70b-chinese-chat/tree/epoch_2))
- learning rate: 1.5e-6
- learning rate scheduler type: cosine
- Warmup ratio: 0.1
- cutoff len (i.e. context length): 8192
- orpo beta (i.e. $\lambda$ in the ORPO paper): 0.05
- global batch size: 128
- fine-tuning type: full parameters
- optimizer: paged_adamw_32bit
# 2. Benchmark Results
We utilize C-Eval [2] and CMMLU [3] to assess the performance of LLMs in Chinese. The results of ChatGPT and GPT-4 are borrowed from the [C-Eval leaderboard](https://cevalbenchmark.com/static/leaderboard.html) and [CMMLU leaderboard](https://github.com/haonan-li/CMMLU#five-shot) accessed on May 10, 2024.
| Model | C-Eval Avg (Test Set) | C-Eval Hard Avg (Test Set) | CMMLU Acc |
| --------------------------- | --------------------- | -------------------------- | --------- |
| ChatGPT | 54.4 | 41.4 | 55.51 |
| GPT-4 | 68.7 | 54.9 | 70.95 |
| **Llama3-70B-Chinese-Chat** | 66.1 | 55.2 | 70.28 |
C-Eval Hard is a distinct benchmark that comprises 8 difficult subjects in math, physics, and chemistry from C-Eval. [2]
[2] Huang, Yuzhen, et al. "C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models." *Advances in Neural Information Processing Systems* 36 (2024).
[3] Li, Haonan, et al. "Cmmlu: Measuring massive multitask language understanding in chinese." *arXiv preprint arXiv:2306.09212* (2023).
# 3. Usage
```python
from transformers import AutoTokenizer, AutoModelForCausalLM
model_id = "shenzhi-wang/Llama3-70B-Chinese-Chat"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
model_id, torch_dtype="auto", device_map="auto"
)
messages = [
{"role": "user", "content": "写一首诗吧"},
]
input_ids = tokenizer.apply_chat_template(
messages, add_generation_prompt=True, return_tensors="pt"
).to(model.device)
outputs = model.generate(
input_ids,
max_new_tokens=8192,
do_sample=True,
temperature=0.6,
top_p=0.9,
)
response = outputs[0][input_ids.shape[-1]:]
print(tokenizer.decode(response, skip_special_tokens=True))
```
# 4. Examples
The following are some examples generated by Llama3-70B-Chinese-Chat.
<details>
<summary><b>Role playing [CLICK TO EXPAND]</b></summary>
**User:**
```
请你现在扮演艾萨克·牛顿，17世纪和18世纪之交的英国物理学家、数学家，也是现代物理学的奠基人之一。
```
**Assistant:**
当然，我可以尝试以艾萨克·牛顿（Isaac Newton）的视角来回答您的问题。不过，请记住我的知识截止日期是2023年，某些后来的科学发展可能与我所处时代的理解有所不同。
现在，您有什么问题需要向我这个17世纪和18世纪之交的物理学家、数学家请教吗？
**User:**
```
牛顿先生，我想询问您关于相对论的概念，您认为它与您的经典力学有什么冲突或补充？尤其是相对论中关于时间和空间的相对性理论与您的绝对时空观点相比，您如何看待这一点？
```
**Assistant:**
啊，相对论，这是一个在我时代之后很久才被提出和发展起来的概念。尽管如此，我可以尝试从我的知识和理解出发来讨论这个问题。
根据我的理论，时间和空间是绝对的，即它们独立于观察者存在，并且以一种普适且一致的方式流逝。但是在相对论中，这种观点被颠覆了。爱因斯坦提出的狭义相对论表明，时间和空间并非固定不变，而是与观察者的参考系有关。当物体以接近光速的速度移动时，它们所经历的时间流逝会有所不同，这就是我们所熟知的“时间膨胀”现象。
这种观点在当时是革命性的，因为它挑战了人们长期以来对于宇宙的基本理解。从某种角度来说，相对论补充了我的经典力学，因为它提供了一种新的视角来看待物体运动和引力的本质，尤其是在极端条件下，比如高速或强重力场。
尽管我无法亲自见证相对论的发展，但如果我能够了解这些概念，我相信它们将为科学界带来了巨大的启示。它向我们展示了物理定律在不同尺度下的多样性，以及在更深层次上理解自然世界的必要性。这正是科学进步的本质——不断地挑战和完善我们
首批 Llama3 70B 中文大模型，在C-Eval和CMMLU的表现超过了ChatGPT，与GPT-4持平。.zip （51个子文件） 
Llama3-70B-Chinese-Chat Llama3-70B-Chinese-Chat 
model-00011-of-00030.safetensors 135B model-00019-of-00030.safetensors 135B model-00007-of-00030.safetensors 135B model-00003-of-00030.safetensors 135B model-00004-of-00030.safetensors 135B model-00002-of-00030.safetensors 135B .gitattributes 50B model-00028-of-00030.safetensors 135B model-00022-of-00030.safetensors 135B LICENSE 8KB tokenizer.json 8.66MB model-00023-of-00030.safetensors 135B .git index 4KB HEAD 23B refs heads master 41B tags objects pack pack-0be7d0249dce67eb38c288fd4febc08033a62139.idx 4KB pack-0be7d0249dce67eb38c288fd4febc08033a62139.pack 2.31MB info packed-refs 162B FETCH_HEAD 259B logs HEAD 130B refs heads master 130B remotes origin master 144B epoch_2 144B hooks config 266B branches model-00014-of-00030.safetensors 135B generation_config.json 146B model-00016-of-00030.safetensors 135B model.safetensors.index.json 58KB model-00017-of-00030.safetensors 135B model-00026-of-00030.safetensors 135B model-00018-of-00030.safetensors 135B model-00021-of-00030.safetensors 135B model-00029-of-00030.safetensors 135B model-00020-of-00030.safetensors 135B model-00013-of-00030.safetensors 135B config.json 648B model-00012-of-00030.safetensors 135B model-00010-of-00030.safetensors 135B model-00001-of-00030.safetensors 135B tokenizer_config.json 50KB model-00008-of-00030.safetensors 135B model-00006-of-00030.safetensors 135B model-00009-of-00030.safetensors 135B special_tokens_map.json 96B model-00025-of-00030.safetensors 135B README.md 60KB model-00005-of-00030.safetensors 135B model-00030-of-00030.safetensors 135B model-00015-of-00030.safetensors 135B model-00024-of-00030.safetensors 135B model-00027-of-00030.safetensors 135B资源评论
