《AI大模型应用》-本项目旨在分享大模型相关技术原理以及实战经验 .zip

## 全量微调 生成用于Ascend芯片分布式通信的芯片资源信息配置文件（RANK_TABLE_FILE）。 Ascend HCCL RANK_TABLE_FILE 文件提供Ascend分布式训练作业的集群信息。 ``` # 如生成8卡的rank_table_file > python ./mindformers/tools/hccl_tools.py --device_num "[0,8)" start ./mindformers/tools/hccl_tools.py visible_devices:['0', '1', '2', '3', '4', '5', '6', '7'] server_id:192.168.1.196 device_num_list: [0, 1, 2, 3, 4, 5, 6, 7] rank_id:0, device_id:0, device_ip:192.168.100.101 rank_id:1, device_id:1, device_ip:192.168.101.101 rank_id:2, device_id:2, device_ip:192.168.102.101 rank_id:3, device_id:3, device_ip:192.168.103.101 rank_id:4, device_id:4, device_ip:192.168.100.100 rank_id:5, device_id:5, device_ip:192.168.101.100 rank_id:6, device_id:6, device_ip:192.168.102.100 rank_id:7, device_id:7, device_ip:192.168.103.100 Completed: hccl file was save in : /root/workspace/code/mindformers/hccl_8p_01234567_192.168.1.196.json ``` ### 修改配置 ``` cd /root/workspace/code/mindformers vim configs/glm/run_glm_6b_finetune.yaml ``` ### 启动训练任务 ``` > bash run_distribute.sh /root/workspace/code/mindformers/hccl_8p_01234567_192.168.1.196.json ../configs/glm/run_glm_6b_finetune.yaml '[0,8]' finetune start training for rank 0, device 0 start training for rank 1, device 1 start training for rank 2, device 2 start training for rank 3, device 3 start training for rank 4, device 4 start training for rank 5, device 5 start training for rank 6, device 6 start training for rank 7, device 7 ``` 部分训练日志如下所示： ``` ... [INFO] 2023-07-11 10:35:39,223 [run_mindformer.py:71] main: moe config is: <mindformers.modules.transformer.moe.MoEConfig object at 0xffff10297b10> [INFO] 2023-07-11 10:35:39,223 [/root/workspace/code/mindformers/scripts/mf_parallel1/mindformers/trainer/base_trainer.py:75] __init__: Now Running Task is: text_generation, Model is: glm_6b [INFO] 2023-07-11 10:35:39,224 [/root/workspace/code/mindformers/scripts/mf_parallel1/mindformers/trainer/base_trainer.py:177] _check_global_batch_size_for_auto_parallel: The current parallel mode is semi_auto_parallel, full batch is True,so global batch size will be changed: global_batch_size = batch_size * data_parallel * micro_batch_interleave_num = 8 * 1 * 1 = 8 [INFO] 2023-07-11 10:35:39,224 [/root/workspace/code/mindformers/scripts/mf_parallel1/mindformers/trainer/base_trainer.py:514] training_process: .........Build Dataset For Train.......... [INFO] 2023-07-11 10:35:39,224 [/root/workspace/code/mindformers/scripts/mf_parallel1/mindformers/trainer/base_trainer.py:268] create_train_dataset: .........Build Dataset From Config.......... [INFO] 2023-07-11 10:35:39,224 [/root/workspace/code/mindformers/scripts/mf_parallel1/mindformers/dataset/causal_language_model_dataset.py:98] __new__: Now Create Causal Language Model Dataset. [INFO] 2023-07-11 10:35:39,224 [/root/workspace/code/mindformers/scripts/mf_parallel1/mindformers/dataset/base_dataset.py:50] init_dataset_config: Now the semi auto parallel mode is used and full_batch is True,and the shuffle of the dataset is required to be False,so as to ensure that the data loaded on each card is consistent and to avoid the problem of non-convergence of loss. [INFO] 2023-07-11 10:35:39,231 [/root/workspace/code/mindformers/scripts/mf_parallel1/mindformers/trainer/utils.py:133] check_runner_config: Will be Training epochs:1, sink_size:4 [INFO] 2023-07-11 10:35:39,231 [/root/workspace/code/mindformers/scripts/mf_parallel1/mindformers/trainer/utils.py:134] check_runner_config: Create training dataset finish, dataset size:125 [INFO] 2023-07-11 10:35:39,231 [/root/workspace/code/mindformers/scripts/mf_parallel1/mindformers/trainer/base_trainer.py:521] training_process: .........Build Net For Train.......... [INFO] 2023-07-11 10:35:39,231 [/root/workspace/code/mindformers/scripts/mf_parallel1/mindformers/trainer/base_trainer.py:282] create_network: .........Build Network From Config.......... [INFO] 2023-07-11 10:38:43,280 [/root/workspace/code/mindformers/scripts/mf_parallel1/mindformers/models/base_model.py:80] load_checkpoint: weights in /root/workspace/model/chatglm-convert/ms_glm_6b.ckpt are loaded [INFO] 2023-07-11 10:38:43,299 [/root/workspace/code/mindformers/scripts/mf_parallel1/mindformers/trainer/base_trainer.py:425] count_parameters: Network Parameters: 6707 M. [INFO] 2023-07-11 10:38:43,299 [/root/workspace/code/mindformers/scripts/mf_parallel1/mindformers/trainer/base_trainer.py:544] training_process: .........Build Optimizer For Train.......... [INFO] 2023-07-11 10:38:43,299 [/root/workspace/code/mindformers/scripts/mf_parallel1/mindformers/trainer/base_trainer.py:321] create_optimizer_scheduler: .........Build Optimizer From Config.......... [INFO] 2023-07-11 10:38:43,299 [/root/workspace/code/mindformers/scripts/mf_parallel1/mindformers/trainer/base_trainer.py:354] create_lr_scheduler: .........Build LR Schedule From Config.......... [WARNING] 2023-07-11 10:38:43,306 [/root/workspace/code/mindformers/scripts/mf_parallel1/mindformers/trainer/optimizer_grouped_parameters.py:74] get_optimizer_grouped_parameters: dynamic_lr_schedule will be reset and invalid when layer_scale is False. ... [INFO] 2023-07-11 10:38:43,568 [/root/workspace/code/mindformers/scripts/mf_parallel1/mindformers/trainer/base_trainer.py:550] training_process: .........Build Running Wrapper From Config For Train.......... [INFO] 2023-07-11 10:38:43,568 [/root/workspace/code/mindformers/scripts/mf_parallel1/mindformers/trainer/base_trainer.py:391] create_model_wrapper: .........Build Model Wrapper for Train From Config.......... [INFO] 2023-07-11 10:38:43,582 [/root/workspace/code/mindformers/scripts/mf_parallel1/mindformers/trainer/base_trainer.py:562] training_process: .........Starting Init Train Model.......... [INFO] 2023-07-11 10:38:43,583 [/root/workspace/code/mindformers/scripts/mf_parallel1/mindformers/trainer/base_trainer.py:581] training_process: .........Build Callbacks For Train.......... [INFO] 2023-07-11 10:38:43,583 [/root/workspace/code/mindformers/scripts/mf_parallel1/mindformers/trainer/base_trainer.py:400] create_callbacks: .........Build Callbacks for Train From Config.......... [INFO] 2023-07-11 10:38:43,584 [/root/workspace/code/mindformers/scripts/mf_parallel1/mindformers/core/callback/callback.py:340] __init__: Integrated_save is changed to False when using auto_parallel. [INFO] 2023-07-11 10:38:43,585 [/root/workspace/code/mindformers/scripts/mf_parallel1/mindformers/trainer/base_trainer.py:609] training_process: .........Starting Training Model.......... [INFO] 2023-07-11 10:38:43,585 [/root/workspace/code/mindformers/scripts/mf_parallel1/mindformers/trainer/base_trainer.py:610] training_process: .........Model Compiling, Please Wait a Moment........... [INFO] 2023-07-11 10:47:36,427 [/root/workspace/code/mindformers/scripts/mf_parallel1/mindformers/core/callback/callback.py:269] print_output_info: Epoch:[ 1/ 1], step:[ 4/ 125], loss:[2.244/2.244], time:507844.205 ms, lr:[0.], overflow cond: True, loss_scale: 268435460.0 [INFO] 2023-07-11 10:47:37,342 [/root/workspace/code/mindformers/scripts/mf_parallel1/mindformers/core/callback/callback.py:146] epoch_end: Per sink_size step time: 533756.177 ms, per step time: 133439.044 ms, avg loss: 2.244 [INFO] 2023-07-11 10:47:44,861 [/root/workspace/code/mindformers/scripts/mf_parallel1/mindformers/core/callback/callback.py:269] print_output_info: Epoch:[ 1/ 1], step:[ 8/ 125], loss:[2.499/2.499], time:7480.938 ms, lr:[0.], overflow cond: True, loss_scale: 16777216.0 [INFO] 2023-07-11 10:47:44,874 [/root/workspace/code/mindformers/scripts/mf_parallel1/mindformers/core/callback/callback.py:146] epoch_end: Per sink_size step time: 7518.224 ms, per step time: 1879.556 ms, avg loss: 2.499 ... [INFO] 2023-07-11 10:48:35,199 [/root/workspace/code/mindformers/scripts/mf_parallel1/mindformers/core/callback/callback.py:146] epoch_end: Per sink_size st