gnh1201/Open-Llama
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Merge pull request #29 from s-JoL/dev

Browse Source 
update readme
This commit is contained in:
s-JoL 2023-04-28 19:50:29 +08:00 committed by GitHub
commit 0fd7dbd636
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
2 changed files with 13 additions and 2 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

7
README.md
View File

@ -2,7 +2,7 @@
* @Author: LiangSong(sl12160010@gmail.com)
* @Date: 2023-03-10 21:18:35
* @LastEditors: LiangSong(sl12160010@gmail.com)
* @LastEditTime: 2023-04-16 23:49:06
* @LastEditTime: 2023-04-28 19:49:29
* @FilePath: /Open-Llama/README.md
* @Description:
*
@ -37,6 +37,7 @@ pip install git+https://github.com/s-JoL/transformers.git@dev
![image4](assets/multiturn_chat.jpeg)
## **更新**
[2023.4.28] Release v2.0
本次更新主要包含以下几个方面，相对于v1版本提升有效训练速度50%，其中pad从30%减少至5%，训练速度从3200token/s提升至3600token/s。0.95 * 3600/(0.7 * 3200)=1.527
1. 使用HuggingFace的datasets库进行数据读取，具体流程如下
@ -49,6 +50,10 @@ pip install git+https://github.com/s-JoL/transformers.git@dev
4. 提供更方便的配置，可见configs/pretrain_config.yaml
5. 提供基于其他预训练模型补充词表，继续预训练功能
[2023.4.16] Release v1.0
提供基础的预训练和指令微调代码，训练速度达到Llama原文速度。预训练和指令微调后的模型已经开源在HuggingFace。
v1版代码可见https://github.com/s-JoL/Open-Llama/tree/v1.0
## **特性**

8
README_en.md
View File

@ -2,7 +2,7 @@
* @Author: LiangSong(sl12160010@gmail.com)
* @Date: 2023-03-10 21:18:35
* @LastEditors: LiangSong(sl12160010@gmail.com)
* @LastEditTime: 2023-04-16 23:49:28
* @LastEditTime: 2023-04-28 19:49:24
* @FilePath: /Open-Llama/README_en.md
* @Description:
*
@ -38,6 +38,8 @@ Below is a display of the model's multi-turn dialogue ability regarding code:
## **Updates**
[2023.4.28] Release v2.0
This update mainly includes the following aspects, increasing the effective training speed by 50% compared to the v1 version, reducing padding from 30% to 5%, and improving training speed from 3200 tokens/s to 3600 tokens/s. 0.95 * 3600 / (0.7 * 3200) = 1.527
1. Use HuggingFace's datasets library for data reading, with the process as follows:
@ -50,6 +52,10 @@ This update mainly includes the following aspects, increasing the effective trai
4. Provide more convenient configuration, see configs/pretrain_config.yaml
5. Provide functionality to continue pre-training based on other pre-trained models and supplementing vocabulary
[2023.4.16] Release v1.0
Basic pre-training and instruction fine-tuning codes are provided, with a training speed comparable to that of the original Llama. The pre-trained and fine-tuned models are already open-sourced on HuggingFace.
v1 version code can be seen at https://github.com/s-JoL/Open-Llama/tree/v1.0
## **Features**