This commit is contained in:
LiangSong 2023-04-28 15:05:33 +08:00
parent 0fdca8b949
commit 2fd13ff075
2 changed files with 3 additions and 3 deletions

View File

@ -42,7 +42,7 @@ pip install git+https://github.com/s-JoL/transformers.git@dev
1. 使用HuggingFace的datasets库进行数据读取具体流程如下
1. 使用transform函数将不同数据集的数据统一格式为{'text': 'xxx'}
2. 使用Tokenizer进行分词
3. 对长序列进行采样,目前提供三种模式,分别是:截断/采样参考Gopher论文/切分
3. 对长序列进行采样,目前提供三种模式,分别是:截断/采样(参考[Gopher论文](https://arxiv.org/abs/2112.11446)/切分
4. 可选对来自不同doc的文本进行拼接。减少了数据中的pad加速训练在v1版本中pad占比为30%使用拼接后pad占比降低为5%。
2. 加入Trainer对于预训练和指令微调都可以复用见solver/trainer.py
3. 统一预训练和指令微调训练入口为train_lm.py

View File

@ -43,7 +43,7 @@ This update mainly includes the following aspects, increasing the effective trai
1. Use HuggingFace's datasets library for data reading, with the process as follows:
1. Use the transform function to unify data formats from different datasets to {'text': 'xxx'}
2. Tokenize using Tokenizer
3. Sample long sequences; currently, three modes are provided: truncation, sampling (refer to the Gopher paper), and splitting
3. Sample long sequences; currently, three modes are provided: truncation, sampling (refer to the [Gopher paper](https://arxiv.org/abs/2112.11446)), and splitting
4. Optional: concatenate texts from different docs, reducing padding in the data and accelerating training. In the v1 version, padding accounted for 30%; after concatenation, padding is reduced to 5%.
2. Add Trainer, which can be reused for both pre-training and instruction fine-tuning, see solver/trainer.py
3. Unify the pre-training and instruction fine-tuning training entry to train_lm.py