gnh1201/Open-Llama
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Update README.md

Browse Source
This commit is contained in:
s-JoL 2023-04-11 15:59:02 +08:00 committed by GitHub
parent d2632467ec
commit 4cb94d2687
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 1 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
README.md
View File

@ -100,6 +100,7 @@ WuDao
The Pile
{'text': 'some text', 'meta': {'pile_set_name': 'Github'}}
```
验证数据完整性可见 [issue](https://github.com/s-JoL/Open-Llama/issues/5)
### 数据读取
数据读取相关代码可见dataset目录，其中包含根据下载的数据集使用SentencePiece训练分词模型，以及根据分词器构建DataLoader。
@ -113,7 +114,6 @@ python3 dataset/train_tokenizer.py
```bash
python3 dataset/pretrain_dataset.py
```
验证数据完整性可见 [issue](https://github.com/s-JoL/Open-Llama/issues/5)
### 模型结构
我们基于Transformers库中的[Llama](https://github.com/facebookresearch/llama)参考论文原文中的2.4 Efficient implementation一节进行了修改，
同时还参考了一些其他论文引入了一些优化。具体来说，我们引入了由META开源的[xformers库](https://github.com/facebookresearch/xformers)中的memory_efficient_attention操作来进行