gnh1201/Open-Llama
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Update README.md

Browse Source
This commit is contained in:
s-JoL 2023-04-11 15:51:43 +08:00 committed by GitHub
parent 5f7a4a69d3
commit be2f0960c7
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 2 additions and 2 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
README.md
View File

@ -113,7 +113,7 @@ python3 dataset/train_tokenizer.py
```bash ```bash
python3 dataset/pretrain_dataset.py python3 dataset/pretrain_dataset.py
``` ```
验证数据完整性可见 [issue](https://github.com/s-JoL/Open-Llama/issues/5)
### 模型结构 ### 模型结构
我们基于Transformers库中的[Llama](https://github.com/facebookresearch/llama)参考论文原文中的2.4 Efficient implementation一节进行了修改， 我们基于Transformers库中的[Llama](https://github.com/facebookresearch/llama)参考论文原文中的2.4 Efficient implementation一节进行了修改，
同时还参考了一些其他论文引入了一些优化。具体来说，我们引入了由META开源的[xformers库](https://github.com/facebookresearch/xformers)中的memory_efficient_attention操作来进行 同时还参考了一些其他论文引入了一些优化。具体来说，我们引入了由META开源的[xformers库](https://github.com/facebookresearch/xformers)中的memory_efficient_attention操作来进行
@ -362,4 +362,4 @@ English text.
13. Gopher, Constructing Token Sequences 13. Gopher, Constructing Token Sequences
和GPT3中的避免mask的方法类似 和GPT3中的避免mask的方法类似
--> -->