gnh1201/Open-Llama
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

disable concat docs

Browse Source
This commit is contained in:
LiangSong 2023-04-15 19:35:24 +08:00
parent 0ee9612f40
commit b21441b14b
2 changed files with 4 additions and 6 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

5
inctruction_tuning.py
View File

@ -2,7 +2,7 @@
Author: LiangSong(sl12160010@gmail.com) Author: LiangSong(sl12160010@gmail.com)
Date: 2023-03-30 21:35:01 Date: 2023-03-30 21:35:01
LastEditors: LiangSong(sl12160010@gmail.com) LastEditors: LiangSong(sl12160010@gmail.com)
LastEditTime: 2023-04-06 03:35:31 LastEditTime: 2023-04-15 19:34:59
FilePath: /Open-Llama/inctruction_tuning.py FilePath: /Open-Llama/inctruction_tuning.py
Description: Description:
@ -59,8 +59,7 @@ transform_dict = {
data_set = DataIter( data_set = DataIter(
paths, paths,
transform_dict=transform_dict, transform_dict=transform_dict,
concat_docs=True, concat_docs=False,
max_length=max_length,
process_index=accelerator.process_index, process_index=accelerator.process_index,
num_processes=accelerator.num_processes, num_processes=accelerator.num_processes,
) )

5
pretrain_llama.py
View File

@ -2,7 +2,7 @@
Author: LiangSong(sl12160010@gmail.com) Author: LiangSong(sl12160010@gmail.com)
Date: 2023-03-17 14:27:28 Date: 2023-03-17 14:27:28
LastEditors: LiangSong(sl12160010@gmail.com) LastEditors: LiangSong(sl12160010@gmail.com)
LastEditTime: 2023-04-05 22:46:31 LastEditTime: 2023-04-15 19:35:06
FilePath: /Open-Llama/pretrain_llama.py FilePath: /Open-Llama/pretrain_llama.py
Description: Description:
pretrain GPT pretrain GPT
@ -51,8 +51,7 @@ transform_dict = {
data_set = DataIter( data_set = DataIter(
paths, paths,
transform_dict=transform_dict, transform_dict=transform_dict,
concat_docs=True, concat_docs=False,
max_length=max_length,
process_index=accelerator.process_index, process_index=accelerator.process_index,
num_processes=accelerator.num_processes, num_processes=accelerator.num_processes,
) )