LiangSong
|
97aff0e051
|
use split_dataset_by_node instead accelerate.prepare to accelerate data loading by 50%
|
2023-04-27 00:04:11 +08:00 |
|
LiangSong
|
0377b43628
|
update tokenizer to LlamaTokenizer
|
2023-04-26 18:53:30 +08:00 |
|
LiangSong
|
f41f5558ec
|
update header
|
2023-04-24 23:19:07 +08:00 |
|
LiangSong
|
f8f4cde228
|
using huggingface datasets to accelerate training, using open-llama to pretrain
|
2023-04-24 19:13:53 +08:00 |
|
LiangSong
|
3f62a23ee2
|
update format
|
2023-04-12 22:16:15 +08:00 |
|
LiangSong
|
a4aa109dd3
|
add trainer and utils
|
2023-04-12 17:59:05 +08:00 |
|