Hugging Face's logo

shisa-ai
/
shisa-v2.1-lfm2-1.2b

Text Generation
Safetensors
Japanese
English
lfm2
shisa
axolotl
trl
conversational
Model card Files
xet
Community
shisa-v2.1-lfm2-1.2b / debug.log
leonardlin's picture
leonardlin
Upload folder using huggingface_hub
69710b3 verified
raw
history blame contribute delete
34.7 kB
 [2025-11-06 08:51:25,486] [WARNING] [py.warnings._showwarnmsg:112] [PID:3076758] /root/miniforge3/envs/axolotl/lib/python3.12/site-packages/deepspeed/runtime/zero/partition_parameters.py:240: UserWarning: expandable_segments not supported on this platform (Triggered internally at /pytorch/c10/hip/HIPAllocatorConfig.h:36.)
tensor: Tensor = fn(*args, **kwargs)
[2025-11-06 08:51:32,980] [WARNING] [py.warnings._showwarnmsg:112] [PID:3076758] /root/miniforge3/envs/axolotl/lib/python3.12/site-packages/torch/distributed/distributed_c10d.py:4807: UserWarning: No device id is provided via `init_process_group` or `barrier `. Using the current device set by the user.
warnings.warn( # warn only once
Extracting prompt in train dataset (num_proc=160): 0%| | 0/105170 [00:00<?, ? examples/s] Extracting prompt in train dataset (num_proc=160): 0%| | 110/105170 [00:03<1:03:35, 27.53 examples/s] Extracting prompt in train dataset (num_proc=160): 4%|█▎ | 3951/105170 [00:04<01:15, 1338.94 examples/s] Extracting prompt in train dataset (num_proc=160): 6%|█▉ | 5842/105170 [00:04<00:46, 2139.82 examples/s] Extracting prompt in train dataset (num_proc=160): 7%|██▍ | 7581/105170 [00:04<00:32, 3039.11 examples/s] Extracting prompt in train dataset (num_proc=160): 9%|███ | 9307/105170 [00:04<00:24, 3949.89 examples/s] Extracting prompt in train dataset (num_proc=160): 13%|████▏ | 13382/105170 [00:04<00:12, 7534.85 examples/s] Extracting prompt in train dataset (num_proc=160): 16%|█████▏ | 17022/105170 [00:04<00:07, 11051.92 examples/s] Extracting prompt in train dataset (num_proc=160): 19%|██████ | 19824/105170 [00:04<00:06, 12624.91 examples/s] Extracting prompt in train dataset (num_proc=160): 21%|██████▊ | 22283/105170 [00:05<00:06, 13352.98 examples/s] Extracting prompt in train dataset (num_proc=160): 23%|███████▍ | 24460/105170 [00:05<00:05, 13662.30 examples/s] Extracting prompt in train dataset (num_proc=160): 25%|████████ | 26478/105170 [00:05<00:05, 13868.55 examples/s] Extracting prompt in train dataset (num_proc=160): 28%|████████▊ | 29156/105170 [00:05<00:04, 16444.30 examples/s] Extracting prompt in train dataset (num_proc=160): 30%|█████████▌ | 31230/105170 [00:05<00:04, 16927.48 examples/s] Extracting prompt in train dataset (num_proc=160): 32%|██████████▏ | 33410/105170 [00:05<00:04, 14539.05 examples/s] Extracting prompt in train dataset (num_proc=160): 49%|███████████████▋ | 51665/105170 [00:05<00:01, 50088.28 examples/s] Extracting prompt in train dataset (num_proc=160): 66%|█████████████████████▏ | 69590/105170 [00:05<00:00, 79793.06 examples/s] Extracting prompt in train dataset (num_proc=160): 90%|███████████████████████████▊ | 94350/105170 [00:06<00:00, 121502.26 examples/s] Extracting prompt in train dataset (num_proc=160): 100%|███████████████████████████████| 105170/105170 [00:06<00:00, 15164.58 examples/s]
Applying chat template to train dataset (num_proc=160): 0%| | 0/105170 [00:00<?, ? examples/s] Applying chat template to train dataset (num_proc=160): 0%| | 50/105170 [00:04<2:30:25, 11.65 examples/s] Applying chat template to train dataset (num_proc=160): 1%|▏ | 675/105170 [00:04<08:15, 210.77 examples/s] Applying chat template to train dataset (num_proc=160): 2%|▌ | 1938/105170 [00:04<02:17, 750.78 examples/s] Applying chat template to train dataset (num_proc=160): 3%|▊ | 3050/105170 [00:04<01:16, 1341.84 examples/s] Applying chat template to train dataset (num_proc=160): 5%|█▍ | 5273/105170 [00:04<00:34, 2935.94 examples/s] Applying chat template to train dataset (num_proc=160): 6%|█▊ | 6618/105170 [00:04<00:25, 3798.52 examples/s] Applying chat template to train dataset (num_proc=160): 9%|██▍ | 8989/105170 [00:05<00:16, 5978.80 examples/s] Applying chat template to train dataset (num_proc=160): 11%|███▏ | 11951/105170 [00:05<00:10, 9212.12 examples/s] Applying chat template to train dataset (num_proc=160): 13%|███▋ | 14157/105170 [00:05<00:08, 11311.43 examples/s] Applying chat template to train dataset (num_proc=160): 15%|████▏ | 16107/105170 [00:05<00:07, 12101.63 examples/s] Applying chat template to train dataset (num_proc=160): 17%|████▌ | 17962/105170 [00:05<00:06, 12748.17 examples/s] Applying chat template to train dataset (num_proc=160): 20%|█████▎ | 20806/105170 [00:05<00:05, 14755.22 examples/s] Applying chat template to train dataset (num_proc=160): 22%|█████▊ | 22626/105170 [00:05<00:05, 13836.62 examples/s] Applying chat template to train dataset (num_proc=160): 23%|██████▏ | 24225/105170 [00:05<00:05, 13621.09 examples/s] Applying chat template to train dataset (num_proc=160): 24%|██████▌ | 25754/105170 [00:06<00:06, 13176.93 examples/s] Applying chat template to train dataset (num_proc=160): 27%|███████▎ | 28443/105170 [00:06<00:04, 15705.45 examples/s] Applying chat template to train dataset (num_proc=160): 29%|███████▋ | 30166/105170 [00:06<00:04, 15903.40 examples/s] Applying chat template to train dataset (num_proc=160): 30%|████████▏ | 31893/105170 [00:06<00:05, 13227.29 examples/s] Applying chat template to train dataset (num_proc=160): 33%|████████▊ | 34205/105170 [00:06<00:04, 15487.94 examples/s] Applying chat template to train dataset (num_proc=160): 34%|█████████▏ | 35947/105170 [00:06<00:05, 13727.27 examples/s] Applying chat template to train dataset (num_proc=160): 36%|█████████▋ | 37785/105170 [00:06<00:04, 13706.64 examples/s] Applying chat template to train dataset (num_proc=160): 39%|██████████▌ | 41145/105170 [00:06<00:03, 17628.22 examples/s] Applying chat template to train dataset (num_proc=160): 41%|███████████ | 43030/105170 [00:07<00:03, 16625.80 examples/s] Applying chat template to train dataset (num_proc=160): 43%|███████████▌ | 44862/105170 [00:07<00:04, 14248.72 examples/s] Applying chat template to train dataset (num_proc=160): 44%|███████████▉ | 46451/105170 [00:07<00:04, 12630.48 examples/s] Applying chat template to train dataset (num_proc=160): 46%|████████████▌ | 48782/105170 [00:07<00:03, 14955.29 examples/s] Applying chat template to train dataset (num_proc=160): 48%|████████████▉ | 50448/105170 [00:07<00:03, 15266.67 examples/s] Applying chat template to train dataset (num_proc=160): 50%|█████████████▍ | 52509/105170 [00:07<00:03, 16492.22 examples/s] Applying chat template to train dataset (num_proc=160): 52%|█████████████▉ | 54268/105170 [00:07<00:03, 13054.62 examples/s] Applying chat template to train dataset (num_proc=160): 53%|██████████████▎ | 55781/105170 [00:08<00:03, 12473.00 examples/s] Applying chat template to train dataset (num_proc=160): 55%|██████████████▊ | 57497/105170 [00:08<00:03, 12879.94 examples/s] Applying chat template to train dataset (num_proc=160): 56%|███████████████ | 58880/105170 [00:08<00:03, 12737.76 examples/s] Applying chat template to train dataset (num_proc=160): 59%|███████████████▊ | 61705/105170 [00:08<00:02, 16520.53 examples/s] Applying chat template to train dataset (num_proc=160): 60%|████████████████▎ | 63508/105170 [00:08<00:02, 16895.09 examples/s] Applying chat template to train dataset (num_proc=160): 62%|████████████████▊ | 65430/105170 [00:08<00:02, 16404.91 examples/s] Applying chat template to train dataset (num_proc=160): 65%|█████████████████▌ | 68209/105170 [00:08<00:02, 17411.58 examples/s] Applying chat template to train dataset (num_proc=160): 67%|█████████████████▉ | 70007/105170 [00:08<00:02, 15183.55 examples/s] Applying chat template to train dataset (num_proc=160): 68%|██████████████████▍ | 71610/105170 [00:09<00:02, 14578.54 examples/s] Applying chat template to train dataset (num_proc=160): 70%|██████████████████▉ | 73807/105170 [00:09<00:02, 14997.86 examples/s] Applying chat template to train dataset (num_proc=160): 72%|███████████████████▎ | 75370/105170 [00:09<00:02, 13514.99 examples/s] Applying chat template to train dataset (num_proc=160): 73%|███████████████████▋ | 76794/105170 [00:09<00:02, 13564.88 examples/s] Applying chat template to train dataset (num_proc=160): 74%|████████████████████ | 78181/105170 [00:09<00:02, 12037.23 examples/s] Applying chat template to train dataset (num_proc=160): 77%|████████████████████▊ | 81009/105170 [00:09<00:01, 15904.44 examples/s] Applying chat template to train dataset (num_proc=160): 79%|█████████████████████▏ | 82739/105170 [00:09<00:01, 12437.93 examples/s] Applying chat template to train dataset (num_proc=160): 80%|█████████████████████▋ | 84495/105170 [00:10<00:01, 12873.48 examples/s] Applying chat template to train dataset (num_proc=160): 82%|██████████████████████▏ | 86312/105170 [00:10<00:01, 13198.06 examples/s] Applying chat template to train dataset (num_proc=160): 85%|██████████████████████▉ | 89292/105170 [00:10<00:00, 17001.18 examples/s] Applying chat template to train dataset (num_proc=160): 87%|███████████████████████▍ | 91165/105170 [00:10<00:00, 16199.78 examples/s] Applying chat template to train dataset (num_proc=160): 88%|███████████████████████▊ | 92909/105170 [00:10<00:00, 14763.83 examples/s] Applying chat template to train dataset (num_proc=160): 91%|████████████████████████▋ | 96038/105170 [00:10<00:00, 18747.94 examples/s] Applying chat template to train dataset (num_proc=160): 93%|█████████████████████████▏ | 98086/105170 [00:10<00:00, 18120.33 examples/s] Applying chat template to train dataset (num_proc=160): 95%|████████████████████████▋ | 100039/105170 [00:10<00:00, 16410.97 examples/s] Applying chat template to train dataset (num_proc=160): 97%|█████████████████████████▏| 101785/105170 [00:11<00:00, 14486.47 examples/s] Applying chat template to train dataset (num_proc=160): 98%|█████████████████████████▌| 103358/105170 [00:11<00:00, 13775.91 examples/s] Applying chat template to train dataset (num_proc=160): 100%|█████████████████████████▉| 104843/105170 [00:11<00:00, 10274.28 examples/s] Applying chat template to train dataset (num_proc=160): 100%|███████████████████████████| 105170/105170 [00:12<00:00, 8452.21 examples/s]
Tokenizing train dataset (num_proc=160): 0%| | 0/105170 [00:00<?, ? examples/s] Tokenizing train dataset (num_proc=160): 0%| | 10/105170 [00:06<17:47:56, 1.64 examples/s] Tokenizing train dataset (num_proc=160): 0%| | 32/105170 [00:06<4:24:34, 6.62 examples/s] Tokenizing train dataset (num_proc=160): 0%| | 56/105170 [00:06<2:05:39, 13.94 examples/s] Tokenizing train dataset (num_proc=160): 0%| | 116/105170 [00:06<45:03, 38.85 examples/s] Tokenizing train dataset (num_proc=160): 0%| | 257/105170 [00:06<15:24, 113.44 examples/s] Tokenizing train dataset (num_proc=160): 0%|▏ | 307/105170 [00:06<12:32, 139.32 examples/s] Tokenizing train dataset (num_proc=160): 1%|▎ | 581/105170 [00:06<04:51, 358.72 examples/s] Tokenizing train dataset (num_proc=160): 1%|▍ | 897/105170 [00:07<02:55, 593.51 examples/s] Tokenizing train dataset (num_proc=160): 1%|▌ | 1198/105170 [00:07<01:59, 869.76 examples/s] Tokenizing train dataset (num_proc=160): 2%|▋ | 1769/105170 [00:07<01:07, 1535.37 examples/s] Tokenizing train dataset (num_proc=160): 2%|▊ | 2024/105170 [00:07<01:00, 1706.84 examples/s] Tokenizing train dataset (num_proc=160): 2%|█ | 2549/105170 [00:07<00:43, 2384.35 examples/s] Tokenizing train dataset (num_proc=160): 3%|█▏ | 2884/105170 [00:07<00:40, 2545.10 examples/s] Tokenizing train dataset (num_proc=160): 3%|█▍ | 3320/105170 [00:07<00:39, 2563.07 examples/s] Tokenizing train dataset (num_proc=160): 4%|█▌ | 3682/105170 [00:07<00:36, 2798.12 examples/s] Tokenizing train dataset (num_proc=160): 4%|█▊ | 4242/105170 [00:08<00:29, 3474.91 examples/s] Tokenizing train dataset (num_proc=160): 5%|██ | 4846/105170 [00:08<00:33, 3020.68 examples/s] Tokenizing train dataset (num_proc=160): 5%|██▍ | 5699/105170 [00:08<00:28, 3521.63 examples/s] Tokenizing train dataset (num_proc=160): 6%|██▌ | 6152/105170 [00:08<00:29, 3345.92 examples/s] Tokenizing train dataset (num_proc=160): 6%|██▊ | 6677/105170 [00:08<00:26, 3738.30 examples/s] Tokenizing train dataset (num_proc=160): 7%|███ | 7316/105170 [00:08<00:22, 4337.63 examples/s] Tokenizing train dataset (num_proc=160): 7%|███▎ | 7793/105170 [00:09<00:35, 2726.37 examples/s] Tokenizing train dataset (num_proc=160): 8%|███▍ | 8167/105170 [00:09<00:38, 2540.90 examples/s] Tokenizing train dataset (num_proc=160): 8%|███▌ | 8496/105170 [00:09<00:37, 2549.51 examples/s] Tokenizing train dataset (num_proc=160): 9%|███▊ | 9005/105170 [00:09<00:32, 2994.71 examples/s] Tokenizing train dataset (num_proc=160): 9%|███▉ | 9537/105170 [00:09<00:31, 3079.02 examples/s] Tokenizing train dataset (num_proc=160): 9%|████▏ | 9899/105170 [00:09<00:29, 3194.26 examples/s] Tokenizing train dataset (num_proc=160): 10%|████▏ | 10249/105170 [00:10<00:39, 2401.80 examples/s] Tokenizing train dataset (num_proc=160): 10%|████▎ | 10535/105170 [00:10<00:42, 2228.54 examples/s] Tokenizing train dataset (num_proc=160): 10%|████▍ | 10790/105170 [00:10<01:02, 1506.25 examples/s] Tokenizing train dataset (num_proc=160): 10%|████▍ | 10989/105170 [00:10<01:00, 1561.95 examples/s] Tokenizing train dataset (num_proc=160): 11%|████▌ | 11186/105170 [00:11<01:33, 1000.43 examples/s] Tokenizing train dataset (num_proc=160): 11%|████▋ | 11353/105170 [00:11<01:36, 972.19 examples/s] Tokenizing train dataset (num_proc=160): 11%|████▊ | 11501/105170 [00:11<01:43, 902.76 examples/s] Tokenizing train dataset (num_proc=160): 11%|████▊ | 11616/105170 [00:11<01:43, 903.41 examples/s] Tokenizing train dataset (num_proc=160): 11%|████▉ | 11730/105170 [00:11<01:41, 916.36 examples/s] Tokenizing train dataset (num_proc=160): 11%|████▉ | 12013/105170 [00:11<01:17, 1199.45 examples/s] Tokenizing train dataset (num_proc=160): 12%|████▉ | 12149/105170 [00:12<01:20, 1162.14 examples/s] Tokenizing train dataset (num_proc=160): 12%|█████ | 12366/105170 [00:12<01:12, 1285.80 examples/s] Tokenizing train dataset (num_proc=160): 12%|█████▏ | 12704/105170 [00:12<00:56, 1633.30 examples/s] Tokenizing train dataset (num_proc=160): 12%|█████▎ | 12876/105170 [00:12<00:59, 1551.22 examples/s] Tokenizing train dataset (num_proc=160): 12%|█████▎ | 13119/105170 [00:12<00:56, 1632.35 examples/s] Tokenizing train dataset (num_proc=160): 13%|█████▍ | 13380/105170 [00:12<00:54, 1681.67 examples/s] Tokenizing train dataset (num_proc=160): 13%|█████▌ | 13562/105170 [00:12<00:56, 1611.60 examples/s] Tokenizing train dataset (num_proc=160): 13%|█████▋ | 13808/105170 [00:13<00:56, 1603.03 examples/s] Tokenizing train dataset (num_proc=160): 13%|█████▋ | 14061/105170 [00:13<01:01, 1493.05 examples/s] Tokenizing train dataset (num_proc=160): 14%|█████▊ | 14266/105170 [00:13<01:08, 1320.66 examples/s] Tokenizing train dataset (num_proc=160): 14%|█████▉ | 14516/105170 [00:13<01:09, 1313.18 examples/s] Tokenizing train dataset (num_proc=160): 14%|██████▏ | 15086/105170 [00:13<00:43, 2083.48 examples/s] Tokenizing train dataset (num_proc=160): 15%|██████▎ | 15328/105170 [00:13<00:43, 2047.49 examples/s] Tokenizing train dataset (num_proc=160): 15%|██████▎ | 15564/105170 [00:13<00:42, 2090.18 examples/s] Tokenizing train dataset (num_proc=160): 15%|██████▍ | 15792/105170 [00:14<00:43, 2065.01 examples/s] Tokenizing train dataset (num_proc=160): 15%|██████▌ | 16009/105170 [00:14<00:55, 1599.65 examples/s] Tokenizing train dataset (num_proc=160): 15%|██████▊ | 16198/105170 [00:14<01:57, 756.36 examples/s] Tokenizing train dataset (num_proc=160): 16%|██████▊ | 16339/105170 [00:15<01:59, 745.22 examples/s] Tokenizing train dataset (num_proc=160): 16%|██████▉ | 16537/105170 [00:15<01:48, 818.59 examples/s] Tokenizing train dataset (num_proc=160): 16%|███████ | 16809/105170 [00:15<01:33, 944.74 examples/s] Tokenizing train dataset (num_proc=160): 16%|██████▉ | 17000/105170 [00:15<01:21, 1081.32 examples/s] Tokenizing train dataset (num_proc=160): 16%|███████▏ | 17326/105170 [00:16<01:27, 999.61 examples/s] Tokenizing train dataset (num_proc=160): 17%|███████▎ | 17485/105170 [00:16<01:46, 824.02 examples/s] Tokenizing train dataset (num_proc=160): 17%|███████▍ | 17733/105170 [00:16<01:42, 854.27 examples/s] Tokenizing train dataset (num_proc=160): 17%|███████▍ | 17843/105170 [00:16<01:50, 793.28 examples/s] Tokenizing train dataset (num_proc=160): 17%|███████▍ | 18174/105170 [00:16<01:21, 1071.36 examples/s] Tokenizing train dataset (num_proc=160): 17%|███████▌ | 18361/105170 [00:17<01:12, 1201.67 examples/s] Tokenizing train dataset (num_proc=160): 18%|███████▌ | 18541/105170 [00:17<01:06, 1303.76 examples/s] Tokenizing train dataset (num_proc=160): 18%|███████▋ | 18702/105170 [00:17<01:15, 1145.99 examples/s] Tokenizing train dataset (num_proc=160): 18%|███████▉ | 18836/105170 [00:17<01:28, 973.70 examples/s] Tokenizing train dataset (num_proc=160): 18%|███████▊ | 19037/105170 [00:17<01:15, 1140.69 examples/s] Tokenizing train dataset (num_proc=160): 18%|███████▉ | 19280/105170 [00:17<01:02, 1383.82 examples/s] Tokenizing train dataset (num_proc=160): 19%|███████▉ | 19550/105170 [00:17<00:50, 1682.84 examples/s] Tokenizing train dataset (num_proc=160): 19%|████████ | 19746/105170 [00:18<00:56, 1520.65 examples/s] Tokenizing train dataset (num_proc=160): 19%|████████▏ | 19919/105170 [00:18<00:58, 1445.12 examples/s] Tokenizing train dataset (num_proc=160): 19%|████████▏ | 20078/105170 [00:18<01:03, 1337.83 examples/s] Tokenizing train dataset (num_proc=160): 19%|████████▎ | 20222/105170 [00:18<01:06, 1273.43 examples/s] Tokenizing train dataset (num_proc=160): 19%|████████▎ | 20357/105170 [00:18<01:11, 1188.40 examples/s] Tokenizing train dataset (num_proc=160): 19%|████████▍ | 20494/105170 [00:18<01:11, 1180.03 examples/s] Tokenizing train dataset (num_proc=160): 20%|████████▌ | 20815/105170 [00:19<01:12, 1160.02 examples/s] Tokenizing train dataset (num_proc=160): 20%|████████▊ | 20934/105170 [00:19<02:04, 675.69 examples/s] Tokenizing train dataset (num_proc=160): 20%|████████▊ | 21191/105170 [00:19<01:59, 701.94 examples/s] Tokenizing train dataset (num_proc=160): 20%|████████▉ | 21407/105170 [00:20<02:05, 667.83 examples/s] Tokenizing train dataset (num_proc=160): 21%|█████████ | 21653/105170 [00:20<01:57, 710.42 examples/s] Tokenizing train dataset (num_proc=160): 21%|█████████ | 21743/105170 [00:20<02:07, 656.76 examples/s] Tokenizing train dataset (num_proc=160): 21%|█████████▏ | 21918/105170 [00:20<01:54, 726.36 examples/s] Tokenizing train dataset (num_proc=160): 21%|█████████▎ | 22238/105170 [00:21<01:28, 932.79 examples/s] Tokenizing train dataset (num_proc=160): 21%|█████████▎ | 22364/105170 [00:21<01:31, 900.12 examples/s] Tokenizing train dataset (num_proc=160): 21%|█████████▍ | 22533/105170 [00:21<01:23, 994.27 examples/s] Tokenizing train dataset (num_proc=160): 22%|█████████▎ | 22784/105170 [00:21<01:10, 1163.91 examples/s] Tokenizing train dataset (num_proc=160): 22%|█████████▍ | 22983/105170 [00:21<01:04, 1274.29 examples/s] Tokenizing train dataset (num_proc=160): 22%|█████████▋ | 23124/105170 [00:21<01:26, 950.38 examples/s] Tokenizing train dataset (num_proc=160): 22%|█████████▌ | 23267/105170 [00:22<01:20, 1014.10 examples/s] Tokenizing train dataset (num_proc=160): 22%|█████████▌ | 23482/105170 [00:22<01:13, 1111.32 examples/s] Tokenizing train dataset (num_proc=160): 22%|█████████▋ | 23604/105170 [00:22<01:19, 1032.16 examples/s] Tokenizing train dataset (num_proc=160): 23%|█████████▉ | 23714/105170 [00:22<01:42, 796.32 examples/s] Tokenizing train dataset (num_proc=160): 23%|█████████▉ | 23823/105170 [00:22<01:44, 775.80 examples/s] Tokenizing train dataset (num_proc=160): 23%|██████████ | 23981/105170 [00:22<01:33, 864.36 examples/s] Tokenizing train dataset (num_proc=160): 23%|██████████ | 24162/105170 [00:23<01:29, 907.95 examples/s] Tokenizing train dataset (num_proc=160): 23%|█████████▉ | 24391/105170 [00:23<01:13, 1102.93 examples/s] Tokenizing train dataset (num_proc=160): 23%|██████████ | 24582/105170 [00:23<01:10, 1141.02 examples/s] Tokenizing train dataset (num_proc=160): 23%|██████████ | 24703/105170 [00:23<01:14, 1076.54 examples/s] Tokenizing train dataset (num_proc=160): 24%|██████████▍ | 24819/105170 [00:23<01:23, 960.79 examples/s] Tokenizing train dataset (num_proc=160): 24%|██████████▍ | 24949/105170 [00:23<01:23, 957.49 examples/s] Tokenizing train dataset (num_proc=160): 24%|██████████▎ | 25256/105170 [00:23<00:56, 1415.02 examples/s] Tokenizing train dataset (num_proc=160): 24%|██████████▍ | 25441/105170 [00:24<00:53, 1502.64 examples/s] Tokenizing train dataset (num_proc=160): 24%|██████████▍ | 25622/105170 [00:24<00:53, 1498.80 examples/s] Tokenizing train dataset (num_proc=160): 25%|██████████▌ | 25781/105170 [00:24<00:54, 1452.65 examples/s] Tokenizing train dataset (num_proc=160): 25%|██████████▌ | 25964/105170 [00:24<01:01, 1281.67 examples/s] Tokenizing train dataset (num_proc=160): 25%|██████████▋ | 26105/105170 [00:24<01:11, 1101.67 examples/s] Tokenizing train dataset (num_proc=160): 25%|██████████▊ | 26326/105170 [00:24<00:58, 1342.69 examples/s] Tokenizing train dataset (num_proc=160): 25%|███████████ | 26477/105170 [00:25<01:26, 911.55 examples/s] Tokenizing train dataset (num_proc=160): 25%|███████████▏ | 26599/105170 [00:25<01:37, 802.05 examples/s] Tokenizing train dataset (num_proc=160): 26%|██████████▉ | 26863/105170 [00:25<01:11, 1092.16 examples/s] Tokenizing train dataset (num_proc=160): 26%|███████████ | 27099/105170 [00:25<01:03, 1233.73 examples/s] Tokenizing train dataset (num_proc=160): 26%|███████████▏ | 27398/105170 [00:25<00:55, 1402.88 examples/s] Tokenizing train dataset (num_proc=160): 26%|███████████▎ | 27554/105170 [00:25<00:57, 1351.88 examples/s] Tokenizing train dataset (num_proc=160): 26%|███████████▎ | 27699/105170 [00:25<00:56, 1368.98 examples/s] Tokenizing train dataset (num_proc=160): 27%|███████████▍ | 27962/105170 [00:26<00:48, 1595.83 examples/s] Tokenizing train dataset (num_proc=160): 27%|███████████▌ | 28344/105170 [00:26<00:36, 2115.14 examples/s] Tokenizing train dataset (num_proc=160): 27%|███████████▋ | 28574/105170 [00:26<01:06, 1155.71 examples/s] Tokenizing train dataset (num_proc=160): 28%|████████████ | 29556/105170 [00:26<00:29, 2568.18 examples/s] Tokenizing train dataset (num_proc=160): 30%|█████████████ | 31856/105170 [00:26<00:11, 6432.02 examples/s] Tokenizing train dataset (num_proc=160): 32%|█████████████▌ | 33316/105170 [00:26<00:08, 8155.63 examples/s] Tokenizing train dataset (num_proc=160): 33%|██████████████ | 34439/105170 [00:27<00:08, 7881.01 examples/s] Tokenizing train dataset (num_proc=160): 34%|██████████████▍ | 35442/105170 [00:27<00:08, 7904.14 examples/s] Tokenizing train dataset (num_proc=160): 35%|██████████████▉ | 36388/105170 [00:27<00:08, 8146.29 examples/s] Tokenizing train dataset (num_proc=160): 37%|███████████████▋ | 39383/105170 [00:27<00:04, 13491.93 examples/s] Tokenizing train dataset (num_proc=160): 39%|████████████████▍ | 41266/105170 [00:27<00:04, 14882.03 examples/s] Tokenizing train dataset (num_proc=160): 42%|█████████████████▋ | 44153/105170 [00:27<00:03, 18649.57 examples/s] Tokenizing train dataset (num_proc=160): 46%|███████████████████ | 47866/105170 [00:27<00:02, 23808.81 examples/s] Tokenizing train dataset (num_proc=160): 50%|████████████████████▉ | 52351/105170 [00:27<00:01, 29802.64 examples/s] Tokenizing train dataset (num_proc=160): 53%|██████████████████████▏ | 55542/105170 [00:27<00:01, 29908.27 examples/s] Tokenizing train dataset (num_proc=160): 56%|███████████████████████▍ | 58625/105170 [00:28<00:01, 25757.21 examples/s] Tokenizing train dataset (num_proc=160): 60%|█████████████████████████▎ | 63427/105170 [00:28<00:01, 31557.28 examples/s] Tokenizing train dataset (num_proc=160): 66%|███████████████████████████▋ | 69218/105170 [00:28<00:00, 38704.85 examples/s] Tokenizing train dataset (num_proc=160): 75%|███████████████████████████████▍ | 78776/105170 [00:28<00:00, 54475.47 examples/s] Tokenizing train dataset (num_proc=160): 86%|████████████████████████████████████ | 90193/105170 [00:28<00:00, 70913.96 examples/s] Tokenizing train dataset (num_proc=160): 93%|██████████████████████████████████████▉ | 97524/105170 [00:28<00:00, 69379.22 examples/s] Tokenizing train dataset (num_proc=160): 100%|██████████████████████████████████████████| 105170/105170 [00:29<00:00, 3577.76 examples/s]