Pile_training_log.txt

NeoXArgs.configure_distributed_args() using world size: 64 and model-parallel size: 1 
> building HFTokenizer tokenizer ...
 > padded vocab (size: 50277) with 27 dummy tokens (new size: 50304)
> setting tensorboard ...
> initializing torch distributed ...
> initializing model parallel with size 1
MPU DP: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63]
MPU PP: [0]
MPU PP: [1]
MPU PP: [2]
MPU PP: [3]
MPU PP: [4]
MPU PP: [5]
MPU PP: [6]
MPU PP: [7]
MPU PP: [8]
MPU PP: [9]
MPU PP: [10]
MPU PP: [11]
MPU PP: [12]
MPU PP: [13]
MPU PP: [14]
MPU PP: [15]
MPU PP: [16]
MPU PP: [17]
MPU PP: [18]
MPU PP: [19]
MPU PP: [20]
MPU PP: [21]
MPU PP: [22]
MPU PP: [23]
MPU PP: [24]
MPU PP: [25]
MPU PP: [26]
MPU PP: [27]
MPU PP: [28]
MPU PP: [29]
MPU PP: [30]
MPU PP: [31]
MPU PP: [32]
MPU PP: [33]
MPU PP: [34]
MPU PP: [35]
MPU PP: [36]
MPU PP: [37]
MPU PP: [38]
MPU PP: [39]
MPU PP: [40]
MPU PP: [41]
MPU PP: [42]
MPU PP: [43]
MPU PP: [44]
MPU PP: [45]
MPU PP: [46]
MPU PP: [47]
MPU PP: [48]
MPU PP: [49]
MPU PP: [50]
MPU PP: [51]
MPU PP: [52]
MPU PP: [53]
MPU PP: [54]
MPU PP: [55]
MPU PP: [56]
MPU PP: [57]
MPU PP: [58]
MPU PP: [59]
MPU PP: [60]
MPU PP: [61]
MPU PP: [62]
MPU PP: [63]
MPU MP: [0]
MPU MP: [1]
MPU MP: [2]
MPU MP: [3]
MPU MP: [4]
MPU MP: [5]
MPU MP: [6]
MPU MP: [7]
MPU MP: [8]
MPU MP: [9]
MPU MP: [10]
MPU MP: [11]
MPU MP: [12]
MPU MP: [13]
MPU MP: [14]
MPU MP: [15]
MPU MP: [16]
MPU MP: [17]
MPU MP: [18]
MPU MP: [19]
MPU MP: [20]
MPU MP: [21]
MPU MP: [22]
MPU MP: [23]
MPU MP: [24]
MPU MP: [25]
MPU MP: [26]
MPU MP: [27]
MPU MP: [28]
MPU MP: [29]
MPU MP: [30]
MPU MP: [31]
MPU MP: [32]
MPU MP: [33]
MPU MP: [34]
MPU MP: [35]
MPU MP: [36]
MPU MP: [37]
MPU MP: [38]
MPU MP: [39]
MPU MP: [40]
MPU MP: [41]
MPU MP: [42]
MPU MP: [43]
MPU MP: [44]
MPU MP: [45]
MPU MP: [46]
MPU MP: [47]
MPU MP: [48]
MPU MP: [49]
MPU MP: [50]
MPU MP: [51]
MPU MP: [52]
MPU MP: [53]
MPU MP: [54]
MPU MP: [55]
MPU MP: [56]
MPU MP: [57]
MPU MP: [58]
MPU MP: [59]
MPU MP: [60]
MPU MP: [61]
MPU MP: [62]
MPU MP: [63]
> setting random seeds to 1234 ...
building GPT2 model ...
SEED_LAYERS=False BASE_SEED=1234 SEED_FN=None
Using topology: {ProcessCoord(pipe=0, data=0, model=0): 0, ProcessCoord(pipe=0, data=1, model=0): 1, ProcessCoord(pipe=0, data=2, model=0): 2, ProcessCoord(pipe=0, data=3, model=0): 3, ProcessCoord(pipe=0, data=4, model=0): 4, ProcessCoord(pipe=0, data=5, model=0): 5, ProcessCoord(pipe=0, data=6, model=0): 6, ProcessCoord(pipe=0, data=7, model=0): 7, ProcessCoord(pipe=0, data=8, model=0): 8, ProcessCoord(pipe=0, data=9, model=0): 9, ProcessCoord(pipe=0, data=10, model=0): 10, ProcessCoord(pipe=0, data=11, model=0): 11, ProcessCoord(pipe=0, data=12, model=0): 12, ProcessCoord(pipe=0, data=13, model=0): 13, ProcessCoord(pipe=0, data=14, model=0): 14, ProcessCoord(pipe=0, data=15, model=0): 15, ProcessCoord(pipe=0, data=16, model=0): 16, ProcessCoord(pipe=0, data=17, model=0): 17, ProcessCoord(pipe=0, data=18, model=0): 18, ProcessCoord(pipe=0, data=19, model=0): 19, ProcessCoord(pipe=0, data=20, model=0): 20, ProcessCoord(pipe=0, data=21, model=0): 21, ProcessCoord(pipe=0, data=22, model=0): 22, ProcessCoord(pipe=0, data=23, model=0): 23, ProcessCoord(pipe=0, data=24, model=0): 24, ProcessCoord(pipe=0, data=25, model=0): 25, ProcessCoord(pipe=0, data=26, model=0): 26, ProcessCoord(pipe=0, data=27, model=0): 27, ProcessCoord(pipe=0, data=28, model=0): 28, ProcessCoord(pipe=0, data=29, model=0): 29, ProcessCoord(pipe=0, data=30, model=0): 30, ProcessCoord(pipe=0, data=31, model=0): 31, ProcessCoord(pipe=0, data=32, model=0): 32, ProcessCoord(pipe=0, data=33, model=0): 33, ProcessCoord(pipe=0, data=34, model=0): 34, ProcessCoord(pipe=0, data=35, model=0): 35, ProcessCoord(pipe=0, data=36, model=0): 36, ProcessCoord(pipe=0, data=37, model=0): 37, ProcessCoord(pipe=0, data=38, model=0): 38, ProcessCoord(pipe=0, data=39, model=0): 39, ProcessCoord(pipe=0, data=40, model=0): 40, ProcessCoord(pipe=0, data=41, model=0): 41, ProcessCoord(pipe=0, data=42, model=0): 42, ProcessCoord(pipe=0, data=43, model=0): 43, ProcessCoord(pipe=0, data=44, model=0): 44, ProcessCoord(pipe=0, data=45, model=0): 45, ProcessCoord(pipe=0, data=46, model=0): 46, ProcessCoord(pipe=0, data=47, model=0): 47, ProcessCoord(pipe=0, data=48, model=0): 48, ProcessCoord(pipe=0, data=49, model=0): 49, ProcessCoord(pipe=0, data=50, model=0): 50, ProcessCoord(pipe=0, data=51, model=0): 51, ProcessCoord(pipe=0, data=52, model=0): 52, ProcessCoord(pipe=0, data=53, model=0): 53, ProcessCoord(pipe=0, data=54, model=0): 54, ProcessCoord(pipe=0, data=55, model=0): 55, ProcessCoord(pipe=0, data=56, model=0): 56, ProcessCoord(pipe=0, data=57, model=0): 57, ProcessCoord(pipe=0, data=58, model=0): 58, ProcessCoord(pipe=0, data=59, model=0): 59, ProcessCoord(pipe=0, data=60, model=0): 60, ProcessCoord(pipe=0, data=61, model=0): 61, ProcessCoord(pipe=0, data=62, model=0): 62, ProcessCoord(pipe=0, data=63, model=0): 63}
stage=0 layers=37
     0: EmbeddingPipe
     1: _pre_transformer_block
     2: ParallelFlownetLayerPipe
     3: ParallelFlownetLayerPipe
     4: ParallelFlownetLayerPipe
     5: ParallelFlownetLayerPipe
     6: ParallelFlownetLayerPipe
     7: ParallelFlownetLayerPipe
     8: ParallelFlownetLayerPipe
     9: ParallelFlownetLayerPipe
    10: ParallelFlownetLayerPipe
    11: ParallelFlownetLayerPipe
    12: ParallelFlownetLayerPipe
    13: ParallelFlownetLayerPipe
    14: ParallelFlownetLayerPipe
    15: ParallelFlownetLayerPipe
    16: ParallelFlownetLayerPipe
    17: ParallelFlownetLayerPipe
    18: ParallelFlownetLayerPipe
    19: ParallelFlownetLayerPipe
    20: ParallelFlownetLayerPipe
    21: ParallelFlownetLayerPipe
    22: ParallelFlownetLayerPipe
    23: ParallelFlownetLayerPipe
    24: ParallelFlownetLayerPipe
    25: ParallelFlownetLayerPipe
    26: ParallelFlownetLayerPipe
    27: ParallelFlownetLayerPipe
    28: ParallelFlownetLayerPipe
    29: ParallelFlownetLayerPipe
    30: ParallelFlownetLayerPipe
    31: ParallelFlownetLayerPipe
    32: ParallelFlownetLayerPipe
    33: ParallelFlownetLayerPipe
    34: _post_transformer_block
    35: NormPipe
    36: EmbeddingPipe
  loss: partial
Configuring Optimizer type: Adam with params: {'lr': 0.0006, 'betas': [0.9, 0.95], 'eps': 1e-08}
> learning rate decay style: cosine
DeepSpeed is enabled.
 > number of parameters on model parallel rank 0: 903252480
 > total params: 903,252,480
Unable to load checkpoint.
Loading checkpoint and starting from iteration 0
> building train, validation, and test datasets ...
    reading sizes...
    reading pointers...
    reading document index...
    creating numpy buffer of mmap...
    creating memory view of numpy buffer...
 > dataset split:
    train:
     document indices in [0, 130154259) total of 130154259 documents
    validation:
     document indices in [130154259, 134183803) total of 4029544 documents
    test:
     document indices in [134183803, 134318121) total of 134318 documents
 > loading doc-idx mapping from /u/wangh/workspace/dataset/language_dataset/pile/pile_0.87_deduped_text_document_train_indexmap_146432000ns_2048sl_1234s_doc_idx.npy
 > loading sample-idx mapping from /u/wangh/workspace/dataset/language_dataset/pile/pile_0.87_deduped_text_document_train_indexmap_146432000ns_2048sl_1234s_sample_idx.npy
 > loading shuffle-idx mapping from /u/wangh/workspace/dataset/language_dataset/pile/pile_0.87_deduped_text_document_train_indexmap_146432000ns_2048sl_1234s_shuffle_idx.npy
    loaded indexed file in 0.317 seconds
    total number of samples: 195915016
    total number of epochs: 2
 > loading doc-idx mapping from /u/wangh/workspace/dataset/language_dataset/pile/pile_0.87_deduped_text_document_valid_indexmap_2969600ns_2048sl_1234s_doc_idx.npy
 > loading sample-idx mapping from /u/wangh/workspace/dataset/language_dataset/pile/pile_0.87_deduped_text_document_valid_indexmap_2969600ns_2048sl_1234s_sample_idx.npy
 > loading shuffle-idx mapping from /u/wangh/workspace/dataset/language_dataset/pile/pile_0.87_deduped_text_document_valid_indexmap_2969600ns_2048sl_1234s_shuffle_idx.npy
    loaded indexed file in 0.112 seconds
    total number of samples: 3097460
    total number of epochs: 1
 > loading doc-idx mapping from /u/wangh/workspace/dataset/language_dataset/pile/pile_0.87_deduped_text_document_test_indexmap_102400ns_2048sl_1234s_doc_idx.npy
 > loading sample-idx mapping from /u/wangh/workspace/dataset/language_dataset/pile/pile_0.87_deduped_text_document_test_indexmap_102400ns_2048sl_1234s_sample_idx.npy
 > loading shuffle-idx mapping from /u/wangh/workspace/dataset/language_dataset/pile/pile_0.87_deduped_text_document_test_indexmap_102400ns_2048sl_1234s_shuffle_idx.npy
    loaded indexed file in 0.079 seconds
    total number of samples: 102462
    total number of epochs: 1
setting training data start iteration to 0
setting validation data start iteration to 0
done with setups ...
time (ms) | model and optimizer: 3443.03 | train/valid/test data iterators: 5438.07
training ...
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step0
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step1
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step2
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step4
 samples/sec: 346.770 | iteration      500/  143000 | elapsed time per iteration (ms): 2953.0 | learning rate: 2.098E-04 | approx flops per GPU: 75.0TFLOPS | lm_loss: 6.376106E+00 | loss scale: 4096.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
after 500 iterations memory (MB) | allocated: 3623.939453125 | max allocated: 11453.5703125 | reserved: 13716.0 | max reserved: 13716.0
time (ms) | forward: 616.85 | backward: 2216.34 | backward-backward: 2216.27 | backward-allreduce: 0.00 | optimizer: 79.68 | batch generator: 3.13
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step8
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step16
 samples/sec: 352.134 | iteration     1000/  143000 | elapsed time per iteration (ms): 2908.0 | learning rate: 4.196E-04 | approx flops per GPU: 76.2TFLOPS | lm_loss: 3.774834E+00 | loss scale: 8192.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 610.82 | backward: 2207.45 | backward-backward: 2207.38 | backward-allreduce: 0.00 | optimizer: 79.73 | batch generator: 2.35
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step32
 samples/sec: 352.377 | iteration     1500/  143000 | elapsed time per iteration (ms): 2906.0 | learning rate: 6.000E-04 | approx flops per GPU: 76.2TFLOPS | lm_loss: 3.132075E+00 | loss scale: 8192.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 610.61 | backward: 2210.01 | backward-backward: 2209.94 | backward-allreduce: 0.00 | optimizer: 79.72 | batch generator: 2.42
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step64
 samples/sec: 352.701 | iteration     2000/  143000 | elapsed time per iteration (ms): 2903.3 | learning rate: 6.000E-04 | approx flops per GPU: 76.3TFLOPS | lm_loss: 2.867739E+00 | loss scale: 16384.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 610.86 | backward: 2206.59 | backward-backward: 2206.51 | backward-allreduce: 0.00 | optimizer: 79.71 | batch generator: 2.43
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step128
 samples/sec: 352.790 | iteration     2500/  143000 | elapsed time per iteration (ms): 2902.6 | learning rate: 5.999E-04 | approx flops per GPU: 76.3TFLOPS | lm_loss: 2.722186E+00 | loss scale: 16384.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 610.26 | backward: 2206.18 | backward-backward: 2206.11 | backward-allreduce: 0.00 | optimizer: 79.65 | batch generator: 2.37
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step256
 samples/sec: 349.630 | iteration     3000/  143000 | elapsed time per iteration (ms): 2928.8 | learning rate: 5.998E-04 | approx flops per GPU: 75.7TFLOPS | lm_loss: 2.633660E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 609.85 | backward: 2233.73 | backward-backward: 2233.65 | backward-allreduce: 0.00 | optimizer: 79.63 | batch generator: 2.30
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step500
 samples/sec: 352.908 | iteration     3500/  143000 | elapsed time per iteration (ms): 2901.6 | learning rate: 5.997E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.576675E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 609.49 | backward: 2206.91 | backward-backward: 2206.84 | backward-allreduce: 0.00 | optimizer: 79.65 | batch generator: 2.28
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step512
 samples/sec: 352.294 | iteration     4000/  143000 | elapsed time per iteration (ms): 2906.7 | learning rate: 5.996E-04 | approx flops per GPU: 76.2TFLOPS | lm_loss: 2.532798E+00 | loss scale: 65536.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 609.75 | backward: 2211.48 | backward-backward: 2211.41 | backward-allreduce: 0.00 | optimizer: 79.65 | batch generator: 2.28
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step1000
 samples/sec: 353.086 | iteration     4500/  143000 | elapsed time per iteration (ms): 2900.1 | learning rate: 5.994E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.499583E+00 | loss scale: 65536.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 609.59 | backward: 2205.48 | backward-backward: 2205.41 | backward-allreduce: 0.00 | optimizer: 79.66 | batch generator: 2.29
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step1500
 samples/sec: 353.079 | iteration     5000/  143000 | elapsed time per iteration (ms): 2900.2 | learning rate: 5.992E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.472691E+00 | loss scale: 32768.0 | number of skipped iterations:   2 | number of nan iterations:   0 |
time (ms) | forward: 610.12 | backward: 2205.31 | backward-backward: 2205.24 | backward-allreduce: 0.00 | optimizer: 79.31 | batch generator: 2.31
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step2000
---------------------------------------------------------------------------------------------------------
 validation results at iteration 5000 | lm_loss value: 2.411079E+00 | lm_loss_ppl value: 1.114599E+01 | 
---------------------------------------------------------------------------------------------------------
 samples/sec: 338.121 | iteration     5500/  143000 | elapsed time per iteration (ms): 3028.5 | learning rate: 5.989E-04 | approx flops per GPU: 73.2TFLOPS | lm_loss: 2.450628E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 610.69 | backward: 2204.89 | backward-backward: 2204.82 | backward-allreduce: 0.00 | optimizer: 79.71 | batch generator: 2.74
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step2500
 samples/sec: 352.386 | iteration     6000/  143000 | elapsed time per iteration (ms): 2905.9 | learning rate: 5.986E-04 | approx flops per GPU: 76.2TFLOPS | lm_loss: 2.431382E+00 | loss scale: 65536.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 609.51 | backward: 2211.14 | backward-backward: 2211.07 | backward-allreduce: 0.00 | optimizer: 79.61 | batch generator: 2.29
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step3000
 samples/sec: 352.866 | iteration     6500/  143000 | elapsed time per iteration (ms): 2902.0 | learning rate: 5.983E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.413347E+00 | loss scale: 65536.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 609.32 | backward: 2207.21 | backward-backward: 2207.13 | backward-allreduce: 0.00 | optimizer: 79.59 | batch generator: 2.26
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step3500
 samples/sec: 353.044 | iteration     7000/  143000 | elapsed time per iteration (ms): 2900.5 | learning rate: 5.979E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.399783E+00 | loss scale: 65536.0 | number of skipped iterations:   2 | number of nan iterations:   0 |
time (ms) | forward: 609.57 | backward: 2206.24 | backward-backward: 2206.17 | backward-allreduce: 0.00 | optimizer: 79.28 | batch generator: 2.33
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step4000
 samples/sec: 353.070 | iteration     7500/  143000 | elapsed time per iteration (ms): 2900.3 | learning rate: 5.976E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.388037E+00 | loss scale: 65536.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 609.42 | backward: 2205.85 | backward-backward: 2205.78 | backward-allreduce: 0.00 | optimizer: 79.60 | batch generator: 2.35
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step4500
 samples/sec: 350.078 | iteration     8000/  143000 | elapsed time per iteration (ms): 2925.1 | learning rate: 5.971E-04 | approx flops per GPU: 75.7TFLOPS | lm_loss: 2.376209E+00 | loss scale: 131072.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 610.41 | backward: 2229.35 | backward-backward: 2229.28 | backward-allreduce: 0.00 | optimizer: 79.72 | batch generator: 2.39
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step5000
 samples/sec: 353.087 | iteration     8500/  143000 | elapsed time per iteration (ms): 2900.1 | learning rate: 5.967E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.366794E+00 | loss scale: 65536.0 | number of skipped iterations:   2 | number of nan iterations:   0 |
time (ms) | forward: 610.16 | backward: 2204.86 | backward-backward: 2204.79 | backward-allreduce: 0.00 | optimizer: 79.36 | batch generator: 2.39
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step5500
 samples/sec: 352.920 | iteration     9000/  143000 | elapsed time per iteration (ms): 2901.5 | learning rate: 5.962E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.356318E+00 | loss scale: 32768.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 610.31 | backward: 2205.70 | backward-backward: 2205.62 | backward-allreduce: 0.00 | optimizer: 79.48 | batch generator: 2.37
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step6000
 samples/sec: 353.012 | iteration     9500/  143000 | elapsed time per iteration (ms): 2900.7 | learning rate: 5.957E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.347873E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 610.18 | backward: 2205.41 | backward-backward: 2205.33 | backward-allreduce: 0.00 | optimizer: 79.59 | batch generator: 2.42
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step6500
 samples/sec: 351.834 | iteration    10000/  143000 | elapsed time per iteration (ms): 2910.5 | learning rate: 5.951E-04 | approx flops per GPU: 76.1TFLOPS | lm_loss: 2.339790E+00 | loss scale: 65536.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 610.60 | backward: 2214.82 | backward-backward: 2214.74 | backward-allreduce: 0.00 | optimizer: 79.46 | batch generator: 2.62
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step7000
----------------------------------------------------------------------------------------------------------
 validation results at iteration 10000 | lm_loss value: 2.289003E+00 | lm_loss_ppl value: 9.865094E+00 | 
----------------------------------------------------------------------------------------------------------
 samples/sec: 338.135 | iteration    10500/  143000 | elapsed time per iteration (ms): 3028.4 | learning rate: 5.946E-04 | approx flops per GPU: 73.2TFLOPS | lm_loss: 2.333742E+00 | loss scale: 65536.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 610.31 | backward: 2205.22 | backward-backward: 2205.15 | backward-allreduce: 0.00 | optimizer: 79.62 | batch generator: 2.78
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step7500
 samples/sec: 352.921 | iteration    11000/  143000 | elapsed time per iteration (ms): 2901.5 | learning rate: 5.939E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.327170E+00 | loss scale: 131072.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 610.67 | backward: 2205.55 | backward-backward: 2205.48 | backward-allreduce: 0.00 | optimizer: 79.70 | batch generator: 2.40
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step8000
 samples/sec: 353.044 | iteration    11500/  143000 | elapsed time per iteration (ms): 2900.5 | learning rate: 5.933E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.319703E+00 | loss scale: 32768.0 | number of skipped iterations:   3 | number of nan iterations:   0 |
time (ms) | forward: 610.05 | backward: 2205.54 | backward-backward: 2205.46 | backward-allreduce: 0.00 | optimizer: 79.16 | batch generator: 2.37
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step8500
 samples/sec: 352.928 | iteration    12000/  143000 | elapsed time per iteration (ms): 2901.4 | learning rate: 5.926E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.316716E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 610.27 | backward: 2206.04 | backward-backward: 2205.97 | backward-allreduce: 0.00 | optimizer: 79.57 | batch generator: 2.37
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step9000
 samples/sec: 352.587 | iteration    12500/  143000 | elapsed time per iteration (ms): 2904.2 | learning rate: 5.919E-04 | approx flops per GPU: 76.3TFLOPS | lm_loss: 2.309714E+00 | loss scale: 65536.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 609.70 | backward: 2209.49 | backward-backward: 2209.42 | backward-allreduce: 0.00 | optimizer: 79.60 | batch generator: 2.36
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step9500
 samples/sec: 352.916 | iteration    13000/  143000 | elapsed time per iteration (ms): 2901.5 | learning rate: 5.912E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.304286E+00 | loss scale: 32768.0 | number of skipped iterations:   2 | number of nan iterations:   0 |
time (ms) | forward: 610.50 | backward: 2205.78 | backward-backward: 2205.71 | backward-allreduce: 0.00 | optimizer: 79.33 | batch generator: 2.41
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step10000
 samples/sec: 352.269 | iteration    13500/  143000 | elapsed time per iteration (ms): 2906.9 | learning rate: 5.904E-04 | approx flops per GPU: 76.2TFLOPS | lm_loss: 2.299688E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 610.55 | backward: 2211.04 | backward-backward: 2210.97 | backward-allreduce: 0.00 | optimizer: 79.64 | batch generator: 2.43
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step10500
 samples/sec: 352.845 | iteration    14000/  143000 | elapsed time per iteration (ms): 2902.1 | learning rate: 5.896E-04 | approx flops per GPU: 76.3TFLOPS | lm_loss: 2.296754E+00 | loss scale: 16384.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 610.46 | backward: 2206.06 | backward-backward: 2205.98 | backward-allreduce: 0.00 | optimizer: 79.46 | batch generator: 2.38
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step11000
 samples/sec: 352.970 | iteration    14500/  143000 | elapsed time per iteration (ms): 2901.1 | learning rate: 5.887E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.291494E+00 | loss scale: 16384.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 609.69 | backward: 2206.35 | backward-backward: 2206.28 | backward-allreduce: 0.00 | optimizer: 79.58 | batch generator: 2.34
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step11500
 samples/sec: 352.892 | iteration    15000/  143000 | elapsed time per iteration (ms): 2901.7 | learning rate: 5.879E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.288373E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 609.95 | backward: 2206.62 | backward-backward: 2206.54 | backward-allreduce: 0.00 | optimizer: 79.58 | batch generator: 2.35
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step12000
----------------------------------------------------------------------------------------------------------
 validation results at iteration 15000 | lm_loss value: 2.235826E+00 | lm_loss_ppl value: 9.354208E+00 | 
----------------------------------------------------------------------------------------------------------
 samples/sec: 338.137 | iteration    15500/  143000 | elapsed time per iteration (ms): 3028.4 | learning rate: 5.870E-04 | approx flops per GPU: 73.2TFLOPS | lm_loss: 2.282740E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 609.83 | backward: 2205.90 | backward-backward: 2205.83 | backward-allreduce: 0.00 | optimizer: 79.63 | batch generator: 2.80
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step12500
 samples/sec: 352.894 | iteration    16000/  143000 | elapsed time per iteration (ms): 2901.7 | learning rate: 5.860E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.279487E+00 | loss scale: 65536.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 610.46 | backward: 2206.08 | backward-backward: 2206.01 | backward-allreduce: 0.00 | optimizer: 79.61 | batch generator: 2.42
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step13000
 samples/sec: 352.987 | iteration    16500/  143000 | elapsed time per iteration (ms): 2901.0 | learning rate: 5.851E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.277207E+00 | loss scale: 65536.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 610.07 | backward: 2205.59 | backward-backward: 2205.52 | backward-allreduce: 0.00 | optimizer: 79.68 | batch generator: 2.34
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step13500
 samples/sec: 352.909 | iteration    17000/  143000 | elapsed time per iteration (ms): 2901.6 | learning rate: 5.841E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.272039E+00 | loss scale: 131072.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 610.39 | backward: 2206.05 | backward-backward: 2205.98 | backward-allreduce: 0.00 | optimizer: 79.44 | batch generator: 2.34
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step14000
 samples/sec: 353.060 | iteration    17500/  143000 | elapsed time per iteration (ms): 2900.4 | learning rate: 5.830E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.269935E+00 | loss scale: 32768.0 | number of skipped iterations:   2 | number of nan iterations:   0 |
time (ms) | forward: 610.54 | backward: 2204.93 | backward-backward: 2204.86 | backward-allreduce: 0.00 | optimizer: 79.30 | batch generator: 2.36
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step14500
 samples/sec: 352.898 | iteration    18000/  143000 | elapsed time per iteration (ms): 2901.7 | learning rate: 5.820E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.265185E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 609.88 | backward: 2206.55 | backward-backward: 2206.48 | backward-allreduce: 0.00 | optimizer: 79.58 | batch generator: 2.29
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step15000
 samples/sec: 353.008 | iteration    18500/  143000 | elapsed time per iteration (ms): 2900.8 | learning rate: 5.809E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.264649E+00 | loss scale: 65536.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 610.41 | backward: 2205.34 | backward-backward: 2205.27 | backward-allreduce: 0.00 | optimizer: 79.61 | batch generator: 2.32
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step15500
 samples/sec: 352.880 | iteration    19000/  143000 | elapsed time per iteration (ms): 2901.8 | learning rate: 5.798E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.260992E+00 | loss scale: 65536.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 609.88 | backward: 2206.13 | backward-backward: 2206.06 | backward-allreduce: 0.00 | optimizer: 79.59 | batch generator: 2.29
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step16000
 samples/sec: 353.063 | iteration    19500/  143000 | elapsed time per iteration (ms): 2900.3 | learning rate: 5.786E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.258127E+00 | loss scale: 65536.0 | number of skipped iterations:   2 | number of nan iterations:   0 |
time (ms) | forward: 609.66 | backward: 2205.94 | backward-backward: 2205.87 | backward-allreduce: 0.00 | optimizer: 79.32 | batch generator: 2.27
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step16500
 samples/sec: 352.917 | iteration    20000/  143000 | elapsed time per iteration (ms): 2901.5 | learning rate: 5.774E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.254986E+00 | loss scale: 32768.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 609.85 | backward: 2206.64 | backward-backward: 2206.57 | backward-allreduce: 0.00 | optimizer: 79.49 | batch generator: 2.28
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step17000
----------------------------------------------------------------------------------------------------------
 validation results at iteration 20000 | lm_loss value: 2.205550E+00 | lm_loss_ppl value: 9.075239E+00 | 
----------------------------------------------------------------------------------------------------------
 samples/sec: 335.977 | iteration    20500/  143000 | elapsed time per iteration (ms): 3047.8 | learning rate: 5.762E-04 | approx flops per GPU: 72.7TFLOPS | lm_loss: 2.253958E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 609.77 | backward: 2205.69 | backward-backward: 2205.62 | backward-allreduce: 0.00 | optimizer: 79.78 | batch generator: 6.67
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step17500
 samples/sec: 352.881 | iteration    21000/  143000 | elapsed time per iteration (ms): 2901.8 | learning rate: 5.750E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.249801E+00 | loss scale: 65536.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 609.97 | backward: 2206.61 | backward-backward: 2206.53 | backward-allreduce: 0.00 | optimizer: 79.76 | batch generator: 2.31
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step18000
 samples/sec: 351.926 | iteration    21500/  143000 | elapsed time per iteration (ms): 2909.7 | learning rate: 5.737E-04 | approx flops per GPU: 76.1TFLOPS | lm_loss: 2.249513E+00 | loss scale: 65536.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 617.36 | backward: 2206.78 | backward-backward: 2206.71 | backward-allreduce: 0.00 | optimizer: 79.67 | batch generator: 9.92
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step18500
 samples/sec: 352.934 | iteration    22000/  143000 | elapsed time per iteration (ms): 2901.4 | learning rate: 5.724E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.247319E+00 | loss scale: 131072.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 609.90 | backward: 2206.51 | backward-backward: 2206.44 | backward-allreduce: 0.00 | optimizer: 79.48 | batch generator: 2.31
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step19000
 samples/sec: 353.014 | iteration    22500/  143000 | elapsed time per iteration (ms): 2900.7 | learning rate: 5.711E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.245720E+00 | loss scale: 65536.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 609.74 | backward: 2205.88 | backward-backward: 2205.81 | backward-allreduce: 0.00 | optimizer: 79.49 | batch generator: 2.36
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step19500
 samples/sec: 352.890 | iteration    23000/  143000 | elapsed time per iteration (ms): 2901.8 | learning rate: 5.697E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.243116E+00 | loss scale: 32768.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 610.26 | backward: 2206.25 | backward-backward: 2206.17 | backward-allreduce: 0.00 | optimizer: 79.52 | batch generator: 2.38
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step20000
 samples/sec: 352.974 | iteration    23500/  143000 | elapsed time per iteration (ms): 2901.1 | learning rate: 5.683E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.241707E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 609.76 | backward: 2206.25 | backward-backward: 2206.18 | backward-allreduce: 0.00 | optimizer: 79.62 | batch generator: 2.36
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step20500
 samples/sec: 352.923 | iteration    24000/  143000 | elapsed time per iteration (ms): 2901.5 | learning rate: 5.669E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.242003E+00 | loss scale: 65536.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 609.99 | backward: 2206.31 | backward-backward: 2206.24 | backward-allreduce: 0.00 | optimizer: 79.65 | batch generator: 2.40
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step21000
 samples/sec: 352.905 | iteration    24500/  143000 | elapsed time per iteration (ms): 2901.6 | learning rate: 5.655E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.237709E+00 | loss scale: 65536.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 609.82 | backward: 2206.67 | backward-backward: 2206.60 | backward-allreduce: 0.00 | optimizer: 79.65 | batch generator: 2.38
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step21500
 samples/sec: 352.945 | iteration    25000/  143000 | elapsed time per iteration (ms): 2901.3 | learning rate: 5.640E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.236009E+00 | loss scale: 65536.0 | number of skipped iterations:   2 | number of nan iterations:   0 |
time (ms) | forward: 610.49 | backward: 2205.98 | backward-backward: 2205.91 | backward-allreduce: 0.00 | optimizer: 79.34 | batch generator: 2.40
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step22000
----------------------------------------------------------------------------------------------------------
 validation results at iteration 25000 | lm_loss value: 2.192039E+00 | lm_loss_ppl value: 8.953446E+00 | 
----------------------------------------------------------------------------------------------------------
 samples/sec: 337.969 | iteration    25500/  143000 | elapsed time per iteration (ms): 3029.9 | learning rate: 5.625E-04 | approx flops per GPU: 73.1TFLOPS | lm_loss: 2.234498E+00 | loss scale: 65536.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 610.67 | backward: 2206.27 | backward-backward: 2206.20 | backward-allreduce: 0.00 | optimizer: 79.67 | batch generator: 2.79
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step22500
 samples/sec: 352.955 | iteration    26000/  143000 | elapsed time per iteration (ms): 2901.2 | learning rate: 5.609E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.233741E+00 | loss scale: 131072.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 610.84 | backward: 2205.22 | backward-backward: 2205.14 | backward-allreduce: 0.00 | optimizer: 79.50 | batch generator: 2.36
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step23000
 samples/sec: 352.802 | iteration    26500/  143000 | elapsed time per iteration (ms): 2902.5 | learning rate: 5.594E-04 | approx flops per GPU: 76.3TFLOPS | lm_loss: 2.230550E+00 | loss scale: 65536.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 610.58 | backward: 2206.58 | backward-backward: 2206.51 | backward-allreduce: 0.00 | optimizer: 79.53 | batch generator: 2.35
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step23500
 samples/sec: 352.196 | iteration    27000/  143000 | elapsed time per iteration (ms): 2907.5 | learning rate: 5.578E-04 | approx flops per GPU: 76.2TFLOPS | lm_loss: 2.229256E+00 | loss scale: 32768.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 610.68 | backward: 2211.71 | backward-backward: 2211.64 | backward-allreduce: 0.00 | optimizer: 79.54 | batch generator: 2.32
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step24000
 samples/sec: 352.787 | iteration    27500/  143000 | elapsed time per iteration (ms): 2902.6 | learning rate: 5.562E-04 | approx flops per GPU: 76.3TFLOPS | lm_loss: 2.227592E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 610.56 | backward: 2206.68 | backward-backward: 2206.61 | backward-allreduce: 0.00 | optimizer: 79.85 | batch generator: 2.36
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step24500
 samples/sec: 352.828 | iteration    28000/  143000 | elapsed time per iteration (ms): 2902.3 | learning rate: 5.545E-04 | approx flops per GPU: 76.3TFLOPS | lm_loss: 2.226016E+00 | loss scale: 65536.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 610.37 | backward: 2206.40 | backward-backward: 2206.33 | backward-allreduce: 0.00 | optimizer: 79.85 | batch generator: 2.33
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step25000
 samples/sec: 352.809 | iteration    28500/  143000 | elapsed time per iteration (ms): 2902.4 | learning rate: 5.528E-04 | approx flops per GPU: 76.3TFLOPS | lm_loss: 2.224761E+00 | loss scale: 32768.0 | number of skipped iterations:   2 | number of nan iterations:   0 |
time (ms) | forward: 610.65 | backward: 2206.90 | backward-backward: 2206.83 | backward-allreduce: 0.00 | optimizer: 79.42 | batch generator: 2.39
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step25500
 samples/sec: 352.819 | iteration    29000/  143000 | elapsed time per iteration (ms): 2902.3 | learning rate: 5.511E-04 | approx flops per GPU: 76.3TFLOPS | lm_loss: 2.222556E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 610.84 | backward: 2206.07 | backward-backward: 2206.00 | backward-allreduce: 0.00 | optimizer: 79.79 | batch generator: 2.41
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step26000
 samples/sec: 352.801 | iteration    29500/  143000 | elapsed time per iteration (ms): 2902.5 | learning rate: 5.494E-04 | approx flops per GPU: 76.3TFLOPS | lm_loss: 2.222526E+00 | loss scale: 16384.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 610.60 | backward: 2206.68 | backward-backward: 2206.61 | backward-allreduce: 0.00 | optimizer: 79.65 | batch generator: 2.35
 samples/sec: 352.290 | iteration    30000/  143000 | elapsed time per iteration (ms): 2906.7 | learning rate: 5.476E-04 | approx flops per GPU: 76.2TFLOPS | lm_loss: 2.219844E+00 | loss scale: 16384.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
after 30000 iterations memory (MB) | allocated: 3623.939453125 | max allocated: 11451.5068359375 | reserved: 13710.0 | max reserved: 13710.0
time (ms) | forward: 617.72 | backward: 2207.65 | backward-backward: 2207.58 | backward-allreduce: 0.00 | optimizer: 79.96 | batch generator: 5.17
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step27000
----------------------------------------------------------------------------------------------------------
 validation results at iteration 30000 | lm_loss value: 2.171570E+00 | lm_loss_ppl value: 8.772044E+00 | 
----------------------------------------------------------------------------------------------------------
 samples/sec: 337.953 | iteration    30500/  143000 | elapsed time per iteration (ms): 3030.0 | learning rate: 5.459E-04 | approx flops per GPU: 73.1TFLOPS | lm_loss: 2.218577E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 609.60 | backward: 2205.52 | backward-backward: 2205.44 | backward-allreduce: 0.00 | optimizer: 79.96 | batch generator: 2.65
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step27500
 samples/sec: 353.089 | iteration    31000/  143000 | elapsed time per iteration (ms): 2900.1 | learning rate: 5.440E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.218100E+00 | loss scale: 32768.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 609.25 | backward: 2205.63 | backward-backward: 2205.56 | backward-allreduce: 0.00 | optimizer: 79.80 | batch generator: 2.25
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step28000
 samples/sec: 353.054 | iteration    31500/  143000 | elapsed time per iteration (ms): 2900.4 | learning rate: 5.422E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.214409E+00 | loss scale: 16384.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 609.42 | backward: 2205.83 | backward-backward: 2205.76 | backward-allreduce: 0.00 | optimizer: 79.81 | batch generator: 2.27
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step28500
 samples/sec: 353.107 | iteration    32000/  143000 | elapsed time per iteration (ms): 2900.0 | learning rate: 5.403E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.214736E+00 | loss scale: 16384.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 609.32 | backward: 2205.26 | backward-backward: 2205.19 | backward-allreduce: 0.00 | optimizer: 80.00 | batch generator: 2.25
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step29000
 samples/sec: 353.060 | iteration    32500/  143000 | elapsed time per iteration (ms): 2900.4 | learning rate: 5.385E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.213073E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 609.50 | backward: 2205.52 | backward-backward: 2205.45 | backward-allreduce: 0.00 | optimizer: 79.97 | batch generator: 2.25
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step29500
 samples/sec: 353.124 | iteration    33000/  143000 | elapsed time per iteration (ms): 2899.8 | learning rate: 5.365E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.213183E+00 | loss scale: 32768.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 609.34 | backward: 2205.10 | backward-backward: 2205.03 | backward-allreduce: 0.00 | optimizer: 79.82 | batch generator: 2.25
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step30000
 samples/sec: 353.007 | iteration    33500/  143000 | elapsed time per iteration (ms): 2900.8 | learning rate: 5.346E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.210692E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 609.95 | backward: 2205.39 | backward-backward: 2205.31 | backward-allreduce: 0.00 | optimizer: 80.00 | batch generator: 2.30
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step30500
 samples/sec: 353.089 | iteration    34000/  143000 | elapsed time per iteration (ms): 2900.1 | learning rate: 5.326E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.211081E+00 | loss scale: 16384.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 609.58 | backward: 2205.18 | backward-backward: 2205.11 | backward-allreduce: 0.00 | optimizer: 79.84 | batch generator: 2.26
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step31000
 samples/sec: 353.024 | iteration    34500/  143000 | elapsed time per iteration (ms): 2900.7 | learning rate: 5.306E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.208467E+00 | loss scale: 16384.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 609.54 | backward: 2205.59 | backward-backward: 2205.51 | backward-allreduce: 0.00 | optimizer: 79.96 | batch generator: 2.26
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step31500
 samples/sec: 353.171 | iteration    35000/  143000 | elapsed time per iteration (ms): 2899.4 | learning rate: 5.286E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.206911E+00 | loss scale: 32768.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 609.37 | backward: 2204.80 | backward-backward: 2204.73 | backward-allreduce: 0.00 | optimizer: 79.83 | batch generator: 2.29
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step32000
----------------------------------------------------------------------------------------------------------
 validation results at iteration 35000 | lm_loss value: 2.161903E+00 | lm_loss_ppl value: 8.687654E+00 | 
----------------------------------------------------------------------------------------------------------
 samples/sec: 338.232 | iteration    35500/  143000 | elapsed time per iteration (ms): 3027.5 | learning rate: 5.266E-04 | approx flops per GPU: 73.2TFLOPS | lm_loss: 2.205729E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 609.79 | backward: 2204.63 | backward-backward: 2204.55 | backward-allreduce: 0.00 | optimizer: 80.01 | batch generator: 2.69
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step32500
 samples/sec: 353.216 | iteration    36000/  143000 | elapsed time per iteration (ms): 2899.1 | learning rate: 5.245E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.204554E+00 | loss scale: 16384.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 609.47 | backward: 2204.01 | backward-backward: 2203.94 | backward-allreduce: 0.00 | optimizer: 79.82 | batch generator: 2.26
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step33000
 samples/sec: 353.212 | iteration    36500/  143000 | elapsed time per iteration (ms): 2899.1 | learning rate: 5.224E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.202843E+00 | loss scale: 16384.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 609.73 | backward: 2204.00 | backward-backward: 2203.92 | backward-allreduce: 0.00 | optimizer: 79.94 | batch generator: 2.30
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step33500
 samples/sec: 353.282 | iteration    37000/  143000 | elapsed time per iteration (ms): 2898.5 | learning rate: 5.203E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.202124E+00 | loss scale: 32768.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 609.55 | backward: 2203.61 | backward-backward: 2203.53 | backward-allreduce: 0.00 | optimizer: 79.83 | batch generator: 2.27
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step34000
 samples/sec: 352.516 | iteration    37500/  143000 | elapsed time per iteration (ms): 2904.8 | learning rate: 5.182E-04 | approx flops per GPU: 76.3TFLOPS | lm_loss: 2.201600E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 609.81 | backward: 2209.54 | backward-backward: 2209.47 | backward-allreduce: 0.00 | optimizer: 79.92 | batch generator: 2.33
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step34500
 samples/sec: 353.205 | iteration    38000/  143000 | elapsed time per iteration (ms): 2899.2 | learning rate: 5.160E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.201355E+00 | loss scale: 16384.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 609.55 | backward: 2204.24 | backward-backward: 2204.17 | backward-allreduce: 0.00 | optimizer: 79.80 | batch generator: 2.27
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step35000
 samples/sec: 353.133 | iteration    38500/  143000 | elapsed time per iteration (ms): 2899.8 | learning rate: 5.139E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.200124E+00 | loss scale: 16384.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 609.78 | backward: 2204.53 | backward-backward: 2204.45 | backward-allreduce: 0.00 | optimizer: 79.96 | batch generator: 2.29
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step35500
 samples/sec: 353.179 | iteration    39000/  143000 | elapsed time per iteration (ms): 2899.4 | learning rate: 5.116E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.200712E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 609.59 | backward: 2204.03 | backward-backward: 2203.96 | backward-allreduce: 0.00 | optimizer: 80.06 | batch generator: 2.26
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step36000
 samples/sec: 353.148 | iteration    39500/  143000 | elapsed time per iteration (ms): 2899.6 | learning rate: 5.094E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.197089E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 609.78 | backward: 2204.26 | backward-backward: 2204.18 | backward-allreduce: 0.00 | optimizer: 80.01 | batch generator: 2.29
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step36500
 samples/sec: 353.290 | iteration    40000/  143000 | elapsed time per iteration (ms): 2898.5 | learning rate: 5.072E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.195795E+00 | loss scale: 32768.0 | number of skipped iterations:   2 | number of nan iterations:   0 |
time (ms) | forward: 609.56 | backward: 2203.77 | backward-backward: 2203.70 | backward-allreduce: 0.00 | optimizer: 79.66 | batch generator: 2.27
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step37000
----------------------------------------------------------------------------------------------------------
 validation results at iteration 40000 | lm_loss value: 2.145396E+00 | lm_loss_ppl value: 8.545420E+00 | 
----------------------------------------------------------------------------------------------------------
 samples/sec: 338.263 | iteration    40500/  143000 | elapsed time per iteration (ms): 3027.2 | learning rate: 5.049E-04 | approx flops per GPU: 73.2TFLOPS | lm_loss: 2.194291E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 610.04 | backward: 2204.35 | backward-backward: 2204.27 | backward-allreduce: 0.00 | optimizer: 80.01 | batch generator: 2.70
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step37500
 samples/sec: 353.221 | iteration    41000/  143000 | elapsed time per iteration (ms): 2899.0 | learning rate: 5.026E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.194061E+00 | loss scale: 8192.0 | number of skipped iterations:   2 | number of nan iterations:   0 |
time (ms) | forward: 609.87 | backward: 2204.04 | backward-backward: 2203.96 | backward-allreduce: 0.00 | optimizer: 79.68 | batch generator: 2.31
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step38000
 samples/sec: 353.054 | iteration    41500/  143000 | elapsed time per iteration (ms): 2900.4 | learning rate: 5.003E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.193298E+00 | loss scale: 8192.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 610.12 | backward: 2204.52 | backward-backward: 2204.45 | backward-allreduce: 0.00 | optimizer: 80.00 | batch generator: 2.34
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step38500
 samples/sec: 353.114 | iteration    42000/  143000 | elapsed time per iteration (ms): 2899.9 | learning rate: 4.980E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.191050E+00 | loss scale: 16384.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 609.98 | backward: 2204.12 | backward-backward: 2204.04 | backward-allreduce: 0.00 | optimizer: 80.02 | batch generator: 2.38
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step39000
 samples/sec: 353.115 | iteration    42500/  143000 | elapsed time per iteration (ms): 2899.9 | learning rate: 4.956E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.191622E+00 | loss scale: 16384.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 610.17 | backward: 2204.08 | backward-backward: 2204.01 | backward-allreduce: 0.00 | optimizer: 80.00 | batch generator: 2.39
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step39500
 samples/sec: 353.193 | iteration    43000/  143000 | elapsed time per iteration (ms): 2899.3 | learning rate: 4.932E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.189374E+00 | loss scale: 16384.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 609.07 | backward: 2204.72 | backward-backward: 2204.65 | backward-allreduce: 0.00 | optimizer: 79.83 | batch generator: 2.34
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step40000
 samples/sec: 353.062 | iteration    43500/  143000 | elapsed time per iteration (ms): 2900.3 | learning rate: 4.909E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.189497E+00 | loss scale: 16384.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 610.04 | backward: 2204.63 | backward-backward: 2204.56 | backward-allreduce: 0.00 | optimizer: 80.00 | batch generator: 2.44
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step40500
 samples/sec: 353.130 | iteration    44000/  143000 | elapsed time per iteration (ms): 2899.8 | learning rate: 4.884E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.186470E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 609.15 | backward: 2204.83 | backward-backward: 2204.76 | backward-allreduce: 0.00 | optimizer: 79.97 | batch generator: 2.40
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step41000
 samples/sec: 353.135 | iteration    44500/  143000 | elapsed time per iteration (ms): 2899.7 | learning rate: 4.860E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.185729E+00 | loss scale: 32768.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 609.37 | backward: 2204.91 | backward-backward: 2204.83 | backward-allreduce: 0.00 | optimizer: 79.83 | batch generator: 2.43
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step41500
 samples/sec: 353.202 | iteration    45000/  143000 | elapsed time per iteration (ms): 2899.2 | learning rate: 4.836E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.185482E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 609.06 | backward: 2204.52 | backward-backward: 2204.45 | backward-allreduce: 0.00 | optimizer: 79.95 | batch generator: 2.32
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step42000
----------------------------------------------------------------------------------------------------------
 validation results at iteration 45000 | lm_loss value: 2.138723E+00 | lm_loss_ppl value: 8.488594E+00 | 
----------------------------------------------------------------------------------------------------------
 samples/sec: 338.317 | iteration    45500/  143000 | elapsed time per iteration (ms): 3026.7 | learning rate: 4.811E-04 | approx flops per GPU: 73.2TFLOPS | lm_loss: 2.183970E+00 | loss scale: 65536.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 609.23 | backward: 2204.88 | backward-backward: 2204.81 | backward-allreduce: 0.00 | optimizer: 79.82 | batch generator: 2.69
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step42500
 samples/sec: 353.201 | iteration    46000/  143000 | elapsed time per iteration (ms): 2899.2 | learning rate: 4.786E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.182952E+00 | loss scale: 32768.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 609.04 | backward: 2204.59 | backward-backward: 2204.52 | backward-allreduce: 0.00 | optimizer: 79.79 | batch generator: 2.30
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step43000
 samples/sec: 353.141 | iteration    46500/  143000 | elapsed time per iteration (ms): 2899.7 | learning rate: 4.761E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.181557E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 609.25 | backward: 2204.76 | backward-backward: 2204.69 | backward-allreduce: 0.00 | optimizer: 79.97 | batch generator: 2.33
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step43500
 samples/sec: 353.227 | iteration    47000/  143000 | elapsed time per iteration (ms): 2899.0 | learning rate: 4.736E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.180339E+00 | loss scale: 65536.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 609.04 | backward: 2204.50 | backward-backward: 2204.42 | backward-allreduce: 0.00 | optimizer: 79.94 | batch generator: 2.31
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step44000
 samples/sec: 353.206 | iteration    47500/  143000 | elapsed time per iteration (ms): 2899.2 | learning rate: 4.710E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.180814E+00 | loss scale: 65536.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 609.59 | backward: 2204.34 | backward-backward: 2204.27 | backward-allreduce: 0.00 | optimizer: 79.82 | batch generator: 2.35
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step44500
 samples/sec: 353.223 | iteration    48000/  143000 | elapsed time per iteration (ms): 2899.0 | learning rate: 4.685E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.179525E+00 | loss scale: 32768.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 609.96 | backward: 2203.51 | backward-backward: 2203.44 | backward-allreduce: 0.00 | optimizer: 79.87 | batch generator: 2.33
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step45000
 samples/sec: 353.153 | iteration    48500/  143000 | elapsed time per iteration (ms): 2899.6 | learning rate: 4.659E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.178233E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 610.21 | backward: 2203.92 | backward-backward: 2203.85 | backward-allreduce: 0.00 | optimizer: 79.99 | batch generator: 2.39
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step45500
 samples/sec: 353.266 | iteration    49000/  143000 | elapsed time per iteration (ms): 2898.7 | learning rate: 4.633E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.178110E+00 | loss scale: 16384.0 | number of skipped iterations:   3 | number of nan iterations:   0 |
time (ms) | forward: 610.00 | backward: 2203.66 | backward-backward: 2203.59 | backward-allreduce: 0.00 | optimizer: 79.49 | batch generator: 2.34
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step46000
 samples/sec: 353.088 | iteration    49500/  143000 | elapsed time per iteration (ms): 2900.1 | learning rate: 4.607E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.177734E+00 | loss scale: 16384.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 610.21 | backward: 2204.44 | backward-backward: 2204.36 | backward-allreduce: 0.00 | optimizer: 79.98 | batch generator: 2.39
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step46500
 samples/sec: 353.138 | iteration    50000/  143000 | elapsed time per iteration (ms): 2899.7 | learning rate: 4.581E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.174503E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 610.04 | backward: 2203.97 | backward-backward: 2203.90 | backward-allreduce: 0.00 | optimizer: 80.00 | batch generator: 2.36
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step47000
----------------------------------------------------------------------------------------------------------
 validation results at iteration 50000 | lm_loss value: 2.130526E+00 | lm_loss_ppl value: 8.419295E+00 | 
----------------------------------------------------------------------------------------------------------
 samples/sec: 338.257 | iteration    50500/  143000 | elapsed time per iteration (ms): 3027.3 | learning rate: 4.554E-04 | approx flops per GPU: 73.2TFLOPS | lm_loss: 2.175228E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 610.28 | backward: 2203.95 | backward-backward: 2203.88 | backward-allreduce: 0.00 | optimizer: 80.03 | batch generator: 2.77
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step47500
 samples/sec: 353.186 | iteration    51000/  143000 | elapsed time per iteration (ms): 2899.3 | learning rate: 4.527E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.173995E+00 | loss scale: 65536.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 609.98 | backward: 2203.62 | backward-backward: 2203.55 | backward-allreduce: 0.00 | optimizer: 79.99 | batch generator: 2.34
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step48000
 samples/sec: 352.949 | iteration    51500/  143000 | elapsed time per iteration (ms): 2901.3 | learning rate: 4.501E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.171973E+00 | loss scale: 32768.0 | number of skipped iterations:   2 | number of nan iterations:   0 |
time (ms) | forward: 610.30 | backward: 2204.71 | backward-backward: 2204.64 | backward-allreduce: 0.00 | optimizer: 80.28 | batch generator: 2.45
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step48500
 samples/sec: 353.153 | iteration    52000/  143000 | elapsed time per iteration (ms): 2899.6 | learning rate: 4.474E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.171407E+00 | loss scale: 16384.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 610.09 | backward: 2204.00 | backward-backward: 2203.93 | backward-allreduce: 0.00 | optimizer: 79.89 | batch generator: 2.42
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step49000
 samples/sec: 353.082 | iteration    52500/  143000 | elapsed time per iteration (ms): 2900.2 | learning rate: 4.447E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.173119E+00 | loss scale: 8192.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 610.28 | backward: 2204.49 | backward-backward: 2204.41 | backward-allreduce: 0.00 | optimizer: 79.85 | batch generator: 2.41
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step49500
 samples/sec: 353.104 | iteration    53000/  143000 | elapsed time per iteration (ms): 2900.0 | learning rate: 4.420E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.170325E+00 | loss scale: 8192.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 610.08 | backward: 2204.37 | backward-backward: 2204.30 | backward-allreduce: 0.00 | optimizer: 79.98 | batch generator: 2.36
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step50000
 samples/sec: 353.031 | iteration    53500/  143000 | elapsed time per iteration (ms): 2900.6 | learning rate: 4.392E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.169852E+00 | loss scale: 16384.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 610.22 | backward: 2204.69 | backward-backward: 2204.62 | backward-allreduce: 0.00 | optimizer: 79.99 | batch generator: 2.36
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step50500
 samples/sec: 353.111 | iteration    54000/  143000 | elapsed time per iteration (ms): 2899.9 | learning rate: 4.365E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.168136E+00 | loss scale: 16384.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 609.97 | backward: 2204.53 | backward-backward: 2204.46 | backward-allreduce: 0.00 | optimizer: 79.87 | batch generator: 2.35
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step51000
 samples/sec: 353.074 | iteration    54500/  143000 | elapsed time per iteration (ms): 2900.2 | learning rate: 4.337E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.167851E+00 | loss scale: 16384.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 610.24 | backward: 2204.42 | backward-backward: 2204.34 | backward-allreduce: 0.00 | optimizer: 79.99 | batch generator: 2.40
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step51500
 samples/sec: 353.128 | iteration    55000/  143000 | elapsed time per iteration (ms): 2899.8 | learning rate: 4.310E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.166386E+00 | loss scale: 32768.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 609.86 | backward: 2204.40 | backward-backward: 2204.33 | backward-allreduce: 0.00 | optimizer: 79.85 | batch generator: 2.36
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step52000
----------------------------------------------------------------------------------------------------------
 validation results at iteration 55000 | lm_loss value: 2.120372E+00 | lm_loss_ppl value: 8.334240E+00 | 
----------------------------------------------------------------------------------------------------------
 samples/sec: 338.261 | iteration    55500/  143000 | elapsed time per iteration (ms): 3027.2 | learning rate: 4.282E-04 | approx flops per GPU: 73.2TFLOPS | lm_loss: 2.166138E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 609.92 | backward: 2204.37 | backward-backward: 2204.29 | backward-allreduce: 0.00 | optimizer: 80.07 | batch generator: 2.70
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step52500
 samples/sec: 353.047 | iteration    56000/  143000 | elapsed time per iteration (ms): 2900.5 | learning rate: 4.254E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.164145E+00 | loss scale: 65536.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 609.80 | backward: 2205.04 | backward-backward: 2204.96 | backward-allreduce: 0.00 | optimizer: 79.90 | batch generator: 2.36
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step53000
 samples/sec: 352.928 | iteration    56500/  143000 | elapsed time per iteration (ms): 2901.4 | learning rate: 4.226E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.163596E+00 | loss scale: 32768.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 609.90 | backward: 2204.34 | backward-backward: 2204.27 | backward-allreduce: 0.00 | optimizer: 79.86 | batch generator: 2.35
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step53500
 samples/sec: 353.058 | iteration    57000/  143000 | elapsed time per iteration (ms): 2900.4 | learning rate: 4.198E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.162641E+00 | loss scale: 16384.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 609.08 | backward: 2205.96 | backward-backward: 2205.88 | backward-allreduce: 0.00 | optimizer: 79.88 | batch generator: 2.28
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step54000
 samples/sec: 353.075 | iteration    57500/  143000 | elapsed time per iteration (ms): 2900.2 | learning rate: 4.170E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.162332E+00 | loss scale: 16384.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 609.29 | backward: 2205.46 | backward-backward: 2205.39 | backward-allreduce: 0.00 | optimizer: 80.01 | batch generator: 2.32
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step54500
 samples/sec: 353.009 | iteration    58000/  143000 | elapsed time per iteration (ms): 2900.8 | learning rate: 4.141E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.162577E+00 | loss scale: 8192.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 609.11 | backward: 2206.19 | backward-backward: 2206.12 | backward-allreduce: 0.00 | optimizer: 79.84 | batch generator: 2.32
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step55000
 samples/sec: 353.009 | iteration    58500/  143000 | elapsed time per iteration (ms): 2900.8 | learning rate: 4.113E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.157890E+00 | loss scale: 8192.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 609.29 | backward: 2205.56 | backward-backward: 2205.49 | backward-allreduce: 0.00 | optimizer: 79.99 | batch generator: 2.29
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step55500
 samples/sec: 353.007 | iteration    59000/  143000 | elapsed time per iteration (ms): 2900.8 | learning rate: 4.084E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.160383E+00 | loss scale: 16384.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 609.06 | backward: 2206.43 | backward-backward: 2206.36 | backward-allreduce: 0.00 | optimizer: 79.82 | batch generator: 2.24
 samples/sec: 352.461 | iteration    59500/  143000 | elapsed time per iteration (ms): 2905.3 | learning rate: 4.055E-04 | approx flops per GPU: 76.3TFLOPS | lm_loss: 2.158436E+00 | loss scale: 16384.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
after 59500 iterations memory (MB) | allocated: 3623.939453125 | max allocated: 11451.5068359375 | reserved: 13710.0 | max reserved: 13710.0
time (ms) | forward: 612.28 | backward: 2212.08 | backward-backward: 2212.00 | backward-allreduce: 0.00 | optimizer: 79.56 | batch generator: 2.38
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step56500
 samples/sec: 352.901 | iteration    60000/  143000 | elapsed time per iteration (ms): 2901.7 | learning rate: 4.027E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.158576E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 610.15 | backward: 2205.89 | backward-backward: 2205.81 | backward-allreduce: 0.00 | optimizer: 79.58 | batch generator: 2.29
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step57000
----------------------------------------------------------------------------------------------------------
 validation results at iteration 60000 | lm_loss value: 2.111970E+00 | lm_loss_ppl value: 8.264506E+00 | 
----------------------------------------------------------------------------------------------------------
 samples/sec: 338.134 | iteration    60500/  143000 | elapsed time per iteration (ms): 3028.4 | learning rate: 3.998E-04 | approx flops per GPU: 73.2TFLOPS | lm_loss: 2.156245E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 609.96 | backward: 2205.88 | backward-backward: 2205.81 | backward-allreduce: 0.00 | optimizer: 79.61 | batch generator: 2.68
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step57500
 samples/sec: 352.976 | iteration    61000/  143000 | elapsed time per iteration (ms): 2901.0 | learning rate: 3.969E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.155501E+00 | loss scale: 16384.0 | number of skipped iterations:   2 | number of nan iterations:   0 |
time (ms) | forward: 610.04 | backward: 2206.12 | backward-backward: 2206.05 | backward-allreduce: 0.00 | optimizer: 79.28 | batch generator: 2.30
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step58000
 samples/sec: 353.005 | iteration    61500/  143000 | elapsed time per iteration (ms): 2900.8 | learning rate: 3.940E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.157141E+00 | loss scale: 16384.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 609.63 | backward: 2206.19 | backward-backward: 2206.12 | backward-allreduce: 0.00 | optimizer: 79.60 | batch generator: 2.30
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step58500
 samples/sec: 352.935 | iteration    62000/  143000 | elapsed time per iteration (ms): 2901.4 | learning rate: 3.911E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.151945E+00 | loss scale: 32768.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 609.81 | backward: 2206.55 | backward-backward: 2206.48 | backward-allreduce: 0.00 | optimizer: 79.48 | batch generator: 2.30
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step59000
 samples/sec: 353.034 | iteration    62500/  143000 | elapsed time per iteration (ms): 2900.6 | learning rate: 3.881E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.153375E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 609.26 | backward: 2206.32 | backward-backward: 2206.25 | backward-allreduce: 0.00 | optimizer: 79.57 | batch generator: 2.28
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step59500
 samples/sec: 352.947 | iteration    63000/  143000 | elapsed time per iteration (ms): 2901.3 | learning rate: 3.852E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.149092E+00 | loss scale: 65536.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 609.42 | backward: 2206.85 | backward-backward: 2206.78 | backward-allreduce: 0.00 | optimizer: 79.58 | batch generator: 2.27
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step60000
 samples/sec: 353.107 | iteration    63500/  143000 | elapsed time per iteration (ms): 2900.0 | learning rate: 3.823E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.150221E+00 | loss scale: 65536.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 609.62 | backward: 2205.37 | backward-backward: 2205.30 | backward-allreduce: 0.00 | optimizer: 79.44 | batch generator: 2.27
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step60500
 samples/sec: 353.025 | iteration    64000/  143000 | elapsed time per iteration (ms): 2900.6 | learning rate: 3.793E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.151737E+00 | loss scale: 65536.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 609.86 | backward: 2205.68 | backward-backward: 2205.61 | backward-allreduce: 0.00 | optimizer: 79.58 | batch generator: 2.28
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step61000
 samples/sec: 353.219 | iteration    64500/  143000 | elapsed time per iteration (ms): 2899.1 | learning rate: 3.764E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.147897E+00 | loss scale: 65536.0 | number of skipped iterations:   2 | number of nan iterations:   0 |
time (ms) | forward: 609.77 | backward: 2204.34 | backward-backward: 2204.27 | backward-allreduce: 0.00 | optimizer: 79.26 | batch generator: 2.34
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step61500
 samples/sec: 353.069 | iteration    65000/  143000 | elapsed time per iteration (ms): 2900.3 | learning rate: 3.735E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.149497E+00 | loss scale: 32768.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 609.94 | backward: 2205.06 | backward-backward: 2204.99 | backward-allreduce: 0.00 | optimizer: 79.48 | batch generator: 2.33
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step62000
----------------------------------------------------------------------------------------------------------
 validation results at iteration 65000 | lm_loss value: 2.102544E+00 | lm_loss_ppl value: 8.186968E+00 | 
----------------------------------------------------------------------------------------------------------
 samples/sec: 338.324 | iteration    65500/  143000 | elapsed time per iteration (ms): 3026.7 | learning rate: 3.705E-04 | approx flops per GPU: 73.2TFLOPS | lm_loss: 2.147435E+00 | loss scale: 16384.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 609.95 | backward: 2204.30 | backward-backward: 2204.23 | backward-allreduce: 0.00 | optimizer: 79.50 | batch generator: 2.73
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step62500
 samples/sec: 353.106 | iteration    66000/  143000 | elapsed time per iteration (ms): 2900.0 | learning rate: 3.675E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.146142E+00 | loss scale: 16384.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 610.14 | backward: 2204.75 | backward-backward: 2204.68 | backward-allreduce: 0.00 | optimizer: 79.59 | batch generator: 2.35
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step63000
 samples/sec: 353.163 | iteration    66500/  143000 | elapsed time per iteration (ms): 2899.5 | learning rate: 3.646E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.153322E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 609.93 | backward: 2204.37 | backward-backward: 2204.30 | backward-allreduce: 0.00 | optimizer: 79.57 | batch generator: 2.35
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step63500
 samples/sec: 353.119 | iteration    67000/  143000 | elapsed time per iteration (ms): 2899.9 | learning rate: 3.616E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.146076E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 610.00 | backward: 2204.76 | backward-backward: 2204.69 | backward-allreduce: 0.00 | optimizer: 79.56 | batch generator: 2.32
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step64000
 samples/sec: 353.197 | iteration    67500/  143000 | elapsed time per iteration (ms): 2899.2 | learning rate: 3.586E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.145408E+00 | loss scale: 32768.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 609.70 | backward: 2204.53 | backward-backward: 2204.46 | backward-allreduce: 0.00 | optimizer: 79.41 | batch generator: 2.34
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step64500
 samples/sec: 353.109 | iteration    68000/  143000 | elapsed time per iteration (ms): 2900.0 | learning rate: 3.557E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.141154E+00 | loss scale: 16384.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 610.22 | backward: 2204.85 | backward-backward: 2204.78 | backward-allreduce: 0.00 | optimizer: 79.44 | batch generator: 2.34
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step65000
 samples/sec: 353.140 | iteration    68500/  143000 | elapsed time per iteration (ms): 2899.7 | learning rate: 3.527E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.142211E+00 | loss scale: 16384.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 609.89 | backward: 2204.71 | backward-backward: 2204.64 | backward-allreduce: 0.00 | optimizer: 79.57 | batch generator: 2.33
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step65500
 samples/sec: 353.111 | iteration    69000/  143000 | elapsed time per iteration (ms): 2899.9 | learning rate: 3.497E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.140988E+00 | loss scale: 16384.0 | number of skipped iterations:   2 | number of nan iterations:   0 |
time (ms) | forward: 610.09 | backward: 2205.02 | backward-backward: 2204.95 | backward-allreduce: 0.00 | optimizer: 79.26 | batch generator: 2.35
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step66000
 samples/sec: 353.136 | iteration    69500/  143000 | elapsed time per iteration (ms): 2899.7 | learning rate: 3.467E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.141401E+00 | loss scale: 8192.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 609.92 | backward: 2204.74 | backward-backward: 2204.67 | backward-allreduce: 0.00 | optimizer: 79.40 | batch generator: 2.35
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step66500
 samples/sec: 353.052 | iteration    70000/  143000 | elapsed time per iteration (ms): 2900.4 | learning rate: 3.437E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.142745E+00 | loss scale: 8192.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 610.18 | backward: 2205.15 | backward-backward: 2205.08 | backward-allreduce: 0.00 | optimizer: 79.55 | batch generator: 2.36
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step67000
----------------------------------------------------------------------------------------------------------
 validation results at iteration 70000 | lm_loss value: 2.095315E+00 | lm_loss_ppl value: 8.128005E+00 | 
----------------------------------------------------------------------------------------------------------
 samples/sec: 338.207 | iteration    70500/  143000 | elapsed time per iteration (ms): 3027.7 | learning rate: 3.407E-04 | approx flops per GPU: 73.2TFLOPS | lm_loss: 2.141310E+00 | loss scale: 16384.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 610.08 | backward: 2205.06 | backward-backward: 2204.99 | backward-allreduce: 0.00 | optimizer: 79.61 | batch generator: 2.76
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step67500
 samples/sec: 352.648 | iteration    71000/  143000 | elapsed time per iteration (ms): 2903.7 | learning rate: 3.377E-04 | approx flops per GPU: 76.3TFLOPS | lm_loss: 2.139759E+00 | loss scale: 16384.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 610.59 | backward: 2204.78 | backward-backward: 2204.71 | backward-allreduce: 0.00 | optimizer: 79.55 | batch generator: 2.40
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step68000
 samples/sec: 344.420 | iteration    71500/  143000 | elapsed time per iteration (ms): 2973.1 | learning rate: 3.348E-04 | approx flops per GPU: 74.5TFLOPS | lm_loss: 2.137903E+00 | loss scale: 16384.0 | number of skipped iterations:   2 | number of nan iterations:   0 |
time (ms) | forward: 610.05 | backward: 2204.68 | backward-backward: 2204.61 | backward-allreduce: 0.00 | optimizer: 79.25 | batch generator: 2.36
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step68500
 samples/sec: 353.074 | iteration    72000/  143000 | elapsed time per iteration (ms): 2900.2 | learning rate: 3.318E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.141233E+00 | loss scale: 8192.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 610.01 | backward: 2205.31 | backward-backward: 2205.23 | backward-allreduce: 0.00 | optimizer: 79.44 | batch generator: 2.38
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step69000
 samples/sec: 353.094 | iteration    72500/  143000 | elapsed time per iteration (ms): 2900.1 | learning rate: 3.288E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.136018E+00 | loss scale: 8192.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 609.69 | backward: 2205.20 | backward-backward: 2205.13 | backward-allreduce: 0.00 | optimizer: 79.61 | batch generator: 2.37
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step69500
 samples/sec: 353.016 | iteration    73000/  143000 | elapsed time per iteration (ms): 2900.7 | learning rate: 3.258E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.136095E+00 | loss scale: 16384.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 610.22 | backward: 2205.11 | backward-backward: 2205.04 | backward-allreduce: 0.00 | optimizer: 79.59 | batch generator: 2.35
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step70000
 samples/sec: 353.134 | iteration    73500/  143000 | elapsed time per iteration (ms): 2899.8 | learning rate: 3.228E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.136326E+00 | loss scale: 16384.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 610.06 | backward: 2204.52 | backward-backward: 2204.45 | backward-allreduce: 0.00 | optimizer: 79.59 | batch generator: 2.36
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step70500
 samples/sec: 353.105 | iteration    74000/  143000 | elapsed time per iteration (ms): 2900.0 | learning rate: 3.198E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.134349E+00 | loss scale: 32768.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 610.22 | backward: 2204.81 | backward-backward: 2204.74 | backward-allreduce: 0.00 | optimizer: 79.43 | batch generator: 2.36
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step71000
 samples/sec: 353.159 | iteration    74500/  143000 | elapsed time per iteration (ms): 2899.5 | learning rate: 3.168E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.133106E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 610.04 | backward: 2204.34 | backward-backward: 2204.27 | backward-allreduce: 0.00 | optimizer: 79.58 | batch generator: 2.34
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step71500
 samples/sec: 353.099 | iteration    75000/  143000 | elapsed time per iteration (ms): 2900.0 | learning rate: 3.138E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.132001E+00 | loss scale: 65536.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 610.22 | backward: 2204.73 | backward-backward: 2204.66 | backward-allreduce: 0.00 | optimizer: 79.56 | batch generator: 2.33
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step72000
----------------------------------------------------------------------------------------------------------
 validation results at iteration 75000 | lm_loss value: 2.086347E+00 | lm_loss_ppl value: 8.055432E+00 | 
----------------------------------------------------------------------------------------------------------
 samples/sec: 338.216 | iteration    75500/  143000 | elapsed time per iteration (ms): 3027.7 | learning rate: 3.108E-04 | approx flops per GPU: 73.2TFLOPS | lm_loss: 2.129546E+00 | loss scale: 65536.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 610.07 | backward: 2204.26 | backward-backward: 2204.19 | backward-allreduce: 0.00 | optimizer: 79.44 | batch generator: 2.74
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step72500
 samples/sec: 353.084 | iteration    76000/  143000 | elapsed time per iteration (ms): 2900.2 | learning rate: 3.078E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.130378E+00 | loss scale: 32768.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 610.35 | backward: 2204.71 | backward-backward: 2204.64 | backward-allreduce: 0.00 | optimizer: 79.42 | batch generator: 2.39
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step73000
 samples/sec: 353.130 | iteration    76500/  143000 | elapsed time per iteration (ms): 2899.8 | learning rate: 3.049E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.129836E+00 | loss scale: 16384.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 610.09 | backward: 2204.74 | backward-backward: 2204.66 | backward-allreduce: 0.00 | optimizer: 79.44 | batch generator: 2.39
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step73500
 samples/sec: 353.016 | iteration    77000/  143000 | elapsed time per iteration (ms): 2900.7 | learning rate: 3.019E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.129329E+00 | loss scale: 16384.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 610.32 | backward: 2205.14 | backward-backward: 2205.07 | backward-allreduce: 0.00 | optimizer: 79.64 | batch generator: 2.38
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step74000
 samples/sec: 353.189 | iteration    77500/  143000 | elapsed time per iteration (ms): 2899.3 | learning rate: 2.989E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.127165E+00 | loss scale: 32768.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 610.12 | backward: 2204.10 | backward-backward: 2204.03 | backward-allreduce: 0.00 | optimizer: 79.45 | batch generator: 2.39
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step74500
 samples/sec: 352.115 | iteration    78000/  143000 | elapsed time per iteration (ms): 2908.1 | learning rate: 2.959E-04 | approx flops per GPU: 76.2TFLOPS | lm_loss: 2.125618E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 612.20 | backward: 2210.20 | backward-backward: 2210.13 | backward-allreduce: 0.00 | optimizer: 79.98 | batch generator: 2.80
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step75000
 samples/sec: 352.992 | iteration    78500/  143000 | elapsed time per iteration (ms): 2900.9 | learning rate: 2.930E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.124752E+00 | loss scale: 65536.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 610.11 | backward: 2203.71 | backward-backward: 2203.64 | backward-allreduce: 0.00 | optimizer: 79.58 | batch generator: 2.39
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step75500
 samples/sec: 353.035 | iteration    79000/  143000 | elapsed time per iteration (ms): 2900.6 | learning rate: 2.900E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.124064E+00 | loss scale: 65536.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 610.27 | backward: 2205.40 | backward-backward: 2205.33 | backward-allreduce: 0.00 | optimizer: 79.43 | batch generator: 2.37
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step76000
 samples/sec: 353.161 | iteration    79500/  143000 | elapsed time per iteration (ms): 2899.5 | learning rate: 2.870E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.124916E+00 | loss scale: 32768.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 610.13 | backward: 2204.26 | backward-backward: 2204.19 | backward-allreduce: 0.00 | optimizer: 79.43 | batch generator: 2.40
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step76500
 samples/sec: 353.076 | iteration    80000/  143000 | elapsed time per iteration (ms): 2900.2 | learning rate: 2.841E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.123121E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 610.38 | backward: 2204.68 | backward-backward: 2204.61 | backward-allreduce: 0.00 | optimizer: 79.59 | batch generator: 2.39
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step77000
----------------------------------------------------------------------------------------------------------
 validation results at iteration 80000 | lm_loss value: 2.076035E+00 | lm_loss_ppl value: 7.972794E+00 | 
----------------------------------------------------------------------------------------------------------
 samples/sec: 338.235 | iteration    80500/  143000 | elapsed time per iteration (ms): 3027.5 | learning rate: 2.812E-04 | approx flops per GPU: 73.2TFLOPS | lm_loss: 2.123672E+00 | loss scale: 16384.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 610.02 | backward: 2204.81 | backward-backward: 2204.74 | backward-allreduce: 0.00 | optimizer: 79.61 | batch generator: 2.72
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step77500
 samples/sec: 353.067 | iteration    81000/  143000 | elapsed time per iteration (ms): 2900.3 | learning rate: 2.782E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.121719E+00 | loss scale: 16384.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 609.56 | backward: 2205.67 | backward-backward: 2205.60 | backward-allreduce: 0.00 | optimizer: 79.61 | batch generator: 2.35
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step78000
 samples/sec: 353.116 | iteration    81500/  143000 | elapsed time per iteration (ms): 2899.9 | learning rate: 2.753E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.122009E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 609.43 | backward: 2205.23 | backward-backward: 2205.16 | backward-allreduce: 0.00 | optimizer: 79.61 | batch generator: 2.31
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step78500
 samples/sec: 353.068 | iteration    82000/  143000 | elapsed time per iteration (ms): 2900.3 | learning rate: 2.724E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.120729E+00 | loss scale: 16384.0 | number of skipped iterations:   2 | number of nan iterations:   0 |
time (ms) | forward: 609.54 | backward: 2206.03 | backward-backward: 2205.95 | backward-allreduce: 0.00 | optimizer: 79.31 | batch generator: 2.31
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step79000
 samples/sec: 353.094 | iteration    82500/  143000 | elapsed time per iteration (ms): 2900.1 | learning rate: 2.694E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.118500E+00 | loss scale: 16384.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 609.46 | backward: 2205.51 | backward-backward: 2205.44 | backward-allreduce: 0.00 | optimizer: 79.62 | batch generator: 2.30
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step79500
 samples/sec: 353.043 | iteration    83000/  143000 | elapsed time per iteration (ms): 2900.5 | learning rate: 2.665E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.117742E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 609.96 | backward: 2205.44 | backward-backward: 2205.37 | backward-allreduce: 0.00 | optimizer: 79.60 | batch generator: 2.32
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step80000
 samples/sec: 353.113 | iteration    83500/  143000 | elapsed time per iteration (ms): 2899.9 | learning rate: 2.636E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.119061E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 610.22 | backward: 2204.66 | backward-backward: 2204.58 | backward-allreduce: 0.00 | optimizer: 79.60 | batch generator: 2.35
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step80500
 samples/sec: 353.104 | iteration    84000/  143000 | elapsed time per iteration (ms): 2900.0 | learning rate: 2.607E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.117008E+00 | loss scale: 32768.0 | number of skipped iterations:   2 | number of nan iterations:   0 |
time (ms) | forward: 610.06 | backward: 2205.13 | backward-backward: 2205.06 | backward-allreduce: 0.00 | optimizer: 79.29 | batch generator: 2.34
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step81000
 samples/sec: 353.060 | iteration    84500/  143000 | elapsed time per iteration (ms): 2900.4 | learning rate: 2.578E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.114706E+00 | loss scale: 16384.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 610.05 | backward: 2205.29 | backward-backward: 2205.22 | backward-allreduce: 0.00 | optimizer: 79.46 | batch generator: 2.34
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step81500
 samples/sec: 353.105 | iteration    85000/  143000 | elapsed time per iteration (ms): 2900.0 | learning rate: 2.550E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.114689E+00 | loss scale: 16384.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 610.21 | backward: 2204.64 | backward-backward: 2204.57 | backward-allreduce: 0.00 | optimizer: 79.60 | batch generator: 2.34
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step82000
----------------------------------------------------------------------------------------------------------
 validation results at iteration 85000 | lm_loss value: 2.068848E+00 | lm_loss_ppl value: 7.915702E+00 | 
----------------------------------------------------------------------------------------------------------
 samples/sec: 338.150 | iteration    85500/  143000 | elapsed time per iteration (ms): 3028.2 | learning rate: 2.521E-04 | approx flops per GPU: 73.2TFLOPS | lm_loss: 2.112462E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 610.06 | backward: 2205.70 | backward-backward: 2205.62 | backward-allreduce: 0.00 | optimizer: 79.64 | batch generator: 2.76
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step82500
 samples/sec: 353.077 | iteration    86000/  143000 | elapsed time per iteration (ms): 2900.2 | learning rate: 2.492E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.112222E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 610.21 | backward: 2204.92 | backward-backward: 2204.84 | backward-allreduce: 0.00 | optimizer: 79.61 | batch generator: 2.35
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step83000
 samples/sec: 353.025 | iteration    86500/  143000 | elapsed time per iteration (ms): 2900.6 | learning rate: 2.464E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.113981E+00 | loss scale: 32768.0 | number of skipped iterations:   2 | number of nan iterations:   0 |
time (ms) | forward: 609.99 | backward: 2205.67 | backward-backward: 2205.60 | backward-allreduce: 0.00 | optimizer: 79.29 | batch generator: 2.33
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step83500
 samples/sec: 353.073 | iteration    87000/  143000 | elapsed time per iteration (ms): 2900.2 | learning rate: 2.435E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.111177E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 610.18 | backward: 2204.98 | backward-backward: 2204.91 | backward-allreduce: 0.00 | optimizer: 79.62 | batch generator: 2.33
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step84000
 samples/sec: 352.984 | iteration    87500/  143000 | elapsed time per iteration (ms): 2901.0 | learning rate: 2.407E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.109004E+00 | loss scale: 16384.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 609.99 | backward: 2205.97 | backward-backward: 2205.90 | backward-allreduce: 0.00 | optimizer: 79.43 | batch generator: 2.34
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step84500
 samples/sec: 353.019 | iteration    88000/  143000 | elapsed time per iteration (ms): 2900.7 | learning rate: 2.379E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.110119E+00 | loss scale: 16384.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 610.07 | backward: 2205.51 | backward-backward: 2205.43 | backward-allreduce: 0.00 | optimizer: 79.60 | batch generator: 2.33
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step85000
 samples/sec: 352.963 | iteration    88500/  143000 | elapsed time per iteration (ms): 2901.2 | learning rate: 2.351E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.108484E+00 | loss scale: 32768.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 609.67 | backward: 2206.40 | backward-backward: 2206.32 | backward-allreduce: 0.00 | optimizer: 79.45 | batch generator: 2.32
 samples/sec: 352.470 | iteration    89000/  143000 | elapsed time per iteration (ms): 2905.2 | learning rate: 2.323E-04 | approx flops per GPU: 76.3TFLOPS | lm_loss: 2.108585E+00 | loss scale: 16384.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
after 89000 iterations memory (MB) | allocated: 3746.75439453125 | max allocated: 11575.26025390625 | reserved: 13710.0 | max reserved: 13710.0
time (ms) | forward: 616.27 | backward: 2208.27 | backward-backward: 2208.19 | backward-allreduce: 0.00 | optimizer: 79.34 | batch generator: 2.30
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step86000
 samples/sec: 352.898 | iteration    89500/  143000 | elapsed time per iteration (ms): 2901.7 | learning rate: 2.295E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.106150E+00 | loss scale: 16384.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 609.45 | backward: 2207.18 | backward-backward: 2207.11 | backward-allreduce: 0.00 | optimizer: 79.51 | batch generator: 2.25
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step86500
 samples/sec: 352.933 | iteration    90000/  143000 | elapsed time per iteration (ms): 2901.4 | learning rate: 2.267E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.105758E+00 | loss scale: 16384.0 | number of skipped iterations:   2 | number of nan iterations:   0 |
time (ms) | forward: 609.27 | backward: 2207.37 | backward-backward: 2207.29 | backward-allreduce: 0.00 | optimizer: 79.23 | batch generator: 2.28
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step87000
----------------------------------------------------------------------------------------------------------
 validation results at iteration 90000 | lm_loss value: 2.062608E+00 | lm_loss_ppl value: 7.866459E+00 | 
----------------------------------------------------------------------------------------------------------
 samples/sec: 338.063 | iteration    90500/  143000 | elapsed time per iteration (ms): 3029.0 | learning rate: 2.240E-04 | approx flops per GPU: 73.1TFLOPS | lm_loss: 2.104899E+00 | loss scale: 16384.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 609.41 | backward: 2207.09 | backward-backward: 2207.02 | backward-allreduce: 0.00 | optimizer: 79.54 | batch generator: 2.58
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step87500
 samples/sec: 353.067 | iteration    91000/  143000 | elapsed time per iteration (ms): 2900.3 | learning rate: 2.212E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.103225E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 609.30 | backward: 2205.97 | backward-backward: 2205.90 | backward-allreduce: 0.00 | optimizer: 79.53 | batch generator: 2.24
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step88000
 samples/sec: 352.923 | iteration    91500/  143000 | elapsed time per iteration (ms): 2901.5 | learning rate: 2.185E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.103656E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 609.50 | backward: 2206.87 | backward-backward: 2206.80 | backward-allreduce: 0.00 | optimizer: 79.57 | batch generator: 2.29
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step88500
 samples/sec: 352.970 | iteration    92000/  143000 | elapsed time per iteration (ms): 2901.1 | learning rate: 2.158E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.102270E+00 | loss scale: 8192.0 | number of skipped iterations:   3 | number of nan iterations:   0 |
time (ms) | forward: 609.18 | backward: 2207.41 | backward-backward: 2207.33 | backward-allreduce: 0.00 | optimizer: 79.08 | batch generator: 2.20
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step89000
 samples/sec: 352.823 | iteration    92500/  143000 | elapsed time per iteration (ms): 2902.3 | learning rate: 2.131E-04 | approx flops per GPU: 76.3TFLOPS | lm_loss: 2.101580E+00 | loss scale: 8192.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 609.45 | backward: 2207.84 | backward-backward: 2207.77 | backward-allreduce: 0.00 | optimizer: 79.56 | batch generator: 2.20
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step89500
 samples/sec: 352.670 | iteration    93000/  143000 | elapsed time per iteration (ms): 2903.6 | learning rate: 2.104E-04 | approx flops per GPU: 76.3TFLOPS | lm_loss: 2.102271E+00 | loss scale: 16384.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 609.20 | backward: 2208.67 | backward-backward: 2208.60 | backward-allreduce: 0.00 | optimizer: 80.12 | batch generator: 2.23
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step90000
 samples/sec: 352.881 | iteration    93500/  143000 | elapsed time per iteration (ms): 2901.8 | learning rate: 2.077E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.098684E+00 | loss scale: 16384.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 609.18 | backward: 2207.48 | backward-backward: 2207.41 | backward-allreduce: 0.00 | optimizer: 79.55 | batch generator: 2.27
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step90500
 samples/sec: 352.937 | iteration    94000/  143000 | elapsed time per iteration (ms): 2901.4 | learning rate: 2.050E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.098608E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 609.02 | backward: 2207.43 | backward-backward: 2207.36 | backward-allreduce: 0.00 | optimizer: 79.56 | batch generator: 2.24
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step91000
 samples/sec: 352.822 | iteration    94500/  143000 | elapsed time per iteration (ms): 2902.3 | learning rate: 2.024E-04 | approx flops per GPU: 76.3TFLOPS | lm_loss: 2.099217E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 609.26 | backward: 2207.85 | backward-backward: 2207.78 | backward-allreduce: 0.00 | optimizer: 79.77 | batch generator: 2.29
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step91500
 samples/sec: 352.954 | iteration    95000/  143000 | elapsed time per iteration (ms): 2901.2 | learning rate: 1.998E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.096398E+00 | loss scale: 65536.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 609.07 | backward: 2207.19 | backward-backward: 2207.12 | backward-allreduce: 0.00 | optimizer: 79.53 | batch generator: 2.29
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step92000
----------------------------------------------------------------------------------------------------------
 validation results at iteration 95000 | lm_loss value: 2.051986E+00 | lm_loss_ppl value: 7.783340E+00 | 
----------------------------------------------------------------------------------------------------------
 samples/sec: 338.135 | iteration    95500/  143000 | elapsed time per iteration (ms): 3028.4 | learning rate: 1.972E-04 | approx flops per GPU: 73.2TFLOPS | lm_loss: 2.096801E+00 | loss scale: 32768.0 | number of skipped iterations:   2 | number of nan iterations:   0 |
time (ms) | forward: 609.28 | backward: 2206.82 | backward-backward: 2206.75 | backward-allreduce: 0.00 | optimizer: 79.24 | batch generator: 2.65
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step92500
 samples/sec: 353.224 | iteration    96000/  143000 | elapsed time per iteration (ms): 2899.0 | learning rate: 1.946E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.097106E+00 | loss scale: 16384.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 609.15 | backward: 2205.10 | backward-backward: 2205.03 | backward-allreduce: 0.00 | optimizer: 79.38 | batch generator: 2.26
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step93000
 samples/sec: 353.030 | iteration    96500/  143000 | elapsed time per iteration (ms): 2900.6 | learning rate: 1.920E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.094771E+00 | loss scale: 16384.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 609.29 | backward: 2206.26 | backward-backward: 2206.19 | backward-allreduce: 0.00 | optimizer: 79.51 | batch generator: 2.23
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step93500
 samples/sec: 353.177 | iteration    97000/  143000 | elapsed time per iteration (ms): 2899.4 | learning rate: 1.894E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.093601E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 609.13 | backward: 2205.23 | backward-backward: 2205.16 | backward-allreduce: 0.00 | optimizer: 79.53 | batch generator: 2.24
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step94000
 samples/sec: 353.113 | iteration    97500/  143000 | elapsed time per iteration (ms): 2899.9 | learning rate: 1.869E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.093295E+00 | loss scale: 32768.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 609.35 | backward: 2205.70 | backward-backward: 2205.63 | backward-allreduce: 0.00 | optimizer: 79.38 | batch generator: 2.24
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step94500
 samples/sec: 352.656 | iteration    98000/  143000 | elapsed time per iteration (ms): 2903.7 | learning rate: 1.843E-04 | approx flops per GPU: 76.3TFLOPS | lm_loss: 2.091473E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 609.11 | backward: 2205.78 | backward-backward: 2205.71 | backward-allreduce: 0.00 | optimizer: 79.54 | batch generator: 2.24
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step95000
 samples/sec: 353.158 | iteration    98500/  143000 | elapsed time per iteration (ms): 2899.6 | learning rate: 1.818E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.090431E+00 | loss scale: 32768.0 | number of skipped iterations:   2 | number of nan iterations:   0 |
time (ms) | forward: 609.32 | backward: 2205.61 | backward-backward: 2205.54 | backward-allreduce: 0.00 | optimizer: 79.22 | batch generator: 2.23
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step95500
 samples/sec: 353.155 | iteration    99000/  143000 | elapsed time per iteration (ms): 2899.6 | learning rate: 1.793E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.090558E+00 | loss scale: 16384.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 609.13 | backward: 2205.53 | backward-backward: 2205.46 | backward-allreduce: 0.00 | optimizer: 79.37 | batch generator: 2.22
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step96000
 samples/sec: 353.124 | iteration    99500/  143000 | elapsed time per iteration (ms): 2899.8 | learning rate: 1.769E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.091439E+00 | loss scale: 16384.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 609.39 | backward: 2205.55 | backward-backward: 2205.48 | backward-allreduce: 0.00 | optimizer: 79.54 | batch generator: 2.24
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step96500
 samples/sec: 352.840 | iteration   100000/  143000 | elapsed time per iteration (ms): 2902.2 | learning rate: 1.744E-04 | approx flops per GPU: 76.3TFLOPS | lm_loss: 2.089060E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 609.18 | backward: 2207.83 | backward-backward: 2207.76 | backward-allreduce: 0.00 | optimizer: 79.58 | batch generator: 2.25
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step97000
-----------------------------------------------------------------------------------------------------------
 validation results at iteration 100000 | lm_loss value: 2.045376E+00 | lm_loss_ppl value: 7.732068E+00 | 
-----------------------------------------------------------------------------------------------------------
 samples/sec: 338.133 | iteration   100500/  143000 | elapsed time per iteration (ms): 3028.4 | learning rate: 1.720E-04 | approx flops per GPU: 73.2TFLOPS | lm_loss: 2.087061E+00 | loss scale: 32768.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 609.37 | backward: 2206.72 | backward-backward: 2206.64 | backward-allreduce: 0.00 | optimizer: 79.45 | batch generator: 2.63
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step97500
 samples/sec: 353.093 | iteration   101000/  143000 | elapsed time per iteration (ms): 2900.1 | learning rate: 1.696E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.086114E+00 | loss scale: 16384.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 609.11 | backward: 2206.10 | backward-backward: 2206.02 | backward-allreduce: 0.00 | optimizer: 79.40 | batch generator: 2.23
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step98000
 samples/sec: 353.044 | iteration   101500/  143000 | elapsed time per iteration (ms): 2900.5 | learning rate: 1.672E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.086666E+00 | loss scale: 16384.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 609.33 | backward: 2205.98 | backward-backward: 2205.91 | backward-allreduce: 0.00 | optimizer: 79.52 | batch generator: 2.25
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step98500
 samples/sec: 353.053 | iteration   102000/  143000 | elapsed time per iteration (ms): 2900.4 | learning rate: 1.648E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.085056E+00 | loss scale: 8192.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 609.14 | backward: 2206.50 | backward-backward: 2206.43 | backward-allreduce: 0.00 | optimizer: 79.37 | batch generator: 2.25
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step99000
 samples/sec: 353.010 | iteration   102500/  143000 | elapsed time per iteration (ms): 2900.8 | learning rate: 1.624E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.084948E+00 | loss scale: 8192.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 609.32 | backward: 2206.03 | backward-backward: 2205.96 | backward-allreduce: 0.00 | optimizer: 79.52 | batch generator: 2.29
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step99500
 samples/sec: 353.093 | iteration   103000/  143000 | elapsed time per iteration (ms): 2900.1 | learning rate: 1.601E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.083879E+00 | loss scale: 16384.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 609.21 | backward: 2205.75 | backward-backward: 2205.68 | backward-allreduce: 0.00 | optimizer: 79.69 | batch generator: 2.33
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step100000
 samples/sec: 352.955 | iteration   103500/  143000 | elapsed time per iteration (ms): 2901.2 | learning rate: 1.578E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.083274E+00 | loss scale: 16384.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 609.38 | backward: 2206.88 | backward-backward: 2206.81 | backward-allreduce: 0.00 | optimizer: 79.50 | batch generator: 2.28
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step100500
 samples/sec: 353.067 | iteration   104000/  143000 | elapsed time per iteration (ms): 2900.3 | learning rate: 1.555E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.082632E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 609.18 | backward: 2206.20 | backward-backward: 2206.13 | backward-allreduce: 0.00 | optimizer: 79.50 | batch generator: 2.26
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step101000
 samples/sec: 353.020 | iteration   104500/  143000 | elapsed time per iteration (ms): 2900.7 | learning rate: 1.532E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.081282E+00 | loss scale: 32768.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 609.35 | backward: 2206.60 | backward-backward: 2206.53 | backward-allreduce: 0.00 | optimizer: 79.37 | batch generator: 2.25
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step101500
 samples/sec: 353.071 | iteration   105000/  143000 | elapsed time per iteration (ms): 2900.3 | learning rate: 1.509E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.082003E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 609.17 | backward: 2206.15 | backward-backward: 2206.08 | backward-allreduce: 0.00 | optimizer: 79.53 | batch generator: 2.25
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step102000
-----------------------------------------------------------------------------------------------------------
 validation results at iteration 105000 | lm_loss value: 2.036453E+00 | lm_loss_ppl value: 7.663379E+00 | 
-----------------------------------------------------------------------------------------------------------
 samples/sec: 338.098 | iteration   105500/  143000 | elapsed time per iteration (ms): 3028.7 | learning rate: 1.487E-04 | approx flops per GPU: 73.2TFLOPS | lm_loss: 2.080755E+00 | loss scale: 16384.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 609.44 | backward: 2206.85 | backward-backward: 2206.77 | backward-allreduce: 0.00 | optimizer: 79.57 | batch generator: 2.64
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step102500
 samples/sec: 352.999 | iteration   106000/  143000 | elapsed time per iteration (ms): 2900.9 | learning rate: 1.465E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.078865E+00 | loss scale: 16384.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 609.23 | backward: 2206.51 | backward-backward: 2206.44 | backward-allreduce: 0.00 | optimizer: 79.53 | batch generator: 2.26
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step103000
 samples/sec: 353.002 | iteration   106500/  143000 | elapsed time per iteration (ms): 2900.8 | learning rate: 1.443E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.078916E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 609.50 | backward: 2206.11 | backward-backward: 2206.04 | backward-allreduce: 0.00 | optimizer: 79.71 | batch generator: 2.32
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step103500
 samples/sec: 353.200 | iteration   107000/  143000 | elapsed time per iteration (ms): 2899.2 | learning rate: 1.422E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.078951E+00 | loss scale: 32768.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 609.27 | backward: 2205.09 | backward-backward: 2205.02 | backward-allreduce: 0.00 | optimizer: 79.39 | batch generator: 2.31
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step104000
 samples/sec: 353.138 | iteration   107500/  143000 | elapsed time per iteration (ms): 2899.7 | learning rate: 1.400E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.077511E+00 | loss scale: 16384.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 609.43 | backward: 2205.37 | backward-backward: 2205.29 | backward-allreduce: 0.00 | optimizer: 79.37 | batch generator: 2.30
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step104500
 samples/sec: 353.109 | iteration   108000/  143000 | elapsed time per iteration (ms): 2900.0 | learning rate: 1.379E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.075537E+00 | loss scale: 16384.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 609.30 | backward: 2205.52 | backward-backward: 2205.45 | backward-allreduce: 0.00 | optimizer: 79.52 | batch generator: 2.31
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step105000
 samples/sec: 353.074 | iteration   108500/  143000 | elapsed time per iteration (ms): 2900.2 | learning rate: 1.358E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.075166E+00 | loss scale: 32768.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 609.45 | backward: 2205.40 | backward-backward: 2205.33 | backward-allreduce: 0.00 | optimizer: 79.39 | batch generator: 2.28
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step105500
 samples/sec: 353.125 | iteration   109000/  143000 | elapsed time per iteration (ms): 2899.8 | learning rate: 1.338E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.074447E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 609.20 | backward: 2205.49 | backward-backward: 2205.42 | backward-allreduce: 0.00 | optimizer: 79.67 | batch generator: 2.23
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step106000
 samples/sec: 353.017 | iteration   109500/  143000 | elapsed time per iteration (ms): 2900.7 | learning rate: 1.317E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.073662E+00 | loss scale: 16384.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 609.41 | backward: 2206.34 | backward-backward: 2206.26 | backward-allreduce: 0.00 | optimizer: 79.38 | batch generator: 2.24
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step106500
 samples/sec: 352.979 | iteration   110000/  143000 | elapsed time per iteration (ms): 2901.0 | learning rate: 1.297E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.075984E+00 | loss scale: 16384.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 609.31 | backward: 2206.66 | backward-backward: 2206.58 | backward-allreduce: 0.00 | optimizer: 79.57 | batch generator: 2.29
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step107000
-----------------------------------------------------------------------------------------------------------
 validation results at iteration 110000 | lm_loss value: 2.029312E+00 | lm_loss_ppl value: 7.608849E+00 | 
-----------------------------------------------------------------------------------------------------------
 samples/sec: 338.003 | iteration   110500/  143000 | elapsed time per iteration (ms): 3029.6 | learning rate: 1.277E-04 | approx flops per GPU: 73.1TFLOPS | lm_loss: 2.073421E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 609.47 | backward: 2207.30 | backward-backward: 2207.23 | backward-allreduce: 0.00 | optimizer: 79.75 | batch generator: 2.65
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step107500
 samples/sec: 353.136 | iteration   111000/  143000 | elapsed time per iteration (ms): 2899.7 | learning rate: 1.257E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.072429E+00 | loss scale: 32768.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 609.23 | backward: 2205.60 | backward-backward: 2205.53 | backward-allreduce: 0.00 | optimizer: 79.40 | batch generator: 2.21
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step108000
 samples/sec: 353.108 | iteration   111500/  143000 | elapsed time per iteration (ms): 2900.0 | learning rate: 1.238E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.070852E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 609.38 | backward: 2205.52 | backward-backward: 2205.45 | backward-allreduce: 0.00 | optimizer: 79.55 | batch generator: 2.21
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step108500
 samples/sec: 353.153 | iteration   112000/  143000 | elapsed time per iteration (ms): 2899.6 | learning rate: 1.219E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.067711E+00 | loss scale: 16384.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 609.25 | backward: 2205.53 | backward-backward: 2205.46 | backward-allreduce: 0.00 | optimizer: 79.38 | batch generator: 2.26
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step109000
 samples/sec: 353.072 | iteration   112500/  143000 | elapsed time per iteration (ms): 2900.3 | learning rate: 1.200E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.069139E+00 | loss scale: 16384.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 609.40 | backward: 2205.82 | backward-backward: 2205.75 | backward-allreduce: 0.00 | optimizer: 79.54 | batch generator: 2.22
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step109500
 samples/sec: 353.082 | iteration   113000/  143000 | elapsed time per iteration (ms): 2900.2 | learning rate: 1.181E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.066094E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 609.35 | backward: 2205.73 | backward-backward: 2205.66 | backward-allreduce: 0.00 | optimizer: 79.53 | batch generator: 2.27
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step110000
 samples/sec: 353.081 | iteration   113500/  143000 | elapsed time per iteration (ms): 2900.2 | learning rate: 1.163E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.068288E+00 | loss scale: 16384.0 | number of skipped iterations:   2 | number of nan iterations:   0 |
time (ms) | forward: 609.54 | backward: 2205.97 | backward-backward: 2205.89 | backward-allreduce: 0.00 | optimizer: 79.22 | batch generator: 2.23
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step110500
 samples/sec: 353.020 | iteration   114000/  143000 | elapsed time per iteration (ms): 2900.7 | learning rate: 1.144E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.066782E+00 | loss scale: 16384.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 609.39 | backward: 2205.94 | backward-backward: 2205.87 | backward-allreduce: 0.00 | optimizer: 79.76 | batch generator: 2.27
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step111000
 samples/sec: 353.041 | iteration   114500/  143000 | elapsed time per iteration (ms): 2900.5 | learning rate: 1.127E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.066362E+00 | loss scale: 32768.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 609.49 | backward: 2206.20 | backward-backward: 2206.13 | backward-allreduce: 0.00 | optimizer: 79.37 | batch generator: 2.22
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step111500
 samples/sec: 353.061 | iteration   115000/  143000 | elapsed time per iteration (ms): 2900.3 | learning rate: 1.109E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.065361E+00 | loss scale: 16384.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 609.38 | backward: 2205.99 | backward-backward: 2205.92 | backward-allreduce: 0.00 | optimizer: 79.37 | batch generator: 2.26
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step112000
-----------------------------------------------------------------------------------------------------------
 validation results at iteration 115000 | lm_loss value: 2.016774E+00 | lm_loss_ppl value: 7.514043E+00 | 
-----------------------------------------------------------------------------------------------------------
 samples/sec: 338.209 | iteration   115500/  143000 | elapsed time per iteration (ms): 3027.7 | learning rate: 1.092E-04 | approx flops per GPU: 73.2TFLOPS | lm_loss: 2.063500E+00 | loss scale: 16384.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 609.42 | backward: 2205.58 | backward-backward: 2205.51 | backward-allreduce: 0.00 | optimizer: 79.72 | batch generator: 2.64
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step112500
 samples/sec: 353.004 | iteration   116000/  143000 | elapsed time per iteration (ms): 2900.8 | learning rate: 1.075E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.064721E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 609.42 | backward: 2206.27 | backward-backward: 2206.20 | backward-allreduce: 0.00 | optimizer: 79.55 | batch generator: 2.27
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step113000
 samples/sec: 353.084 | iteration   116500/  143000 | elapsed time per iteration (ms): 2900.2 | learning rate: 1.058E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.064561E+00 | loss scale: 16384.0 | number of skipped iterations:   2 | number of nan iterations:   0 |
time (ms) | forward: 609.59 | backward: 2205.72 | backward-backward: 2205.65 | backward-allreduce: 0.00 | optimizer: 79.24 | batch generator: 2.27
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step113500
 samples/sec: 352.924 | iteration   117000/  143000 | elapsed time per iteration (ms): 2901.5 | learning rate: 1.041E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.061940E+00 | loss scale: 16384.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 609.42 | backward: 2206.61 | backward-backward: 2206.54 | backward-allreduce: 0.00 | optimizer: 79.56 | batch generator: 2.27
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step114000
 samples/sec: 353.029 | iteration   117500/  143000 | elapsed time per iteration (ms): 2900.6 | learning rate: 1.025E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.062451E+00 | loss scale: 8192.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 609.61 | backward: 2206.10 | backward-backward: 2206.03 | backward-allreduce: 0.00 | optimizer: 79.37 | batch generator: 2.27
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step114500
 samples/sec: 352.905 | iteration   118000/  143000 | elapsed time per iteration (ms): 2901.6 | learning rate: 1.009E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.061479E+00 | loss scale: 8192.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 609.44 | backward: 2207.04 | backward-backward: 2206.97 | backward-allreduce: 0.00 | optimizer: 79.55 | batch generator: 2.28
 samples/sec: 352.810 | iteration   118500/  143000 | elapsed time per iteration (ms): 2902.4 | learning rate: 9.933E-05 | approx flops per GPU: 76.3TFLOPS | lm_loss: 2.060447E+00 | loss scale: 16384.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
after 118500 iterations memory (MB) | allocated: 3623.939453125 | max allocated: 11451.5068359375 | reserved: 13710.0 | max reserved: 13710.0
time (ms) | forward: 611.70 | backward: 2209.83 | backward-backward: 2209.76 | backward-allreduce: 0.00 | optimizer: 79.55 | batch generator: 2.32
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step115500
 samples/sec: 353.042 | iteration   119000/  143000 | elapsed time per iteration (ms): 2900.5 | learning rate: 9.778E-05 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.058811E+00 | loss scale: 16384.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 610.03 | backward: 2205.13 | backward-backward: 2205.06 | backward-allreduce: 0.00 | optimizer: 79.54 | batch generator: 2.27
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step116000
 samples/sec: 353.040 | iteration   119500/  143000 | elapsed time per iteration (ms): 2900.5 | learning rate: 9.627E-05 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.059651E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 609.77 | backward: 2205.63 | backward-backward: 2205.56 | backward-allreduce: 0.00 | optimizer: 79.65 | batch generator: 2.28
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step116500
 samples/sec: 353.094 | iteration   120000/  143000 | elapsed time per iteration (ms): 2900.1 | learning rate: 9.478E-05 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.058310E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 609.97 | backward: 2205.14 | backward-backward: 2205.07 | backward-allreduce: 0.00 | optimizer: 79.53 | batch generator: 2.30
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step117000
-----------------------------------------------------------------------------------------------------------
 validation results at iteration 120000 | lm_loss value: 2.013359E+00 | lm_loss_ppl value: 7.488429E+00 | 
-----------------------------------------------------------------------------------------------------------
 samples/sec: 338.314 | iteration   120500/  143000 | elapsed time per iteration (ms): 3026.8 | learning rate: 9.334E-05 | approx flops per GPU: 73.2TFLOPS | lm_loss: 2.056560E+00 | loss scale: 16384.0 | number of skipped iterations:   3 | number of nan iterations:   0 |
time (ms) | forward: 610.09 | backward: 2204.59 | backward-backward: 2204.52 | backward-allreduce: 0.00 | optimizer: 79.06 | batch generator: 2.76
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step117500
 samples/sec: 353.022 | iteration   121000/  143000 | elapsed time per iteration (ms): 2900.7 | learning rate: 9.191E-05 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.057362E+00 | loss scale: 16384.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 610.17 | backward: 2205.14 | backward-backward: 2205.07 | backward-allreduce: 0.00 | optimizer: 79.54 | batch generator: 2.37
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step118000
 samples/sec: 353.183 | iteration   121500/  143000 | elapsed time per iteration (ms): 2899.4 | learning rate: 9.051E-05 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.056591E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 609.83 | backward: 2204.49 | backward-backward: 2204.42 | backward-allreduce: 0.00 | optimizer: 79.54 | batch generator: 2.30
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step118500
 samples/sec: 353.131 | iteration   122000/  143000 | elapsed time per iteration (ms): 2899.8 | learning rate: 8.914E-05 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.055705E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 610.11 | backward: 2204.43 | backward-backward: 2204.36 | backward-allreduce: 0.00 | optimizer: 79.68 | batch generator: 2.31
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step119000
 samples/sec: 353.233 | iteration   122500/  143000 | elapsed time per iteration (ms): 2898.9 | learning rate: 8.781E-05 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.056121E+00 | loss scale: 32768.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 609.95 | backward: 2204.14 | backward-backward: 2204.07 | backward-allreduce: 0.00 | optimizer: 79.40 | batch generator: 2.31
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step119500
 samples/sec: 353.173 | iteration   123000/  143000 | elapsed time per iteration (ms): 2899.4 | learning rate: 8.650E-05 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.056422E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 610.10 | backward: 2204.31 | backward-backward: 2204.24 | backward-allreduce: 0.00 | optimizer: 79.52 | batch generator: 2.34
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step120000
 samples/sec: 353.259 | iteration   123500/  143000 | elapsed time per iteration (ms): 2898.7 | learning rate: 8.522E-05 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.054773E+00 | loss scale: 16384.0 | number of skipped iterations:   3 | number of nan iterations:   0 |
time (ms) | forward: 609.95 | backward: 2204.21 | backward-backward: 2204.14 | backward-allreduce: 0.00 | optimizer: 79.19 | batch generator: 2.34
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step120500
 samples/sec: 353.137 | iteration   124000/  143000 | elapsed time per iteration (ms): 2899.7 | learning rate: 8.398E-05 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.053937E+00 | loss scale: 16384.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 610.21 | backward: 2204.34 | backward-backward: 2204.27 | backward-allreduce: 0.00 | optimizer: 79.51 | batch generator: 2.34
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step121000
 samples/sec: 353.167 | iteration   124500/  143000 | elapsed time per iteration (ms): 2899.5 | learning rate: 8.276E-05 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.052470E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 609.99 | backward: 2204.39 | backward-backward: 2204.32 | backward-allreduce: 0.00 | optimizer: 79.67 | batch generator: 2.30
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step121500
 samples/sec: 353.132 | iteration   125000/  143000 | elapsed time per iteration (ms): 2899.8 | learning rate: 8.157E-05 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.053600E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 609.88 | backward: 2204.79 | backward-backward: 2204.72 | backward-allreduce: 0.00 | optimizer: 79.54 | batch generator: 2.30
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step122000
-----------------------------------------------------------------------------------------------------------
 validation results at iteration 125000 | lm_loss value: 2.008578E+00 | lm_loss_ppl value: 7.452709E+00 | 
-----------------------------------------------------------------------------------------------------------
 samples/sec: 338.435 | iteration   125500/  143000 | elapsed time per iteration (ms): 3025.7 | learning rate: 8.041E-05 | approx flops per GPU: 73.2TFLOPS | lm_loss: 2.052524E+00 | loss scale: 65536.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 609.53 | backward: 2203.95 | backward-backward: 2203.88 | backward-allreduce: 0.00 | optimizer: 79.37 | batch generator: 2.65
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step122500
 samples/sec: 353.238 | iteration   126000/  143000 | elapsed time per iteration (ms): 2898.9 | learning rate: 7.929E-05 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.051470E+00 | loss scale: 16384.0 | number of skipped iterations:   2 | number of nan iterations:   0 |
time (ms) | forward: 609.68 | backward: 2204.45 | backward-backward: 2204.38 | backward-allreduce: 0.00 | optimizer: 79.32 | batch generator: 2.26
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step123000
 samples/sec: 353.280 | iteration   126500/  143000 | elapsed time per iteration (ms): 2898.5 | learning rate: 7.819E-05 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.051416E+00 | loss scale: 16384.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 609.48 | backward: 2203.99 | backward-backward: 2203.92 | backward-allreduce: 0.00 | optimizer: 79.50 | batch generator: 2.23
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step123500
 samples/sec: 353.207 | iteration   127000/  143000 | elapsed time per iteration (ms): 2899.2 | learning rate: 7.713E-05 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.051840E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 609.66 | backward: 2204.34 | backward-backward: 2204.27 | backward-allreduce: 0.00 | optimizer: 79.53 | batch generator: 2.26
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step124000
 samples/sec: 353.283 | iteration   127500/  143000 | elapsed time per iteration (ms): 2898.5 | learning rate: 7.609E-05 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.050111E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 609.53 | backward: 2203.98 | backward-backward: 2203.91 | backward-allreduce: 0.00 | optimizer: 79.50 | batch generator: 2.25
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step124500
 samples/sec: 353.206 | iteration   128000/  143000 | elapsed time per iteration (ms): 2899.2 | learning rate: 7.509E-05 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.051122E+00 | loss scale: 65536.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 609.78 | backward: 2204.40 | backward-backward: 2204.33 | backward-allreduce: 0.00 | optimizer: 79.52 | batch generator: 2.25
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step125000
 samples/sec: 353.334 | iteration   128500/  143000 | elapsed time per iteration (ms): 2898.1 | learning rate: 7.412E-05 | approx flops per GPU: 76.5TFLOPS | lm_loss: 2.049028E+00 | loss scale: 32768.0 | number of skipped iterations:   2 | number of nan iterations:   0 |
time (ms) | forward: 609.05 | backward: 2204.39 | backward-backward: 2204.32 | backward-allreduce: 0.00 | optimizer: 79.20 | batch generator: 2.22
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step125500
 samples/sec: 353.224 | iteration   129000/  143000 | elapsed time per iteration (ms): 2899.0 | learning rate: 7.318E-05 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.049796E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 609.32 | backward: 2204.66 | backward-backward: 2204.60 | backward-allreduce: 0.00 | optimizer: 79.51 | batch generator: 2.22
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step126000
 samples/sec: 353.378 | iteration   129500/  143000 | elapsed time per iteration (ms): 2897.7 | learning rate: 7.228E-05 | approx flops per GPU: 76.5TFLOPS | lm_loss: 2.046924E+00 | loss scale: 32768.0 | number of skipped iterations:   2 | number of nan iterations:   0 |
time (ms) | forward: 609.03 | backward: 2204.02 | backward-backward: 2203.95 | backward-allreduce: 0.00 | optimizer: 79.30 | batch generator: 2.22
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step126500
 samples/sec: 353.229 | iteration   130000/  143000 | elapsed time per iteration (ms): 2899.0 | learning rate: 7.140E-05 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.048169E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 609.67 | backward: 2204.25 | backward-backward: 2204.18 | backward-allreduce: 0.00 | optimizer: 79.56 | batch generator: 2.25
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step127000
-----------------------------------------------------------------------------------------------------------
 validation results at iteration 130000 | lm_loss value: 2.005995E+00 | lm_loss_ppl value: 7.433485E+00 | 
-----------------------------------------------------------------------------------------------------------
 samples/sec: 338.434 | iteration   130500/  143000 | elapsed time per iteration (ms): 3025.7 | learning rate: 7.056E-05 | approx flops per GPU: 73.2TFLOPS | lm_loss: 2.048540E+00 | loss scale: 32768.0 | number of skipped iterations:   2 | number of nan iterations:   0 |
time (ms) | forward: 609.73 | backward: 2204.00 | backward-backward: 2203.93 | backward-allreduce: 0.00 | optimizer: 79.24 | batch generator: 2.63
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step127500
 samples/sec: 353.168 | iteration   131000/  143000 | elapsed time per iteration (ms): 2899.5 | learning rate: 6.974E-05 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.048352E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 609.99 | backward: 2204.34 | backward-backward: 2204.27 | backward-allreduce: 0.00 | optimizer: 79.54 | batch generator: 2.33
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step128000
 samples/sec: 353.268 | iteration   131500/  143000 | elapsed time per iteration (ms): 2898.7 | learning rate: 6.896E-05 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.046191E+00 | loss scale: 65536.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 609.81 | backward: 2203.79 | backward-backward: 2203.72 | backward-allreduce: 0.00 | optimizer: 79.55 | batch generator: 2.32
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step128500
 samples/sec: 352.743 | iteration   132000/  143000 | elapsed time per iteration (ms): 2903.0 | learning rate: 6.822E-05 | approx flops per GPU: 76.3TFLOPS | lm_loss: 2.046235E+00 | loss scale: 32768.0 | number of skipped iterations:   2 | number of nan iterations:   0 |
time (ms) | forward: 609.96 | backward: 2208.10 | backward-backward: 2208.03 | backward-allreduce: 0.00 | optimizer: 79.23 | batch generator: 2.33
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step129000
 samples/sec: 353.251 | iteration   132500/  143000 | elapsed time per iteration (ms): 2898.8 | learning rate: 6.750E-05 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.046550E+00 | loss scale: 16384.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 609.71 | backward: 2204.34 | backward-backward: 2204.27 | backward-allreduce: 0.00 | optimizer: 79.36 | batch generator: 2.29
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step129500
 samples/sec: 353.110 | iteration   133000/  143000 | elapsed time per iteration (ms): 2899.9 | learning rate: 6.681E-05 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.045019E+00 | loss scale: 16384.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 609.84 | backward: 2205.06 | backward-backward: 2204.99 | backward-allreduce: 0.00 | optimizer: 79.49 | batch generator: 2.25
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step130000
 samples/sec: 352.687 | iteration   133500/  143000 | elapsed time per iteration (ms): 2903.4 | learning rate: 6.616E-05 | approx flops per GPU: 76.3TFLOPS | lm_loss: 2.044678E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 612.96 | backward: 2205.08 | backward-backward: 2205.01 | backward-allreduce: 0.00 | optimizer: 79.77 | batch generator: 2.40
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step130500
 samples/sec: 353.041 | iteration   134000/  143000 | elapsed time per iteration (ms): 2900.5 | learning rate: 6.554E-05 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.043597E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 609.57 | backward: 2205.57 | backward-backward: 2205.50 | backward-allreduce: 0.00 | optimizer: 79.50 | batch generator: 2.26
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step131000
 samples/sec: 353.273 | iteration   134500/  143000 | elapsed time per iteration (ms): 2898.6 | learning rate: 6.496E-05 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.043592E+00 | loss scale: 65536.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 608.95 | backward: 2204.99 | backward-backward: 2204.92 | backward-allreduce: 0.00 | optimizer: 79.33 | batch generator: 2.22
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step131500
 samples/sec: 353.157 | iteration   135000/  143000 | elapsed time per iteration (ms): 2899.6 | learning rate: 6.440E-05 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.043424E+00 | loss scale: 32768.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 609.14 | backward: 2205.64 | backward-backward: 2205.57 | backward-allreduce: 0.00 | optimizer: 79.33 | batch generator: 2.22
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step132000
-----------------------------------------------------------------------------------------------------------
 validation results at iteration 135000 | lm_loss value: 2.000182E+00 | lm_loss_ppl value: 7.390400E+00 | 
-----------------------------------------------------------------------------------------------------------
 samples/sec: 338.294 | iteration   135500/  143000 | elapsed time per iteration (ms): 3027.0 | learning rate: 6.388E-05 | approx flops per GPU: 73.2TFLOPS | lm_loss: 2.042341E+00 | loss scale: 16384.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 609.49 | backward: 2205.20 | backward-backward: 2205.13 | backward-allreduce: 0.00 | optimizer: 79.46 | batch generator: 2.62
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step132500
 samples/sec: 353.116 | iteration   136000/  143000 | elapsed time per iteration (ms): 2899.9 | learning rate: 6.339E-05 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.044158E+00 | loss scale: 16384.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 609.83 | backward: 2205.09 | backward-backward: 2205.02 | backward-allreduce: 0.00 | optimizer: 79.54 | batch generator: 2.26
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step133000
 samples/sec: 353.214 | iteration   136500/  143000 | elapsed time per iteration (ms): 2899.1 | learning rate: 6.293E-05 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.041776E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 609.19 | backward: 2204.87 | backward-backward: 2204.80 | backward-allreduce: 0.00 | optimizer: 79.53 | batch generator: 2.23
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step133500
 samples/sec: 352.813 | iteration   137000/  143000 | elapsed time per iteration (ms): 2902.4 | learning rate: 6.251E-05 | approx flops per GPU: 76.3TFLOPS | lm_loss: 2.043309E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 609.14 | backward: 2206.15 | backward-backward: 2206.08 | backward-allreduce: 0.00 | optimizer: 79.52 | batch generator: 2.24
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step134000
 samples/sec: 353.197 | iteration   137500/  143000 | elapsed time per iteration (ms): 2899.2 | learning rate: 6.212E-05 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.040981E+00 | loss scale: 32768.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 608.93 | backward: 2205.39 | backward-backward: 2205.32 | backward-allreduce: 0.00 | optimizer: 79.34 | batch generator: 2.22
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step134500
 samples/sec: 353.091 | iteration   138000/  143000 | elapsed time per iteration (ms): 2900.1 | learning rate: 6.176E-05 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.040547E+00 | loss scale: 8192.0 | number of skipped iterations:   2 | number of nan iterations:   0 |
time (ms) | forward: 609.11 | backward: 2206.17 | backward-backward: 2206.11 | backward-allreduce: 0.00 | optimizer: 79.30 | batch generator: 2.22
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step135000
 samples/sec: 353.149 | iteration   138500/  143000 | elapsed time per iteration (ms): 2899.6 | learning rate: 6.144E-05 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.042986E+00 | loss scale: 8192.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 609.07 | backward: 2205.53 | backward-backward: 2205.46 | backward-allreduce: 0.00 | optimizer: 79.61 | batch generator: 2.24
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step135500
 samples/sec: 353.067 | iteration   139000/  143000 | elapsed time per iteration (ms): 2900.3 | learning rate: 6.115E-05 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.041756E+00 | loss scale: 16384.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 609.23 | backward: 2205.98 | backward-backward: 2205.91 | backward-allreduce: 0.00 | optimizer: 79.64 | batch generator: 2.26
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step136000
 samples/sec: 353.187 | iteration   139500/  143000 | elapsed time per iteration (ms): 2899.3 | learning rate: 6.089E-05 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.040638E+00 | loss scale: 16384.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 609.07 | backward: 2205.21 | backward-backward: 2205.14 | backward-allreduce: 0.00 | optimizer: 79.65 | batch generator: 2.30
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step136500
 samples/sec: 353.088 | iteration   140000/  143000 | elapsed time per iteration (ms): 2900.1 | learning rate: 6.066E-05 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.039759E+00 | loss scale: 32768.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 609.19 | backward: 2205.90 | backward-backward: 2205.83 | backward-allreduce: 0.00 | optimizer: 79.58 | batch generator: 2.24
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step137000
-----------------------------------------------------------------------------------------------------------
 validation results at iteration 140000 | lm_loss value: 1.998048E+00 | lm_loss_ppl value: 7.374645E+00 | 
-----------------------------------------------------------------------------------------------------------
 samples/sec: 338.334 | iteration   140500/  143000 | elapsed time per iteration (ms): 3026.6 | learning rate: 6.047E-05 | approx flops per GPU: 73.2TFLOPS | lm_loss: 2.040289E+00 | loss scale: 32768.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 609.00 | backward: 2205.39 | backward-backward: 2205.32 | backward-allreduce: 0.00 | optimizer: 79.36 | batch generator: 2.58
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step137500
 samples/sec: 353.029 | iteration   141000/  143000 | elapsed time per iteration (ms): 2900.6 | learning rate: 6.031E-05 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.040376E+00 | loss scale: 16384.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 609.20 | backward: 2206.47 | backward-backward: 2206.40 | backward-allreduce: 0.00 | optimizer: 79.36 | batch generator: 2.22
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step138000
 samples/sec: 353.137 | iteration   141500/  143000 | elapsed time per iteration (ms): 2899.7 | learning rate: 6.018E-05 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.039720E+00 | loss scale: 16384.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 609.03 | backward: 2205.80 | backward-backward: 2205.73 | backward-allreduce: 0.00 | optimizer: 79.49 | batch generator: 2.23
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step138500
 samples/sec: 353.059 | iteration   142000/  143000 | elapsed time per iteration (ms): 2900.4 | learning rate: 6.009E-05 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.041119E+00 | loss scale: 16384.0 | number of skipped iterations:   2 | number of nan iterations:   0 |
time (ms) | forward: 609.17 | backward: 2206.66 | backward-backward: 2206.59 | backward-allreduce: 0.00 | optimizer: 79.17 | batch generator: 2.21
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step139000
 samples/sec: 353.093 | iteration   142500/  143000 | elapsed time per iteration (ms): 2900.1 | learning rate: 6.003E-05 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.039393E+00 | loss scale: 8192.0 | number of skipped iterations:   1 | number of nan iterations:   0 |
time (ms) | forward: 609.09 | backward: 2206.25 | backward-backward: 2206.18 | backward-allreduce: 0.00 | optimizer: 79.39 | batch generator: 2.22
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step139500
 samples/sec: 352.952 | iteration   143000/  143000 | elapsed time per iteration (ms): 2901.2 | learning rate: 6.000E-05 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.039866E+00 | loss scale: 8192.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
time (ms) | forward: 609.83 | backward: 2206.30 | backward-backward: 2206.23 | backward-allreduce: 0.00 | optimizer: 79.50 | batch generator: 2.25
---------------------------------------------------------------------------------------------------------------------------
 validation results at the end of training for val data | lm_loss value: 1.999463E+00 | lm_loss_ppl value: 7.385092E+00 | 
---------------------------------------------------------------------------------------------------------------------------
WARNING: Deleting old checkpoints: 
	/u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step140000
----------------------------------------------------------------------------------------------------------------------
 test results at the end of training for test data | lm_loss value: 1.992931E+00 | lm_loss_ppl value: 7.337004E+00 | 
----------------------------------------------------------------------------------------------------------------------