vit5-base-finetune-visp-paraphrase-lora

This model is a fine-tuned version of VietAI/vit5-base on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 0.9093

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-05
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 2
• total_train_batch_size: 16
• optimizer: Use OptimizerNames.ADAMW_8BIT with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_ratio: 0.1
• num_epochs: 5
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss
1.2555	0.1378	3500	1.1542
1.1306	0.2757	7000	1.0637
1.0952	0.4135	10500	1.0265
1.0649	0.5513	14000	1.0053
1.0381	0.6891	17500	0.9880
1.0283	0.8270	21000	0.9768
1.0088	0.9648	24500	0.9663
0.9947	1.1026	28000	0.9581
0.9922	1.2404	31500	0.9511
0.9782	1.3782	35000	0.9457
0.9738	1.5161	38500	0.9402
0.9631	1.6539	42000	0.9416
0.9591	1.7917	45500	0.9339
0.9547	1.9295	49000	0.9313
0.9348	2.0673	52500	0.9283
0.9503	2.2052	56000	0.9259
0.9408	2.3430	59500	0.9268
0.9345	2.4808	63000	0.9222
0.9238	2.6186	66500	0.9197
0.9293	2.7565	70000	0.9181
0.9306	2.8943	73500	0.9168
0.9136	3.0321	77000	0.9153
0.9182	3.1699	80500	0.9147
0.9153	3.3077	84000	0.9144
0.9091	3.4456	87500	0.9153
0.9078	3.5834	91000	0.9132
0.9111	3.7212	94500	0.9110
0.9054	3.8590	98000	0.9110
0.9077	3.9969	101500	0.9102
0.8956	4.1347	105000	0.9109
0.8996	4.2725	108500	0.9095
0.9072	4.4103	112000	0.9096
0.9044	4.5482	115500	0.9093
0.9048	4.6860	119000	0.9093
0.9085	4.8238	122500	0.9093
0.9077	4.9616	126000	0.9093

Framework versions

• PEFT 0.10.0
• Transformers 4.49.0
• Pytorch 2.4.1+cu118
• Datasets 4.1.1
• Tokenizers 0.21.0