youth-ai-initiative/Meta-Llama-3.1-Math-QA-finetuning-Group-3

Meta-Llama-3.1-Math-QA-finetuning-Group-3

This model is a fine-tuned version of Meta-Llama-3.1-8B on the MetaMathQA dataset for mathematical reasoning tasks. Training Details

Method: QLoRA (4-bit quantization with LoRA adapters) Framework: Unsloth for memory and time efficient fine-tuning Dataset: 50,000 randomly selected samples from MetaMathQA (seed=42) Hardware: Google Colab T4 GPU

Hyperparameters

# QLoRA Configuration
load_in_4bit = True
lora_r = 16
lora_alpha = 16
lora_dropout = 0

# Training Configuration
num_train_epochs = 5
max_steps = 50
learning_rate = 5e-5
per_device_train_batch_size = 2
gradient_accumulation_steps = 4
warmup_steps = 5
weight_decay = 0.001
lr_scheduler_type = "linear"
optim = "adamw_8bit"
seed = 3407

Training Results

• 50th Epoch training loss: 0.551400