# How to resume training from lora checkpoint

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-resume-training-from-lora-checkpoint%2F95514%3E%5C
**Category:** Beginners\
**Created:** [July 4, 2024, 12:08pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-resume-training-from-lora-checkpoint%2F95514 "2024-07-04T12:08:49Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![ankitMLAI](https://avatars.discourse-cdn.com/v4/letter/a/258eb7/32.png) [@ankitMLAI](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FankitMLAI)%5C
**Post date:** [July 4, 2024, 12:08pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-resume-training-from-lora-checkpoint%2F95514%2F1 "2024-07-04T12:08:49Z")

</div>

model / tokenizer= “Mistral model”

checkpoint\_path = “model/checkpoint-1000”

lora\_r = 16  
lora\_alpha = 64  
lora\_dropout = 0.1  
lora\_target\_modules = [  
“q\_proj”,  
“up\_proj”,  
“o\_proj”,  
“k\_proj”,  
“down\_proj”,  
“gate\_proj”,  
“v\_proj”,  
]

```
peft_config = LoraConfig(
    r=lora_r,
    lora_alpha=lora_alpha,
    lora_dropout=lora_dropout,
    target_modules=lora_target_modules,
    bias="none",
    task_type="CAUSAL_LM",
)
training_arguments = TrainingArguments(
    per_device_train_batch_size=1,
    gradient_accumulation_steps=32,
    optim="paged_adamw_32bit",
    logging_steps=1,
    learning_rate=1e-4,
    fp16=True,
    max_grad_norm=0.3,
    num_train_epochs=10,
    save_steps=100,
    evaluation_strategy="steps",
    eval_steps=0.2,
    warmup_ratio=0.05,
    save_strategy="steps",
    group_by_length="steps",
    output_dir=OUTPUT_DIR,
    report_to="tensorboard",
    save_safetensors=True,
    lr_scheduler_type="cosine",
    seed=42,
)

```

trainer = SFTTrainer(  
model=model,  
train\_dataset=train\_dataset[‘train’],  
eval\_dataset=val\_dataset[‘train’],  
peft\_config=peft\_config,  
dataset\_text\_field=“text”,  
max\_seq\_length=1024,  
tokenizer=tokenizer,  
args=training\_arguments,

)
