# Correct way to save/load adapters and checkpoints in PEFT

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcorrect-way-to-save-load-adapters-and-checkpoints-in-peft%2F77836%3E%5C
**Category:** 🤗Transformers\
**Created:** [March 18, 2024, 1:38am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcorrect-way-to-save-load-adapters-and-checkpoints-in-peft%2F77836 "2024-03-18T01:38:46Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![remorax98](https://avatars.discourse-cdn.com/v4/letter/r/57b2e6/32.png) [@remorax98](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fremorax98)%5C
**Post date:** [March 18, 2024, 1:38am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcorrect-way-to-save-load-adapters-and-checkpoints-in-peft%2F77836%2F1 "2024-03-18T01:38:46Z")

</div>

Hi,

It is not clear to me what is the correct way to save/load a PEFT checkpoint, as well as the final fine-tuned model. There have been reports of `trainer.resume_from_checkpoint` not working as expected [[1]](https://github.com/huggingface/transformers/issues/24330)[[2]](https://github.com/huggingface/transformers/issues/24252)[[3]](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fretraining-peft-model%2F43829)%2C each of which have very few replies, or do not seem to have any sort of consensus. Proposed solutions range from `trainer.save_model`, to `trainer.save_state` to `resume_from_checkpoint=True` to `model.save_pretrained` [(PEFT docs)](https://huggingface.co/docs/transformers/main/en/peft) to even a very complicated procedure of merging and saving the model [[4]](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fretraining-peft-model%2F43829%2F3).

It is very confusing trying to figure out the correct solution between these, especially if `resume_from_checkpoint` can be buggy. Loading/saving models should really not be this confusing, so can we resolve once and for all what is the officially recommended (+tested) way of saving/loading adapters, as well as individual checkpoints during training? Can we update the HF docs accordingly, and simplify this process?

@sgugger

---

<div class="post-metadata">

**Author:** ![not-lain](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/not-lain/32/59092_2.png) [@not-lain](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fnot-lain)%5C
**Post date:** [March 22, 2024, 5:46am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcorrect-way-to-save-load-adapters-and-checkpoints-in-peft%2F77836%2F2 "2024-03-22T05:46:46Z")

</div>

Hi @remorax98  
unfortunately sylvian is no longer a part of 🤗 (just wanted to clarify this since there are lots of people who kept tagging him)

also for loading your PEFT model for continuous training there is a very easy parameter around this it’s called [`is_trainable`](https://huggingface.co/docs/peft/en/package_reference/peft_model#peft.PeftModel.from_pretrained.is_trainable) which should allow you to load your PEFT model in a trainable state, then you can continue your training easily.  
**how to use**  
for my repo in [`not-lain/Gemma-2b-Peft-finetuning`](https://huggingface.co/not-lain/Gemma-2b-Peft-finetuning) all i have to do is

```python
# most of this code is from the button at the top right corner on 🤗
from peft import PeftModel, PeftConfig
from transformers import AutoModelForCausalLM

config = PeftConfig.from_pretrained("not-lain/Gemma-2b-Peft-finetuning")
model = AutoModelForCausalLM.from_pretrained("google/gemma-2b")
model = PeftModel.from_pretrained(model, 
"not-lain/Gemma-2b-Peft-finetuning",
 is_trainable=True # 👈 here
)
# check if it's working
model.print_trainable_parameters()
# >>> trainable params: 9,805,824 || all params: 2,515,978,240 || trainable%: 0.3897420034920493

```

heart react this comment if this solved the problem for you 🤗

---

<div class="post-metadata">

**Author:** ![remorax98](https://avatars.discourse-cdn.com/v4/letter/r/57b2e6/32.png) [@remorax98](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fremorax98)%5C
**Post date:** [March 22, 2024, 9:51am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcorrect-way-to-save-load-adapters-and-checkpoints-in-peft%2F77836%2F3 "2024-03-22T09:51:52Z")

</div>

Hi @not-lain thanks for letting me know about Sylvain! Sorry, my bad.

Regarding this parameter, this is good to know and quite useful for me. But it does not answer my specific question - I am more concerned with the proper way for saving adapters and checkpoints in HF, and the lack of clarity in documentation regarding the same.

Thanks anyway!

---

<div class="post-metadata">

**Author:** ![not-lain](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/not-lain/32/59092_2.png) [@not-lain](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fnot-lain)%5C
**Post date:** [March 22, 2024, 5:34pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcorrect-way-to-save-load-adapters-and-checkpoints-in-peft%2F77836%2F4 "2024-03-22T17:34:54Z")

</div>

don’t mention it @remorax98.  
I also made this notebook for you explaining all steps from the initial training then recalling the model and continue training, it took me a lot of time to get it done, but hope this helps you out.

> **[Google Colaboratory](https://colab.research.google.com/drive/12pMorxvLV-VwjuNBM76L4xXnzVYg57iB?usp=sharing)**

if this notebook helped you clarify how to use PEFT, please consider marking this conversation as solved 🤗

regards,  
hafedh hichri

---

<div class="post-metadata">

**Author:** ![adiudiun](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/adiudiun/32/25966_2.png) [@adiudiun](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fadiudiun)%5C
**Post date:** [April 18, 2024, 4:07pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcorrect-way-to-save-load-adapters-and-checkpoints-in-peft%2F77836%2F5 "2024-04-18T16:07:29Z")

</div>

Guys can you help with proper examples of storing it to the file system instead of pushing to service? I tried  
`trainer.model.save_pretrained("shake_adapter")`  
and then

```auto
shmodel = PeftModel.from_pretrained(
    base_model,
    "shake_adapter",
    is_trainable=True
)

```

but it does not update my base model at all

---

<div class="post-metadata">

**Author:** ![not-lain](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/not-lain/32/59092_2.png) [@not-lain](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fnot-lain)%5C
**Post date:** [April 21, 2024, 3:14am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcorrect-way-to-save-load-adapters-and-checkpoints-in-peft%2F77836%2F7 "2024-04-21T03:14:39Z")

</div>

@adiudiun the way PEFT works is it uses adapters to train the model  
the image below explains how you can visualize lora.

I think what you’re looking for is the `save_embedding_layers`, could you try setting it manually to `True` ?

> **[Models](https://huggingface.co/docs/peft/en/package_reference/peft_model#peft.PeftModel.save_pretrained.save_embedding_layers)**
>
> We’re on a journey to advance and democratize artificial intelligence through open source and open science.

![image](https://us1.discourse-cdn.com/hellohellohello/original/3X/6/d/6d7f689445e50ea5c21f349a6a03e3b1ae9bf6a4.png)

---

<div class="post-metadata">

**Author:** ![akashghimire](https://avatars.discourse-cdn.com/v4/letter/a/c6cbf5/32.png) [@akashghimire](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fakashghimire)%5C
**Post date:** [May 16, 2024, 6:43pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcorrect-way-to-save-load-adapters-and-checkpoints-in-peft%2F77836%2F8 "2024-05-16T18:43:23Z")

</div>

@adiudiun, I had the same problem. The correct way is to first load the `base_model` using `AutoModel.from_pretrained()`. Then, load the adapter config using `PeftConfig.from_pretrained('saved_dire')`. Finally, load the `peft_model` as follows:

# Load the entire model with adapters

peft\_model = PeftModel.from\_pretrained(base\_model, saved\_dire)

# Load adapter1 and adapter2

peft\_model.load\_adapter(saved\_dire + ‘/adapter1’, adapter\_name=‘adapter1’)

peft\_model.load\_adapter(saved\_dire + ‘/adapter2’, adapter\_name=‘adapter2’)

Here is the screenshot of that part:

 ![image](https://us1.discourse-cdn.com/hellohellohello/original/3X/d/2/d233a75b20e994755d9200ee3b603aa5507e2630.png)

Also, here is the link to the turtorial,  
[https://github.com/akashghimireOfficial/HuggingFace\_101/blob/master/Turtorial/12\_more\_about\_\_get\_peft\_model.ipynb](https://github.com/akashghimireOfficial/HuggingFace_101/blob/master/Turtorial/12_more_about__get_peft_model.ipynb)

---

<div class="post-metadata">

**Author:** ![roman174](https://avatars.discourse-cdn.com/v4/letter/r/278dde/32.png) [@roman174](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Froman174)%5C
**Post date:** [July 7, 2024, 4:35pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcorrect-way-to-save-load-adapters-and-checkpoints-in-peft%2F77836%2F9 "2024-07-07T16:35:45Z")

</div>

Hi, for what you have defined config when you don’t use it?

---

<div class="post-metadata">

**Author:** ![YuchengShi](https://avatars.discourse-cdn.com/v4/letter/y/c6cbf5/32.png) [@YuchengShi](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FYuchengShi)%5C
**Post date:** [August 15, 2024, 3:33pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcorrect-way-to-save-load-adapters-and-checkpoints-in-peft%2F77836%2F10 "2024-08-15T15:33:06Z")

</div>

Thank you for your effort. It is working now.

---

<div class="post-metadata">

**Author:** ![AceMB](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/acemb/32/53564_2.png) [@AceMB](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FAceMB)%5C
**Post date:** [September 7, 2025, 11:56am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcorrect-way-to-save-load-adapters-and-checkpoints-in-peft%2F77836%2F12 "2025-09-07T11:56:03Z")

</div>

from peft import PeftModel, PeftConfig  
from unsloth import FastLanguageModel  
import torch  
max\_seq\_length = 4096 # Can increase for longer reasoning traces  
lora\_rank = 32 # Larger rank = smarter, but slower

model, tokenizer = FastLanguageModel.from\_pretrained(  
model\_name = “unsloth/Qwen3-4B-Instruct-2507”,  
max\_seq\_length = max\_seq\_length,  
load\_in\_4bit = True, # False for LoRA 16bit  
#fast\_inference = Tr, # Enable vLLM fast inference  
#max\_lora\_rank = lora\_rank,  
#gpu\_memory\_utilization = 0.7, # Reduce if out of memory  
)  
model = PeftModel.from\_pretrained(model,  
“/kaggle/input/qwen3-4b-instruct-lora/Qwen3\_(4B)-Instruct\_lora\_model”,  
is\_trainable=True # 👈 here  
)  
…  
RuntimeError: element 0 of tensors does not require grad and does not have a grad\_fn

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C
**Post date:** [September 8, 2025, 12:06am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcorrect-way-to-save-load-adapters-and-checkpoints-in-peft%2F77836%2F13 "2025-09-08T00:06:14Z")

</div>

> PeftModel  
> FastLanguageModel

It’s probably good [not to mix `PEFTModel` and `FastLanguageModel`](https://huggingface.co/docs/trl/en/unsloth_integration#installation).

Or [element 0 of tensors does not require grad and does not have a grad\_fn #187](https://github.com/huggingface/peft/issues/137)
