# How to load a model fine-tuned with QLoRA

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-load-a-model-fine-tuned-with-qlora%2F78157%3E%5C
**Category:** 🤗Transformers\
**Created:** [March 19, 2024, 7:54pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-load-a-model-fine-tuned-with-qlora%2F78157 "2024-03-19T19:54:35Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![AnsonKw](https://avatars.discourse-cdn.com/v4/letter/a/3e96dc/32.png) [@AnsonKw](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FAnsonKw)%5C
**Post date:** [March 19, 2024, 7:54pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-load-a-model-fine-tuned-with-qlora%2F78157%2F1 "2024-03-19T19:54:35Z")

</div>

I fine-tuned a mistral 7b with QLoRA that initiated as the following code during training:

```auto
## TRAINING
bnb_config = BitsAndBytesConfig(  
    load_in_4bit= True,
    bnb_4bit_quant_type= "nf4",
    bnb_4bit_compute_dtype= torch.bfloat16,
    bnb_4bit_use_double_quant= True,
)

model = AutoModelForSequenceClassification.from_pretrained(
    "mistralai/Mistral-7B-v0.1",
    quantization_config=bnb_config,
    torch_dtype=torch.bfloat16,
)

model.gradient_checkpointing_enable()
model = prepare_model_for_kbit_training(model, use_gradient_checkpointing = False)
peft_config = LoraConfig(...)
model = get_peft_model(model, peft_config)

```

In inferences, I tried to load the model using the same setting as training:

```auto
## INFERENCING
bnb_config = BitsAndBytesConfig(  
    load_in_4bit= True,
    bnb_4bit_quant_type= "nf4",
    bnb_4bit_compute_dtype= torch.bfloat16,
    bnb_4bit_use_double_quant= True,
)

base_model = AutoModelForSequenceClassification.from_pretrained(
    "mistralai/Mistral-7B-v0.1",
    quantization_config=bnb_config,
    torch_dtype=torch.bfloat16,
)
model = PeftModel.from_pretrained(base_model, "saved-adapter-path").eval()

```

However, the inference process is very slow.

I found that loading the model only with load\_in\_4bit=True like the following. The inference process is much faster.

```auto
base_model = AutoModelForSequenceClassification.from_pretrained(
    "mistralai/Mistral-7B-v0.1",
    bnb_config = BitsAndBytesConfig(load_in_4bit= True)
)
model = PeftModel.from_pretrained(base_model, "saved-adapter-path").eval()

```

Isn’t using double quant and bfloat supposed to be faster and use less memory when compared to the default setting?

What should be the correct way to load the model in my case?

---

<div class="post-metadata">

**Author:** ![nielsr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nielsr/32/39617_2.png) [@nielsr](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fnielsr)%5C
**Post date:** [March 19, 2024, 8:17pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-load-a-model-fine-tuned-with-qlora%2F78157%2F2 "2024-03-19T20:17:04Z")

</div>

Hi,

Refer to my [demo notebook](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/Mistral/Supervised_fine_tuning_(SFT)_of_an_LLM_using_Hugging_Face_tooling.ipynb) on fine-tuning Mistral-7B, it includes an inference section.

In summary, one can simply use the Auto classes (like AutoModelForCausalLM) to load models fine-tuned with Q-LoRa, thanks to the [PEFT integration in Transformers](https://huggingface.co/docs/peft/tutorial/peft_integrations#transformers). It will automatically load the base model + adapter weights.

Optionally, you can then call the `merge_and_unload()` method on it to merge the adapters into the base model.

---

<div class="post-metadata">

**Author:** ![blesspearl](https://avatars.discourse-cdn.com/v4/letter/b/e68b1a/32.png) [@blesspearl](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fblesspearl)%5C
**Post date:** [July 29, 2024, 9:01pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-load-a-model-fine-tuned-with-qlora%2F78157%2F3 "2024-07-29T21:01:11Z")

</div>

Hi please how do i perform the uploading of the model to the hub where it can be downloaded and used by others
