# How to generate using a fine-tuned qlora cast to bfloat16

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-generate-using-a-fine-tuned-qlora-cast-to-bfloat16%2F73428%3E%5C
**Category:** Beginners\
**Created:** [February 16, 2024, 5:20am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-generate-using-a-fine-tuned-qlora-cast-to-bfloat16%2F73428 "2024-02-16T05:20:17Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![jmeld](https://avatars.discourse-cdn.com/v4/letter/j/8dc957/32.png) [@jmeld](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fjmeld)%5C
**Post date:** [February 16, 2024, 5:20am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-generate-using-a-fine-tuned-qlora-cast-to-bfloat16%2F73428%2F1 "2024-02-16T05:20:17Z")

</div>

I tried to fine-tune a small model as lightly as I could just to test my workflow, and I can’t get past an error.

My quantization config is as follows:

```auto
quantization_config= BitsAndBytesConfig(
                load_in_4bit= True,
                bnb_4bit_compute_dtype= torch.bfloat16,
                bnb_4bit_use_double_quant= True,
                bnb_4bit_quant_type="nf4",
            )

```

LORA Config:

```auto
peft_params = LoraConfig(
    lora_alpha=16,
    lora_dropout=0.1,
    r=64,
    bias="none",
    task_type="CAUSAL_LM",
)

```

Trainer config:

```auto
training_params = TrainingArguments(
    output_dir="./results",
    num_train_epochs=1,
    per_device_train_batch_size=1,
    gradient_accumulation_steps=16,
    optim="adamw_torch",
    save_steps=25,
    logging_steps=25,
    learning_rate=1e-4,
    weight_decay=0.001,
    fp16=False,
    bf16=True,
    max_grad_norm=0.3,
    max_steps=-1,
    warmup_ratio=0.03,
    group_by_length=True,
    lr_scheduler_type="constant",
    report_to="tensorboard"
)

```

After a brief fine-tuning of just a few dataset examples, I try to generate some demo completions. I’m just trying to make sure everything works. I used the following commands:

```auto
prompt_text= "Hello Hello Hello"
input_ids= tokenizer(prompt_text, return_tensors="pt", padding=True, truncation=True)["input_ids"]
input_ids= input_ids.to(model.device)
output= model.generate(input_ids)
generated_text= tokenizer.decode(output[0], skip_special_tokens=True)
print(generated_text)

```

And I get the following error seemingly no matter what I do:

```auto
---------------------------------------------------------------------------
RuntimeError Traceback (most recent call last)
<ipython-input-51-1b75867bada9> in <cell line: 4>()
      2 input_ids= tokenizer(prompt_text, return_tensors="pt", padding=True, truncation=True)["input_ids"]
      3 input_ids= input_ids.to(model.device)
----> 4 output= model.generate(input_ids)
      5 generated_text= tokenizer.decode(output[0], skip_special_tokens=True)
      6 print(generated_text)

10 frames
/usr/local/lib/python3.10/dist-packages/torch/nn/modules/linear.py in forward(self, input)
    112 
    113 def forward(self, input: Tensor) -> Tensor:
--> 114 return F.linear(input, self.weight, self.bias)
    115 
    116 def extra_repr(self) -> str:

RuntimeError: expected mat1 and mat2 to have the same dtype, but got: float != c10::BFloat16

```

I’ve tried re-casting the input\_ids to bfloat16, but I still get the same error. I can’t seem to get it to work. Any advice?

---

<div class="post-metadata">

**Author:** ![UserDAN](https://avatars.discourse-cdn.com/v4/letter/u/9fc29f/32.png) [@UserDAN](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FUserDAN)%5C
**Post date:** [April 6, 2024, 3:05am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-generate-using-a-fine-tuned-qlora-cast-to-bfloat16%2F73428%2F2 "2024-04-06T03:05:17Z")

</div>

Did you find a solution for this issue??
