# PEFT LoRA GPT-NeoX - LoraConfig

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpeft-lora-gpt-neox-loraconfig%2F35790%3E%5C
**Category:** 🤗Transformers\
**Created:** [April 6, 2023, 2:54pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpeft-lora-gpt-neox-loraconfig%2F35790 "2023-04-06T14:54:50Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![eusip](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/eusip/32/2714_2.png) [@eusip](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Feusip)%5C
**Post date:** [April 6, 2023, 2:54pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpeft-lora-gpt-neox-loraconfig%2F35790%2F1 "2023-04-06T14:54:50Z")

</div>

I have written a training script that makes use of the Accelerate and PEFT libraries to finetune GPT-NeoX and while the training loop runs without error the script does not generate an adapter\_config.json file. My intuition is that there is something about my LoraConfig object that is not properly parameterized resulting in a silent failure.

Does anyone have any suggestions on how to best parameterize LoraConfig for GPT-NEoX family of models?

My current LoraConfig:

```auto
    peft_config = LoraConfig(
        r=16,
        lora_alpha=32,
        lora_dropout=0.05,
        target_modules = ["query_key_value", "xxx"],
        bias="none",
        task_type=TaskType.CAUSAL_LM,
        fan_in_fan_out=False,
    )

```

My full training script can be found [here](https://gist.github.com/eusip/de8fadb761741b56d5d9a6232bf979ed).

---

<div class="post-metadata">

**Author:** ![ybelkada](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/ybelkada/32/24289_2.png) [@ybelkada](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fybelkada)%5C
**Post date:** [April 6, 2023, 3:00pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpeft-lora-gpt-neox-loraconfig%2F35790%2F2 "2023-04-06T15:00:19Z")

</div>

Hello @eusip!  
Thanks for the issue!  
Indeed you need to slightly tweak the trainer to add a callback to properly save your Peft models, please have a look at what have been suggested in [Incorrect Saving Peft Models using HuggingFace Trainer · Issue #96 · huggingface/peft · GitHub](https://github.com/huggingface/peft/issues/96) and let us know if this works!

---

<div class="post-metadata">

**Author:** ![eusip](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/eusip/32/2714_2.png) [@eusip](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Feusip)%5C
**Post date:** [April 6, 2023, 4:08pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpeft-lora-gpt-neox-loraconfig%2F35790%2F3 "2023-04-06T16:08:26Z")

</div>

@ybelkada again you have saved the day! Thanks for your help!
