# Model is getting loaded unevenly on GPUs

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fmodel-is-getting-loaded-unevenly-on-gpus%2F96722%3E%5C
**Category:** 🤗Transformers\
**Created:** [July 11, 2024, 3:47am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fmodel-is-getting-loaded-unevenly-on-gpus%2F96722 "2024-07-11T03:47:31Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![abpani1994](https://avatars.discourse-cdn.com/v4/letter/a/ec9cab/32.png) [@abpani1994](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fabpani1994)%5C
**Post date:** [July 11, 2024, 3:47am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fmodel-is-getting-loaded-unevenly-on-gpus%2F96722%2F1 "2024-07-11T03:47:32Z")

</div>

I am trying to finetune Llama3 8B model using peft qlora.  
Loading gptq model with automodelforcasualLM which gets loaded unevenly in the GPUs which prevents me from using batch size of more than 1.  
I am using context length of 8192 or 4096.  
if I use 4096 I cant go more than 2 batchsize  
 ![Screenshot 2024-07-08 at 7.31.22 PM](https://us1.discourse-cdn.com/hellohellohello/original/3X/b/d/bd04ccc368e18fe4a00e9cd85aee3c48516a44a2.png)

Please help.

---

<div class="post-metadata">

**Author:** ![abpani1994](https://avatars.discourse-cdn.com/v4/letter/a/ec9cab/32.png) [@abpani1994](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fabpani1994)%5C
**Post date:** [July 11, 2024, 3:50am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fmodel-is-getting-loaded-unevenly-on-gpus%2F96722%2F2 "2024-07-11T03:50:05Z")

</div>

![Screenshot 2024-06-22 at 1.45.26 PM](https://us1.discourse-cdn.com/hellohellohello/original/3X/0/3/03c35656b3b183aec040ffe898384de81614ce9f.png)  
Another example.Happening with all models either gptq or bnb.
