# Ref\_model in DPOTrainer

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fref-model-in-dpotrainer%2F82341%3E%5C
**Category:** 🤗Transformers\
**Created:** [April 18, 2024, 4:40am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fref-model-in-dpotrainer%2F82341 "2024-04-18T04:40:53Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![mrin](https://avatars.discourse-cdn.com/v4/letter/m/3e96dc/32.png) [@mrin](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmrin)%5C
**Post date:** [April 18, 2024, 4:40am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fref-model-in-dpotrainer%2F82341%2F1 "2024-04-18T04:40:53Z")

</div>

Hi, I’m using Mistral-7B-Instruct-v0.1 model with peft loaded in the DPOTrainer model.  
now when I’m giving ref\_model by creating a copy of this PeftLoaded model by create\_reference\_model() fn then the below piece of code is running infinitely and my Jupyter Lab got freezed. not responding anymore.  
but by passing ref\_model = None it’s running peacefully.

Now I want to know if I don’t pass the ref\_model rather than passing, is there anything will be affected, I know DPOtrainer will create a copy on it’s own for a ref\_model, but why passing ref\_model is not working or is there any impact.

```auto
trainer = DPOTrainer(
    model = combined_model,
    ref_model=None,
    args=args,
    train_dataset=small_dataset,
    eval_dataset=eval_dataset,
    tokenizer=combined_tokenizer,
    beta=dpo_args["beta"],
    loss_type=dpo_args["loss_type"],
    max_length = 512,
    max_prompt_length = 128,
)

```
