# DPO Training ruins my model’s conversational coherence

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdpo-training-ruins-my-model-s-conversational-coherence%2F160733%3E%5C
**Category:** Intermediate\
**Created:** [June 26, 2025, 7:13am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdpo-training-ruins-my-model-s-conversational-coherence%2F160733 "2025-06-26T07:13:47Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![enxi15](https://avatars.discourse-cdn.com/v4/letter/e/e99b99/32.png) [@enxi15](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fenxi15)%5C
**Post date:** [June 26, 2025, 7:13am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdpo-training-ruins-my-model-s-conversational-coherence%2F160733%2F1 "2025-06-26T07:13:47Z")

</div>

Hi everyone,

I’m currently fine-tuning a chatbot. My pipeline first applies SFT to establish the desired style, then incorporates DPO training (with a mixed-in SFT loss for stability) to help the model understand its capability boundaries — e.g., to avoid making unrealistic promises like “I can help you turn on the air conditioner.”

The SFT phase works fine; however, once I apply DPO, the model’s behavior completely collapses. Specifically: with a system prompt, the model begins producing incoherent or repetitive output after a few regular turns. Without a system prompt, the degradation is even worse — output becomes pure noise or completely unreasonable for most of the time.  
I’ve used DPO in other contexts, and while results can vary, I’ve never seen it completely destroy a model’s ability to hold a coherent conversation.

Some additional details:

-I’ve tried both my own custom trainer and existing frameworks like Swift, with similar outcomes.

-My training data follows standard DPO format, containing: conversation history, instruction, chosen, and rejected. (Note: system prompts are not included in training data.)

-Every assistant’s response is taken into account when calculating the loss. I also tried the regular way, which is to only consider the last round but didn’t see anything changed.

- I did my experiments on 7B and 32B models; nothing really changed.

Has anyone encountered similar issues, or do you have any insights on what might be going wrong?

Any insight would be incredibly appreciated. Thank you!

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C
**Post date:** [June 26, 2025, 7:51am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdpo-training-ruins-my-model-s-conversational-coherence%2F160733%2F2 "2025-06-26T07:51:21Z")

</div>

This issue might be similar.

> <https://github.com/huggingface/trl/issues/1025>
>
> Hi, I am running some tests with DPOTrainer to see how it works but I have encou…ntered some problems during the inference phase of the generated model. In details, this is the pipeline of operations I performed: 
> 1. I pre-trained from scratch a T5 model on natural language (English language). For this operation, I followed the instructions of the \[Hugging Face library.\](https://github.com/huggingface/transformers/tree/main/examples/flax/language-modeling) As for training the tokenizer, this was done using the \[sentencepiece library\](https://github.com/google/sentencepiece). The generated file (extension .model) was then used through the T5Tokenizer class, which allows using the .model file instead of a json file. 
> 
> 2. I fine-tuned T5 using a very trivial dataset such as the following.
> 
> | Input | Target |
> |-------------------------------------------|--------|
> | I love cats | a |
> | The cat is orange | b |
> | The cat is on the table | c |
> | The cat chased the mouse under the table. | d |
>       
> In summary, if there is no word 'the' in the input then the output will be 'a', if there is only one occurrence of 'the' then the output will be 'b', and so on... For fine-tuning, I did not use the SFTTrainer class but the classic Seq2SeqTrainer. 
> 
> 3. Then, I performed the DPO with the same inputs as the dataset present above, but in the JSON format. The code used is the same as the \[example on the repository\](https://github.com/huggingface/trl/blob/main/examples/research\_projects/stack\_llama\_2/scripts/dpo\_llama2.py). In this case, however, we used our finetuned T5 model and tokenizer (with classes T5ForConditionalGeneration, T5Tokenizer, T5Config). You can find the JSON file and the full code at the end of this message.
> 
> The problem arises in the inference phase of the model generated by the DPOTrainer. In fact, for several instances the output generated by the model is 'a a a a a a', ' b b b b b b b', 'c c c c c c c c', and so on... (the number of repetitions of the class is variable). Moreover, this behavior becomes more pronounced as the number of steps increases. Also, as the number of steps increases, words that are part of the train set are generated in the output (e.g., 'aaacat' is generated).
> 
> I cannot figure out what could be the cause of this behavior. By making inference of the simply fine-tuned model, the output generated is as expected (i.e., a class between 'a', 'b', 'c' and 'd'), so the problem is introduced during training with DPO. I also tried to use the pre-trained 't5-small' model / tokenizer instead of the ones trained from scratch, but the problem still persists.
> 
> I look forward to your feedback should more information or snippets of code used be needed.
> 
> 
> 
> \<details\>
> \<summary\>DPO dataset\</summary\>
> \[
> {
> 'prompt': 'I love cats',
> 'chosen': 'a',
> 'rejected': 'b',
> },
> {
> 'prompt': 'I love cats',
> 'chosen': 'a',
> 'rejected': 'c',
> },
> {
> 'prompt': 'I love cats',
> 'chosen': 'a',
> 'rejected': 'd',
> },
> {
> 'prompt': 'The cat is orange',
> 'chosen': 'b',
> 'rejected': 'a',
> },
> {
> 'prompt': 'The cat is orange',
> 'chosen': 'b',
> 'rejected': 'c',
> },
> {
> 'prompt': 'The cat is orange',
> 'chosen': 'b',
> 'rejected': 'd',
> }
> ...
> \]
> \</details\>
> 
> 
> \<details\>
> \<summary\>DPO code\</summary\>
> \`\`\`
> 
> # 0. imports
> import os
> from dataclasses import dataclass, field
> from typing import Dict, Optional
>     
> import torch
> from datasets import Dataset, load\_dataset
> from transformers import AutoModelForCausalLM, AutoTokenizer, HfArgumentParser, TrainingArguments, T5Config, T5Tokenizer, T5ForConditionalGeneration
>     
> from trl import DPOTrainer
>     
>     
> # Define and parse arguments.
> @dataclass
> class ScriptArguments:
> """
> The arguments for the DPO training script.
> """
>     
> # data parameters
> beta: Optional\[float\] = field(default=0.1, metadata={"help": "the beta parameter for DPO loss"})
>     
> # training parameters
> model\_name\_or\_path: Optional\[str\] = field(
> default="../sft/results/final\_checkpoint",
> metadata={"help": "the location of the SFT model name or path"},
> )
> config\_name: Optional\[str\] = field(
> default=None, metadata={"help": "Pretrained config name or path if not the same as model\_name"}
> )
> tokenizer\_name: Optional\[str\] = field(
> default=None, metadata={"help": "Pretrained tokenizer name or path if not the same as model\_name"}
> )
> cache\_dir: Optional\[str\] = field(
> default=None,
> metadata={"help": "Where to store the pretrained models downloaded from huggingface.co"},
> )
> use\_fast\_tokenizer: bool = field(
> default=False,
> metadata={"help": "Whether to use one of the fast tokenizer (backed by the tokenizers library) or not."},
> )
> model\_revision: str = field(
> default="main",
> metadata={"help": "The specific model version to use (can be a branch name, tag name or commit id)."},
> )
> use\_auth\_token: Optional\[str\] = field(
> default=None,
> metadata={
> "help": "Will use the token generated when running \`transformers-cli login\` (necessary to use this script "
> "with private models)."
> },
> )
>     
> train\_file: Optional\[str\] = field(
> default=None, metadata={"help": "The input training data file (a jsonlines or csv file)."}
> )
>     
> eval\_file: Optional\[str\] = field(
> default=None, metadata={"help": "The input eval data file (a jsonlines or csv file)."}
> )
>     
> learning\_rate: Optional\[float\] = field(default=5e-4, metadata={"help": "optimizer learning rate"})
> lr\_scheduler\_type: Optional\[str\] = field(default="cosine", metadata={"help": "the lr scheduler type"})
> warmup\_steps: Optional\[int\] = field(default=100, metadata={"help": "the number of warmup steps"})
> weight\_decay: Optional\[float\] = field(default=0.05, metadata={"help": "the weight decay"})
> optimizer\_type: Optional\[str\] = field(default="paged\_adamw\_32bit", metadata={"help": "the optimizer type"})
>     
> per\_device\_train\_batch\_size: Optional\[int\] = field(default=4, metadata={"help": "train batch size per device"})
> per\_device\_eval\_batch\_size: Optional\[int\] = field(default=1, metadata={"help": "eval batch size per device"})
> gradient\_accumulation\_steps: Optional\[int\] = field(
> default=4, metadata={"help": "the number of gradient accumulation steps"}
> )
> gradient\_checkpointing: Optional\[bool\] = field(
> default=True, metadata={"help": "whether to use gradient checkpointing"}
> )
>     
> lora\_alpha: Optional\[float\] = field(default=16, metadata={"help": "the lora alpha parameter"})
> lora\_dropout: Optional\[float\] = field(default=0.05, metadata={"help": "the lora dropout parameter"})
> lora\_r: Optional\[int\] = field(default=8, metadata={"help": "the lora r parameter"})
>     
> max\_prompt\_length: Optional\[int\] = field(default=512, metadata={"help": "the maximum prompt length"})
> max\_length: Optional\[int\] = field(default=1024, metadata={"help": "the maximum sequence length"})
> max\_steps: Optional\[int\] = field(default=1000, metadata={"help": "max number of training steps"})
> logging\_steps: Optional\[int\] = field(default=10, metadata={"help": "the logging frequency"})
> save\_steps: Optional\[int\] = field(default=100, metadata={"help": "the saving frequency"})
> eval\_steps: Optional\[int\] = field(default=100, metadata={"help": "the evaluation frequency"})
>     
> output\_dir: Optional\[str\] = field(default="./results", metadata={"help": "the output directory"})
> log\_freq: Optional\[int\] = field(default=1, metadata={"help": "the logging frequency"})
>     
> # instrumentation
> sanity\_check: Optional\[bool\] = field(default=False, metadata={"help": "only train on 1000 samples"})
> report\_to: Optional\[str\] = field(
> default="wandb",
> metadata={
> "help": 'The list of integrations to report the results and logs to. Supported platforms are \`"azure\_ml"\`,'
> '\`"comet\_ml"\`, \`"mlflow"\`, \`"neptune"\`, \`"tensorboard"\`,\`"clearml"\` and \`"wandb"\`. '
> 'Use \`"all"\` to report to all integrations installed, \`"none"\` for no integrations.'
> },
> )
> # debug argument for distributed training
> ignore\_bias\_buffers: Optional\[bool\] = field(
> default=False,
> metadata={
> "help": "fix for DDP issues with LM bias/mask buffers - invalid scalar type,\`inplace operation. See"
> "https://github.com/huggingface/transformers/issues/22482#issuecomment-1595790992"
> },
> )
>     
>     
> def convert(
> dataset: Dataset = None,
> sanity\_check: bool = False,
> cache\_dir: str = None,
> num\_proc=24,
> ) -\> Dataset:
> """Load the dataset and convert it to the necessary format.
>     
> The dataset is converted to a dictionary with the following structure:
> {
> 'prompt': List\[str\],
> 'chosen': List\[str\],
> 'rejected': List\[str\],
> }
> """
> original\_columns = dataset.column\_names
>     
> if sanity\_check:
> dataset = dataset.select(range(min(len(dataset), 1000)))
>     
> def return\_prompt\_and\_responses(samples) -\> Dict\[str, str\]:
> return {
> "prompt": samples\["prompt"\],
> "chosen": samples\["chosen"\],
> "rejected": samples\["rejected"\],
> }
>     
> return dataset.map(
> return\_prompt\_and\_responses,
> batched=True,
> num\_proc=num\_proc,
> remove\_columns=original\_columns,
> )
>     
>     
> if \_\_name\_\_ == "\_\_main\_\_":
> parser = HfArgumentParser(ScriptArguments)
> script\_args = parser.parse\_args\_into\_dataclasses()\[0\]
>     
> # 1. load a pretrained model
> config = T5Config.from\_pretrained(
> script\_args.config\_name if script\_args.config\_name else script\_args.model\_name\_or\_path,
> cache\_dir=script\_args.cache\_dir,
> revision=script\_args.model\_revision,
> use\_auth\_token=script\_args.use\_auth\_token,
> )
>     
> tokenizer = T5Tokenizer.from\_pretrained(
> script\_args.tokenizer\_name if script\_args.tokenizer\_name else script\_args.model\_name\_or\_path,
> cache\_dir=script\_args.cache\_dir,
> use\_fast=script\_args.use\_fast\_tokenizer,
> revision=script\_args.model\_revision,
> use\_auth\_token=script\_args.use\_auth\_token,
> )
>     
> model = T5ForConditionalGeneration.from\_pretrained(
> script\_args.model\_name\_or\_path,
> config=config,
> cache\_dir=script\_args.cache\_dir,
> revision=script\_args.model\_revision,
> use\_auth\_token=script\_args.use\_auth\_token,
> )
> model.config.use\_cache = False
>     
> model\_ref = T5ForConditionalGeneration.from\_pretrained(
> script\_args.model\_name\_or\_path,
> config=config,
> cache\_dir=script\_args.cache\_dir,
> revision=script\_args.model\_revision,
> use\_auth\_token=script\_args.use\_auth\_token,
> )
>     
>     
> if script\_args.ignore\_bias\_buffers:
> # torch distributed hack
> model.\_ddp\_params\_and\_buffers\_to\_ignore = \[
> name for name, buffer in model.named\_buffers() if buffer.dtype == torch.bool
> \]
>     
> # 2. Load the dataset and split in train / eval
> train\_dataset = load\_dataset("json", data\_files=script\_args.train\_file, split="train")
>     
> train\_dataset = convert(dataset=train\_dataset, sanity\_check=script\_args.sanity\_check)
> train\_dataset = train\_dataset.filter(
> lambda x: len(x\["prompt"\]) + len(x\["chosen"\]) \<= script\_args.max\_length
> and len(x\["prompt"\]) + len(x\["rejected"\]) \<= script\_args.max\_length
> )
>     
> # 3. Load evaluation dataset
> eval\_dataset = load\_dataset("json", data\_files=script\_args.eval\_file, split="train")
>     
> eval\_dataset = convert(dataset=eval\_dataset, sanity\_check=script\_args.sanity\_check)
> eval\_dataset = eval\_dataset.filter(
> lambda x: len(x\["prompt"\]) + len(x\["chosen"\]) \<= script\_args.max\_length
> and len(x\["prompt"\]) + len(x\["rejected"\]) \<= script\_args.max\_length
> )
>     
> # 4. initialize training arguments:
> training\_args = TrainingArguments(
> per\_device\_train\_batch\_size=script\_args.per\_device\_train\_batch\_size,
> per\_device\_eval\_batch\_size=script\_args.per\_device\_eval\_batch\_size,
> max\_steps=script\_args.max\_steps,
> logging\_steps=script\_args.logging\_steps,
> save\_steps=script\_args.save\_steps,
> gradient\_accumulation\_steps=script\_args.gradient\_accumulation\_steps,
> gradient\_checkpointing=script\_args.gradient\_checkpointing,
> learning\_rate=script\_args.learning\_rate,
> evaluation\_strategy="steps",
> eval\_steps=script\_args.eval\_steps,
> output\_dir=script\_args.output\_dir,
> lr\_scheduler\_type=script\_args.lr\_scheduler\_type,
> warmup\_steps=script\_args.warmup\_steps,
> remove\_unused\_columns=False,
> run\_name="dpo",
> )
>     
>     
> # 5. initialize the DPO trainer
> dpo\_trainer = DPOTrainer(
> model,
> model\_ref,
> args=training\_args,
> beta=script\_args.beta,
> train\_dataset=train\_dataset,
> eval\_dataset=eval\_dataset,
> tokenizer=tokenizer,
> max\_prompt\_length=script\_args.max\_prompt\_length,
> max\_length=script\_args.max\_length,
> )
>     
> # 6. train
> dpo\_trainer.train()
> dpo\_trainer.save\_model(script\_args.output\_dir)
>     
> # 7. save
> output\_dir = os.path.join(script\_args.output\_dir, "final\_checkpoint")
> dpo\_trainer.model.save\_pretrained(output\_dir)
> \`\`\`
> \</details\>

---

<div class="post-metadata">

**Author:** ![AustinAligned](https://avatars.discourse-cdn.com/v4/letter/a/97f17d/32.png) [@AustinAligned](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FAustinAligned)%5C
**Post date:** [July 14, 2026, 2:54pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdpo-training-ruins-my-model-s-conversational-coherence%2F160733%2F4 "2026-07-14T14:54:51Z")

</div>

Hi, I know this thread is a bit older, but did you ever figure out what was causing the collapse? I’m researching how people run preference training on open-weight models, and post-DPO degradation like you described is exactly the kind of thing I’m trying to understand. Curious what you ended up trying, whether it was the dataset, the beta, or something else.

Happy to compare notes here or by email, I’m at [austin@aureliusaligned.ai](mailto:austin@aureliusaligned.ai). Just trying to learn, not selling anything.

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C
**Post date:** [July 15, 2026, 6:17am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdpo-training-ruins-my-model-s-conversational-coherence%2F160733%2F5 "2026-07-15T06:17:15Z")

</div>

Hmm… I don’t know whether the original poster ever pinned down the final cause, but if you’re asking what I’d consider plausible in a case like this:

* * *

First, a small correction to my earlier reply: I am not the original poster, and I cannot see a later message in this thread establishing whether the root cause was the dataset, beta, or something else.

The [old TRL issue I linked](https://github.com/huggingface/trl/issues/1025) only showed a superficially similar repetition/corruption symptom in a very different T5, classification-like setup. I would not treat it as evidence that the same mechanism was responsible here.

For a collapse like this, I would not reduce the investigation to **“dataset versus beta.”** I would separate at least five layers:

1. the **rendered sequence contract** ;
2. the **policy/reference construction** ;
3. the overall **optimization strength** ;
4. the **preference-data geometry and policy support** ;
5. the **saved model and inference path**.

Different trainers can still share several of those layers, so reproducing the symptom in a custom trainer and Swift weakens a purely framework-specific explanation, but does not by itself establish that DPO’s objective or the semantic content of the dataset was the cause.

A compact diagnostic path would be:

```plaintext
A. Does merely loading/wrapping the SFT model in the DPO path change its output?
   Yes → inspect active adapters, model mode, dtype/quantization,
         tokenizer/template, and the inference function
   No → B

B. Does a zero-update save/reload round trip change its output?
   Yes → inspect serialization, base/adapter pairing, merge state,
         tokenizer files, generation config, EOS/stop settings
   No → C

C. Are the final rendered prompt/chosen/rejected token sequences,
   shared prefix, completion masks, EOS tokens, and truncation positions correct?
   No → sequence-contract or preprocessing problem
   Yes → D

D. At step 0, does policy/reference behavior match the intended reference design?
   No → reference or adapter construction problem
   Yes → E

E. What changes first during training?

   chosen log-prob ↓, rejected log-prob ↓, margin ↑
       → relative objective is improving while absolute likelihood is falling;
         inspect pair similarity / likelihood displacement

   chosen log-prob ↑, rejected log-prob ↓, ordinary-chat quality ↓
       → inspect total optimization strength and preference-distribution coverage

   only long or multi-turn conversations degrade
       → inspect truncation, role boundaries, trajectory semantics,
         and context-length coverage

   training-time model is coherent, reloaded model is not
       → return to the artifact/inference path

```

The highest-information controls, before doing a large beta sweep, would therefore be:

- a **load/wrap-only comparison** with no training;
- a **zero-update save/reload comparison** ;
- a dump of the **actual rendered and tokenized pair** , including masks and truncation;
- a **step-zero policy/reference log-prob comparison** ;
- fixed ordinary-chat and capability-boundary evaluations at **SFT, step 1, early checkpoints, and final** ;
- separate tracking of **absolute chosen and rejected log-probs** , not only reward accuracy or margin.

I would also reproduce the failure first with deterministic decoding and exactly the same inference function. Otherwise, training drift and decoding/configuration drift remain confounded.

> **1. Sequence contract and shared-prefix checks**
>
> “Standard DPO format” at the Python/dataset level does not guarantee that the final token sequences are correct.
> 
> As the [Transformers chat-template documentation](https://huggingface.co/docs/transformers/chat_templating) emphasizes, a chat model ultimately consumes a sequence of tokens. The model-specific role and control tokens are part of its learned protocol; using the wrong template, duplicating special tokens, or placing the assistant-generation boundary differently can substantially change behavior.
> 
> For at least one random pair and one of the longest multi-turn pairs, I would inspect:
> 
> ```plaintext
> raw prompt messages
> raw chosen messages
> raw rejected messages
> 
> rendered prompt
> rendered chosen sequence
> rendered rejected sequence
> 
> token IDs
> decoded token-by-token representation
> 
> prompt/completion boundary
> chosen/rejected completion mask
> all loss labels
> 
> BOS location
> EOS locations
> assistant-start/control tokens
> 
> length before truncation
> length after truncation
> which turns were removed
> number of non-masked chosen/rejected tokens that remain
> 
> ```
> 
> A particularly useful invariant is:
> 
> ```plaintext
> chosen_prompt_token_ids == rejected_prompt_token_ids
> 
> ```
> 
> up to the exact point where the two completions are supposed to diverge.
> 
> I would compare the prefix generated in four places:
> 
> ```plaintext
> SFT training
> DPO chosen branch
> DPO rejected branch
> inference
> 
> ```
> 
> A pair can look identical when printed as plain text while differing in model-visible control tokens, EOS placement, or assistant-start tokens.
> 
> For long conversations, it is not enough to ask whether the example exceeded `max_length`. The more useful questions are:
> 
> - Which historical turn was removed?
> - Did truncation remove the part of the context that made the preference meaningful?
> - Did it remove the actual chosen/rejected difference?
> - Did one branch retain a different number of completion tokens?
> - Did an EOS token move into or out of the loss-bearing region?
> - Under all-turn loss, how many assistant tokens were actually supervised?
> - Under last-turn loss, was the final response still paired with the intended history?
> 
> The original report said that system prompts were absent from the DPO data and that inference became even worse without a system prompt. That is a useful clue, but it does not prove that omitting a system prompt caused the collapse. It could indicate format dependence, distribution coverage, or simply that the system prompt partially stabilized an already degraded policy.
> 
> Similarly, trying both all-assistant-turn loss and last-turn-only loss weakens the claim that the choice between those two modes was the sole cause. It does not rule out a shared masking, rendering, or truncation problem.
> 
> The current [TRL DPO documentation](https://huggingface.co/docs/trl/dpo_trainer) recommends an explicit prompt and automatically applies a chat template to conversational preference data. However, the exact behavior is version-dependent, so I would record the trainer version/commit rather than assume current behavior matches a 2025 run.

> **2. Policy/reference and PEFT construction**
>
> DPO depends on four quantities for each pair:
> 
> ```plaintext
> policy log-prob of chosen
> policy log-prob of rejected
> reference log-prob of chosen
> reference log-prob of rejected
> 
> ```
> 
> Therefore, “the policy and reference were configured from the same checkpoint” is weaker evidence than directly observing their outputs on the same tokenized batch.
> 
> If the intended reference is the same SFT policy from which DPO starts, I would expect the step-zero completion log-probs to agree within the numerical tolerance expected from dtype, quantization, and implementation details:
> 
> ```plaintext
> policy_chosen_logp ≈ reference_chosen_logp
> policy_rejected_logp ≈ reference_rejected_logp
> 
> ```
> 
> If a deliberately different reference is being used, exact equality is not expected; the control becomes verifying that the observed difference matches that intended design.
> 
> A substantial unexplained step-zero difference would move the investigation toward:
> 
> - reference using the base model without the SFT adapter;
> - different active adapter names;
> - merged policy versus unmerged reference;
> - reference adapter not being loaded or activated;
> - different token masks or preprocessing;
> - policy in train mode versus reference in eval mode;
> - dtype or quantization differences;
> - stale or incorrectly generated cached reference log-probs.
> 
> This is especially relevant with PEFT. The [TRL v0.18.1 DPO documentation](https://huggingface.co/docs/trl/v0.18.1/dpo_trainer) described multiple ways of constructing an SFT-aware reference, including loading the same adapter twice under separate names. The fact that several valid constructions existed means reference handling was an explicit design choice, not merely an invisible implementation detail.
> 
> I would record:
> 
> ```plaintext
> base model repository and exact revision
> SFT checkpoint
> policy adapter path and name
> reference adapter path and name
> active adapter before every forward pass
> merged or unmerged state
> dtype and quantization
> whether reference log-probs were cached
> exact code/version used to construct the reference
> 
> ```
> 
> Current TRL has evolved substantially since v0.18.1, so present-day defaults should not be projected backward onto an older Swift or custom implementation. Conversely, reproducing an old run on current TRL without recording the version can silently change the reference and preprocessing path.

> **3. Preference-data geometry and policy support**
>
> “Dataset problem” can mean several different things. I would separate at least four possibilities.
> 
> ### A. Chosen/rejected responses are too similar
> 
> Standard DPO optimizes a relative preference margin. It does not require the absolute likelihood of the chosen response to increase.
> 
> The [DPO-Positive paper](https://arxiv.org/abs/2402.13228) demonstrates that the ordinary DPO loss can improve while the likelihood of preferred examples falls, and reports that this occurred particularly often for datasets where chosen and rejected completions had low edit distance.
> 
> The later [likelihood displacement paper](https://arxiv.org/abs/2410.08847) studies a related phenomenon in which probability mass can move away from both members of a semantically similar pair and toward an unintended response, potentially even one with the opposite meaning.
> 
> That is highly relevant as a possible mechanism for capability-boundary pairs such as:
> 
> ```plaintext
> "I can do that."
> "I cannot do that."
> 
> ```
> 
> or pairs differing mainly in:
> 
> ```plaintext
> can / cannot
> will / may
> always / sometimes
> yes / no
> one number
> one modal verb
> one safety qualifier
> 
> ```
> 
> However, it would still be an overreach to infer that this happened in the original run without seeing the pairs and log-prob dynamics.
> 
> Useful dataset summaries would include:
> 
> ```plaintext
> token overlap between chosen and rejected
> normalized edit distance
> length ratio
> fraction of pairs differing mainly by negation
> fraction differing only near the end
> semantic-similarity distribution
> difference-token positions
> 
> ```
> 
> A small manual stratification may be more useful than one global average:
> 
> ```plaintext
> high-overlap / polarity-only pairs
> moderately different pairs
> clearly different responses
> 
> ```
> 
> Then compare the training signature and held-out behavior for those groups.
> 
> ### B. The preference label is ambiguous or contradictory
> 
> A preference pair may be syntactically valid while providing an unclear learning signal.
> 
> Examples include:
> 
> - both answers are acceptable;
> - both answers are poor;
> - the “correct” capability boundary depends on unstated context;
> - duplicate prompts have conflicting labels;
> - a cautious answer is preferred in one example and rejected in a nearly identical one;
> - the label reflects style while the intended target is factual capability;
> - the rejected answer contains both the unwanted promise and otherwise useful content.
> 
> The [Robust DPO paper](https://arxiv.org/abs/2403.00409) formalizes the general problem of incorrect or ambiguous preference feedback. I would not jump directly to a different loss, but I would manually classify a small sample as:
> 
> ```plaintext
> clear preference
> context-dependent preference
> ambiguous preference
> contradictory or mislabeled preference
> 
> ```
> 
> That can distinguish “the dataset is bad” from a more specific and actionable problem.
> 
> ### C. The responses are far from the SFT policy
> 
> If the chosen or rejected responses were produced by a different, much stronger, or differently aligned model, the target policy may assign extremely low initial probability to them.
> 
> The [WPO paper](https://arxiv.org/abs/2406.11827) discusses distributional gaps in off-policy preference optimization. Again, this is not evidence that the original case had that problem, but it suggests recording:
> 
> ```plaintext
> which model or process generated chosen
> which model or process generated rejected
> initial SFT log-prob of chosen
> initial SFT log-prob of rejected
> 
> ```
> 
> If the chosen response is already an extreme outlier under the SFT policy, the pair is different from a local preference correction between two plausible SFT outputs.
> 
> ### D. Length is carrying unintended preference information
> 
> Chosen/rejected length should be summarized as a distribution rather than only an average.
> 
> I would check:
> 
> ```plaintext
> chosen/rejected token-length ratio
> whether chosen is systematically shorter or longer
> whether high-overlap pairs also have a length bias
> sum log-prob and per-token average log-prob
> effective length after truncation
> 
> ```
> 
> Length is probably not enough to explain severe conversational corruption by itself, but it can confound pair comparisons and the interpretation of sequence-level log-probs.
> 
> The important practical point is that “dataset versus beta” is not one comparison. The data side includes at least pair similarity, label clarity, response-source/policy support, length, and the rendered token contract.

> **4. Optimization strength and early capability degradation**
>
> Beta matters, but a beta-only sweep would not isolate the broader optimization problem.
> 
> I would record the following together:
> 
> ```plaintext
> beta
> learning rate
> optimizer and scheduler
> optimizer steps
> processed preference pairs
> processed completion tokens
> effective epochs / average example exposure
> batch size and gradient accumulation
> full fine-tuning or PEFT
> LoRA rank, alpha, target modules
> number and fraction of trainable parameters
> gradient norm
> SFT-loss weight
> 
> ```
> 
> These variables are not perfectly interchangeable, but together they determine how aggressively and where the policy moves.
> 
> The current [TRL DPO documentation](https://huggingface.co/docs/trl/dpo_trainer) exposes separate metrics for:
> 
> ```plaintext
> logps/chosen
> logps/rejected
> rewards/chosen
> rewards/rejected
> rewards/margins
> rewards/accuracies
> entropy
> mean token accuracy
> gradient norm
> processed token count
> 
> ```
> 
> Older TRL versions did not necessarily expose the same standard metrics, so an older run may require custom instrumentation rather than simply reading an existing log.
> 
> I would avoid using only:
> 
> ```plaintext
> training loss
> reward accuracy
> reward margin
> 
> ```
> 
> as evidence that the model is improving.
> 
> A more informative timeline is:
> 
> ```plaintext
> SFT baseline
> DPO model before the first update
> step 1
> first few logging/checkpoint intervals
> mid-run
> final checkpoint
> reloaded final checkpoint
> 
> ```
> 
> At each point, evaluate both:
> 
> 1. a preference-specific holdout;
> 2. a fixed ordinary-chat holdout unrelated to the DPO dataset.
> 
> The [direct-alignment overoptimization study](https://arxiv.org/abs/2406.02900) reports that DPO and related methods can show degradation before completing a single epoch. That does not diagnose this run, but it is a strong reason not to compare only SFT against the final DPO checkpoint.
> 
> The result patterns have different implications:
> 
> | Observation | More consistent with |
> | --- | --- |
> | Quality collapses almost immediately | implementation/reference mismatch, very aggressive updates, or severe data-contract problem |
> | Preference holdout improves while ordinary chat degrades | narrow preference distribution or excessive movement away from general-chat behavior |
> | Both improve initially, then ordinary chat falls | overoptimization / checkpoint selection problem |
> | Only final reloaded model fails | artifact or inference-path problem |
> | Smaller beta helps only when learning rate/exposure also changes | not enough evidence for beta as the isolated cause |
> 
> The original poster also reported mixing in an SFT loss already. Therefore, “add SFT loss” would not be a sufficient answer here. It is still useful as an ablation signal:
> 
> ```plaintext
> Did SFT mixing prevent chosen log-prob from falling?
> Did it delay ordinary-chat degradation?
> Did it preserve style but not multi-turn coherence?
> Did changing its weight alter the first failing checkpoint?
> 
> ```
> 
> If the collapse signature is unchanged, that weakens the idea that simple SFT regularization alone addresses the mechanism.
> 
> I would treat absolute log-probs as necessary diagnostics, but not as standalone quality metrics. Even a seemingly desirable pattern such as chosen increasing and rejected decreasing does not guarantee preserved general conversation quality. The held-out behavior remains the deciding observation.

> **5. Multi-turn and capability-boundary semantics**
>
> The original task was not generic preference tuning; it was teaching the chatbot its capability boundaries and preventing unrealistic promises.
> 
> That makes the **unit of preference** important.
> 
> A chosen/rejected label might refer to:
> 
> ```plaintext
> only the final assistant response
> the entire assistant trajectory
> whether a refusal was correct
> whether the explanation after refusal was useful
> whether the model recovered after the user clarified
> whether the complete conversation remained coherent
> 
> ```
> 
> Those are not equivalent objectives.
> 
> The original poster tried both loss over every assistant response and loss over only the last round. Since neither fixed the issue, I would inspect what the pair-level preference actually represents rather than treating all-turn versus last-turn as the remaining main question.
> 
> Research on [direct multi-turn preference optimization](https://arxiv.org/abs/2406.14868) treats multi-turn trajectories as distinct from ordinary single-completion preference pairs, with additional state and trajectory-length considerations. That does not mean a specialized multi-turn loss is automatically required here. It does mean that “the final answer is preferred” and “the whole interaction is preferred” should not be silently treated as identical supervision.
> 
> For capability-boundary evaluation, I would split the holdout into at least three groups:
> 
> | Group | Desired behavior |
> | --- | --- |
> | Clearly possible and acceptable | comply normally; do not over-refuse |
> | Impossible, unverifiable, unsafe, or outside the model’s agency | decline, qualify, or explain the limitation |
> | Context-dependent or underspecified | ask for clarification or state conditions rather than making a categorical claim |
> 
> This is analogous to the contrastive logic behind [XSTest](https://arxiv.org/abs/2308.01263), which evaluates both prompts that should be refused and superficially similar safe prompts that should not be refused. Capability boundaries are broader than safety, but the same evaluation principle helps prevent one-sided training.
> 
> For multi-turn conversations, I would additionally measure:
> 
> ```plaintext
> Does the model maintain the same capability claim across turns?
> Does it explain a limitation without repeating one canned refusal?
> Can it offer a realistic alternative?
> Can it recover when the user changes the conditions?
> Does it switch from refusal to compliance when the task becomes feasible?
> Does it continue coherently after a normal unrelated turn?
> 
> ```
> 
> That can distinguish several failure modes which look similar in a single screenshot:
> 
> - over-refusal;
> - semantic inversion of a preference;
> - repetitive decoding;
> - context/truncation failure;
> - loss of general conversational ability;
> - inability to revise a capability judgment across turns.

> **6. Save/load, PEFT, and inference-path controls**
>
> I would keep the training and artifact paths separate throughout the investigation.
> 
> A useful comparison matrix is:
> 
> | Model state | Same deterministic inference function |
> | --- | --- |
> | Original SFT checkpoint | evaluate |
> | SFT loaded through the DPO code path | evaluate |
> | SFT wrapped by trainer, no optimizer step | evaluate |
> | Zero-update saved checkpoint, reloaded | evaluate |
> | Early DPO checkpoint in memory | evaluate |
> | Same early checkpoint reloaded | evaluate |
> | Final DPO model in memory | evaluate |
> | Same final checkpoint reloaded | evaluate |
> | Merged export, if used | evaluate |
> 
> This isolates where the first behavioral difference appears.
> 
> For a deterministic sanity check, keep constant:
> 
> ```plaintext
> prompt messages
> chat template
> add_generation_prompt behavior
> tokenizer files
> BOS/EOS/pad tokens
> max_new_tokens
> stopping criteria
> greedy decoding or fixed sampling seed
> dtype
> quantization
> device/backend
> 
> ```
> 
> The [Transformers generation documentation](https://huggingface.co/docs/transformers/main_classes/text_generation) notes that generation behavior is controlled by `GenerationConfig`. A different saved or loaded generation config can therefore change output independently of training.
> 
> With PEFT, the [checkpoint-format documentation](https://huggingface.co/docs/peft/developer_guides/checkpoint) is particularly relevant: an adapter checkpoint normally contains adapter parameters, not the base model itself. Reloading therefore depends on the correct base model and configuration.
> 
> I would preserve:
> 
> ```plaintext
> base model repository
> exact base model revision/commit
> adapter checkpoint
> adapter_config.json
> active adapter name
> merged/unmerged state
> tokenizer revision
> chat template
> generation_config.json
> library versions
> 
> ```
> 
> Common artifact-level confounders include:
> 
> - loading the DPO adapter over a different base revision;
> - loading the base model without the SFT adapter expected by DPO;
> - accidentally activating the reference adapter;
> - merging an adapter twice;
> - comparing a merged model with an unmerged model under different precision;
> - saving only the adapter, then evaluating the base model alone;
> - tokenizer or chat-template files coming from a different repository;
> - generation config changing sampling, EOS, or minimum-length behavior.
> 
> A failure here would not mean DPO training was healthy. It would mean the observed deployed collapse cannot yet be attributed to DPO updates, dataset semantics, or beta.

> **7. How I would interpret the main outcomes**
>
> | Observation | What it supports | What it does **not** establish |
> | --- | --- | --- |
> | Loading/wrapping changes SFT output before training | adapter activation, train/eval mode, dtype, tokenizer/template, or inference differences | dataset or beta as the cause |
> | Zero-update save/reload changes output | serialization, base/adapter pairing, merge state, generation config | DPO optimization failure |
> | Chosen/rejected prompt prefixes differ | rendering or preprocessing contract problem | that the preference labels themselves are wrong |
> | Completion masks or EOS positions are wrong | loss applied to unintended tokens | that correcting masks will necessarily solve every symptom |
> | Same-SFT policy/reference differ substantially at step 0 | reference, adapter, preprocessing, mode, or cache mismatch | which side is correct, or that the mismatch alone explains all degradation |
> | Chosen and rejected absolute log-probs both fall while margin rises | relative improvement with absolute likelihood displacement | that pair similarity is definitely the cause |
> | Chosen rises, rejected falls, ordinary chat falls | optimization/coverage problem despite success on preference pairs | beta alone as the cause |
> | Only long conversations fail | truncation, context coverage, trajectory or role-boundary issue | insufficient model size |
> | Only systemless conversations fail | format or distribution dependence | that every DPO example must contain a system message |
> | Training-time model works, reload fails | artifact/inference path | healthy training in all other respects |
> | 7B and 32B both fail | weakens a simple capacity explanation | proves an implementation-independent DPO failure |
> | Custom trainer and Swift both fail | weakens a single-framework-only explanation | rules out shared preprocessing, reference, optimization, or artifact errors |
> | SFT-loss mixing does not fix it | weakens “just add SFT regularization” | rules out excessive updates or narrow data coverage |
> 
> The general rule I would use is:
> 
> > A symptom can nominate a branch, but it should not be promoted to a root cause until a control separates it from the neighboring branches.

> **8. A compact case record for comparing post-DPO degradation reports**
>
> Since you mentioned researching preference training on open-weight models, these are the fields I would find most useful for comparing cases:
> 
> ```plaintext
> Model:
> Exact base revision:
> Exact SFT checkpoint:
> 
> Trainer/framework:
> Exact version or commit:
> Relevant custom modifications:
> 
> Full fine-tuning or PEFT:
> Trainable parameter count/fraction:
> LoRA rank/alpha/target modules:
> 
> Policy construction:
> Reference construction:
> Active adapters:
> Merged/unmerged state:
> Reference log-probs cached or live:
> 
> Dataset source:
> Chosen response source:
> Rejected response source:
> Number of pairs:
> Clear/ambiguous/conflicting pair sample:
> Chosen/rejected token-overlap distribution:
> Chosen/rejected length distribution:
> 
> Chat template:
> Rendered pair example:
> Prompt-prefix equality check:
> BOS/EOS/control tokens:
> Completion masks:
> Maximum lengths:
> Truncation rule:
> Effective completion-token distribution:
> 
> Beta:
> Learning rate:
> Batch size:
> Gradient accumulation:
> Optimizer steps:
> Effective epochs:
> Processed tokens:
> SFT-loss weight:
> Gradient norm trend:
> 
> Step-zero policy/reference comparison:
> Chosen absolute log-prob trend:
> Rejected absolute log-prob trend:
> Reward-margin trend:
> Entropy/repetition trend:
> 
> Ordinary-chat holdout:
> Capability-boundary holdout:
> Multi-turn holdout:
> First checkpoint showing degradation:
> 
> Original SFT output:
> Trainer-wrapped zero-update output:
> Zero-update reload output:
> Early in-memory output:
> Early reloaded output:
> Final in-memory output:
> Final reloaded output:
> 
> Inference function:
> Generation config:
> Tokenizer revision:
> EOS/pad/stop settings:
> Precision/quantization:
> 
> ```
> 
> Not every report needs every field, but this schema makes “dataset,” “beta,” “reference,” “format,” and “artifact” claims meaningfully comparable.

So my current answer would be: **I cannot tell from the thread whether the original cause was ever identified, and the available observations do not justify choosing dataset or beta as the answer.**

The default route I would use is:

```plaintext
1. establish identical deterministic inference;
2. compare SFT before and after DPO-path wrapping;
3. run a zero-update save/reload control;
4. inspect actual rendered tokens, masks, EOS, and truncation;
5. verify the intended step-zero policy/reference relationship;
6. track chosen and rejected absolute log-probs separately;
7. evaluate ordinary chat and capability boundaries from the first few steps;
8. only then stratify by pair geometry or sweep optimization settings.

```

That sequence should usually reveal whether the case is primarily moving toward a sequence/reference bug, an artifact problem, a preference-data geometry problem, or genuine overoptimization—without assuming in advance that “DPO destroyed the model” and without treating a better reward margin as proof that conversational quality was preserved.
