# DPOConfig - SFT as loss function

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdpoconfig-sft-as-loss-function%2F168490%3E%5C
**Category:** Intermediate\
**Created:** [September 17, 2025, 4:23am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdpoconfig-sft-as-loss-function%2F168490 "2025-09-17T04:23:41Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Kathi00](https://avatars.discourse-cdn.com/v4/letter/k/e99b99/32.png) [@Kathi00](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FKathi00)%5C
**Post date:** [September 17, 2025, 4:23am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdpoconfig-sft-as-loss-function%2F168490%2F1 "2025-09-17T04:23:41Z")

</div>

Hello everyone, new to DPOConfig. I have quick question regarding **`DPOConfig`**. It provides the possibility to choose between different loss functions, one of them being **`sft`**.

Something I don’t understand: DPO is a method based on _“chosen”_ and _“rejected”_, but SFT on the other hand works only with _“chosen”_ without _“rejected”_.

What exactly is then plotted in **`train/reward/rejected`**? How does DPO work with SFT?

Best regards, Kathi

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C
**Post date:** [September 17, 2025, 8:24am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdpoconfig-sft-as-loss-function%2F168490%2F2 "2025-09-17T08:24:46Z")

</div>

> How does DPO work with SFT?

Seems [with some ways](https://huggingface.co/datasets/John6666/forum1/blob/main/dpo_sft.md).

---

<div class="post-metadata">

**Author:** ![Kathi00](https://avatars.discourse-cdn.com/v4/letter/k/e99b99/32.png) [@Kathi00](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FKathi00)%5C
**Post date:** [September 18, 2025, 5:36am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdpoconfig-sft-as-loss-function%2F168490%2F3 "2025-09-18T05:36:04Z")

</div>

Thank you, that is a really detailed description. I have one question: if I work with DPO, because I have three prompts (prompt, chosen, rejected), then ideally the loss _apo\_zero_ should be optimal.

I can see an improvement in the plots for reward/chosen…it looks similar to the SFT reward/chosen. But since it now also takes the rejected examples into consideration, I see negative values, which do not appear in SFT.

However, when I check the model outputs, I notice quite a bit of repetition and randomness. I would have expected it to perform better than SFT, since it has more information to train on. But _apo\_zero_ is not as good as in SFT. Why is it like this?

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C
**Post date:** [September 18, 2025, 6:20am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdpoconfig-sft-as-loss-function%2F168490%2F4 "2025-09-18T06:20:58Z")

</div>

Hmm… idk…

* * *

Short answer: DPO/APO optimize a **pairwise margin** , not absolute likelihood. With `apo_zero` you **push winners up and losers down** , which can widen the margin while **lowering the absolute prob. of some winner tokens** and amplifying verbosity/length biases. That can look better in `rewards/*` plots yet decode worse than SFT and repeat more. This is expected and documented. ([Hugging Face](https://huggingface.co/docs/trl/main/en/dpo_trainer "DPO Trainer"))

Why `apo_zero` can underperform SFT and cause repetition/randomness

1. **Objective mismatch.** Standard DPO-type losses maximize a _difference_; they need not increase the winner’s likelihood everywhere. When chosen/rejected are very similar, vanilla DPO can even reduce chosen likelihood while still improving the pairwise logit. DPOP (a fix) was proposed for this exact failure mode. ([arXiv](https://arxiv.org/pdf/2402.13228 "Fixing Failure Modes of Preference Optimisation with DPO- ..."))

2. **APO anchoring is situational.** `apo_zero` assumes winners are better than your model and thus pushes winners ↑ and losers ↓. If your model is already stronger than many winners, the right variant is `apo_down` (decrease both, penalize losers more). Using the wrong anchor can harm quality. ([Hugging Face](https://huggingface.co/docs/trl/main/en/dpo_trainer "DPO Trainer"))

3. **Preference noise.** Flipped or ambiguous labels make DPO drift. Robust/cDPO variants or label smoothing mitigate this. TRL exposes `label_smoothing` and “Robust DPO.” ([ar5iv](https://ar5iv.org/abs/2403.00409 "[2403.00409] Provably Robust DPO: Aligning Language Models with Noisy Feedback"))

4. **Length/verbosity bias.** DPO magnifies length sensitivity; longer answers win, which increases repetition and randomness at decode time unless controlled. Use LD-DPO/SamPO-style regularizers. TRL ships LD-DPO. ([ACL Anthology](https://aclanthology.org/2024.findings-acl.297/ "Disentangling Length from Quality in Direct Preference ..."))

5. **β and reference sensitivity.** DPO-style methods are sensitive to β and reference choice; poor settings yield unstable gradients and worse decoding. This has been observed empirically in small-compute settings. ([cs224r.stanford.edu](https://cs224r.stanford.edu/projects/pdfs/CS224R_Final_Report__1_%20%281%29.pdf "Direct vs Adversarial Direct Preference Optimization (DPO vs ..."))

6. **It’s a known outcome.** Practitioners have reported “better margins, worse generations” when pairs are too similar or data is small; mixing SFT back in helped. ([GitHub](https://github.com/rasbt/LLMs-from-scratch/discussions/394 "DPO training make model even worse · rasbt LLMs-from-scratch · Discussion #394 · GitHub"))

About the plots you see

- `rewards/chosen` and `rewards/rejected` are **policy–reference log-prob gaps** scaled by β. Negative values only mean “policy assigns lower log-prob than ref on that side,” not that training failed. Under `apo_zero`, pushing losers down often drives `rewards/rejected` sharply negative even if `rewards/chosen` wiggles. ([Hugging Face](https://huggingface.co/docs/trl/main/en/dpo_trainer "DPO Trainer"))

What to change that usually fixes it

- **Pick the right anchor.** If many “winners” are worse than your current model, switch to `loss_type="apo_down"`. If winners are truly better, keep `apo_zero`. ([Hugging Face](https://huggingface.co/docs/trl/main/en/dpo_trainer "DPO Trainer"))

- **Add positivity pressure.** Try DPOP or add an SFT term: `rpo_alpha≈1.0` in TRL or multi-loss `["sigmoid","sft"]`. This counters the “margin-only” failure and stabilizes decoding. ([arXiv](https://arxiv.org/pdf/2402.13228 "Fixing Failure Modes of Preference Optimisation with DPO- ..."))

- **Harden against noise.** Set `label_smoothing>0` or use the robust DPO loss when label flips are likely. ([ar5iv](https://ar5iv.org/abs/2403.00409 "[2403.00409] Provably Robust DPO: Aligning Language Models with Noisy Feedback"))

- **Control verbosity.** Enable LD-DPO (set `ld_alpha`) or adopt a length-regularized scheme like SamPO. ([Hugging Face](https://huggingface.co/docs/trl/main/en/dpo_trainer "DPO Trainer"))

- **Mind decoding.** Evaluate apples-to-apples with stable decoding (lower temperature, nucleus sampling, optional repetition penalty). Many “randomness” complaints come from sampler drift, not the policy. ([Hugging Face](https://huggingface.co/docs/transformers/en/generation_strategies "Generation strategies"))

- **Use ref syncing if drifting.** TRL exposes reference-model syncing callbacks to reduce KL drift during DPO-style training. ([Hugging Face](https://huggingface.co/docs/trl/main/en/dpo_trainer "DPO Trainer"))

---

<div class="post-metadata">

**Author:** ![Kathi00](https://avatars.discourse-cdn.com/v4/letter/k/e99b99/32.png) [@Kathi00](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FKathi00)%5C
**Post date:** [September 21, 2025, 6:47am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdpoconfig-sft-as-loss-function%2F168490%2F5 "2025-09-21T06:47:29Z")

</div>

Oh, okay I get it. Sorry for asking too many questions, and thanks for answering them. One doubt regarding my initial question: I chose DPOConfig with loss\_type=“sft” (I have to and also “apo\_zero). I need to vary beta (and learning rate) to train a model. If I understand correctly, beta does not affect the SFT formula. Based on the plotted formula, the diagnostics will reflect changes in beta because it’s included in it. But does a model trained with different beta values reflect those changes in SFT? No, right?

I trained models with different beta values and don’t see significant changes in their outputs. I’m trying to understand the logic between plots and trained models. The plots aren’t reflecting my model’s output (e.g., I see higher reward/chosen values for higher beta, but my model’s outputs aren’t different from smaller beta…).

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C
**Post date:** [September 21, 2025, 7:34am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdpoconfig-sft-as-loss-function%2F168490%2F6 "2025-09-21T07:34:18Z")

</div>

NP

> But does a model trained with different beta values reflect those changes in SFT? No, right?

Seems [right](https://huggingface.co/datasets/John6666/forum1/blob/main/dpo_sft_2.md).
