# Can SFT and DPO be done at the same time?

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcan-sft-and-dpo-be-done-at-the-same-time%2F88545%3E%5C
**Category:** Beginners\
**Created:** [May 26, 2024, 7:50am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcan-sft-and-dpo-be-done-at-the-same-time%2F88545 "2024-05-26T07:50:22Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Hamana0509](https://avatars.discourse-cdn.com/v4/letter/h/8baadc/32.png) [@Hamana0509](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FHamana0509)%5C
**Post date:** [May 26, 2024, 7:50am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcan-sft-and-dpo-be-done-at-the-same-time%2F88545%2F1 "2024-05-26T07:50:22Z")

</div>

I want to do SFT and DPO at the same time.

What I mean is I want to SFT model a few steps, and then I use that same model to do the DPO.

Or create a Datacollator of about 15 samples to perform SFT. After performing SFT of all 15 samples, I made another Datacollator of about 5 samples to perform DPO, repeating the above process until I had finished running all the data sets.

Does anyone have any ideas that they can suggest to me?

---

<div class="post-metadata">

**Author:** ![wjmcat](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/wjmcat/32/28812_2.png) [@wjmcat](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fwjmcat)%5C
**Post date:** [June 29, 2024, 1:49am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcan-sft-and-dpo-be-done-at-the-same-time%2F88545%2F2 "2024-06-29T01:49:31Z")

</div>

the answer is NO.

I wonder why you ask this question. SFT doesn’t contain reference model, so you just load the only one full-weight model to finetune. But dpo will load the model and forzen it.

difference structure, difference loss, difference codebase. so you should not do it.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/hellohellohello/original/2X/d/de4155eb4aa4108ecb32a1389d7cc37ae69f88b7.png) [@system](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsystem)%5C
**Post date:** [June 29, 2024, 1:50pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcan-sft-and-dpo-be-done-at-the-same-time%2F88545%2F3 "2024-06-29T13:50:14Z")

</div>

Closed on June 29, 2024, 1:50pm UTC

This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.
