# Tiny fine tune messed up the pretrained sd v1-4 model

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftiny-fine-tune-messed-up-the-pretrained-sd-v1-4-model%2F35484%3E%5C
**Category:** 🧨 Diffusers\
**Created:** [April 4, 2023, 5:24am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftiny-fine-tune-messed-up-the-pretrained-sd-v1-4-model%2F35484 "2023-04-04T05:24:31Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![plumblossom](https://avatars.discourse-cdn.com/v4/letter/p/db5fbb/32.png) [@plumblossom](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fplumblossom)%5C
**Post date:** [April 4, 2023, 5:24am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftiny-fine-tune-messed-up-the-pretrained-sd-v1-4-model%2F35484%2F1 "2023-04-04T05:24:31Z")

</div>

newbie to diffusers, i’m following the instruction in [diffusers/examples/text\_to\_image at main · huggingface/diffusers · GitHub](https://github.com/huggingface/diffusers/tree/main/examples/text_to_image)

made some small change in training script and inference

training

```auto
export MODEL_NAME='CompVis/stable-diffusion-v1-4'

export dataset_name="lambdalabs/pokemon-blip-captions"
# using CPU
accelerate launch --mixed_precision="no" train_text_to_image.py \
  --pretrained_model_name_or_path=$MODEL_NAME \
  --dataset_name=$dataset_name \
  --use_ema \
  --resolution=512 --center_crop --random_flip \
  --train_batch_size=1 \
  --gradient_accumulation_steps=4 \
  --gradient_checkpointing \
  --max_train_steps=10 \
  --learning_rate=1e-03 \
  --max_grad_norm=1 \
  --lr_scheduler="constant" --lr_warmup_steps=0 \
  --output_dir="sd-pokemon-model" 

```

inference

```auto
pipe = StableDiffusionPipeline.from_pretrained(model_path, safety_checker=None, requires_safety_checker=False)
pipe = pipe.to("cpu")
# Recommended if your computer has < 64 GB of RAM
pipe.enable_attention_slicing()

```

however, the 10-step continuously training made the model totally messy, the inference just returned something pure noise

question: with pokemon data, even only 10 steps training, the result became totally messy, anything incompatible for fine tune?

btw, changed to 0 steps, everything works fine, the inference returns the good result as pretrained one, so the model loading and saving parts are good.

thanks!

---

<div class="post-metadata">

**Author:** ![sayakpaul](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sayakpaul/32/23772_2.png) [@sayakpaul](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsayakpaul)%5C
**Post date:** [April 6, 2023, 3:20am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftiny-fine-tune-messed-up-the-pretrained-sd-v1-4-model%2F35484%2F2 "2023-04-06T03:20:44Z")

</div>

Could you provide the prompts you used for comparison and the images you got out from the prompts?

0 training step means that there was no fine-tuning. It might so have happened that the pre-trained model generated a plausible image w.r.t the input prompt you had given.

---

<div class="post-metadata">

**Author:** ![plumblossom](https://avatars.discourse-cdn.com/v4/letter/p/db5fbb/32.png) [@plumblossom](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fplumblossom)%5C
**Post date:** [April 8, 2023, 12:41am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftiny-fine-tune-messed-up-the-pretrained-sd-v1-4-model%2F35484%2F3 "2023-04-08T00:41:34Z")

</div>

i’m using random prompts, e.g. “yoda”, “a warrior on horse”, etc.  
the issue here is that, after loading the pretrained model, just a few steps (even with small learning rate in a small dataset) would mess up the entire model, that looks unexpected. i doubt any config or format issue.
