# \[Stable Diffusion\] Error in "In Painting" pipeline

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fstable-diffusion-error-in-in-painting-pipeline%2F23255%3E%5C
**Category:** 🧨 Diffusers\
**Created:** [September 17, 2022, 10:26am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fstable-diffusion-error-in-in-painting-pipeline%2F23255 "2022-09-17T10:26:31Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![CristoJV](https://avatars.discourse-cdn.com/v4/letter/c/71e660/32.png) [@CristoJV](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FCristoJV)%5C
**Post date:** [September 17, 2022, 10:26am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fstable-diffusion-error-in-in-painting-pipeline%2F23255%2F1 "2022-09-17T10:26:31Z")

</div>

Hello!  
Why in the “In Painting” pipeline the masking is done in the latents and not in the decoded VAE versions?

```python
295 latents = (init_latents_proper * mask) + (latents * (1 - mask))

```

If this is correct, How the mask is mapped into the latent space? Are pixels original locations (clustered) reflected on expected positions in the latents space?

Should not be the algoritm the following?

1. Retrieve the latents.
2. Decode the latents.
3. Mix both images (original and decoded version) using the mask.
4. Encode the image to obtain the new latents.

Instead of:

1. Retrieve the latents.
2. Mix the original latents (with noise added corresponding to the timestep) with the latents.

---

<div class="post-metadata">

**Author:** ![anton-l](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/anton-l/32/2771_2.png) [@anton-l](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fanton-l)%5C
**Post date:** [September 21, 2022, 8:19pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fstable-diffusion-error-in-in-painting-pipeline%2F23255%2F2 "2022-09-21T20:19:21Z")

</div>

Hi @CristoJV! As the diffusion process for Stable Diffusion works exclusively with the VAE latents, the masks received by the inpainting pipeline are getting reampled from 512x512 to 64x64 to mask the latents.

---

<div class="post-metadata">

**Author:** ![CristoJV](https://avatars.discourse-cdn.com/v4/letter/c/71e660/32.png) [@CristoJV](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FCristoJV)%5C
**Post date:** [September 22, 2022, 1:05pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fstable-diffusion-error-in-in-painting-pipeline%2F23255%2F3 "2022-09-22T13:05:01Z")

</div>

Hi @anton-I! Thank you for replying.

Should the inpainting pipeline be upgraded to add a further step mixing the original image and the generated but in the image domain.

The issue is that, although the image latents are preserved after the masking, the VAE’s encoding and decoding functions produces losses. For example, after retrieving non-modified faces (not affected by the mask) they look a little bit uglier or distorted. Maybe if there is another further step that mix both images using the mask after the generation the results would improve without interfering the pipeline operation.

---

<div class="post-metadata">

**Author:** ![Emulator000](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/emulator000/32/15575_2.png) [@Emulator000](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FEmulator000)%5C
**Post date:** [April 22, 2023, 11:46am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fstable-diffusion-error-in-in-painting-pipeline%2F23255%2F4 "2023-04-22T11:46:34Z")

</div>

Hi @CristoJV, I’m using SD 1.5 and in my code I just added a post-processing step in order to mix the original untouched image within the result decoded from VAE and the original mask (not downscaled) and I get a better result.

By the way, the output from VAE also differs in saturation and brightness and a slightly difference between the inpainted area and the original image is noticeable.

I’m guessing that the encode-decode process from VAE make the image to loses their original properties.

An idea that I’ll try for sure is dilating a bit the original mask in order to keep other border information from the decoded latent and then blend the luminosity of the latent and the original image; with this trick I think that I could achieve a better result.

By the way I just agree with you that the process should be upgraded in order to get better results.

For what matters, to me the inpainting process with Diffusers is just not usable for processing images with people faces, the distortion is fairly aggressive.

---

<div class="post-metadata">

**Author:** ![rauln](https://avatars.discourse-cdn.com/v4/letter/r/e9a140/32.png) [@rauln](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Frauln)%5C
**Post date:** [June 29, 2023, 10:23pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fstable-diffusion-error-in-in-painting-pipeline%2F23255%2F5 "2023-06-29T22:23:19Z")

</div>

Same problem here. Has anyone found a fix?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/hellohellohello/original/2X/d/de4155eb4aa4108ecb32a1389d7cc37ae69f88b7.png) [@system](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsystem)%5C
**Post date:** [March 7, 2024, 12:52pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fstable-diffusion-error-in-in-painting-pipeline%2F23255%2F6 "2024-03-07T12:52:48Z")

</div>

Closed on March 7, 2024, 12:52pm UTC

This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.
