# Why models(llama in particular) upcasts softmax to fp32?

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwhy-models-llama-in-particular-upcasts-softmax-to-fp32%2F44787%3E%5C
**Category:** 🤗Transformers\
**Created:** [June 28, 2023, 5:19am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwhy-models-llama-in-particular-upcasts-softmax-to-fp32%2F44787 "2023-06-28T05:19:40Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Maykeye](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/maykeye/32/30283_2.png) [@Maykeye](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FMaykeye)%5C
**Post date:** [June 28, 2023, 5:19am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwhy-models-llama-in-particular-upcasts-softmax-to-fp32%2F44787%2F1 "2023-06-28T05:19:40Z")

</div>

Consider the following:

> <https://github.com/huggingface/transformers/blob/12240925cfa29fff932e49927eb9744713ab1018/src/transformers/models/llama/modeling_llama.py#L232>

For my laptop’s 3080Ti it’s difference between getting OoM at ~1K context on open\_llama\_3b loaded in bf16 and not getting OoM, sitting at 14GB VRAM used.

Is bf16 that unstable and prone to returning nans/infs? I removed upscale and didn’t notice the difference(other than not getting OoM).

---

<div class="post-metadata">

**Author:** ![joaogante](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/joaogante/32/20106_2.png) [@joaogante](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fjoaogante)%5C
**Post date:** [June 29, 2023, 10:19am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwhy-models-llama-in-particular-upcasts-softmax-to-fp32%2F44787%2F2 "2023-06-29T10:19:46Z")

</div>

(see this [related GH issue](https://github.com/huggingface/transformers/issues/24519))
