# Attention mask shape (custom attention masking)

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fattention-mask-shape-custom-attention-masking%2F104272%3E%5C
**Category:** 🤗Transformers\
**Created:** [August 27, 2024, 8:53pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fattention-mask-shape-custom-attention-masking%2F104272 "2024-08-27T20:53:24Z")\
**Posts on this page:** 1\
**Showing post:** 2

<div class="post-metadata">

**Author:** ![AnneTheBoo](https://avatars.discourse-cdn.com/v4/letter/a/c5a1d2/32.png) [@AnneTheBoo](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FAnneTheBoo)%5C
**Post date:** [February 17, 2025, 3:38pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fattention-mask-shape-custom-attention-masking%2F104272%2F2 "2025-02-17T15:38:19Z")

</div>

> [@alex-hh](#):
>
> Is this correct? Should this be documented? Is there anything to watch out for when doing this? (I’m interested in providing a custom attention masking pattern to the Llama architecture).

Hi,  
yes I passed a custom mask in 4D and it worked 🙂 (confirmed with the attention scores)

Relating to your question - did you perhaps find out how to answer this question regarding additive/binary attention mask?

> [@Does Llama-2 use additive attention masking?](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdoes-llama-2-use-additive-attention-masking%2F140399)%3A
>
> Hi, I need to build a custom causal attention mask for meta-llama/Llama-2-7b-chat-hf from LlamaForCausalML. Can someone confirm that it uses additive masking (0 or large negative number close to -inf) instead of binary masking (0 or 1) for its attention mechanism? I found that binary masking doesn’t work when I investigated the attention scores, whereas additive masking actually lead to attention scores being 0 in the right dimensions. But on the other hand the documentation says it uses bina…

---

_[View the full topic](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fattention-mask-shape-custom-attention-masking%2F104272)._
