# Visualizing Attention Maps in SwinV2

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fvisualizing-attention-maps-in-swinv2%2F47113%3E%5C
**Category:** Models\
**Created:** [July 17, 2023, 8:00pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fvisualizing-attention-maps-in-swinv2%2F47113 "2023-07-17T20:00:08Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![VedaantJain](https://avatars.discourse-cdn.com/v4/letter/v/8dc957/32.png) [@VedaantJain](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FVedaantJain)%5C
**Post date:** [July 17, 2023, 8:00pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fvisualizing-attention-maps-in-swinv2%2F47113%2F1 "2023-07-17T20:00:08Z")

</div>

I have been trying to visualize attention maps of vision transformers, I was able to do so for ViT using the attention rollout method.

However when I tried doing so for SwinV2, I observed that the shapes of attention states tensors in Swinv2Large model were:  
torch.Size([16, 144, 144])  
torch.Size([4, 144, 144])  
torch.Size([1, 144, 144])  
torch.Size([1, 36, 36])

So, the attention matrices were of different sizes so could not be recursively multiplied to get the attention flow. I could do it for the first three attention states, but I am not sure that it is the right approach.

TLDR: I want to know how to visualize attention maps from SwinV2 transformers given that the shapes of attention are not the same. Is there a paper or a code repository I could refer to?  
Thank you for your help.

---

<div class="post-metadata">

**Author:** ![Dylonsword](https://avatars.discourse-cdn.com/v4/letter/d/8797f3/32.png) [@Dylonsword](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FDylonsword)%5C
**Post date:** [December 29, 2023, 8:07am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fvisualizing-attention-maps-in-swinv2%2F47113%2F2 "2023-12-29T08:07:55Z")

</div>

Hello, Had you deal with this problem?