# Key-value pair from attention layer of GPT2

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fkey-value-pair-from-attention-layer-of-gpt2%2F44803%3E%5C
**Category:** 🤗Transformers\
**Created:** [June 28, 2023, 7:57am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fkey-value-pair-from-attention-layer-of-gpt2%2F44803 "2023-06-28T07:57:11Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![dharmendra](https://avatars.discourse-cdn.com/v4/letter/d/9d8465/32.png) [@dharmendra](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fdharmendra)%5C
**Post date:** [June 28, 2023, 7:57am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fkey-value-pair-from-attention-layer-of-gpt2%2F44803%2F1 "2023-06-28T07:57:11Z")

</div>

I am using gpt2 pre-trained model and want to extract key-value pair of attention layer from all the 12 decoder blocks of this model.  
I am into the understanding of memory augmented large language models.  
is there any way to proceed for this?  
Thanks in advance