# Explicit inputs\_embeds and vocab\_size=1 in GPT2

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fexplicit-inputs-embeds-and-vocab-size-1-in-gpt2%2F72875%3E%5C
**Category:** Beginners\
**Created:** [February 12, 2024, 6:42pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fexplicit-inputs-embeds-and-vocab-size-1-in-gpt2%2F72875 "2024-02-12T18:42:58Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![Jeremy9959](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/jeremy9959/32/23792_2.png) [@Jeremy9959](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJeremy9959)%5C
**Post date:** [February 12, 2024, 6:42pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fexplicit-inputs-embeds-and-vocab-size-1-in-gpt2%2F72875%2F1 "2024-02-12T18:42:58Z")

</div>

Hi all,

If I plan to use a GPT2 Base Model and call the forward method with an explicit inputs\_embeds,  
does that mean I can set vocab\_size=1? So for example if, by some preprocessing,  
I’ve converted my length L text T into a 1 x L x 768 tensor E, then am I right that model(inputs\_embeds=E) combines my explicit E with the positional embeddings in model.wpe but ignores model.wte?

If this is not right, what is the relationship between vocab\_size and explicit embeddings?

Thanks!