# \[Question\] Why does vocab size determine training parameters

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fquestion-why-does-vocab-size-determine-training-parameters%2F8961%3E%5C
**Category:** Beginners\
**Created:** [August 4, 2021, 2:20pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fquestion-why-does-vocab-size-determine-training-parameters%2F8961 "2021-08-04T14:20:07Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Andrija](https://avatars.discourse-cdn.com/v4/letter/a/aca169/32.png) [@Andrija](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FAndrija)%5C
**Post date:** [August 4, 2021, 2:20pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fquestion-why-does-vocab-size-determine-training-parameters%2F8961%2F1 "2021-08-04T14:20:07Z")

</div>

I have a not so smart question: **Why does the vocab size increase training parameters by a lot?**  
The following configuration:

```auto
from transformers import RobertaConfig
config = RobertaConfig(
    vocab_size=48000,
    max_position_embeddings=514,
    num_attention_heads=12,
    num_hidden_layers=6,
    type_vocab_size=1,
)

```

Passed to the model gives 80 mil parameters

```auto
model = RobertaForMaskedLM(config=config)
model.num_parameters() → 80.4 milion

```

If vocab\_size is reduced to 16k,

```auto
model = RobertaForMaskedLM(config=config)
model.num_parameters() → 55 milion

```

The Embedding layers at the start increases training parameters so much?

```auto
        self.word_embeddings = nn.Embedding(config.vocab_size, config.hidden_size, padding_idx=config.pad_token_id)
        self.position_embeddings = nn.Embedding(config.max_position_embeddings, config.hidden_size)
        self.token_type_embeddings = nn.Embedding(config.type_vocab_size, config.hidden_size)

```

The layer embeding takes:

> Embedding(vocab\_size, embedding\_space, input\_length=max\_length))

Where in my case embedding\_space is 768 (where 768\*48000 - 768 \* 16000 ~ 25 mil.)  
Therefore the number of training parameters would be vocab\_size \* embedding\_space?

---

<div class="post-metadata">

**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsgugger)%5C
**Post date:** [August 4, 2021, 3:04pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fquestion-why-does-vocab-size-determine-training-parameters%2F8961%2F2 "2021-08-04T15:04:05Z")

</div>

I’m not sure where you question is, all your math is correct and yes, the embedding matrix is responsible for a looot of the model parameters.

---

<div class="post-metadata">

**Author:** ![Andrija](https://avatars.discourse-cdn.com/v4/letter/a/aca169/32.png) [@Andrija](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FAndrija)%5C
**Post date:** [August 5, 2021, 11:02am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fquestion-why-does-vocab-size-determine-training-parameters%2F8961%2F3 "2021-08-05T11:02:11Z")

</div>

Thanks for confirming my check!
