# Can't load pre-trained tokenizer with additional new tokens

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcant-load-pre-trained-tokenizer-with-additional-new-tokens%2F8966%3E%5C
**Category:** 🤗Transformers\
**Created:** [August 4, 2021, 9:43pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcant-load-pre-trained-tokenizer-with-additional-new-tokens%2F8966 "2021-08-04T21:43:19Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![rlian](https://avatars.discourse-cdn.com/v4/letter/r/a88e57/32.png) [@rlian](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Frlian)%5C
**Post date:** [August 4, 2021, 9:43pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcant-load-pre-trained-tokenizer-with-additional-new-tokens%2F8966%2F1 "2021-08-04T21:43:19Z")

</div>

I first pretrained masked language model by adding additional list of words to the tokenizer. Then I saved the pretrained model and tokenizer.

```auto
tokenizer = AutoTokenizer.from_pretrained('bert-base-uncased')
model = AutoModelForMaskedLM.from_pretrained(
        'bert-base-uncased'
)
tokenizer.add_tokens(list_of_words)
model.resize_token_embeddings(len(tokenizer))

trainer.train()
model_to_save = model.module if hasattr(model, 'module') else model
model_to_save.save_pretrained(data_args.output_file)
tokenizer.save_pretrained(data_args.output_file)

```

After that, I want to load the pre-trained tokenizer and model by

```auto
tokenizer = BertTokenizer.from_pretrained(model_args.model_name_or_path)
encoder = BertModel.from_pretrained(model_args.model_name_or_path, num_labels=num_classes)
tokenizer.add_tokens(list_of_words)
encoder.resize_token_embeddings(len(tokenizer))

```

However, an error occurred as shown below and it seems that the pretrained tokenizer couldn’t be loaded correctly.

```auto
AssertionError: Non-consecutive added token 'IncelTears' found. Should have index 30525 but has index 30526 in saved vocabulary.

```

Does anyone have an idea on this? Thanks a lot.

---

<div class="post-metadata">

**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsgugger)%5C
**Post date:** [August 5, 2021, 6:48am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcant-load-pre-trained-tokenizer-with-additional-new-tokens%2F8966%2F2 "2021-08-05T06:48:11Z")

</div>

Which line threw the error? If it’s `tokenizer.add_tokens(list_of_words)`, it’s because your tokenizer already has those words added from the first sample, so you can’t re-add them.

---

<div class="post-metadata">

**Author:** ![rlian](https://avatars.discourse-cdn.com/v4/letter/r/a88e57/32.png) [@rlian](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Frlian)%5C
**Post date:** [August 5, 2021, 2:36pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcant-load-pre-trained-tokenizer-with-additional-new-tokens%2F8966%2F3 "2021-08-05T14:36:32Z")

</div>

> [@rlian](#):
>
> `tokenizer = BertTokenizer.from_pretrained(model_args.model_name_or_path)`

This line threw the error while loading the pre-trained the tokenizer. I wonder if I have to set **additional\_special\_tokens** while loading?

```auto
tokenizer = BertTokenizer.from_pretrained(model_args.model_name_or_path, additional_special_tokens=list_of_word)

```

---

<div class="post-metadata">

**Author:** ![kaankork](https://avatars.discourse-cdn.com/v4/letter/k/e9bcb4/32.png) [@kaankork](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fkaankork)%5C
**Post date:** [August 10, 2021, 8:19am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcant-load-pre-trained-tokenizer-with-additional-new-tokens%2F8966%2F4 "2021-08-10T08:19:30Z")

</div>

Hi @snugger I thought the original `tokenizer` just skips the words in `list_of_words` if tokenizer already has these words in it.

Is it necessary to pre-process this `list_of_words` before using the `add_tokens` method so that `list_of_words` consists of only new words?
