# \`additional\_special\_tokens\` are not added

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fadditional-special-tokens-are-not-added%2F93192%3E%5C
**Category:** 🤗Tokenizers\
**Created:** [June 20, 2024, 12:55pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fadditional-special-tokens-are-not-added%2F93192 "2024-06-20T12:55:40Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Yao-Lirong](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/yao-lirong/32/28453_2.png) [@Yao-Lirong](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FYao-Lirong)%5C
**Post date:** [June 20, 2024, 12:55pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fadditional-special-tokens-are-not-added%2F93192%2F1 "2024-06-20T12:55:40Z")

</div>

Hi Hugging Face Community,

I have the following questions regarding special tokens:

1. **Why doesn’t `tokenizer.all_special_tokens` include `<image>` token?** I’m using the LLaVA model that has `<image>` as a special token (as defined in [`added_tokens_decoder` of tokenizer\_config.json](https://huggingface.co/llava-hf/llava-v1.6-mistral-7b-hf/blob/main/tokenizer_config.json#L30-L37)). The tokenizer encodes and decodes it indeed as a special token. However, when I load in its tokenizer and call `tokenizer.all_special_tokens` or `tokenizer.additional_special_tokens`, `<image>` token is not included.

2. **Where is `<image>` token loaded?** I looked into the [`tokenizer.from_pretrained` function](https://github.com/huggingface/transformers/blob/6d4306160ab22c54a677e07f88c7d8808b137d38/src/transformers/tokenization_utils_base.py#L2317) but there doesn’t seem to be a place to actually read in the `added_tokens_decoder` where in config file this special token is defined ?

3. **Where is `tokenizer.decode` function defined as a special token?** I tried to break-point into it to find how it skipped `<image>` as a special token but I seem to get into a loop call between `tokenization_utils_base.py` and `tokenization_utils_fast.py`

It would be really helpful if you give an answer to any of these questions. Thank you very much!

---

<div class="post-metadata">

**Author:** ![Yao-Lirong](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/yao-lirong/32/28453_2.png) [@Yao-Lirong](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FYao-Lirong)%5C
**Post date:** [June 20, 2024, 12:58pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fadditional-special-tokens-are-not-added%2F93192%2F2 "2024-06-20T12:58:10Z")

</div>

I’m not allowed to paste more than 2 links in a post so I will provide the codes related to question 3 where I got into a loop.  
[tokenization\_utils\_base.py](https://github.com/huggingface/transformers/blob/6d4306160ab22c54a677e07f88c7d8808b137d38/src/transformers/tokenization_utils_base.py#L3906-L3911) and [tokenization\_utils\_fast.py](https://github.com/huggingface/transformers/blob/6d4306160ab22c54a677e07f88c7d8808b137d38/src/transformers/tokenization_utils_fast.py#L651)
