# How to determine if a token is special

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-determine-if-a-token-is-special%2F152329%3E%5C
**Category:** 🤗Tokenizers\
**Created:** [April 25, 2025, 4:54pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-determine-if-a-token-is-special%2F152329 "2025-04-25T16:54:56Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![BoltzmachineQ](https://avatars.discourse-cdn.com/v4/letter/b/e47c2d/32.png) [@BoltzmachineQ](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FBoltzmachineQ)%5C
**Post date:** [April 25, 2025, 4:54pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-determine-if-a-token-is-special%2F152329%2F1 "2025-04-25T16:54:56Z")

</div>

For example, for llama 3.1 tokenizer, `"<|start_header_id|>` is a special token but is not in `all_special_tokens`.

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C
**Post date:** [April 26, 2025, 5:55am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-determine-if-a-token-is-special%2F152329%2F2 "2025-04-26T05:55:23Z")

</div>

It seems more complicated than I thought, or rather, it seems to be tangled up.

> <https://github.com/meta-llama/llama3/issues/77>
>
> Apologies in case this is documented somewhere and I missed it:
> 
> I notice that… there are 250 "reserved special tokens" defined in the tokenizer. Is there any information available on what these are meant for, and what users are supposed to (not) do with them? For instance, could one use some of these tokens in finetunes (instead of adding additional tokens and resizing the vocabulary), or would that be problematic?
> 
> Thanks so much!

> <https://github.com/huggingface/transformers/issues/16787>
>
> \## Environment info
> \- \`transformers\` version: 4.18.0
> \- Platform: macOS-12.0.1-…arm64-arm-64bit
> \- Python version: 3.8.13
> \- Huggingface\_hub version: 0.4.0
> \- PyTorch version (GPU?): 1.11.0 (False)
> \- Tensorflow version (GPU?): not installed (NA)
> \- Flax version (CPU?/GPU?/TPU?): not installed (NA)
> \- Jax version: not installed
> \- JaxLib version: not installed
> \- Using GPU in script?: No
> \- Using distributed or parallel set-up in script?: No
> 
> \### Who can help
> Tokenizers: @SaulLu
> 
> \## Information
> Using the BERT tokenizer and wanted to add my own special tokens, but am not getting the expected behavior
> 
> \## To reproduce
> 
> \`\`\`
> tokenizer = BertTokenizer.from\_pretrained("bert-base-uncased")
> n\_added\_tokens = tokenizer.add\_tokens(\["\<CONCEPT\>", "\</CONCEPT\>"\], special\_tokens=True)
> print(n\_added\_tokens)
> context = "Hi I am taking \<CONCEPT\> Advil \</CONCEPT\>"
> context\_tokens = tokenizer(context).input\_ids
> print(tokenizer.convert\_ids\_to\_tokens(context\_tokens))
> context = "\<CONCEPT\>"
> context\_tokens = tokenizer(context).input\_ids
> print(tokenizer.convert\_ids\_to\_tokens(context\_tokens))
> \`\`\`
> 
> \*\*Output:\*\*
> \`\`\`
> 2
> \['\[CLS\]', 'hi', 'i', 'am', 'taking', '\<', 'concept', '\>', 'ad', '##vil', '\<', '/', 'concept', '\>', '\[SEP\]'\]
> \['\[CLS\]', '\<CONCEPT\>', '\[SEP\]'\]
> \`\`\`
> 
> \## Expected behavior
> \*\*Expected Output:\*\*
> \`\`\`
> 2
> \['\[CLS\]', 'hi', 'i', 'am', 'taking', '\<CONCEPT\>', 'ad', '##vil', '\</CONCEPT\>', '\[SEP\]'\]
> \['\[CLS\]', '\<CONCEPT\>', '\[SEP\]'\]
> \`\`\`
> 
> Seems like when we pass the special token alone it somehow works, but not in more text .

> <https://stackoverflow.com/questions/73322462/how-to-add-all-standard-special-tokens-to-my-hugging-face-tokenizer-and-model>

---

<div class="post-metadata">

**Author:** ![e950280](https://avatars.discourse-cdn.com/v4/letter/e/3be4f8/32.png) [@e950280](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fe950280)%5C
**Post date:** [April 29, 2025, 2:53am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-determine-if-a-token-is-special%2F152329%2F3 "2025-04-29T02:53:42Z")

</div>

It’s not just the training that’s a black box; for intermediate learners, every step forward is a struggle.

Taking the model accessed via Unsloth as an example, you can find the token with id=“128006” (\<|start\_header\_id|\>) in the tokenizer.json file. One of its attributes is “special”: true.

There is a not-so-smart way to verify this: whether using llama-cpp or directly using transformers, during inference, set skip\_special\_tokens=False and print the output. However, this method might be more suitable when you are adding your own special tokens.  
e.g.` text_output = tokenizer.decode(_[0], skip_special_tokens=False)`

However, what I’m not sure about is why it doesn’t appear in special\_tokens\_map.json. One explanation I’ve heard is that this file only contains certain special tokens, but I’m not completely sure.
