# Convert\_tokens\_to\_ids produces \<unk\>

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fconvert-tokens-to-ids-produces-unk%2F24771%3E%5C
**Category:** 🤗Tokenizers\
**Created:** [October 21, 2022, 10:45am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fconvert-tokens-to-ids-produces-unk%2F24771 "2022-10-21T10:45:46Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![AfonsoSousa](https://avatars.discourse-cdn.com/v4/letter/a/5f9b8f/32.png) [@AfonsoSousa](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FAfonsoSousa)%5C
**Post date:** [October 21, 2022, 10:45am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fconvert-tokens-to-ids-produces-unk%2F24771%2F1 "2022-10-21T10:45:46Z")

</div>

Hi. I am trying to tokenize single words with a Roberta BPE Sub-word tokenizer. I was expecting to have some words with multiple ids, but when that is supposed to be the case, the method `convert_tokens_to_ids` just returns `<unk>`. However, ` __call__ ` from tokenizer produces the multiple ids. To reproduce the problem, run:

```
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("roberta-base")
token_id = tokenizer.convert_tokens_to_ids("exam")
print(f"{token_id} => {tokenizer.decode([token_id])}")
token_ids = tokenizer("exam").input_ids[1:3]
print(f"{token_ids} => {tokenizer.decode(token_ids)}")

```

Is there a way to make `convert_tokens_to_ids` have the same behaviour as `tokenizer(token).input_ids[1:3]`? THanks in advance for any help you can provide.

---

<div class="post-metadata">

**Author:** ![lianghsun](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lianghsun/32/57774_2.png) [@lianghsun](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flianghsun)%5C
**Post date:** [October 25, 2022, 8:01pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fconvert-tokens-to-ids-produces-unk%2F24771%2F2 "2022-10-25T20:01:46Z")

</div>

I think you misunderstand `tokenizer.convert_tokens_to_ids()`. Please note this function is to map **token** to **id** , however _exam_ is **not** a _token_, it is a _word_ instead. You can check by the following code:

```auto
tokenizer.convert_ids_to_token([3463, 424])
> ['ex', 'am'] # exam is tokenized to two token!

```

So, it’s obivously that there is no token `exam` in vocab, but `[UNK]` instead.
