# Same sequence maps to different token ids

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsame-sequence-maps-to-different-token-ids%2F22240%3E%5C
**Category:** Intermediate\
**Created:** [August 29, 2022, 12:48am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsame-sequence-maps-to-different-token-ids%2F22240 "2022-08-29T00:48:48Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![tk2801](https://avatars.discourse-cdn.com/v4/letter/t/a6a055/32.png) [@tk2801](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Ftk2801)%5C
**Post date:** [August 29, 2022, 12:48am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsame-sequence-maps-to-different-token-ids%2F22240%2F1 "2022-08-29T00:48:48Z")

</div>

Is it possible to use the tokenizers package to create a custom tokenizer that can switch it the behavior of the string(token) to id (stoi) when it encounters a special token?  
For example, the sequence ABC exists multiple times in the same example, but the appearance of the sequence before and after the modification token has a different underlying meaning. Accordingly, the tokenizer maps the ABC before the mod token to different ids than the ids after and the back again.  
Ex of tokenization  
`

[bos] **ABC** [mod]_ABC_[endmod] **ABC** [eos]  
input\_ids: 0 **1 2 3** 4 _5 6 7_ 8 **1 2 3** 9  
A larger issue is how would you synchronize the tokenizers, so that the ids for special tokens line up, but the rest are shifted appropriately? I assuming tokenizer has 2 different stoi and itos…