# Continuation token in pertained tokenizer bert-base-chinese

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcontinuation-token-in-pertained-tokenizer-bert-base-chinese%2F218%3E%5C
**Category:** 🤗Tokenizers\
**Created:** [July 11, 2020, 10:21am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcontinuation-token-in-pertained-tokenizer-bert-base-chinese%2F218 "2020-07-11T10:21:02Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![jiayu](https://avatars.discourse-cdn.com/v4/letter/j/7c8e57/32.png) [@jiayu](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fjiayu)%5C
**Post date:** [July 11, 2020, 10:21am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcontinuation-token-in-pertained-tokenizer-bert-base-chinese%2F218%2F1 "2020-07-11T10:21:02Z")

</div>

I am a bit confused on the different tokens (with and without ## i.e. continuation prefix) in `bert-base-chinese`.

```auto
In [1]: from transformers import AutoTokenizer

In [2]: tokenizer = AutoTokenizer.from_pretrained('bert-base-chinese')

In [3]: tokenizer.encode('恋爱ing', add_special_tokens=False)
Out[3]: [2605, 4263, 10139]

In [4]: tokenizer.save_pretrained('tokenizer')

Out[4]:
('tokenizer/vocab.txt',
 'tokenizer/special_tokens_map.json',
 'tokenizer/added_tokens.json')

In [5]: !grep -n ing tokenizer/vocab.txt
8222:##ing
9108:##ting
9310:booking
9383:king
9427:##ling
9536:##ning
9663:shopping
9741:##king
9756:##ding
10062:ling
10070:ｗedding
10140:ing
...

In [6]: !grep -n 爱 tokenizer/vocab.txt
4264:爱
17321:##爱

```

Here it shows that 爱 is tokenized into 4263 instead of 17320, indicating that for this Chinese pre-trained model the two character word is split separately; but then why do we still need the `##爱` token in the vocab?
