# Does AutoTokenizer.from\_pretrained add \[cls\] tokens?

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdoes-autotokenizer-from-pretrained-add-cls-tokens%2F4056%3E%5C
**Category:** 🤗Tokenizers\
**Created:** [March 1, 2021, 2:31pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdoes-autotokenizer-from-pretrained-add-cls-tokens%2F4056 "2021-03-01T14:31:34Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![theudster](https://avatars.discourse-cdn.com/v4/letter/t/97f17d/32.png) [@theudster](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Ftheudster)%5C
**Post date:** [March 1, 2021, 2:31pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdoes-autotokenizer-from-pretrained-add-cls-tokens%2F4056%2F1 "2021-03-01T14:31:34Z")

</div>

Hello,  
I am currently working on a classification problem using ProtBERT and I am following the [Fine-Tuning Tutorial](https://huggingface.co/transformers/training.html). I have called the tokenised using

```
tokenizer = AutoTokenizer.from_pretrained

```

and then tokenised like the tutorial says

```
train_encodings = tokenizer(seq_train, truncation=True, padding=True, 
max_length=1024, return_tensors="pt")

```

Unfortunately, the model doesn’t seem to be learning (I froze the BERT layers). From reading around, I saw that I need to add the [CLS] token and found such an option using

```
tokenised.encode(add_special_tokens=True)

```

Yet the tutorial I am following doesn’t seem to require and I was wondering wyy is there a discrepancy and perhaps maybe this is why my model isn’t learning.

Thank you

---

<div class="post-metadata">

**Author:** ![lewtun](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lewtun/32/30995_2.png) [@lewtun](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flewtun)%5C
**Post date:** [March 1, 2021, 3:10pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdoes-autotokenizer-from-pretrained-add-cls-tokens%2F4056%2F2 "2021-03-01T15:10:43Z")

</div>

Hi @theudster, I’m pretty sure that ProtBERT has a `CLS` token since you can see it in the tokenizer’s special tokens map:

```auto
from transformers import AutoTokenizer, AutoModelForMaskedLM

tokenizer = AutoTokenizer.from_pretrained("Rostlab/prot_bert")
# returns {'unk_token': '[UNK]', 'sep_token': '[SEP]', 'pad_token': '[PAD]', 'cls_token': '[CLS]', 'mask_token': '[MASK]'}
tokenizer.special_tokens_map

```

You can also see it by encoding a text and then decoding it:

```auto
text = "I love Adelaide!"
# add_special_tokens=True is set by default
text_enc = tokenizer.encode(text)

for tok in text_enc:
    print(tok, tokenizer.decode(tok))

```

You say you froze the BERT layers, so I’m wondering how you’re doing fine-tuning? I’ve sometimes found that the tutorials in the docs aren’t always complete, so for fine-tuning with text classification I would recommend following Sylvain’s tutorial here: [https://colab.research.google.com/github/huggingface/notebooks/blob/master/examples/token\_classification.ipynb](https://colab.research.google.com/github/huggingface/notebooks/blob/master/examples/token_classification.ipynb)

---

<div class="post-metadata">

**Author:** ![theudster](https://avatars.discourse-cdn.com/v4/letter/t/97f17d/32.png) [@theudster](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Ftheudster)%5C
**Post date:** [March 1, 2021, 3:31pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdoes-autotokenizer-from-pretrained-add-cls-tokens%2F4056%2F3 "2021-03-01T15:31:21Z")

</div>

Thank you @lewtun.

So are you saying that _tokenized_ adds [CLS]?

Also, I am freezing the BERT layer but fine-tuning the classification layer. The reason I am doing that is because Colab crashes as it runs out of GPU if I don’t. If I try using just the CPU, I get “Your session crashed after using all available RAM.”  
I therefore add

```
for param in model.bert.parameters():
        param.requires_grad = False

```

Which fixes that, but then the model doesn’t seem to learn anything 😔

---

<div class="post-metadata">

**Author:** ![lewtun](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lewtun/32/30995_2.png) [@lewtun](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flewtun)%5C
**Post date:** [March 1, 2021, 8:38pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdoes-autotokenizer-from-pretrained-add-cls-tokens%2F4056%2F4 "2021-03-01T20:38:34Z")

</div>

Yes, the tokenizers in `transformers` add the special tokens by default (see the docs [here](https://huggingface.co/transformers/preprocessing.html#base-use)).

I’m not familiar with ProtBERT but I’m surprised its crashing Colab because the repo has some Colab examples: [ProtTrans/ProtBert-BFD-FineTuning-MS.ipynb at master · agemagician/ProtTrans · GitHub](https://github.com/agemagician/ProtTrans/blob/master/Fine-Tuning/ProtBert-BFD-FineTuning-MS.ipynb)

If you’re still having problems fine-tuning with a GPU, perhaps you can reduce the batch size to avoid the OOM errors

---

<div class="post-metadata">

**Author:** ![theudster](https://avatars.discourse-cdn.com/v4/letter/t/97f17d/32.png) [@theudster](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Ftheudster)%5C
**Post date:** [March 1, 2021, 10:52pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdoes-autotokenizer-from-pretrained-add-cls-tokens%2F4056%2F5 "2021-03-01T22:52:45Z")

</div>

thank you @lewtun. I have gone through their implementation and noticed that their per\_device\_train\_batch\_size is 1. Once I changed that, It works and I am actually getting some amazing results (its on its final epoch now, but currently hitting 80% accuracy 😀 ).

I was hoping you could perhaps explain to me what this per\_device\_train\_batch\_size does and why was that the issue?

---

<div class="post-metadata">

**Author:** ![lewtun](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lewtun/32/30995_2.png) [@lewtun](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flewtun)%5C
**Post date:** [March 2, 2021, 10:18am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdoes-autotokenizer-from-pretrained-add-cls-tokens%2F4056%2F6 "2021-03-02T10:18:50Z")

</div>

the `per_device_train_batch_size` specifies the batch size for the device you are training on (e.g. GPU/TPU/CPU), so if your training set has 1,000 examples and and `per_device_train_batch_size=1` then it will take 1,000 steps to complete one epoch.

by increasing the value of `per_device_train_batch_size` you are able to train faster since it takes less steps to complete each epoch (e.g. if `per_device_train_batch_size=4` then we only need 250 steps / epoch in our example), but this can sometimes lead to worse performance since the gradients are averaged / summed in a minibatch.

in your case, my guess is that with `per_device_train_batch_size=1` you need to train for a very long time to see the model learn anything.

---

<div class="post-metadata">

**Author:** ![theudster](https://avatars.discourse-cdn.com/v4/letter/t/97f17d/32.png) [@theudster](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Ftheudster)%5C
**Post date:** [March 2, 2021, 10:32am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdoes-autotokenizer-from-pretrained-add-cls-tokens%2F4056%2F7 "2021-03-02T10:32:11Z")

</div>

well, on Colab, it took a bit over an hour to train 3 epochs, not sure if that is long or not

---

<div class="post-metadata">

**Author:** ![lewtun](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lewtun/32/30995_2.png) [@lewtun](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flewtun)%5C
**Post date:** [March 2, 2021, 12:48pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdoes-autotokenizer-from-pretrained-add-cls-tokens%2F4056%2F8 "2021-03-02T12:48:27Z")

</div>

oh what i meant by “long” is that you may need to run for many epochs before you start seeing any flattening out of you training / validation loss (i.e “convergence”). if you saw the validation loss drop during those 3 epochs then i am not sure what else might have gone wrong in your ProtBERT example
