# Training BERT from scratch (MLM+NSP) on a new domain

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftraining-bert-from-scratch-mlm-nsp-on-a-new-domain%2F2075%3E%5C
**Category:** 🤗Transformers\
**Created:** [November 15, 2020, 11:01pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftraining-bert-from-scratch-mlm-nsp-on-a-new-domain%2F2075 "2020-11-15T23:01:10Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![rish](https://avatars.discourse-cdn.com/v4/letter/r/46a35a/32.png) [@rish](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Frish)%5C
**Post date:** [November 15, 2020, 11:01pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftraining-bert-from-scratch-mlm-nsp-on-a-new-domain%2F2075%2F1 "2020-11-15T23:01:10Z")

</div>

Hi, I have been trying to train BERT from scratch using the wonderful hugging face library. I am referring to the Language modeling tutorial and have made changes to it for the BERT. As I am running on a completely new domain I have trained my own tokenizer, which trains fine. However, I run into the following error during the training of the model.

)

/usr/local/lib/python3.6/dist-packages/transformers/trainer.py in train(self, model\_path, trial)  
710 self.state.is\_world\_process\_zero = self.is\_world\_process\_zero()  
711  
–\> 712 tr\_loss = torch.tensor(0.0).to(self.args.device)  
713 self.\_logging\_loss\_scalar = 0  
714 self.\_total\_flos = self.state.total\_flos

RuntimeError: CUDA error: device-side assert triggered

From what I understand this is related to some tensor mismatch, however, I unable to resolve this and can’t understand where I am going wrong during the model building. I would really appreciate your help with this.

I have attached the colab notebook for reference

([https://colab.research.google.com/drive/12NHfXeUBo7RBl3Kffa-715i-Zpd0MOzP?usp=sharing](https://colab.research.google.com/drive/12NHfXeUBo7RBl3Kffa-715i-Zpd0MOzP?usp=sharing)).

I am using BertForPreTraining, TextDatasetForNextSentencePrediction, DataCollatorForNextSentencePrediction and BertFasttokeizer.

Environment:  
tokenizers: 0.9.2  
transformers: 3.4.0  
torch: 1.7.0  
CUDA: 10.1

---

<div class="post-metadata">

**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsgugger)%5C
**Post date:** [November 16, 2020, 2:02pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftraining-bert-from-scratch-mlm-nsp-on-a-new-domain%2F2075%2F2 "2020-11-16T14:02:47Z")

</div>

I haven’t run your notebook, but the first thing I’d double check looking at it is that your vocabulary has indeed 50,000 tokens and not more.

---

<div class="post-metadata">

**Author:** ![rish](https://avatars.discourse-cdn.com/v4/letter/r/46a35a/32.png) [@rish](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Frish)%5C
**Post date:** [November 16, 2020, 8:29pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftraining-bert-from-scratch-mlm-nsp-on-a-new-domain%2F2075%2F3 "2020-11-16T20:29:32Z")

</div>

Hi @sgugger, yes the vocabulary has 50000 tokens.

---

<div class="post-metadata">

**Author:** ![rgwatwormhill](https://avatars.discourse-cdn.com/v4/letter/r/a9adbd/32.png) [@rgwatwormhill](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Frgwatwormhill)%5C
**Post date:** [November 17, 2020, 2:49pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftraining-bert-from-scratch-mlm-nsp-on-a-new-domain%2F2075%2F4 "2020-11-17T14:49:19Z")

</div>

Hi, I’m not an expert, but wouldn’t you need to do the MLM training before the NSP training?  
(Does that happen automatically as part of the tokenizer training?)

Have you tried reducing your batch-size? It depends on your text lengths, but 64 might be too big. Finetuning Bert using Colab GPU I found that maxlen x batchsize = 8192 should fit, eg 512 x 16, 128 x 64 - too big causes OutOfMemory.

By the way, is the save\_steps correct?

---

<div class="post-metadata">

**Author:** ![rish](https://avatars.discourse-cdn.com/v4/letter/r/46a35a/32.png) [@rish](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Frish)%5C
**Post date:** [November 17, 2020, 9:24pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftraining-bert-from-scratch-mlm-nsp-on-a-new-domain%2F2075%2F5 "2020-11-17T21:24:43Z")

</div>

Hi @rgwatwormhill, if I read the documentation from huggingface ,  
BertForPreTraining provides two training heads mlm + nsp.([https://huggingface.co/transformers/model\_doc/bert.html#transformers.BertForPreTraining](https://huggingface.co/transformers/model_doc/bert.html#transformers.BertForPreTraining))

Also according to this PR on github ([https://github.com/huggingface/transformers/pull/6644](https://github.com/huggingface/transformers/pull/6644)), the DatasetforNSP and DatacollatorforNSP, both of them take care for MLM +NSP data formatiing. I expected using both (DatasetforNSP and DatacollatorforNSP) with BertForPreTraining should be fine but I may be wrong. Also for the batch size you are correct, it may be too large for the training, however, the error which I am getting has some other problem, and does not successfully go to the training. 😀

---

<div class="post-metadata">

**Author:** ![rgwatwormhill](https://avatars.discourse-cdn.com/v4/letter/r/a9adbd/32.png) [@rgwatwormhill](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Frgwatwormhill)%5C
**Post date:** [November 17, 2020, 10:21pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftraining-bert-from-scratch-mlm-nsp-on-a-new-domain%2F2075%2F6 "2020-11-17T22:21:54Z")

</div>

Why does your notebook config say

> vocab\_size=50\_000,

---

<div class="post-metadata">

**Author:** ![rish](https://avatars.discourse-cdn.com/v4/letter/r/46a35a/32.png) [@rish](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Frish)%5C
**Post date:** [November 17, 2020, 10:46pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftraining-bert-from-scratch-mlm-nsp-on-a-new-domain%2F2075%2F7 "2020-11-17T22:46:20Z")

</div>

I want the BERT model which I am training, to have 50000 vocab size. This vocab is generated using bwpt tokenizer, and the vocab is available at path Bert/voc-vocab.txt.

---

<div class="post-metadata">

**Author:** ![rgwatwormhill](https://avatars.discourse-cdn.com/v4/letter/r/a9adbd/32.png) [@rgwatwormhill](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Frgwatwormhill)%5C
**Post date:** [November 17, 2020, 10:49pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftraining-bert-from-scratch-mlm-nsp-on-a-new-domain%2F2075%2F8 "2020-11-17T22:49:10Z")

</div>

Sorry, I didn’t make my self clear:

it currently says 50\_000  
I think it should say 50000

---

<div class="post-metadata">

**Author:** ![rish](https://avatars.discourse-cdn.com/v4/letter/r/46a35a/32.png) [@rish](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Frish)%5C
**Post date:** [November 17, 2020, 11:04pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftraining-bert-from-scratch-mlm-nsp-on-a-new-domain%2F2075%2F9 "2020-11-17T23:04:35Z")

</div>

> [@rgwatwormhill](#):
>
> 50\_000

I think both should be fine according to pep coding style

---

<div class="post-metadata">

**Author:** ![rgwatwormhill](https://avatars.discourse-cdn.com/v4/letter/r/a9adbd/32.png) [@rgwatwormhill](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Frgwatwormhill)%5C
**Post date:** [November 17, 2020, 11:55pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftraining-bert-from-scratch-mlm-nsp-on-a-new-domain%2F2075%2F10 "2020-11-17T23:55:59Z")

</div>

Oh yes, thank you (I didn’t know that!)

---

<div class="post-metadata">

**Author:** ![kumarme072](https://avatars.discourse-cdn.com/v4/letter/k/d6d6ee/32.png) [@kumarme072](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fkumarme072)%5C
**Post date:** [February 2, 2024, 5:27pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftraining-bert-from-scratch-mlm-nsp-on-a-new-domain%2F2075%2F11 "2024-02-02T17:27:15Z")

</div>

Is there any limit for vocab size ??  
I don’t think so … if there limit please explain why can’t we increase the vocab size more than 50k ?  
What will happen if we do so ???
