# ModernBERT MaskedLM nan training loss

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fmodernbert-maskedlm-nan-training-loss%2F133951%3E%5C
**Category:** Models\
**Created:** [January 2, 2025, 9:08am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fmodernbert-maskedlm-nan-training-loss%2F133951 "2025-01-02T09:08:17Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![yzimmermann](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/yzimmermann/32/38489_2.png) [@yzimmermann](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fyzimmermann)%5C
**Post date:** [January 2, 2025, 9:08am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fmodernbert-maskedlm-nan-training-loss%2F133951%2F1 "2025-01-02T09:08:17Z")

</div>

I have been trying to run pre-training on a fineweb subset with ModernBERT…

First, I tokenize my dataset:

```python
hf_tokenizer = PreTrainedTokenizerFast.from_pretrained("answerdotai/ModernBERT-base")

def tokenize_function(examples):
    return hf_tokenizer(examples["text"],truncation=True)

tokenized_dataset = ds_select.map(
    tokenize_function,
    batched=True, 
    batch_size=1000, 
)

```

Then, I initialize a ModernBERT model:

```python
bert_config = ModernBertConfig(
    global_rope_theta=10000,
    pad_token_id=hf_tokenizer.pad_token_id,
    bos_token_id=hf_tokenizer.bos_token_id,
    eos_token_id=hf_tokenizer.eos_token_id,
    cls_token_id=hf_tokenizer.cls_token_id,
    sep_token_id=hf_tokenizer.sep_token_id,
)
model = ModernBertForMaskedLM(bert_config)

```

I set up a DataCollator with the recommended `mlm_probability`:

```python
data_collator = DataCollatorForLanguageModeling(
    tokenizer=hf_tokenizer, mlm=True, mlm_probability=0.3
)

```

and start the training:

```python
trainer = LoggingTrainer(
    model=model,
    args=training_args,
    train_dataset=split_datasets["train"].shuffle(),
    eval_dataset=split_datasets["test"].shuffle(),
    data_collator=data_collator,
    processing_class=hf_tokenizer,
)
trainer.train()

```

Right on the first example I get a nan loss:

```auto
Loss: tensor([10.8572, nan], device='cuda:0', grad_fn=<GatherBackward>)
Faulty inputs detected:
input_ids: tensor([[50281, 510, 6146, ..., 7355, 50284, 50282],
        [50281, 510, 34461, ..., 50283, 50283, 50283]], device='cuda:0')
attention_mask: tensor([[1, 1, 1, ..., 1, 1, 1],
        [1, 1, 1, ..., 0, 0, 0]], device='cuda:0')
labels: tensor([[-100, -100, -100, ..., -100, 15, -100],
        [-100, -100, -100, ..., -100, -100, -100]], device='cuda:0')
Loss: tensor([nan, nan], device='cuda:0', grad_fn=<GatherBackward>)
Faulty inputs detected:
input_ids: tensor([[50281, 25897, 13, ..., 50283, 50283, 50283],
        [50281, 510, 941, ..., 50284, 15, 50282]], device='cuda:0')
attention_mask: tensor([[1, 1, 1, ..., 0, 0, 0],
        [1, 1, 1, ..., 1, 1, 1]], device='cuda:0')
labels: tensor([[-100, -100, -100, ..., -100, -100, -100],
        [-100, -100, -100, ..., 2774, -100, -100]], device='cuda:0')

```

Notice how the labels don’t seem to be aligned (`50284` vs. `15`)? What am I doing wrong here? I have done pretraining with other models using the transformers library and haven’t run into this kind of problem before. I played around with different optimizer parameters but got the same outcome. I would be thankful for any guidance.

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C
**Post date:** [January 2, 2025, 10:09am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fmodernbert-maskedlm-nan-training-loss%2F133951%2F2 "2025-01-02T10:09:23Z")

</div>

There seems to be a phenomenon where NaN Loss occurs with fp16, but it is unclear whether this is related to the issue.

> <https://github.com/huggingface/transformers/issues/9160>
>
> \## Environment info
> 
>      
> \- \`transformers\` version: 4.0.1 (also reproduced …the same issue with 3.5.1)
> \- Platform: Google Colab
> \- Python version: 3.6
> \- PyTorch version (GPU?): 1.7.0+cu101
> \- Tensorflow version (GPU?): no
> \- Using GPU in script?: Yes
> \- Using distributed or parallel set-up in script?: (don't know. Probably not)
> 
> \### Who can help
> @sgugger
> 
> 
> 
> \## Information
> The problem arises when using:
> \* \[x\] my own modified scripts: (give details below)
> 
> The tasks I am working on is:
> \* \[x\] my own task or dataset: (give details below)
> 
> \*\*Description:\*\*
> I’m trying to fine-tune a pre-trained NLI model (\`ynie/roberta-large-snli\_mnli\_fever\_anli\_R1\_R2\_R3-nli\`) on a dataset of around 276.000 hypothesis-premise pairs. I’m following the instructions from the docs \[here\](https://huggingface.co/transformers/custom\_datasets.html) and \[here\](https://huggingface.co/transformers/training.html). When I run the training, it seems like the fine-tuning works (it does the training and saves the checkpoints), but \`trainer.train()\` and \`trainer.evaluate()\` return "nan" as loss value. 
> 
> \*\*What I've tried:\*\*
> \- I tried using both \`ynie/roberta-large-snli\_mnli\_fever\_anli\_R1\_R2\_R3-nli\` and \`facebook/bart-large-mnli\` to make sure that it's not linked to specific model, but I get the issue for both models
> \- I tried following the advice in this \[related github issue\](https://github.com/huggingface/transformers/issues/1727), but adding \`num\_labels=3\` to the config file does not solve the issue. (I think my issue is different because the models are already fine-tuned on NLI in my case)
> \- I tried many different ways of changing my input data because I suspected that there could be an issue with my input data, but I also couldn't solve it that way.
> \- \*\*The probable source of the issue:\*\* I inspected the prediction output from the model during training and the weird thing is that the prediction value always seems to be "0" (entailment) in 100% of cases (see printed output at the bottom of the code below). This cannot be right. 
> Even weirder: When I first run the model to predict a test sequences before running the trainer, I get normal logits as output. When I run the exact same code block again at the end after having run the trainer, I get \`tensor(\[\[nan, nan, nan\]\]\` as output (see code below). 
> \- I suspect that the source for the 'only 0 prediction output' is that the logits the model returns during training are possibly always \`torch.tensor(\[\[np.nan, np.nan, np.nan\]\])\`. \`torch.tensor(\[\[np.nan, np.nan, np.nan\]\]).argmax(-1)\` returns torch.tensor(0) without triggering an error. The big mystery for me is why the logits would become "nan", because the model does not do that when I use the same input data only outside of the trainer, but something changes once I've run the trainer. 
> 
> =\> I would be very thankful for any help on this! (I've been trying to solve this since two days now)
> Thanks a lot in advance. 
> 
> 
> \## To reproduce
> 
> 
> 
> 
> \### Here is my code: 
> 
> ### load model & tokenize
> from transformers import AutoTokenizer, AutoModelForSequenceClassification
> import torch
> 
> max\_length = 256
> hg\_model\_hub\_name = "ynie/roberta-large-snli\_mnli\_fever\_anli\_R1\_R2\_R3-nli"
> # also tried: hg\_model\_hub\_name = "facebook/bart-large-mnli"
> tokenizer = AutoTokenizer.from\_pretrained(hg\_model\_hub\_name)
> model = AutoModelForSequenceClassification.from\_pretrained(hg\_model\_hub\_name)
> model.config
> 
> device = "cuda" if torch.cuda.is\_available() else "cpu"
> print(f"Device: {device}")
> if device == "cuda":
> model = model.half()
> model.to(device)
> model.train();
> 
> \*\*Running a test inference with the model at this point works fine:\*\*
> \`\`\`
> test\_enc = tokenizer(nli\_train\[0\]\["premise"\], nli\_train\[0\]\["hypothesis"\], return\_tensors="pt", max\_length=max\_length,
> return\_token\_type\_ids=True, truncation=True, padding=True)
> model.eval();
> test\_output\_loss = model(test\_enc\["input\_ids"\].to(device), attention\_mask=test\_enc\["attention\_mask"\].to(device), token\_type\_ids=test\_enc\["token\_type\_ids"\].to(device), labels=torch.tensor(2).to(device))
> print(test\_output\_loss)
> \#output: SequenceClassifierOutput(loss=tensor(2.2168, device='cuda:0', dtype=torch.float16, grad\_fn=\<NllLossBackward\>), logits=tensor(\[\[0.4075, 0.8511, -0.7549\]\], device='cuda:0', dtype=torch.float16,
> grad\_fn=\<AddmmBackward\>), hidden\_states=None, attentions=None)
> \`\`\`
> 
> \*\*Then I continue with preprocessing and training:\*\*
> #... some data preprocessing
> 
> encodings\_train = tokenizer(premise\_train, hypothesis\_train, return\_tensors="pt", max\_length=max\_length,
> return\_token\_type\_ids=True, truncation=False, padding=True)
> encodings\_val = tokenizer(premise\_val, hypothesis\_val, return\_tensors="pt", max\_length=max\_length,
> return\_token\_type\_ids=True, truncation=False, padding=True)
> encodings\_test = tokenizer(premise\_test, hypothesis\_test, return\_tensors="pt", max\_length=max\_length,
> return\_token\_type\_ids=True, truncation=False, padding=True)
> 
> 
> ### create pytorch dataset object
> class XDataset(torch.utils.data.Dataset):
> def \_\_init\_\_(self, encodings, labels):
> self.encodings = encodings
> self.labels = labels
> def \_\_getitem\_\_(self, idx):
> item = {key: torch.as\_tensor(val\[idx\]) for key, val in self.encodings.items()}
> #item = {key: torch.as\_tensor(val\[idx\]).to(device) for key, val in self.encodings.items()}
> item\['labels'\] = torch.as\_tensor(self.labels\[idx\])
> #item\['labels'\] = self.labels\[idx\]
> return item
> def \_\_len\_\_(self):
> return len(self.labels)
> 
> dataset\_train = XDataset(encodings\_train, label\_train)
> dataset\_val = XDataset(encodings\_val, label\_val)
> dataset\_test = XDataset(encodings\_test, label\_test)
> 
> # compute metrics with trainer
> from sklearn.metrics import accuracy\_score, precision\_recall\_fscore\_support
> def compute\_metrics(pred):
> labels = pred.label\_ids
> print(labels)
> preds = pred.predictions.argmax(-1)
> print(preds)
> precision, recall, f1, \_ = precision\_recall\_fscore\_support(labels, preds, average='binary', pos\_label=0)
> acc = accuracy\_score(labels, preds)
> return {
> 'accuracy': acc,
> 'f1': f1,
> 'precision': precision,
> 'recall': recall
> }
> 
> 
> ## training
> from transformers import Trainer, TrainingArguments
> 
> # https://huggingface.co/transformers/main\_classes/trainer.html#transformers.TrainingArguments
> training\_args = TrainingArguments(
> output\_dir='./results', # output directory
> num\_train\_epochs=1, # total number of training epochs
> per\_device\_train\_batch\_size=8, # batch size per device during training
> per\_device\_eval\_batch\_size=8, # batch size for evaluation
> warmup\_steps=500, # number of warmup steps for learning rate scheduler
> weight\_decay=0.01, # strength of weight decay
> logging\_dir='./logs', # directory for storing logs
> logging\_steps=100,
> )
> 
> trainer = Trainer(
> model=model, # the instantiated 🤗 Transformers model to be trained
> args=training\_args, # training arguments, defined above
> train\_dataset=dataset\_train, # training dataset
> eval\_dataset=dataset\_val # evaluation dataset
> )
> 
> trainer.train()
> # output: TrainOutput(global\_step=181, training\_loss=nan)
> trainer.evaluate()
> # output: 
> \[2 2 2 0 0 2 2 2 0 2 0 0 2 2 2 2 0 2 0 2 2 2 2 0 2 0 2 0 0 2 0 0 2 0 0 0 2
> 0 2 0 0 0 0 0 2 0 0 2 2 2 0 2 2 2 2 2 0 0 0 0 2 0 0 0 2 2 0 0 0 2 0 0 0 2
> 2 0 2 0 0 2 2 2 0 2 2 0 0 0 0 0 0 0 2 0 0 0 0 2 0 2 2 0 2 0 0 2 2 2 2 2 2
> 2 0 0 0 0 2 0 0 2 0 0 0 0 2 2 2 0 0 0 0 0 2 0 0 2 0 2 0 2 0 2 0 0 2 2 0 0
> 2 2 2 2 2 2 0 0 2 2 2 2 0 2 0 0 2 2 2 0 0 2 0 2 0 2 0 0 0 0 0 0 2 0 0 2 2
> 0 2 2 2 0 2 2 0 2 2 2 2 2 2 0 0 2 0 0 2 2 0 0 0 2 0 2 2 2 0 0 0 0 0 0 0 0
> 2 0 2 2 2 0 2 0 0 2 0 2 2 0 0 0 0 2 2 2 0 0 0 2 2 2 2 0 2 0 2 2 2\]
> \[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\]
>     
> {'epoch': 1.0,
> 'eval\_accuracy': 0.5137254901960784,
> 'eval\_f1': 0.6787564766839378,
> 'eval\_loss': nan,
> 'eval\_precision': 0.5137254901960784,
> 'eval\_recall': 1.0}
> 
> \*\*Test running the model again after training, returns \`tensor(\[\[nan, nan, nan\]\]\` for some reason:\*\*
> \`\`\`
> test\_enc = tokenizer(nli\_train\[0\]\["premise"\], nli\_train\[0\]\["hypothesis"\], return\_tensors="pt", max\_length=max\_length,
> return\_token\_type\_ids=True, truncation=True, padding=True)
> model.eval();
> test\_output\_loss = model(test\_enc\["input\_ids"\].to(device), attention\_mask=test\_enc\["attention\_mask"\].to(device), token\_type\_ids=test\_enc\["token\_type\_ids"\].to(device), labels=torch.tensor(2).to(device))
> print(test\_output\_loss)
> \#output: SequenceClassifierOutput(loss=tensor(nan, device='cuda:0', dtype=torch.float16, grad\_fn=\<NllLossBackward\>), logits=tensor(\[\[nan, nan, nan\]\], device='cuda:0', dtype=torch.float16,
> grad\_fn=\<AddmmBackward\>), hidden\_states=None, attentions=None)
> \`\`\`
> 
> \[1\]: https://huggingface.co/
> 
> 
> 
> \## Expected behavior
> 
> Model should not return "nan" for logits and return a loss value.

---

<div class="post-metadata">

**Author:** ![yzimmermann](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/yzimmermann/32/38489_2.png) [@yzimmermann](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fyzimmermann)%5C
**Post date:** [January 2, 2025, 10:41am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fmodernbert-maskedlm-nan-training-loss%2F133951%2F4 "2025-01-02T10:41:32Z")

</div>

Thanks for the note, but I’m running this with fp32 right now.

---

<div class="post-metadata">

**Author:** ![dhruvgrammarly](https://avatars.discourse-cdn.com/v4/letter/d/ccd318/32.png) [@dhruvgrammarly](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fdhruvgrammarly)%5C
**Post date:** [January 3, 2025, 5:00pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fmodernbert-maskedlm-nan-training-loss%2F133951%2F5 "2025-01-03T17:00:29Z")

</div>

> [@yzimmermann](#):
>
> `50283`

What does token `50283` correspond to? I don’t know for sure but maybe the padding stuff isn’t working as expected? Having 3 instances of `50283` in a row looks suspicious.

---

<div class="post-metadata">

**Author:** ![akhooli](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/akhooli/32/38599_2.png) [@akhooli](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fakhooli)%5C
**Post date:** [January 5, 2025, 8:41am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fmodernbert-maskedlm-nan-training-loss%2F133951%2F6 "2025-01-05T08:41:54Z")

</div>

50283 is [PAD]. Not sure if this implementation is complete (collator to support dynamic padding, global/local attention etc.). Still, haven’t seen a successful pretraining script.

---

<div class="post-metadata">

**Author:** ![dhruvgrammarly](https://avatars.discourse-cdn.com/v4/letter/d/ccd318/32.png) [@dhruvgrammarly](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fdhruvgrammarly)%5C
**Post date:** [January 6, 2025, 6:29am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fmodernbert-maskedlm-nan-training-loss%2F133951%2F7 "2025-01-06T06:29:27Z")

</div>

Try manually extracting samples from your dataset, and detokenizing them using the tokenizer and inspecting each (token, string, label) tuple and seeing if it matches what you expect. If you can identify the faulty inputs you’ll have something to go on.

---

<div class="post-metadata">

**Author:** ![akhooli](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/akhooli/32/38599_2.png) [@akhooli](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fakhooli)%5C
**Post date:** [January 6, 2025, 12:21pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fmodernbert-maskedlm-nan-training-loss%2F133951%2F8 "2025-01-06T12:21:18Z")

</div>

> [@yzimmermann](#):
>
> `LoggingTrainer`

I used a similar approach and trained a tiny model but also trained my own tokenizer. It completed training successfully. Only other diff I used the Trainer class directly.

---

<div class="post-metadata">

**Author:** ![kendallpark](https://avatars.discourse-cdn.com/v4/letter/k/ec9cab/32.png) [@kendallpark](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fkendallpark)%5C
**Post date:** [January 27, 2025, 4:16pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fmodernbert-maskedlm-nan-training-loss%2F133951%2F9 "2025-01-27T16:16:26Z")

</div>

I am also getting this error. What is weird is that training works on MPS but CUDA yields NaNs during cross entropy loss.

Edit: resolved my issue: [ModernBertModel works on the CPU but fails on the GPU · Issue #174 · AnswerDotAI/ModernBERT · GitHub](https://github.com/AnswerDotAI/ModernBERT/issues/174)
