# Loss behaviour for bert fine-tuning on QNLI

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Floss-behaviour-for-bert-fine-tuning-on-qnli%2F6054%3E%5C
**Category:** Models\
**Created:** [May 7, 2021, 10:26am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Floss-behaviour-for-bert-fine-tuning-on-qnli%2F6054 "2021-05-07T10:26:24Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![axeldinh](https://avatars.discourse-cdn.com/v4/letter/a/fbc32d/32.png) [@axeldinh](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Faxeldinh)%5C
**Post date:** [May 7, 2021, 10:26am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Floss-behaviour-for-bert-fine-tuning-on-qnli%2F6054%2F1 "2021-05-07T10:26:25Z")

</div>

Hello everyone,

I trained Bert on the QNLI dataset for 20 epochs and here are the losses I got:

![Example](https://us1.discourse-cdn.com/hellohellohello/original/2X/2/21fa20c07c4a62f1e45e79616a9ec256ec1446a2.png)

We can see that the training loss is increasing before dropping between each epoch, and I don’t really understand this behaviour, has anyone an idea where it might come from ? Is it normal, or do you think it could come from a problem in my code ?

Also there are “spikes” appearing at the end of each epoch (especially after 8 epochs). I think it comes from the size of my batches. It is quite small, so my last batch contains only 2 samples.

Here are the specification of my training:

- The model I used is “bert-base-cased” which I got pre-trained from the Transformers library, same for the tokenizer.
- I split the training set in a 80/20 ratio to get the validation set.
- I optimized using Adam with a learning rate of 3e-5, nothing else.
- I am evaluating on the validation set 10 times per epoch.

And here is the code I use on each batch for training:

```
    optimizer.zero_grad()
    output = model(input_ids, attention_mask=attention_masks, token_type_ids = 
                   token_type_ids, labels=labels)
    loss = output.loss
    loss.backward()

    optimizer.step()

```

Thank you in advance for your help !

---

<div class="post-metadata">

**Author:** ![lucasval](https://avatars.discourse-cdn.com/v4/letter/l/b9bd4f/32.png) [@lucasval](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flucasval)%5C
**Post date:** [October 15, 2021, 1:17pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Floss-behaviour-for-bert-fine-tuning-on-qnli%2F6054%2F2 "2021-10-15T13:17:52Z")

</div>

Seems like you’re overfitting by many epochs. The original BERT paper suggests fine-tuning for 2 to 4 epochs.

---

<div class="post-metadata">

**Author:** ![nielsr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nielsr/32/39617_2.png) [@nielsr](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fnielsr)%5C
**Post date:** [October 15, 2021, 1:30pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Floss-behaviour-for-bert-fine-tuning-on-qnli%2F6054%2F3 "2021-10-15T13:30:59Z")

</div>

Some sanity checks:

- did you shuffle your training dataloader? (PyTorch’ `DataLoader` class doesn’t shuffle by default)
- I typically use a learning rate of 5e-5, might be worth trying out.

---

<div class="post-metadata">

**Author:** ![axeldinh](https://avatars.discourse-cdn.com/v4/letter/a/fbc32d/32.png) [@axeldinh](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Faxeldinh)%5C
**Post date:** [October 15, 2021, 2:24pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Floss-behaviour-for-bert-fine-tuning-on-qnli%2F6054%2F4 "2021-10-15T14:24:03Z")

</div>

Dear @lucasval and @nielsr,

Thank you for your answers. In fact I already found what was wrong, I forgot to set the model in training mode at the end of my evaluation loop!

Doing so the dropout layers would only be random for the first 1/10th of each epoch, and would be in eval mode for the remaining 9/10th of each epoch, which explains the periodic behavior of the loss.

Again thank you for your replies!

Best regards,  
Axel
