# Regularisation using Bert

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fregularisation-using-bert%2F11891%3E%5C
**Category:** Beginners\
**Created:** [November 16, 2021, 4:17pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fregularisation-using-bert%2F11891 "2021-11-16T16:17:11Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![arame3333](https://avatars.discourse-cdn.com/v4/letter/a/e19adc/32.png) [@arame3333](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Farame3333)%5C
**Post date:** [November 16, 2021, 4:17pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fregularisation-using-bert%2F11891%2F1 "2021-11-16T16:17:11Z")

</div>

I am seeking to predict nationality and sentiment from tweets and I use this code to set the dropout rate. I have tested this code using dropout rates of 0.1, 0.3 and 0.5 but I find the problem with overfitting is not being fixed. I am not sure whether to include the dropout rate for attention, so I commented out that line.

```auto
def load_bert_model():
    # Load BertForSequenceClassification, the pretrained BERT model with a single 
    # linear classification layer on top. 
    Helper.printline(f'Loading {Hyper.model_name_short} model using {Hyper.model_name} ...')
    _config = set_dropout()
    model = BertForSequenceClassification.from_pretrained(
        Hyper.model_name, # Use the 12-layer BERT model, with an uncased vocab.
        config = _config
    )

    return model

def set_dropout():
    config = BertConfig()
    config.num_labels = Hyper.num_labels # Labels are either positive or negative sentiment, and country.
    #config.attention_probs_dropout_prob = Hyper.dropout_rate
    config.hidden_dropout_prob = Hyper.dropout_rate
    config.output_attentions = False # Do not return attentions weights.
    config.output_hidden_states = False # Do not return all hidden-states.
    return config

```

Because the training loss per epoch is decreasing I am thinking it must be possible to improve the testing loss as well, but I am not sure what to try next? I am using 4 epochs, should I stick to one?

 ![lockdown_training_validation_loss](https://us1.discourse-cdn.com/hellohellohello/original/2X/4/43cf93154fe35e913761c3e4d07234ee1350b0ac.png)