Hey, I am trying to figure out how to freeze layers of a model and read that I had to use for param in model.base_model.parameters(): param.requires_grad = False if I wanted to freeze the encoder of a pretrained MLM for example. But how do I use this with the Trainer? I tried the following: …

Looking at the source code of BertForMaskedLM , the base model is the “bert” attribute, not the “base_model” attribute. So if you want to freeze the parameters of the base model before training, you should type for param in model.bert.parameters(): param.requires_grad = False instead.

@nielsr base_model is an attribute that will work on all the PreTraineModel (to make it easy to access the encoder in a generic fashion) :slight_smile: The Trainer puts your model into training mode, so your difference might simply come from that (there are dropouts in the model). You should check …

@sgugger oh didn’t know that, I learn every day!

OMG! This is so obvious and I cant believe I didn’t realize that. Will test and report! Thanks :slight_smile:

@sgugger model.eval() should have done the trick, right? I am afraid the results still don’t match :frowning:

@sgugger Thanks, that was important to know for me so I knew I had to be the one screwing up somewhere else and I did somehow manage that :smile:

Hi, I tried your code, but I am getting this error: AttributeError: 'RobertaForMaskedLM' object has no attribute 'bert'

Hey, yeah, this is because you are using roberta instead of bert, therefore it uses .roberta to store the encoder. I believe there is some model independant keyword like “base_model” or something, but I dont know right now (im on vacation, but maybe you can try or google it). Hope that helps! Best …

Sorry, answered per mail, sgugger literary provided the base_model keyword in this thread, so there you go :wink:

How to freeze layers using trainer?

Beginners

sgugger March 19, 2021, 4:37pm 7

You should inspect the weights to see where they difer then. Trainer will not change the requires_grad value of your parameters.

Freeze Lower Layers with Auto Classification Model

Topic		Replies	Views
How to freeze layers while fine-tuning? 🤗Transformers	2	1016	May 16, 2025
How to freeze some layers of BertModel Beginners	8	17936	August 25, 2022
Gradual Layer Freezing with huggingface model 🤗Transformers	1	944	February 10, 2021
Gradual Unfreezing support for Fine tuning models 🤗Transformers	3	4138	August 26, 2020
Freeze Lower Layers with Auto Classification Model 🤗Transformers	6	18885	May 25, 2023

How to freeze layers using trainer?

Related topics