# Gradual Unfreezing support for Fine tuning models

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fgradual-unfreezing-support-for-fine-tuning-models%2F860%3E%5C
**Category:** 🤗Transformers\
**Created:** [August 25, 2020, 10:48am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fgradual-unfreezing-support-for-fine-tuning-models%2F860 "2020-08-25T10:48:40Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![keyurfaldu](https://avatars.discourse-cdn.com/v4/letter/k/a8b319/32.png) [@keyurfaldu](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fkeyurfaldu)%5C
**Post date:** [August 25, 2020, 10:48am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fgradual-unfreezing-support-for-fine-tuning-models%2F860%2F1 "2020-08-25T10:48:40Z")

</div>

Does huggingface library has support for gradual unfreezing? As thats one of the key strategy to fine tune model effectively for down stream task.

---

<div class="post-metadata">

**Author:** ![prajjwal1](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/prajjwal1/32/2246_2.png) [@prajjwal1](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fprajjwal1)%5C
**Post date:** [August 25, 2020, 11:04am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fgradual-unfreezing-support-for-fine-tuning-models%2F860%2F2 "2020-08-25T11:04:32Z")

</div>

You will have to do that manually. The `Trainer` won’t do it automatically for you. To freeze the pytorch model, you can do something like this:

```auto
model = AutoModel.from_pretrained(model_id)
for param in model.parameters():
    param.requires_grad = False

```

In this way, you’d have more granular control over which layers you want to freeze. For your use case, you’d want to train just the linear layer initially. You can do this by freezing just the encoder

```auto
for param in model.encoder.parameters():
    param.requires_grad = False

```

You’d have to check if `encoder` is an attribute of all models to be fully sure. At later stages, you can turn the bool to True, to train the encoder.

I hope this gives you an idea.

---

<div class="post-metadata">

**Author:** ![sshleifer](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sshleifer/32/3304_2.png) [@sshleifer](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsshleifer)%5C
**Post date:** [August 25, 2020, 3:51pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fgradual-unfreezing-support-for-fine-tuning-models%2F860%2F3 "2020-08-25T15:51:42Z")

</div>

I would be very happy to help support this if anyone has results that demonstrate its effectiveness!  
There is also a `freeze_params` helper function here: [https://github.com/huggingface/transformers/blob/e92efcf7286c955e6901f894be39cf6154af48b7/examples/seq2seq/utils.py#L277](https://github.com/huggingface/transformers/blob/e92efcf7286c955e6901f894be39cf6154af48b7/examples/seq2seq/utils.py#L277)

---

<div class="post-metadata">

**Author:** ![rgwatwormhill](https://avatars.discourse-cdn.com/v4/letter/r/a9adbd/32.png) [@rgwatwormhill](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Frgwatwormhill)%5C
**Post date:** [August 26, 2020, 11:28pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fgradual-unfreezing-support-for-fine-tuning-models%2F860%2F4 "2020-08-26T23:28:04Z")

</div>

hi @sshleifer,

what counts as a “layer” for freezing?

I’ve created a model based on BertModel (with a custom head), and I’d like to freeze the embedding layer and some of the attention layers. I’ve applied

ct = 0  
for child in model.children():  
ct += 1  
if ct \< 11: # this freezes layers 1-10  
for param in child.parameters():  
param.requires\_grad = False

but I’m not sure what that will have frozen.

How can I tell ?
