# How do I change the classification head of a model?

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-do-i-change-the-classification-head-of-a-model%2F4720%3E%5C
**Category:** 🤗Transformers\
**Created:** [March 19, 2021, 2:46pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-do-i-change-the-classification-head-of-a-model%2F4720 "2021-03-19T14:46:59Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![oliverguhr](https://avatars.discourse-cdn.com/v4/letter/o/b487fb/32.png) [@oliverguhr](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Foliverguhr)%5C
**Post date:** [March 19, 2021, 2:46pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-do-i-change-the-classification-head-of-a-model%2F4720%2F1 "2021-03-19T14:46:59Z")

</div>

Hello,  
I like to change the number of labels that a trained model has. I am loading a model that was trained on 17 classes and I like adapt this model to my own task. Now if I simply change the number of labels like this:

```
model_checkpoint ="vblagoje/bert-english-uncased-finetuned-pos" 
model = AutoModelForTokenClassification.from_pretrained(model_checkpoint,num_labels=2)

```

I get an error saying:

```
RuntimeError: Error(s) in loading state_dict for BertForTokenClassification:
        size mismatch for classifier.weight: copying a param with shape torch.Size([17, 768]) from checkpoint, the shape in current model is torch.Size([2, 768]).
        size mismatch for classifier.bias: copying a param with shape torch.Size([17]) from checkpoint, the shape in current model is torch.Size([2]).

```

My question is: How do I replace the classification head?

Thanks a lot 🤗

---

<div class="post-metadata">

**Author:** ![jominmathew](https://avatars.discourse-cdn.com/v4/letter/j/76d3ee/32.png) [@jominmathew](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fjominmathew)%5C
**Post date:** [March 20, 2021, 1:40am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-do-i-change-the-classification-head-of-a-model%2F4720%2F2 "2021-03-20T01:40:10Z")

</div>

The reason is: you are trying to use mode, which has already pretrained on a particular classification task. You have to remove the last part ( classification head) of the model.

This is actually a kind of design fault too. In practice

( BERT base uncased + Classification ) = new Model .

is your model. Now, if you want to reuse them on a different tasks, either use BERT base uncased or extract that part from new Model.

---

<div class="post-metadata">

**Author:** ![oliverguhr](https://avatars.discourse-cdn.com/v4/letter/o/b487fb/32.png) [@oliverguhr](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Foliverguhr)%5C
**Post date:** [March 22, 2021, 10:01am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-do-i-change-the-classification-head-of-a-model%2F4720%2F3 "2021-03-22T10:01:07Z")

</div>

I wanted to test if the training on the POS task results in better scores compared to using just the pain BERT base.

So my question is: How do I extract the “base” part from a trained model and add a new head?

---

<div class="post-metadata">

**Author:** ![jominmathew](https://avatars.discourse-cdn.com/v4/letter/j/76d3ee/32.png) [@jominmathew](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fjominmathew)%5C
**Post date:** [March 22, 2021, 12:35pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-do-i-change-the-classification-head-of-a-model%2F4720%2F4 "2021-03-22T12:35:23Z")

</div>

I am not sure 🙂

---

<div class="post-metadata">

**Author:** ![eddie96](https://avatars.discourse-cdn.com/v4/letter/e/db5fbb/32.png) [@eddie96](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Feddie96)%5C
**Post date:** [March 22, 2021, 1:37pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-do-i-change-the-classification-head-of-a-model%2F4720%2F5 "2021-03-22T13:37:20Z")

</div>

Hi from what i noticed the weights your using are already fine tuned for token classification (the classifier has been trained for said task), i recommend you fine tune on the bert base case as such:

```python
from transformers import AutoModelForTokenClassification
model = AutoModelForTokenClassification('bert-base-uncased', num_labels=2)
# Start your own training

```

or if you want to write your own as requested with a custom classifier head

```python
import torch.nn as nn
from transformers import AutoModel
class PosModel(nn.Module):
    def __init__ (self):
        super(PosModel, self). __init__ ()
        
        self.base_model = AutoModel.from_pretrained('bert-base-uncased')
        self.dropout = nn.Dropout(0.5)
        self.linear = nn.Linear(768, 2) # output features from bert is 768 and 2 is ur number of labels
        
    def forward(self, input_ids, attn_mask):
        outputs = self.base_model(input_ids, attention_mask=attn_mask)
        # You write you new head here
        outputs = self.dropout(outputs[0])
        outputs = self.linear(outputs)
        
        return outputs

model = PosModel()
model.to('cuda')

```

---

<div class="post-metadata">

**Author:** ![eddie96](https://avatars.discourse-cdn.com/v4/letter/e/db5fbb/32.png) [@eddie96](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Feddie96)%5C
**Post date:** [March 22, 2021, 1:38pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-do-i-change-the-classification-head-of-a-model%2F4720%2F6 "2021-03-22T13:38:16Z")

</div>

I am assuming your using pytorch for this.

---

<div class="post-metadata">

**Author:** ![oliverguhr](https://avatars.discourse-cdn.com/v4/letter/o/b487fb/32.png) [@oliverguhr](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Foliverguhr)%5C
**Post date:** [March 22, 2021, 3:31pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-do-i-change-the-classification-head-of-a-model%2F4720%2F7 "2021-03-22T15:31:43Z")

</div>

This is a good solution if you train new a model based on a LM like bert-based-uncased from scratch. I try to replace the classification head of a model. Running your first code with a pre-trained model for token classification will result in an error message (see my sample above).  
Running your second code will add a new classification layer on top of the existing one when you run it with a pos model like “vblagoje/bert-english-uncased-finetuned-pos”.

---

<div class="post-metadata">

**Author:** ![eddie96](https://avatars.discourse-cdn.com/v4/letter/e/db5fbb/32.png) [@eddie96](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Feddie96)%5C
**Post date:** [March 22, 2021, 3:36pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-do-i-change-the-classification-head-of-a-model%2F4720%2F8 "2021-03-22T15:36:14Z")

</div>

Your error basically is a mismatch of the final layer which is the classifier part:

vblagoje/bert-english-uncased-finetuned-pos - this is already finetune weights for 17 classes

bert-base-uncased - you need to use this where isn’t finetuned for anything yet, so this is what you want to use for just fine tuning the 2 classes you have

---

<div class="post-metadata">

**Author:** ![eddie96](https://avatars.discourse-cdn.com/v4/letter/e/db5fbb/32.png) [@eddie96](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Feddie96)%5C
**Post date:** [March 22, 2021, 3:39pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-do-i-change-the-classification-head-of-a-model%2F4720%2F9 "2021-03-22T15:39:23Z")

</div>

Unfortunately the author of the model did not write anything to specify what he fine tuned on so i have no idea myself but this model has already been fine tuned, therefore you cannot continue fine tuning it

```auto
vblagoje/bert-english-uncased-finetuned-pos

```

---

<div class="post-metadata">

**Author:** ![oliverguhr](https://avatars.discourse-cdn.com/v4/letter/o/b487fb/32.png) [@oliverguhr](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Foliverguhr)%5C
**Post date:** [March 22, 2021, 3:57pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-do-i-change-the-classification-head-of-a-model%2F4720%2F10 "2021-03-22T15:57:36Z")

</div>

Technically it should not be an issue to remove a classification head. This is the main idea of transfer learning and people do this all the time with CNNs. I like to check if this works for NLP as well.

---

<div class="post-metadata">

**Author:** ![eddie96](https://avatars.discourse-cdn.com/v4/letter/e/db5fbb/32.png) [@eddie96](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Feddie96)%5C
**Post date:** [March 22, 2021, 4:08pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-do-i-change-the-classification-head-of-a-model%2F4720%2F11 "2021-03-22T16:08:48Z")

</div>

I mean for this case here, when you load the weights it expects the linear layer to have a size of 17 but u specified 2, this is where the error comes from

```auto
size mismatch for classifier.weight: copying a param with shape torch.Size([17, 768]) from checkpoint, the shape in current model is torch.Size([2, 768]).

```

Well technically you can do that by deleting the last couple of layers after loading it, after then fine tuning it. again

```python
# did not test this out
model = AutoModelForTokenClassification.from_pretrained(model_checkpoint,num_labels=17)
model.classifier = nn.Linear(768, 2)
# Run training

```

---

<div class="post-metadata">

**Author:** ![oliverguhr](https://avatars.discourse-cdn.com/v4/letter/o/b487fb/32.png) [@oliverguhr](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Foliverguhr)%5C
**Post date:** [March 22, 2021, 5:48pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-do-i-change-the-classification-head-of-a-model%2F4720%2F12 "2021-03-22T17:48:33Z")

</div>

Well, I tried that and added two lines to set the correct number of labels:

```auto
model.classifier = nn.Linear(786,1)
model.num_labels = 2
model.config.num_labels = 2

```

printing the model shows that this worked.

```
  (dropout): Dropout(p=0.1, inplace=False)
  (classifier): Linear(in_features=786, out_features=1, bias=True)

```

However, running this with the trainer class results in this error:

`mat1 dim 1 must match mat2 dim 0`

I suspect that I missed something or broke some clever autoconfiguration. Thats why I wrote my question here.

---

<div class="post-metadata">

**Author:** ![eddie96](https://avatars.discourse-cdn.com/v4/letter/e/db5fbb/32.png) [@eddie96](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Feddie96)%5C
**Post date:** [March 23, 2021, 4:09am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-do-i-change-the-classification-head-of-a-model%2F4720%2F13 "2021-03-23T04:09:52Z")

</div>

```python
model.classifier = nn.Linear(786,1) # you have 1 class? I think you should change it to 2 
model.num_labels = 2 # while here you specify 2 classes so its a bit confusing

```

Unless you are aiming for a sigmoid function for your last layer is thats why your adding 1 class then i think you need to change to your loss function to bcewithlogitsloss

---

<div class="post-metadata">

**Author:** ![jominmathew](https://avatars.discourse-cdn.com/v4/letter/j/76d3ee/32.png) [@jominmathew](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fjominmathew)%5C
**Post date:** [March 23, 2021, 5:26am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-do-i-change-the-classification-head-of-a-model%2F4720%2F14 "2021-03-23T05:26:45Z")

</div>

Check this. Its neatly implemented here.  
And a pretty neat architecture as well.

model —\> downstream\_task . Well defined.

> <https://github.com/legacyai/tf-transformers/blob/main/src/tf_transformers/notebooks/tutorials/bert_cola_text_classification.ipynb>

---

<div class="post-metadata">

**Author:** ![oliverguhr](https://avatars.discourse-cdn.com/v4/letter/o/b487fb/32.png) [@oliverguhr](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Foliverguhr)%5C
**Post date:** [March 25, 2021, 9:24am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-do-i-change-the-classification-head-of-a-model%2F4720%2F15 "2021-03-25T09:24:12Z")

</div>

Thanks for your help! It was way too late when I wrote this 🙈

It turned out that the easiest way to solve my initial question is this:

```auto
config = AutoConfig.from_pretrained(model_checkpoint)
config.num_labels = 2
model = AutoModelForTokenClassification.from_config(config)

```

---

<div class="post-metadata">

**Author:** ![tolgayan](https://avatars.discourse-cdn.com/v4/letter/t/3ab097/32.png) [@tolgayan](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Ftolgayan)%5C
**Post date:** [July 7, 2021, 12:05pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-do-i-change-the-classification-head-of-a-model%2F4720%2F16 "2021-07-07T12:05:56Z")

</div>

It works, but how this change affects the model architecture, and the results? It would be great if anyone can explain the intuition behind this.

---

<div class="post-metadata">

**Author:** ![Akshayextreme](https://avatars.discourse-cdn.com/v4/letter/a/e56c9b/32.png) [@Akshayextreme](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FAkshayextreme)%5C
**Post date:** [October 5, 2021, 3:42pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-do-i-change-the-classification-head-of-a-model%2F4720%2F17 "2021-10-05T15:42:14Z")

</div>

I don’t think this solved your problem. Initialising model with ‘from\_config’ only changes model configuration and it does not load model weight.

Does anyone know how to solve this problem?

---

<div class="post-metadata">

**Author:** ![savasy](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/savasy/32/39731_2.png) [@savasy](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsavasy)%5C
**Post date:** [October 12, 2021, 1:33pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-do-i-change-the-classification-head-of-a-model%2F4720%2F18 "2021-10-12T13:33:25Z")

</div>

Changing the classification head scenarios suggested above does not work for my case. Instead, I suggest, which works for me, that you can change the body instead of head as follows

`old_model= BertForSequenceClassification.from_pretrained("model-x") new_model=BertForSequenceClassification.from_pretrained("bert-base-uncased", num_labels=HowMany_LABELS_I_WANT) new_model.bert=old_model.bert`  
which works for me

---

<div class="post-metadata">

**Author:** ![nielsr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nielsr/32/39617_2.png) [@nielsr](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fnielsr)%5C
**Post date:** [October 12, 2021, 3:21pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-do-i-change-the-classification-head-of-a-model%2F4720%2F19 "2021-10-12T15:21:36Z")

</div>

This is now possible (thanks to @sgugger) by passing in an additional argument called `ignore_mismatched_sizes`, which you can set to True.

If you have an already fine-tuned model with, let’s say 17 labels, and you want to replace the head with one that has 10 outputs, you can do it as follows:

```auto
from transformers import BertForTokenClassification

model_name = "vblagoje/bert-english-uncased-finetuned-pos"

model = BertForTokenClassification.from_pretrained(model_name, num_labels=10, ignore_mismatched_sizes=True)

```

This will print the following warning:

```auto
Some weights of BertForTokenClassification were not initialized from the model checkpoint at vblagoje/bert-english-uncased-finetuned-pos and are newly initialized because the shapes did not match:
- classifier.weight: found shape torch.Size([17, 768]) in the checkpoint and torch.Size([10, 768]) in the model instantiated
- classifier.bias: found shape torch.Size([17]) in the checkpoint and torch.Size([10]) in the model instantiated
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.

```

---

<div class="post-metadata">

**Author:** ![Aliseyfi](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/aliseyfi/32/17890_2.png) [@Aliseyfi](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FAliseyfi)%5C
**Post date:** [December 21, 2021, 7:18pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-do-i-change-the-classification-head-of-a-model%2F4720%2F20 "2021-12-21T19:18:02Z")

</div>

I guess you’re right. I’m still looking for a solution [HERE](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Freplacing-last-layer-of-a-fine-tuned-model-to-use-different-set-of-labels%2F12995)

[Next page](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-do-i-change-the-classification-head-of-a-model%2F4720.md%3Fpage%3D2)
