# Finetune pretrained BERT for custom regression task
**URL:**
**Author:**  [@slabschonoren](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fslabschonoren)%5C
**Post date:** [May 31, 2022, 11:29am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffinetune-pretrained-bert-for-custom-regression-task%2F18562%2F1 "2022-05-31T11:29:50Z")
I have a list of sentences:
`X = ["Today is Tuesday", "I went to the store", "This is a computer",....]`
and for each sentence the label is vector of 5 floats:
`y = [[1,4,3,1,7], [5,1,2,8,9],[0,1,6,5,2],....]`
I want to finetune BERT (or other sutiable pre-trained LM) with the proper head to predict the labels.
But I couldnt find ant example to something similar.
Can someone please provide a code sample as to how I can do it?
I am really helpless.
Thanks
---
**Author:**  [@nbroad](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fnbroad)%5C
**Post date:** [May 31, 2022, 1:27pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffinetune-pretrained-bert-for-custom-regression-task%2F18562%2F2 "2022-05-31T13:27:00Z")
Does the order of the numbers in the vector matter? Are you able to provide more context about what the vector signifies?
---
**Author:**  [@slabschonoren](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fslabschonoren)%5C
**Post date:** [May 31, 2022, 5:31pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffinetune-pretrained-bert-for-custom-regression-task%2F18562%2F3 "2022-05-31T17:31:18Z")
@nbroad Sure! sorry for being unclear.
The label vec is a representation vector of other higher dimensional data , so the order does matter.
Actually the head I thought would be the most reasonable to add is
`sklearn.linear_model.Ridge`
Which suitable as it is compatible it multiple output for y.
What am I missing is how to add the add on top of BERT and perform the fine-tuning
---
**Author:**  [@nbroad](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fnbroad)%5C
**Post date:** [May 31, 2022, 6:43pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffinetune-pretrained-bert-for-custom-regression-task%2F18562%2F4 "2022-05-31T18:43:02Z")
You might be able to do something like the code below. The labels need to be in the dataset as (n, 5, num\_labels) where n is the total number of examples and num\_labels is the number of labels for the output vector.
It is just doing multiclass classification 5 separate times – one for each position in the vector.
You should be able to use this model in a `Trainer`.
```python
class CustomModel(PreTrainedModel):
def __init__ (self, config):
super(). __init__ (config)
self.backbone = AutoModel.from_config(config)
self.outputs = [nn.Linear(config.hidden_size, num_labels) for _ in range(5)]
def forward(
self,
input_ids,
attention_mask=None,
token_type_ids=None,
position_ids=None,
labels=None,
):
outputs = self.backbone(
input_ids,
attention_mask=attention_mask,
token_type_ids=token_type_ids,
position_ids=position_ids,
)
sequence_output = outputs.last_hidden_state
outputs = [self.outputs[i](sequence_output) for i in range(5)]
# if labels, then we are training
loss = None
if labels is not None:
loss_fn = nn.BCEWithLogitsLoss()
losses = [loss_fn(outputs[i], labels[i]) for i in range(5)]
loss = sum(losses)/len(losses)
return {
"loss": loss,
"logits": outputs
}
# You'll also have to do this when creating the model
# Config is from AutoConfig.from_pretrained(model_path)
# model_path is something like bert-base-cased
def get_pretrained(config, model_path):
model = CustomModel(config)
if model_path.endswith("pytorch_model.bin"):
model.load_state_dict(torch.load(model_path))
else:
model.backbone = AutoModel.from_pretrained(model_path)
return model
```
---
**Author:**  [@slabschonoren](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fslabschonoren)%5C
**Post date:** [May 31, 2022, 7:18pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffinetune-pretrained-bert-for-custom-regression-task%2F18562%2F5 "2022-05-31T19:18:11Z")
@nbroad Sorry not sure I completely understand:
1. My goal is not only to get a prediction but also to fine-tune BERT, how does it happen here?
2. Why the labels are 3D and not just (n,5)? 5 is the number of labels
3. I actually rather use Ridge Regression if possible, is there a way to do so? Also I prefer not to do it 5 separate times - one for each position, as A) my actual label is more around 300 ints per vector B) I want to minimize the Ridge loss altogether
Is this possible?
---
**Author:**  [@nbroad](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fnbroad)%5C
**Post date:** [May 31, 2022, 11:48pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffinetune-pretrained-bert-for-custom-regression-task%2F18562%2F6 "2022-05-31T23:48:21Z")
Ok you could have one linear layer (`nn.Linear(config.hidden_size, 5)`) and just use `MSELoss`. I think that would work. It won’t output ints but you can round
---
**Author:**  [@slabschonoren](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fslabschonoren)%5C
**Post date:** [June 1, 2022, 7:06am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffinetune-pretrained-bert-for-custom-regression-task%2F18562%2F7 "2022-06-01T07:06:59Z")
@nbroad Thanks. Sure floats are fine.
Can u please provide a code example?
I really dont understand how to implement this
---
**Author:**  [@nbroad](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fnbroad)%5C
**Post date:** [June 3, 2022, 1:09am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffinetune-pretrained-bert-for-custom-regression-task%2F18562%2F8 "2022-06-03T01:09:35Z")
```python
class CustomModel(PreTrainedModel):
def __init__ (self, config):
super(). __init__ (config)
self.backbone = AutoModel.from_config(config)
self.output = nn.Linear(config.hidden_size, config.num_labels)
def forward(
self,
input_ids,
attention_mask=None,
token_type_ids=None,
position_ids=None,
labels=None,
):
outputs = self.backbone(
input_ids,
attention_mask=attention_mask,
token_type_ids=token_type_ids,
position_ids=position_ids,
)
sequence_output = outputs.last_hidden_state
outputs = self.output(sequence_output)
# if labels, then we are training
loss = None
if labels is not None:
loss_fn = nn.MSELoss()
loss = loss_fn(outputs, labels)
return {
"loss": loss,
"logits": outputs
}
```
Have your labels be shape (n, 5) where n is the number of samples.
```python
model_name = "roberta-base"
cfg = AutoConfig.from_pretrained(model_name)
cfg.update({
"num_labels": 5
})
model = get_pretrained(cfg, model_name)
# put the model and data into Trainer
```
---
**Author:**  [@avinasht](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Favinasht)%5C
**Post date:** [September 6, 2023, 10:38pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffinetune-pretrained-bert-for-custom-regression-task%2F18562%2F9 "2023-09-06T22:38:37Z")
@nbroad, how to get [CLS] embedding from your code, it says “AttributeError: ‘dict’ object has no attribute ‘last\_hidden\_state’”
---
**Author:**  [@nbroad](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fnbroad)%5C
**Post date:** [September 11, 2023, 10:22pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffinetune-pretrained-bert-for-custom-regression-task%2F18562%2F10 "2023-09-11T22:22:58Z")
There is no CLS embedding, but you can get the model predictions by doing
```python
output = model(**inputs)
predictions = output["logits"]
```
---
**Author:**  [@avinasht](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Favinasht)%5C
**Post date:** [September 12, 2023, 1:17am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffinetune-pretrained-bert-for-custom-regression-task%2F18562%2F11 "2023-09-12T01:17:42Z")
@nbroad , I want to finetune BERT with regression tasks and then use its embedding as a feature for prediction. Where do I need to modify? i need this embedding so that I can apply XGBoost as well