# The size of tensor error while fine tuning whisper

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fthe-size-of-tensor-error-while-fine-tuning-whisper%2F56388%3E%5C
**Category:** Beginners\
**Created:** [September 26, 2023, 7:04am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fthe-size-of-tensor-error-while-fine-tuning-whisper%2F56388 "2023-09-26T07:04:28Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![CKSINGH](https://avatars.discourse-cdn.com/v4/letter/c/e9bcb4/32.png) [@CKSINGH](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FCKSINGH)%5C
**Post date:** [September 26, 2023, 7:04am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fthe-size-of-tensor-error-while-fine-tuning-whisper%2F56388%2F1 "2023-09-26T07:04:28Z")

</div>

I **am using my custom dataset . Now , while running [trainer.train()] , i am getting error as [**  
**RuntimeError: The size of tensor a (517) must match the size of tensor b (448) at non-singleton dimension 1]**

**Please note I am using whisper medium model as seen below :**

from transformers import Seq2SeqTrainingArguments

training\_args = Seq2SeqTrainingArguments(  
output\_dir=“./whisper-medium-hi”, # change to a repo name of your choice  
per\_device\_train\_batch\_size=2,  
gradient\_accumulation\_steps=16, # increase by 2x for every 2x decrease in batch size  
learning\_rate=1e-5,  
warmup\_steps=500,  
max\_steps=4000,  
gradient\_checkpointing=True,  
fp16=True,  
evaluation\_strategy=“steps”,  
per\_device\_eval\_batch\_size=8,  
predict\_with\_generate=True,  
generation\_max\_length=225,  
save\_steps=1000,  
eval\_steps=1000,  
logging\_steps=25,  
report\_to=[“tensorboard”],  
load\_best\_model\_at\_end=True,  
metric\_for\_best\_model=“wer”,  
greater\_is\_better=False,  
push\_to\_hub=True,  
)

**Below is the error stacktrace :**

## `use_cache = True` is incompatible with gradient checkpointing. Setting `use_cache = False`…

[1001/4000 1:31:27 \< 4:34:34, 0.18 it/s, Epoch 1.76/8]  
Step Training Loss Validation Loss  
[151/228 11:43 \< 06:01, 0.21 it/s]

RuntimeError Traceback (most recent call last)  
in \<cell line: 1\>()  
----\> 1 trainer.train()

15 frames  
/usr/local/lib/python3.10/dist-packages/transformers/trainer.py in train(self, resume\_from\_checkpoint, trial, ignore\_keys\_for\_eval, \*\*kwargs)  
1578 # Disable progress bars when uploading models during checkpoints to avoid polluting stdout  
1579 hf\_hub\_utils.disable\_progress\_bars()  
→ 1580 return inner\_training\_loop(  
1581 args=args,  
1582 resume\_from\_checkpoint=resume\_from\_checkpoint,

/usr/local/lib/python3.10/dist-packages/transformers/trainer.py in \_inner\_training\_loop(self, batch\_size, args, resume\_from\_checkpoint, trial, ignore\_keys\_for\_eval)  
1980 self.control = self.callback\_handler.on\_step\_end(args, self.state, self.control)  
1981  
→ 1982 self.\_maybe\_log\_save\_evaluate(tr\_loss, model, trial, epoch, ignore\_keys\_for\_eval)  
1983 else:  
1984 self.control = self.callback\_handler.on\_substep\_end(args, self.state, self.control)

/usr/local/lib/python3.10/dist-packages/transformers/trainer.py in \_maybe\_log\_save\_evaluate(self, tr\_loss, model, trial, epoch, ignore\_keys\_for\_eval)  
2324 metrics.update(dataset\_metrics)  
2325 else:  
→ 2326 metrics = self.evaluate(ignore\_keys=ignore\_keys\_for\_eval)  
2327 self.\_report\_to\_hp\_search(trial, self.state.global\_step, metrics)  
2328

/usr/local/lib/python3.10/dist-packages/transformers/trainer\_seq2seq.py in evaluate(self, eval\_dataset, ignore\_keys, metric\_key\_prefix, \*\*gen\_kwargs)  
163 self.\_gen\_kwargs = gen\_kwargs  
164  
→ 165 return super().evaluate(eval\_dataset, ignore\_keys=ignore\_keys, metric\_key\_prefix=metric\_key\_prefix)  
166  
167 def predict(

/usr/local/lib/python3.10/dist-packages/transformers/trainer.py in evaluate(self, eval\_dataset, ignore\_keys, metric\_key\_prefix)  
3062  
3063 eval\_loop = self.prediction\_loop if self.args.use\_legacy\_prediction\_loop else self.evaluation\_loop  
→ 3064 output = eval\_loop(  
3065 eval\_dataloader,  
3066 description=“Evaluation”,

/usr/local/lib/python3.10/dist-packages/transformers/trainer.py in evaluation\_loop(self, dataloader, description, prediction\_loss\_only, ignore\_keys, metric\_key\_prefix)  
3251  
3252 # Prediction step  
→ 3253 loss, logits, labels = self.prediction\_step(model, inputs, prediction\_loss\_only, ignore\_keys=ignore\_keys)  
3254 main\_input\_name = getattr(self.model, “main\_input\_name”, “input\_ids”)  
3255 inputs\_decode = self.\_prepare\_input(inputs[main\_input\_name]) if args.include\_inputs\_for\_metrics else None

/usr/local/lib/python3.10/dist-packages/transformers/trainer\_seq2seq.py in prediction\_step(self, model, inputs, prediction\_loss\_only, ignore\_keys, \*\*gen\_kwargs)  
310 if has\_labels:  
311 with self.compute\_loss\_context\_manager():  
→ 312 outputs = model(\*\*inputs)  
313 if self.label\_smoother is not None:  
314 loss = self.label\_smoother(outputs, inputs[“labels”]).mean().detach()

/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py in \_call\_impl(self, \*args, \*\*kwargs)  
1499 or \_global\_backward\_pre\_hooks or \_global\_backward\_hooks  
1500 or \_global\_forward\_hooks or \_global\_forward\_pre\_hooks):  
→ 1501 return forward\_call(\*args, \*\*kwargs)  
1502 # Do not call functions when jit is used  
1503 full\_backward\_hooks, non\_full\_backward\_hooks = ,

/usr/local/lib/python3.10/dist-packages/accelerate/utils/operations.py in forward(\*args, \*\*kwargs)  
634  
635 def forward(\*args, \*\*kwargs):  
→ 636 return model\_forward(\*args, \*\*kwargs)  
637  
638 # To act like a decorator so that it can be popped when doing `extract_model_from_parallel`

/usr/local/lib/python3.10/dist-packages/accelerate/utils/operations.py in **call** (self, \*args, \*\*kwargs)  
622  
623 def **call** (self, \*args, \*\*kwargs):  
→ 624 return convert\_to\_fp32(self.model\_forward(\*args, \*\*kwargs))  
625  
626 def **getstate** (self):

/usr/local/lib/python3.10/dist-packages/torch/amp/autocast\_mode.py in decorate\_autocast(\*args, \*\*kwargs)  
12 def decorate\_autocast(\*args, \*\*kwargs):  
13 with autocast\_instance:  
—\> 14 return func(\*args, \*\*kwargs)  
15 decorate\_autocast.\_\_script\_unsupported = ‘@autocast() decorator is not supported in script mode’ # type: ignore[attr-defined]  
16 return decorate\_autocast

/usr/local/lib/python3.10/dist-packages/transformers/models/whisper/modeling\_whisper.py in forward(self, input\_features, attention\_mask, decoder\_input\_ids, decoder\_attention\_mask, head\_mask, decoder\_head\_mask, cross\_attn\_head\_mask, encoder\_outputs, past\_key\_values, decoder\_inputs\_embeds, labels, use\_cache, output\_attentions, output\_hidden\_states, return\_dict)  
1484 )  
1485  
→ 1486 outputs = self.model(  
1487 input\_features,  
1488 attention\_mask=attention\_mask,

/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py in \_call\_impl(self, \*args, \*\*kwargs)  
1499 or \_global\_backward\_pre\_hooks or \_global\_backward\_hooks  
1500 or \_global\_forward\_hooks or \_global\_forward\_pre\_hooks):  
→ 1501 return forward\_call(\*args, \*\*kwargs)  
1502 # Do not call functions when jit is used  
1503 full\_backward\_hooks, non\_full\_backward\_hooks = ,

/usr/local/lib/python3.10/dist-packages/transformers/models/whisper/modeling\_whisper.py in forward(self, input\_features, attention\_mask, decoder\_input\_ids, decoder\_attention\_mask, head\_mask, decoder\_head\_mask, cross\_attn\_head\_mask, encoder\_outputs, past\_key\_values, decoder\_inputs\_embeds, use\_cache, output\_attentions, output\_hidden\_states, return\_dict)  
1360  
1361 # decoder outputs consists of (dec\_features, past\_key\_value, dec\_hidden, dec\_attn)  
→ 1362 decoder\_outputs = self.decoder(  
1363 input\_ids=decoder\_input\_ids,  
1364 attention\_mask=decoder\_attention\_mask,

/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py in \_call\_impl(self, \*args, \*\*kwargs)  
1499 or \_global\_backward\_pre\_hooks or \_global\_backward\_hooks  
1500 or \_global\_forward\_hooks or \_global\_forward\_pre\_hooks):  
→ 1501 return forward\_call(\*args, \*\*kwargs)  
1502 # Do not call functions when jit is used  
1503 full\_backward\_hooks, non\_full\_backward\_hooks = ,

/usr/local/lib/python3.10/dist-packages/transformers/models/whisper/modeling\_whisper.py in forward(self, input\_ids, attention\_mask, encoder\_hidden\_states, head\_mask, cross\_attn\_head\_mask, past\_key\_values, inputs\_embeds, use\_cache, output\_attentions, output\_hidden\_states, return\_dict)  
1124 positions = self.embed\_positions(inputs\_embeds, past\_key\_values\_length=past\_key\_values\_length)  
1125  
→ 1126 hidden\_states = inputs\_embeds + positions  
1127 hidden\_states = nn.functional.dropout(hidden\_states, p=self.dropout, training=self.training)  
1128

RuntimeError: The size of tensor a (517) must match the size of tensor b (448) at non-singleton dimension 1

---

<div class="post-metadata">

**Author:** ![emily18](https://avatars.discourse-cdn.com/v4/letter/e/0ea827/32.png) [@emily18](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Femily18)%5C
**Post date:** [February 13, 2024, 8:01am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fthe-size-of-tensor-error-while-fine-tuning-whisper%2F56388%2F2 "2024-02-13T08:01:13Z")

</div>

did you solve it?
