# Flan T5 fine tuning error

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fflan-t5-fine-tuning-error%2F56803%3E%5C
**Category:** 🤗Transformers\
**Created:** [September 29, 2023, 9:11am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fflan-t5-fine-tuning-error%2F56803 "2023-09-29T09:11:58Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![Indroganguly](https://avatars.discourse-cdn.com/v4/letter/i/ccd318/32.png) [@Indroganguly](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FIndroganguly)%5C
**Post date:** [September 29, 2023, 9:11am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fflan-t5-fine-tuning-error%2F56803%2F1 "2023-09-29T09:11:58Z")

</div>

Hi All,

I have participated in the kaggle LLM science competition for that i am using Flan T5 to fine tune my model using lora. I am facing issue when i am trying to train my model, below is my error.

> ValueError: Caught ValueError in replica 0 on device 0.  
> Original Traceback (most recent call last):  
> File “/opt/conda/lib/python3.10/site-packages/torch/nn/parallel/parallel\_apply.py”, line 64, in \_worker  
> output = module(\*input, \*\*kwargs)  
> File “/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py”, line 1501, in \_call\_impl  
> return forward\_call(\*args, \*\*kwargs)  
> File “/opt/conda/lib/python3.10/site-packages/peft/peft\_model.py”, line 1080, in forward  
> return self.base\_model(  
> File “/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py”, line 1501, in \_call\_impl  
> return forward\_call(\*args, \*\*kwargs)  
> File “/opt/conda/lib/python3.10/site-packages/transformers/models/t5/modeling\_t5.py”, line 1709, in forward  
> encoder\_outputs = self.encoder(  
> File “/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py”, line 1501, in \_call\_impl  
> return forward\_call(\*args, \*\*kwargs)  
> File “/opt/conda/lib/python3.10/site-packages/transformers/models/t5/modeling\_t5.py”, line 1023, in forward  
> batch\_size, seq\_length = input\_shape  
> ValueError: too many values to unpack (expected 2)

Below is my code i am using to train my model.

> import torch  
> from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig  
> import pandas as pd  
> from string import Template  
> from pathlib import Path
> 
> import os
> 
> import warnings  
> warnings.simplefilter(“ignore”)
> 
> from tqdm.notebook import tqdm
> 
> # for training
> 
> from peft import LoraConfig, get\_peft\_model, TaskType  
> from transformers import TrainingArguments  
> from trl import SFTTrainer, DataCollatorForCompletionOnlyLM
> 
> # for traing set
> 
> from datasets import load\_dataset,Dataset  
> from langchain.prompts import PromptTemplate  
> import matplotlib.pyplot as plt  
> import bitsandbytes as bnb  
> import numpy as np
> 
> from IPython.display import Markdown, display  
> from transformers import AutoModelForSeq2SeqLM, AutoTokenizer, GenerationConfig, TrainingArguments, Trainer  
> import torch  
> import time  
> import pandas as pd  
> import numpy as np
> 
> llm = “/kaggle/input/googleflan-t5-base/flan-t5-base”
> 
> # load training data
> 
> train\_dataset = load\_dataset(path=“csv”, data\_files=[“/kaggle/input/kaggle-llm-science-exam/train.csv”])  
> test\_dataset = load\_dataset(“csv”, data\_files=[“/kaggle/input/kaggle-llm-science-exam/test.csv”])
> 
> original\_model=AutoModelForSeq2SeqLM.from\_pretrained(llm,torch\_dtype=torch.bfloat16)  
> tokenizer=AutoTokenizer.from\_pretrained(llm)
> 
> lora\_config = LoraConfig(  
> r=32,  
> lora\_alpha=32,  
> target\_modules=[“q”, “v”],  
> lora\_dropout=0.05,  
> bias=“none”,  
> task\_type=TaskType.SEQ\_2\_SEQ\_LM  
> )
> 
> def format\_text(example):  
> “”" fill inputs in promt for a sample “”"
> 
> ```
> template="""Answer the following multiple choice question by giving the most appropriate response. Answer should be one among [A,B,C,D,E]
> Question: {prompt}\n
> A) {a}\n
> B) {b}\n
> C) {c}\n
> D) {d}\n
> E) {e}\n
> 
> ### Answer:
> """
> 
> prompt = PromptTemplate(template=template,input_variables=['prompt', 'a', 'b', 'c', 'd', 'e'])
> text = prompt.format(prompt=example['prompt'], 
> a=example['A'], 
> b=example['B'], 
> c=example['C'], 
> d=example['D'], 
> e=example['E'])
> 
> example['input_ids'] = tokenizer(text, padding="max_length", truncation=True, return_tensors="pt").input_ids
> example['labels'] = tokenizer(example['answer'], padding='longest', truncation=True,return_tensors="pt").input_ids
> 
> return example
> 
> ```
> 
> train\_dataset = train\_dataset.map(format\_text)  
> tokenized\_dataset = train\_dataset.remove\_columns([‘id’, ‘prompt’, ‘A’, ‘B’,‘C’,‘D’,‘E’,‘answer’])  
> peft\_model = get\_peft\_model(original\_model,  
> lora\_config)  
> print(print\_number\_of\_trainable\_model\_parameters(peft\_model))
> 
> output\_dir = f’/kaggle/working/peft-dialogue-training-{str(int(time.time()))}’  
> os.environ[“WANDB\_DISABLED”] = “true”  
> peft\_training\_args = TrainingArguments(  
> output\_dir=output\_dir,  
> auto\_find\_batch\_size=True,  
> learning\_rate=1e-3, # Higher learning rate than full fine-tuning.  
> num\_train\_epochs=1,  
> logging\_steps=1,  
> max\_steps=1  
> )
> 
> peft\_trainer = Trainer(  
> model=peft\_model,  
> args=peft\_training\_args,  
> train\_dataset=tokenized\_dataset[“train”]  
> )
> 
> peft\_trainer.train()

Kaggle link can be found here [Kaggle - LLM Science Exam](https://www.kaggle.com/competitions/kaggle-llm-science-exam)

Any help is appreciated.