# Struggle with finetuneing flan-t5-xxl using deepspeed
**URL:**
**Author:**  [@vivianzhao24](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fvivianzhao24)%5C
**Post date:** [May 30, 2023, 5:23pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fstruggle-with-finetuneing-flan-t5-xxl-using-deepspeed%2F41522%2F1 "2023-05-30T17:23:10Z")
Hi,
I am a researcher at Children’s hospital of Philadelphia. We are currently working on automated processing electronic heath records. I attempted to finetune flan-t5-xxl model for translation. However, I kept running into cuda out of memory problem. I am using 4\*A100 GPU each with 80G, the four GPUs are on a same node. I also requested 800G CPU memory which is across multi-nodes.
I have successfully trained a flan-t5-xl model using 4 GPUs. When I attempt to train a xxl model, I set offload to CPU and using ZeRO Stage 3. The same code crashes right before training. Could you help?
Can anyone help? I highly appreciate it!
‘’’
Here is my transformer env:
‘’’
- `transformers` version: 4.27.2
- Platform: Linux-5.14.0-162.6.1.el9\_1.x86\_64-x86\_64-with-glibc2.34
- Python version: 3.11.2
- Huggingface\_hub version: 0.13.4
- PyTorch version (GPU?): 2.0.0+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?:
- Using distributed or parallel set-up in script?:
‘’’
My config file is as follows
‘’’
{
“bf16”: {
“enabled”: “True”,
“loss\_scale”: 0,
“loss\_scale\_window”: 1000,
“initial\_scale\_power”: 500,
“hysteresis”: 4,
“min\_loss\_scale”: 1
},
“optimizer”: {
“type”: “AdamW”,
“params”: {
“lr”: “auto”,
“betas”: “auto”,
“eps”: “auto”,
“weight\_decay”: “auto”
}
},
“scheduler”: {
“type”: “WarmupLR”,
“params”: {
“warmup\_min\_lr”: “auto”,
“warmup\_max\_lr”: “auto”,
“warmup\_num\_steps”: “auto”
}
},
“zero\_optimization”: {
“stage”: 3,
“offload\_optimizer”: {
“device”: “cpu”,
“pin\_memory”: true
},
“offload\_param”: {
“device”: “cpu”,
“pin\_memory”: true
},
“overlap\_comm”: true,
“contiguous\_gradients”: true,
“sub\_group\_size”: 1e9,
“reduce\_bucket\_size”: “auto”,
“stage3\_prefetch\_bucket\_size”: “auto”,
“stage3\_param\_persistence\_threshold”: “auto”,
“stage3\_max\_live\_parameters”: 1e9,
“stage3\_max\_reuse\_distance”: 1e9,
“stage3\_gather\_16bit\_weights\_on\_model\_save”: true
},
“gradient\_accumulation\_steps”: “auto”,
“gradient\_clipping”: “auto”,
“steps\_per\_print”: 2000,
“train\_batch\_size”: “auto”,
“train\_micro\_batch\_size\_per\_gpu”: “auto”,
“wall\_clock\_breakdown”: false
}
‘’’
Here is my python code
‘’’
ROOT\_DIR = “.”
train\_tsvfile = os.path.join(ROOT\_DIR,
‘Clinical-Genetics-NLP-Training-Data/Physical-Exam/Partitioned/Assembled-Span-Annotated-Training-Data-Train.tsv’)
dev\_tsvfile = os.path.join(ROOT\_DIR,
‘Clinical-Genetics-NLP-Training-Data/Physical-Exam/Partitioned/Assembled-Span-Annotated-Training-Data-Dev.tsv’)
model\_name = ‘google/flan-t5-xl’
train\_inputs, train\_targets, id2name = translater\_train\_data(ROOT\_DIR, train\_tsvfile, 3)
dev\_inputs, dev\_targets = translater\_dev\_data(dev\_tsvfile, id2name)
mydataset = create\_my\_dataset(train\_inputs, train\_targets, dev\_inputs, dev\_targets)
prefix = "translate EHR to HPO: "
tokenizer = T5Tokenizer.from\_pretrained(model\_name)
def preprocess\_function(examples):
inputs = [prefix + example[‘EHR’] for example in examples[‘translation’]]
targets = [example[‘HPO’] for example in examples[‘translation’]]
model\_inputs = tokenizer(inputs, max\_length=512, padding=‘max\_length’, truncation=True)
labels = tokenizer(targets, max\_length=512, padding=‘max\_length’, truncation=True)
labels[“input\_ids”] = [[(l if l != tokenizer.pad\_token\_id else -100) for l in label] for label in labels[“input\_ids”]]
model\_inputs[“labels”] = labels[“input\_ids”]
return model\_inputs
tokenized\_encodings = mydataset.map(preprocess\_function, batched=True)
model = T5ForConditionalGeneration.from\_pretrained(model\_name)
data\_collator = DataCollatorForSeq2Seq(tokenizer=tokenizer, model=model\_name)
metric = evaluate.load(“sacrebleu”)
def postprocess\_text(preds, labels):
preds = [pred.strip() for pred in preds]
labels = [[label.strip()] for label in labels]
return preds, labels
def compute\_metrics(eval\_preds):
preds, labels = eval\_preds
if isinstance(preds, tuple):
preds = preds[0]
decoded\_preds = tokenizer.batch\_decode(preds, skip\_special\_tokens=True)
```
labels = np.where(labels != -100, labels, tokenizer.pad_token_id)
decoded_labels = tokenizer.batch_decode(labels, skip_special_tokens=True)
decoded_preds, decoded_labels = postprocess_text(decoded_preds, decoded_labels)
result = metric.compute(predictions=decoded_preds, references=decoded_labels)
result = {"bleu": result["score"]}
prediction_lens = [np.count_nonzero(pred != tokenizer.pad_token_id) for pred in preds]
result["gen_len"] = np.mean(prediction_lens)
result = {k: round(v, 4) for k, v in result.items()}
return result
```
training\_args = Seq2SeqTrainingArguments(
output\_dir=“my\_model”,
evaluation\_strategy=“epoch”,
learning\_rate=2e-5,
per\_device\_train\_batch\_size=4,
per\_device\_eval\_batch\_size=4,
weight\_decay=0.01,
save\_total\_limit=2,
num\_train\_epochs=2,
predict\_with\_generate=True,
no\_deprecation\_warning=True,
push\_to\_hub=False,
)
trainer = Seq2SeqTrainer(
model=model,
args=training\_args,
train\_dataset=tokenized\_encodings[“train”],
eval\_dataset=tokenized\_encodings[“validation”],
tokenizer=tokenizer,
data\_collator=data\_collator,
compute\_metrics=compute\_metrics,
)
trainer.train()
‘’’
Error Message
‘’’
---
**Author:**  [@pogpog](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpogpog)%5C
**Post date:** [March 12, 2024, 5:59pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fstruggle-with-finetuneing-flan-t5-xxl-using-deepspeed%2F41522%2F2 "2024-03-12T17:59:19Z")
In your python code:
```auto
def compute_metrics(eval_preds):
...
per_device_train_batch_size=4,
per_device_eval_batch_size=4,
```
It may help to reduce these values, or to add the `auto_find_batch_size` parameter, which I believe is used in addition to the others, to reduce until a good fit is found.
---
**Author:**  [@vivianzhao24](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fvivianzhao24)%5C
**Post date:** [March 12, 2024, 6:24pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fstruggle-with-finetuneing-flan-t5-xxl-using-deepspeed%2F41522%2F3 "2024-03-12T18:24:03Z")
@pogpog Thank you very much for your reply. I have solved solved this problem. But I really appreciate your help.
---
**Author:**  [@system](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsystem)%5C
**Post date:** [March 13, 2024, 6:24am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fstruggle-with-finetuneing-flan-t5-xxl-using-deepspeed%2F41522%2F4 "2024-03-13T06:24:15Z")
Closed on March 13, 2024, 6:24am UTC
This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.