# CUDA out of memory when running on multiple GPUs

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcuda-out-of-memory-when-running-on-multiple-gpus%2F19457%3E%5C
**Category:** Beginners\
**Created:** [June 22, 2022, 9:55am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcuda-out-of-memory-when-running-on-multiple-gpus%2F19457 "2022-06-22T09:55:13Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![jabudu](https://avatars.discourse-cdn.com/v4/letter/j/f4b2a3/32.png) [@jabudu](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fjabudu)%5C
**Post date:** [June 22, 2022, 9:55am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcuda-out-of-memory-when-running-on-multiple-gpus%2F19457%2F1 "2022-06-22T09:55:13Z")

</div>

Hey guys,

I’m running some basic tests for preparing a fine tuning job for mT5-large using aws Sagemaker. With a `p3.2xlarge` instance (1 Tesla V100-GPU) and the following setting, the job runs successfully:

```auto
per_device_train_batch_size=16
gradient_accumulation_steps=8
gradient_checkpointing=True

```

But if I use `p3.8xlarge` (4 GPUs) with the setting

```auto
per_device_train_batch_size=4
gradient_accumulation_steps=8
gradient_checkpointing=True

```

I run into a memory error. Any idea how I should adapt the parameters?
