# Accelerator.device always show xla:0 not opus

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Faccelerator-device-always-show-xla-0-not-opus%2F95137%3E%5C
**Category:** 🤗Accelerate\
**Created:** [July 2, 2024, 12:16pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Faccelerator-device-always-show-xla-0-not-opus%2F95137 "2024-07-02T12:16:05Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![chaeunl](https://avatars.discourse-cdn.com/v4/letter/c/c77e96/32.png) [@chaeunl](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fchaeunl)%5C
**Post date:** [July 2, 2024, 12:16pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Faccelerator-device-always-show-xla-0-not-opus%2F95137%2F1 "2024-07-02T12:16:05Z")

</div>

Hi,

I am trying to run [textural\_inversion.py](https://github.com/huggingface/diffusers/blob/main/examples/textual_inversion/textual_inversion.py)) script.

Before running the script, I followed the instruction for setting accelerate configuration as default: `accelerate config default`

The following is the result of `accelerate env`:

```auto
- `Accelerate` version: 0.30.1
- Platform: Linux-5.15.0-113-generic-x86_64-with-glibc2.31
- `accelerate` bash location: /usr/local/bin/accelerate
- Python version: 3.10.14
- Numpy version: 1.26.4
- PyTorch version (GPU?): 2.1.2+cu118 (True)
- PyTorch XPU available: False
- PyTorch NPU available: False
- PyTorch MLU available: False
- System RAM: 1007.53 GB
- GPU type: NVIDIA A100 80GB PCIe
- `Accelerate` default config:
        - compute_environment: LOCAL_MACHINE
        - distributed_type: MULTI_GPU
        - mixed_precision: no
        - use_cpu: False
        - debug: False
        - num_processes: 8
        - machine_rank: 0
        - num_machines: 1
        - rdzv_backend: static
        - same_network: False
        - main_training_function: main
        - enable_cpu_affinity: False
        - downcast_bf16: False
        - tpu_use_cluster: False
        - tpu_use_sudo: False

```

When I run the script and it printed out that codes run only on the device `xla:0`, but my local machine only has 8 A100 GPUs.

I am now running the script in the docker container that has following torch libs:

```auto
torch 2.1.2+cu118
torch-xla 2.1.0
torchmetrics 1.0.3
torchvision 0.16.2+cu118

```

Outside of the container, the version of Nvidia Cuda driver is 11.4.:

```auto
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.256.02 Driver Version: 470.256.02 CUDA Version: 11.8 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 NVIDIA A100 80G... Off | 00000000:1B:00.0 Off | 0 |
| N/A 37C P0 41W / 300W | 0MiB / 80994MiB | 0% Default |
| | | Disabled |
+-------------------------------+----------------------+----------------------+
| 1 NVIDIA A100 80G... Off | 00000000:1C:00.0 Off | 0 |
| N/A 37C P0 44W / 300W | 0MiB / 80994MiB | 0% Default |
| | | Disabled |
+-------------------------------+----------------------+----------------------+
| 2 NVIDIA A100 80G... Off | 00000000:1D:00.0 Off | 0 |
| N/A 37C P0 43W / 300W | 0MiB / 80994MiB | 0% Default |
| | | Disabled |
+-------------------------------+----------------------+----------------------+
| 3 NVIDIA A100 80G... Off | 00000000:1E:00.0 Off | 0 |
| N/A 36C P0 44W / 300W | 0MiB / 80994MiB | 0% Default |
| | | Disabled |
+-------------------------------+----------------------+----------------------+
| 4 NVIDIA A100 80G... Off | 00000000:3D:00.0 Off | 0 |
| N/A 34C P0 40W / 300W | 0MiB / 80994MiB | 0% Default |
| | | Disabled |
+-------------------------------+----------------------+----------------------+
| 5 NVIDIA A100 80G... Off | 00000000:3F:00.0 Off | 0 |
| N/A 36C P0 41W / 300W | 0MiB / 80994MiB | 0% Default |
| | | Disabled |
+-------------------------------+----------------------+----------------------+
| 6 NVIDIA A100 80G... Off | 00000000:40:00.0 Off | 0 |
| N/A 36C P0 45W / 300W | 0MiB / 80994MiB | 0% Default |
| | | Disabled |
+-------------------------------+----------------------+----------------------+
| 7 NVIDIA A100 80G... Off | 00000000:41:00.0 Off | 0 |
| N/A 36C P0 44W / 300W | 0MiB / 80994MiB | 0% Default |
| | | Disabled |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| No running processes found |
+-----------------------------------------------------------------------------+

```

To inspect what’s going on, the following is the result when I ran `accelerate test`:

```auto
Running: accelerate-launch /usr/local/lib/python3.10/dist-packages/accelerate/test_utils/scripts/test_script.py
stderr: WARNING:root:PJRT is now the default runtime. For more information, see https://github.com/pytorch/xla/blob/master/docs/pjrt.md
stderr: WARNING:root:Defaulting to PJRT_DEVICE=CPU
stderr: WARNING:root:PJRT is now the default runtime. For more information, see https://github.com/pytorch/xla/blob/master/docs/pjrt.md
stderr: WARNING:root:Defaulting to PJRT_DEVICE=CPU
stderr: WARNING:root:PJRT is now the default runtime. For more information, see https://github.com/pytorch/xla/blob/master/docs/pjrt.md
stderr: WARNING:root:Defaulting to PJRT_DEVICE=CPU
stderr: WARNING:root:PJRT is now the default runtime. For more information, see https://github.com/pytorch/xla/blob/master/docs/pjrt.md
stderr: WARNING:root:Defaulting to PJRT_DEVICE=CPU
stdout: Distributed environment: XLA Backend: xla
stdout: Num processes: 1
stdout: Process index: 0
stdout: Local process index: 5
stdout: Device: xla:0
stdout: 
stdout: Mixed precision type: no
stdout: 
stderr: WARNING:root:PJRT is now the default runtime. For more information, see https://github.com/pytorch/xla/blob/master/docs/pjrt.md
stderr: WARNING:root:Defaulting to PJRT_DEVICE=CPU
stderr: WARNING:root:PJRT is now the default runtime. For more information, see https://github.com/pytorch/xla/blob/master/docs/pjrt.md
stderr: WARNING:root:Defaulting to PJRT_DEVICE=CPU
stdout: Distributed environment: XLA Backend: xla
stdout: Num processes: 1
stdout: Process index: 0
stdout: Local process index: 7
stdout: Device: xla:0
stdout: 
stdout: Mixed precision type: no
stdout: 
stderr: WARNING:root:PJRT is now the default runtime. For more information, see https://github.com/pytorch/xla/blob/master/docs/pjrt.md
stderr: WARNING:root:Defaulting to PJRT_DEVICE=CPU
stdout: Distributed environment: XLA Backend: xla
stdout: Num processes: 1
stdout: Process index: 0
stdout: Local process index: 6
stdout: Device: xla:0
stdout: 
stdout: Mixed precision type: no
stdout: 
stdout: 
stdout: **Test process execution**
stdout: 
stdout: **Test process execution**
stdout: Distributed environment: XLA Backend: xla
stdout: Num processes: 1
stdout: Process index: 0
stdout: Local process index: 2
stdout: Device: xla:0
stdout: 
stdout: Mixed precision type: no
stdout: 
stdout: Distributed environment: XLA Backend: xla
stdout: Num processes: 1
stdout: Process index: 0
stdout: Local process index: 3
stdout: Device: xla:0
stdout: 
stdout: Mixed precision type: no
stdout: 
stdout: Distributed environment: XLA Backend: xla
stdout: Num processes: 1
stdout: Process index: 0
stdout: Local process index: 1
stdout: Device: xla:0
stdout: 
stdout: Mixed precision type: no
stdout: 
stdout: 
stdout: **Test process execution**
stdout: Distributed environment: XLA Backend: xla
stdout: Num processes: 1
stdout: Process index: 0
stdout: Local process index: 4
stdout: Device: xla:0
stdout: 
stdout: Mixed precision type: no
stdout: 
stdout: 
stdout: **Test process execution**
stderr: WARNING:root:PJRT is now the default runtime. For more information, see https://github.com/pytorch/xla/blob/master/docs/pjrt.md
stderr: WARNING:root:Defaulting to PJRT_DEVICE=CPU
stdout: 
stdout: **Test process execution**
stdout: 
stdout: **Test process execution**
stdout: **Initialization**
stdout: Testing, testing. 1, 2, 3.
stdout: Distributed environment: XLA Backend: xla
stdout: Num processes: 1
stdout: Process index: 0
stdout: Local process index: 0
stdout: Device: xla:0
stdout: 
stdout: Mixed precision type: no
stdout: 
stdout: 
stdout: **Test process execution**
stdout: 
stdout: **Test process execution**
stderr: Traceback (most recent call last):
stderr: File "/usr/local/lib/python3.10/dist-packages/accelerate/test_utils/scripts/test_script.py", line 804, in <module>
stderr: Traceback (most recent call last):
stderr: File "/usr/local/lib/python3.10/dist-packages/accelerate/test_utils/scripts/test_script.py", line 804, in <module>
stderr: main()
stderr: File "/usr/local/lib/python3.10/dist-packages/accelerate/test_utils/scripts/test_script.py", line 746, in main
stderr: main()
stderr: File "/usr/local/lib/python3.10/dist-packages/accelerate/test_utils/scripts/test_script.py", line 746, in main
stderr: process_execution_check()
stderr: File "/usr/local/lib/python3.10/dist-packages/accelerate/test_utils/scripts/test_script.py", line 135, in process_execution_check
stderr: process_execution_check()
stderr: File "/usr/local/lib/python3.10/dist-packages/accelerate/test_utils/scripts/test_script.py", line 102, in process_execution_check
stderr: assert f.getvalue().rstrip() == ""
stderr: with open(path) as f:
stderr: AssertionError
stderr: FileNotFoundError: [Errno 2] No such file or directory: 'check_main_process_first.txt'
stderr: Traceback (most recent call last):
stderr: File "/usr/local/lib/python3.10/dist-packages/accelerate/test_utils/scripts/test_script.py", line 804, in <module>
stderr: main()
stderr: File "/usr/local/lib/python3.10/dist-packages/accelerate/test_utils/scripts/test_script.py", line 746, in main
stderr: process_execution_check()
stderr: File "/usr/local/lib/python3.10/dist-packages/accelerate/test_utils/scripts/test_script.py", line 135, in process_execution_check
stderr: Traceback (most recent call last):
stderr: File "/usr/local/lib/python3.10/dist-packages/accelerate/test_utils/scripts/test_script.py", line 804, in <module>
stderr: assert f.getvalue().rstrip() == ""
stderr: AssertionError
stderr: main()
stderr: File "/usr/local/lib/python3.10/dist-packages/accelerate/test_utils/scripts/test_script.py", line 746, in main
stderr: process_execution_check()
stderr: File "/usr/local/lib/python3.10/dist-packages/accelerate/test_utils/scripts/test_script.py", line 102, in process_execution_check
stderr: with open(path) as f:
stderr: FileNotFoundError: [Errno 2] No such file or directory: 'check_main_process_first.txt'
stderr: Traceback (most recent call last):
stderr: File "/usr/local/lib/python3.10/dist-packages/accelerate/test_utils/scripts/test_script.py", line 804, in <module>
stderr: main()
stderr: File "/usr/local/lib/python3.10/dist-packages/accelerate/test_utils/scripts/test_script.py", line 746, in main
stderr: process_execution_check()
stderr: File "/usr/local/lib/python3.10/dist-packages/accelerate/test_utils/scripts/test_script.py", line 135, in process_execution_check
stderr: assert f.getvalue().rstrip() == ""
stderr: AssertionError
stderr: Traceback (most recent call last):
stderr: File "/usr/local/lib/python3.10/dist-packages/accelerate/test_utils/scripts/test_script.py", line 804, in <module>
stderr: main()
stderr: File "/usr/local/lib/python3.10/dist-packages/accelerate/test_utils/scripts/test_script.py", line 746, in main
stderr: process_execution_check()
stderr: File "/usr/local/lib/python3.10/dist-packages/accelerate/test_utils/scripts/test_script.py", line 135, in process_execution_check
stderr: assert f.getvalue().rstrip() == ""
stderr: AssertionError
stderr: Traceback (most recent call last):
stderr: File "/usr/local/lib/python3.10/dist-packages/accelerate/test_utils/scripts/test_script.py", line 804, in <module>
stderr: main()
stderr: File "/usr/local/lib/python3.10/dist-packages/accelerate/test_utils/scripts/test_script.py", line 746, in main
stderr: process_execution_check()
stderr: File "/usr/local/lib/python3.10/dist-packages/accelerate/test_utils/scripts/test_script.py", line 135, in process_execution_check
stderr: assert f.getvalue().rstrip() == ""
stderr: AssertionError
stdout: 
stdout: **Test split between processes as a list**
stdout: 
stdout: **Test split between processes as a dict**
stdout: 
stdout: **Test split between processes as a tensor**
stdout: 
stdout: **Test split between processes as a datasets.Dataset**
stdout: 
stdout: **Test random number generator synchronization**
stdout: All rng are properly synched.
stdout: 
stdout: **DataLoader integration test**
stdout: 0 tensor([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17,
stdout: 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31],
stdout: device='xla:0') <class 'accelerate.data_loader.MpDeviceLoaderWrapper'>
stdout: Non-shuffled dataloader passing.
stdout: Shuffled dataloader passing.
stdout: 
stdout: **Training integration test**
stdout: Model dtype: torch.float32, torch.float32. Input dtype: torch.float32
stdout: Model dtype: torch.float32, torch.float32. Input dtype: torch.float32
stdout: Training yielded the same results on one CPU or distributed setup with no batch split.
stdout: Model dtype: torch.float32, torch.float32. Input dtype: torch.float32
stdout: Training yielded the same results on one CPU or distributes setup with batch split.
stdout: FP16 training check.
stderr: /usr/local/lib/python3.10/dist-packages/torch/amp/autocast_mode.py:315: UserWarning: In XLA autocast, but the target dtype is not supported. Disabling autocast.
stderr: XLA Autocast only supports dtype of torch.bfloat16 currently.
stderr: warnings.warn(error_message)
stdout: Model dtype: torch.float32, torch.float32. Input dtype: torch.float32
stdout: Keep fp32 wrapper check.
stdout: BF16 training check.
stdout: Model dtype: torch.float32, torch.float32. Input dtype: torch.float32
stdout: Model dtype: torch.float32, torch.float32. Input dtype: torch.float32
stdout: Model dtype: torch.float32, torch.float32. Input dtype: torch.float32
stdout: Training yielded the same results on one CPU or distributed setup with no batch split.
stdout: Model dtype: torch.float32, torch.float32. Input dtype: torch.float32
stdout: Training yielded the same results on one CPU or distributes setup with batch split.
stdout: FP16 training check.
stdout: Model dtype: torch.float32, torch.float32. Input dtype: torch.float32
stdout: Keep fp32 wrapper check.
stdout: BF16 training check.
stdout: Model dtype: torch.float32, torch.float32. Input dtype: torch.float32
stdout: 
stdout: **Breakpoint trigger test**
stderr: [2024-07-02 12:14:36,020] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 129704 closing signal SIGTERM
stderr: [2024-07-02 12:14:36,437] torch.distributed.elastic.multiprocessing.api: [ERROR] failed (exitcode: 1) local_rank: 1 (pid: 129705) of binary: /usr/bin/python3.10
stderr: Traceback (most recent call last):
stderr: File "/usr/local/bin/accelerate-launch", line 8, in <module>
stderr: sys.exit(main())
stderr: File "/usr/local/lib/python3.10/dist-packages/accelerate/commands/launch.py", line 1088, in main
stderr: launch_command(args)
stderr: File "/usr/local/lib/python3.10/dist-packages/accelerate/commands/launch.py", line 1073, in launch_command
stderr: multi_gpu_launcher(args)
stderr: File "/usr/local/lib/python3.10/dist-packages/accelerate/commands/launch.py", line 718, in multi_gpu_launcher
stderr: distrib_run.run(args)
stderr: File "/usr/local/lib/python3.10/dist-packages/torch/distributed/run.py", line 797, in run
stderr: elastic_launch(
stderr: File "/usr/local/lib/python3.10/dist-packages/torch/distributed/launcher/api.py", line 134, in __call__
stderr: return launch_agent(self._config, self._entrypoint, list(args))
stderr: File "/usr/local/lib/python3.10/dist-packages/torch/distributed/launcher/api.py", line 264, in launch_agent
stderr: raise ChildFailedError(
stderr: torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
stderr: ============================================================
stderr: /usr/local/lib/python3.10/dist-packages/accelerate/test_utils/scripts/test_script.py FAILED
stderr: ------------------------------------------------------------
stderr: Failures:
stderr: [1]:
stderr: time : 2024-07-02_12:14:36
stderr: host : 4f95f1ff1378
stderr: rank : 2 (local_rank: 2)
stderr: exitcode : 1 (pid: 129706)
stderr: error_file: <N/A>
stderr: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
stderr: [2]:
stderr: time : 2024-07-02_12:14:36
stderr: host : 4f95f1ff1378
stderr: rank : 3 (local_rank: 3)
stderr: exitcode : 1 (pid: 129707)
stderr: error_file: <N/A>
stderr: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
stderr: [3]:
stderr: time : 2024-07-02_12:14:36
stderr: host : 4f95f1ff1378
stderr: rank : 4 (local_rank: 4)
stderr: exitcode : 1 (pid: 129708)
stderr: error_file: <N/A>
stderr: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
stderr: [4]:
stderr: time : 2024-07-02_12:14:36
stderr: host : 4f95f1ff1378
stderr: rank : 5 (local_rank: 5)
stderr: exitcode : 1 (pid: 129709)
stderr: error_file: <N/A>
stderr: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
stderr: [5]:
stderr: time : 2024-07-02_12:14:36
stderr: host : 4f95f1ff1378
stderr: rank : 6 (local_rank: 6)
stderr: exitcode : 1 (pid: 129710)
stderr: error_file: <N/A>
stderr: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
stderr: [6]:
stderr: time : 2024-07-02_12:14:36
stderr: host : 4f95f1ff1378
stderr: rank : 7 (local_rank: 7)
stderr: exitcode : 1 (pid: 129711)
stderr: error_file: <N/A>
stderr: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
stderr: ------------------------------------------------------------
stderr: Root Cause (first observed failure):
stderr: [0]:
stderr: time : 2024-07-02_12:14:36
stderr: host : 4f95f1ff1378
stderr: rank : 1 (local_rank: 1)
stderr: exitcode : 1 (pid: 129705)
stderr: error_file: <N/A>
stderr: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
stderr: ============================================================
Traceback (most recent call last):
  File "/usr/local/bin/accelerate", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.10/dist-packages/accelerate/commands/accelerate_cli.py", line 46, in main
    args.func(args)
  File "/usr/local/lib/python3.10/dist-packages/accelerate/commands/test.py", line 53, in test_command
    result = execute_subprocess_async(cmd)
  File "/usr/local/lib/python3.10/dist-packages/accelerate/test_utils/testing.py", line 555, in execute_subprocess_async
    raise RuntimeError(
RuntimeError: 'accelerate-launch /usr/local/lib/python3.10/dist-packages/accelerate/test_utils/scripts/test_script.py' failed with returncode 1

The combined stderr from workers follows:
WARNING:root:PJRT is now the default runtime. For more information, see https://github.com/pytorch/xla/blob/master/docs/pjrt.md
WARNING:root:Defaulting to PJRT_DEVICE=CPU
WARNING:root:PJRT is now the default runtime. For more information, see https://github.com/pytorch/xla/blob/master/docs/pjrt.md
WARNING:root:Defaulting to PJRT_DEVICE=CPU
WARNING:root:PJRT is now the default runtime. For more information, see https://github.com/pytorch/xla/blob/master/docs/pjrt.md
WARNING:root:Defaulting to PJRT_DEVICE=CPU
WARNING:root:PJRT is now the default runtime. For more information, see https://github.com/pytorch/xla/blob/master/docs/pjrt.md
WARNING:root:Defaulting to PJRT_DEVICE=CPU
WARNING:root:PJRT is now the default runtime. For more information, see https://github.com/pytorch/xla/blob/master/docs/pjrt.md
WARNING:root:Defaulting to PJRT_DEVICE=CPU
WARNING:root:PJRT is now the default runtime. For more information, see https://github.com/pytorch/xla/blob/master/docs/pjrt.md
WARNING:root:Defaulting to PJRT_DEVICE=CPU
WARNING:root:PJRT is now the default runtime. For more information, see https://github.com/pytorch/xla/blob/master/docs/pjrt.md
WARNING:root:Defaulting to PJRT_DEVICE=CPU
WARNING:root:PJRT is now the default runtime. For more information, see https://github.com/pytorch/xla/blob/master/docs/pjrt.md
WARNING:root:Defaulting to PJRT_DEVICE=CPU
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/accelerate/test_utils/scripts/test_script.py", line 804, in <module>
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/accelerate/test_utils/scripts/test_script.py", line 804, in <module>
    main()
  File "/usr/local/lib/python3.10/dist-packages/accelerate/test_utils/scripts/test_script.py", line 746, in main
    main()
  File "/usr/local/lib/python3.10/dist-packages/accelerate/test_utils/scripts/test_script.py", line 746, in main
    process_execution_check()
  File "/usr/local/lib/python3.10/dist-packages/accelerate/test_utils/scripts/test_script.py", line 135, in process_execution_check
    process_execution_check()
  File "/usr/local/lib/python3.10/dist-packages/accelerate/test_utils/scripts/test_script.py", line 102, in process_execution_check
    assert f.getvalue().rstrip() == ""
    with open(path) as f:
AssertionError
FileNotFoundError: [Errno 2] No such file or directory: 'check_main_process_first.txt'
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/accelerate/test_utils/scripts/test_script.py", line 804, in <module>
    main()
  File "/usr/local/lib/python3.10/dist-packages/accelerate/test_utils/scripts/test_script.py", line 746, in main
    process_execution_check()
  File "/usr/local/lib/python3.10/dist-packages/accelerate/test_utils/scripts/test_script.py", line 135, in process_execution_check
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/accelerate/test_utils/scripts/test_script.py", line 804, in <module>
    assert f.getvalue().rstrip() == ""
AssertionError
    main()
  File "/usr/local/lib/python3.10/dist-packages/accelerate/test_utils/scripts/test_script.py", line 746, in main
    process_execution_check()
  File "/usr/local/lib/python3.10/dist-packages/accelerate/test_utils/scripts/test_script.py", line 102, in process_execution_check
    with open(path) as f:
FileNotFoundError: [Errno 2] No such file or directory: 'check_main_process_first.txt'
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/accelerate/test_utils/scripts/test_script.py", line 804, in <module>
    main()
  File "/usr/local/lib/python3.10/dist-packages/accelerate/test_utils/scripts/test_script.py", line 746, in main
    process_execution_check()
  File "/usr/local/lib/python3.10/dist-packages/accelerate/test_utils/scripts/test_script.py", line 135, in process_execution_check
    assert f.getvalue().rstrip() == ""
AssertionError
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/accelerate/test_utils/scripts/test_script.py", line 804, in <module>
    main()
  File "/usr/local/lib/python3.10/dist-packages/accelerate/test_utils/scripts/test_script.py", line 746, in main
    process_execution_check()
  File "/usr/local/lib/python3.10/dist-packages/accelerate/test_utils/scripts/test_script.py", line 135, in process_execution_check
    assert f.getvalue().rstrip() == ""
AssertionError
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/accelerate/test_utils/scripts/test_script.py", line 804, in <module>
    main()
  File "/usr/local/lib/python3.10/dist-packages/accelerate/test_utils/scripts/test_script.py", line 746, in main
    process_execution_check()
  File "/usr/local/lib/python3.10/dist-packages/accelerate/test_utils/scripts/test_script.py", line 135, in process_execution_check
    assert f.getvalue().rstrip() == ""
AssertionError
/usr/local/lib/python3.10/dist-packages/torch/amp/autocast_mode.py:315: UserWarning: In XLA autocast, but the target dtype is not supported. Disabling autocast.
XLA Autocast only supports dtype of torch.bfloat16 currently.
  warnings.warn(error_message)
[2024-07-02 12:14:36,020] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 129704 closing signal SIGTERM
[2024-07-02 12:14:36,437] torch.distributed.elastic.multiprocessing.api: [ERROR] failed (exitcode: 1) local_rank: 1 (pid: 129705) of binary: /usr/bin/python3.10
Traceback (most recent call last):
  File "/usr/local/bin/accelerate-launch", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.10/dist-packages/accelerate/commands/launch.py", line 1088, in main
    launch_command(args)
  File "/usr/local/lib/python3.10/dist-packages/accelerate/commands/launch.py", line 1073, in launch_command
    multi_gpu_launcher(args)
  File "/usr/local/lib/python3.10/dist-packages/accelerate/commands/launch.py", line 718, in multi_gpu_launcher
    distrib_run.run(args)
  File "/usr/local/lib/python3.10/dist-packages/torch/distributed/run.py", line 797, in run
    elastic_launch(
  File "/usr/local/lib/python3.10/dist-packages/torch/distributed/launcher/api.py", line 134, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/usr/local/lib/python3.10/dist-packages/torch/distributed/launcher/api.py", line 264, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
============================================================
/usr/local/lib/python3.10/dist-packages/accelerate/test_utils/scripts/test_script.py FAILED
------------------------------------------------------------
Failures:
[1]:
  time : 2024-07-02_12:14:36
  host : 4f95f1ff1378
  rank : 2 (local_rank: 2)
  exitcode : 1 (pid: 129706)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[2]:
  time : 2024-07-02_12:14:36
  host : 4f95f1ff1378
  rank : 3 (local_rank: 3)
  exitcode : 1 (pid: 129707)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[3]:
  time : 2024-07-02_12:14:36
  host : 4f95f1ff1378
  rank : 4 (local_rank: 4)
  exitcode : 1 (pid: 129708)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[4]:
  time : 2024-07-02_12:14:36
  host : 4f95f1ff1378
  rank : 5 (local_rank: 5)
  exitcode : 1 (pid: 129709)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[5]:
  time : 2024-07-02_12:14:36
  host : 4f95f1ff1378
  rank : 6 (local_rank: 6)
  exitcode : 1 (pid: 129710)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[6]:
  time : 2024-07-02_12:14:36
  host : 4f95f1ff1378
  rank : 7 (local_rank: 7)
  exitcode : 1 (pid: 129711)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time : 2024-07-02_12:14:36
  host : 4f95f1ff1378
  rank : 1 (local_rank: 1)
  exitcode : 1 (pid: 129705)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================

```

How can I resolve this issue?  
I am looking for a way to run the script on GPUs.
