# No GPUs found in a machine definitely with GPUs

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fno-gpus-found-in-a-machine-definitely-with-gpus%2F32769%3E%5C
**Category:** 🤗Accelerate\
**Created:** [March 1, 2023, 4:33am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fno-gpus-found-in-a-machine-definitely-with-gpus%2F32769 "2023-03-01T04:33:13Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![Trusure](https://avatars.discourse-cdn.com/v4/letter/t/df705f/32.png) [@Trusure](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FTrusure)%5C
**Post date:** [March 1, 2023, 4:33am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fno-gpus-found-in-a-machine-definitely-with-gpus%2F32769%2F1 "2023-03-01T04:33:13Z")

</div>

the error is below:

```auto
stderr: ╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
stderr: │ /home/chenzhixuan/anaconda3/lib/python3.9/site-packages/accelerate/test_utils/scripts/test_scrip │
stderr: │ t.py:336 in <module> │
stderr: │ │
stderr: │ 333 │
stderr: │ 334 │
stderr: │ 335 if __name__ == " __main__": │
stderr: │ ❱ 336 │ main() │
stderr: │ 337 │
stderr: │ │
stderr: │ /home/chenzhixuan/anaconda3/lib/python3.9/site-packages/accelerate/test_utils/scripts/test_scrip │
stderr: │ t.py:305 in main │
stderr: │ │
stderr: │ 302 │
stderr: │ 303 │
stderr: │ 304 def main(): │
stderr: │ ❱ 305 │ accelerator = Accelerator() │
stderr: │ 306 │ state = accelerator.state │
stderr: │ 307 │ if state.local_process_index == 0: │
stderr: │ 308 │ │ print(" **Initialization**") │
stderr: │ │
stderr: │ /home/chenzhixuan/anaconda3/lib/python3.9/site-packages/accelerate/accelerator.py:323 in │
stderr: │ __init__ │
stderr: │ │
stderr: │ 320 │ │ │ │ │ │ self.init_handler = handler │
stderr: │ 321 │ │ │
stderr: │ 322 │ │ kwargs = self.init_handler.to_kwargs() if self.init_handler is not None else {} │
stderr: │ ❱ 323 │ │ self.state = AcceleratorState( │
stderr: │ 324 │ │ │ mixed_precision=mixed_precision, │
stderr: │ 325 │ │ │ cpu=cpu, │
stderr: │ 326 │ │ │ dynamo_backend=dynamo_backend, │
stderr: │ │
stderr: │ /home/chenzhixuan/anaconda3/lib/python3.9/site-packages/accelerate/state.py:162 in __init__ │
stderr: │ │
stderr: │ 159 │ │ │ elif int(os.environ.get("LOCAL_RANK", -1)) != -1 and not cpu: │
stderr: │ 160 │ │ │ │ self.distributed_type = DistributedType.MULTI_GPU │
stderr: │ 161 │ │ │ │ if not torch.distributed.is_initialized(): │
stderr: │ ❱ 162 │ │ │ │ │ torch.distributed.init_process_group(backend="nccl", **kwargs) │
stderr: │ 163 │ │ │ │ │ self.backend = "nccl" │
stderr: │ 164 │ │ │ │ self.num_processes = torch.distributed.get_world_size() │
stderr: │ 165 │ │ │ │ self.process_index = torch.distributed.get_rank() │
stderr: │ │
stderr: │ /home/chenzhixuan/anaconda3/lib/python3.9/site-packages/torch/distributed/distributed_c10d.py:76 │
stderr: │ 1 in init_process_group │
stderr: │ │
stderr: │ 758 │ │ │ # different systems (e.g. RPC) in case the store is multi-tenant. │
stderr: │ 759 │ │ │ store = PrefixStore("default_pg", store) │
stderr: │ 760 │ │ │
stderr: │ ❱ 761 │ │ default_pg = _new_process_group_helper( │
stderr: │ 762 │ │ │ world_size, │
stderr: │ 763 │ │ │ rank, │
stderr: │ 764 │ │ │ [], │
stderr: │ │
stderr: │ /home/chenzhixuan/anaconda3/lib/python3.9/site-packages/torch/distributed/distributed_c10d.py:89 │
stderr: │ 7 in _new_process_group_helper │
stderr: │ │
stderr: │ 894 │ │ │ │ pg_options.is_high_priority_stream = False │
stderr: │ 895 │ │ │ │ pg_options._timeout = timeout │
stderr: │ 896 │ │ │ │
stderr: │ ❱ 897 │ │ │ pg = ProcessGroupNCCL(prefix_store, group_rank, group_size, pg_options) │
stderr: │ 898 │ │ │ # In debug mode and if GLOO is available, wrap in a wrapper PG that │
stderr: │ 899 │ │ │ # enables enhanced collective checking for debugability. │
stderr: │ 900 │ │ │ if get_debug_level() == DebugLevel.DETAIL: │
stderr: ╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
stderr: RuntimeError: ProcessGroupNCCL is only supported with GPUs, no GPUs found!
stderr: ╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
stderr: │ /home/chenzhixuan/anaconda3/lib/python3.9/site-packages/accelerate/test_utils/scripts/test_scrip │
stderr: │ t.py:336 in <module> │
stderr: │ │
stderr: │ 333 │
stderr: │ 334 │
stderr: │ 335 if __name__ == " __main__": │
stderr: │ ❱ 336 │ main() │
stderr: │ 337 │
stderr: │ │
stderr: │ /home/chenzhixuan/anaconda3/lib/python3.9/site-packages/accelerate/test_utils/scripts/test_scrip │
stderr: │ t.py:305 in main │
stderr: │ │
stderr: │ 302 │
stderr: │ 303 │
stderr: │ 304 def main(): │
stderr: │ ❱ 305 │ accelerator = Accelerator() │
stderr: │ 306 │ state = accelerator.state │
stderr: │ 307 │ if state.local_process_index == 0: │
stderr: │ 308 │ │ print(" **Initialization**") │
stderr: │ │
stderr: │ /home/chenzhixuan/anaconda3/lib/python3.9/site-packages/accelerate/accelerator.py:323 in │
stderr: │ __init__ │
stderr: │ │
stderr: │ 320 │ │ │ │ │ │ self.init_handler = handler │
stderr: │ 321 │ │ │
stderr: │ 322 │ │ kwargs = self.init_handler.to_kwargs() if self.init_handler is not None else {} │
stderr: │ ❱ 323 │ │ self.state = AcceleratorState( │
stderr: │ 324 │ │ │ mixed_precision=mixed_precision, │
stderr: │ 325 │ │ │ cpu=cpu, │
stderr: │ 326 │ │ │ dynamo_backend=dynamo_backend, │
stderr: │ │
stderr: │ /home/chenzhixuan/anaconda3/lib/python3.9/site-packages/accelerate/state.py:162 in __init__ │
stderr: │ │
stderr: │ 159 │ │ │ elif int(os.environ.get("LOCAL_RANK", -1)) != -1 and not cpu: │
stderr: │ 160 │ │ │ │ self.distributed_type = DistributedType.MULTI_GPU │
stderr: │ 161 │ │ │ │ if not torch.distributed.is_initialized(): │
stderr: │ ❱ 162 │ │ │ │ │ torch.distributed.init_process_group(backend="nccl", **kwargs) │
stderr: │ 163 │ │ │ │ │ self.backend = "nccl" │
stderr: │ 164 │ │ │ │ self.num_processes = torch.distributed.get_world_size() │
stderr: │ 165 │ │ │ │ self.process_index = torch.distributed.get_rank() │
stderr: │ │
stderr: │ /home/chenzhixuan/anaconda3/lib/python3.9/site-packages/torch/distributed/distributed_c10d.py:76 │
stderr: │ 1 in init_process_group │
stderr: │ │
stderr: │ 758 │ │ │ # different systems (e.g. RPC) in case the store is multi-tenant. │
stderr: │ 759 │ │ │ store = PrefixStore("default_pg", store) │
stderr: │ 760 │ │ │
stderr: │ ❱ 761 │ │ default_pg = _new_process_group_helper( │
stderr: │ 762 │ │ │ world_size, │
stderr: │ 763 │ │ │ rank, │
stderr: │ 764 │ │ │ [], │
stderr: │ │
stderr: │ /home/chenzhixuan/anaconda3/lib/python3.9/site-packages/torch/distributed/distributed_c10d.py:89 │
stderr: │ 7 in _new_process_group_helper │
stderr: │ │
stderr: │ 894 │ │ │ │ pg_options.is_high_priority_stream = False │
stderr: │ 895 │ │ │ │ pg_options._timeout = timeout │
stderr: │ 896 │ │ │ │
stderr: │ ❱ 897 │ │ │ pg = ProcessGroupNCCL(prefix_store, group_rank, group_size, pg_options) │
stderr: │ 898 │ │ │ # In debug mode and if GLOO is available, wrap in a wrapper PG that │
stderr: │ 899 │ │ │ # enables enhanced collective checking for debugability. │
stderr: │ 900 │ │ │ if get_debug_level() == DebugLevel.DETAIL: │
stderr: ╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
stderr: RuntimeError: ProcessGroupNCCL is only supported with GPUs, no GPUs found!

```

but my config is multi-GPU distributed mode:

```auto
- `Accelerate` version: 0.16.0
- Platform: Linux-5.15.0-58-generic-x86_64-with-glibc2.31
- Python version: 3.9.13
- Numpy version: 1.21.5
- PyTorch version (GPU?): 1.13.0+cu116 (True)
- `Accelerate` default config:
        - compute_environment: LOCAL_MACHINE
        - distributed_type: MULTI_GPU
        - mixed_precision: fp16
        - use_cpu: False
        - dynamo_backend: NO
        - num_processes: 2
        - machine_rank: 0
        - num_machines: 1
        - gpu_ids: [0,1]
        - rdzv_backend: static
        - same_network: True
        - main_training_function: main
        - deepspeed_config: {}
        - fsdp_config: {}
        - megatron_lm_config: {}
        - downcast_bf16: no

```

---

<div class="post-metadata">

**Author:** ![violasara](https://avatars.discourse-cdn.com/v4/letter/v/a9adbd/32.png) [@violasara](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fviolasara)%5C
**Post date:** [March 23, 2023, 7:47am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fno-gpus-found-in-a-machine-definitely-with-gpus%2F32769%2F2 "2023-03-23T07:47:10Z")

</div>

I’m having the same problem. I use the following test-script:

```auto
from accelerate import Accelerator
import torch

if __name__ == " __main__":
    print("Cuda support:", torch.cuda.is_available(),":", torch.cuda.device_count(), "devices")
    accelerator = Accelerator()
    print(accelerator.state)

```

Started without accelerate, the output is

```auto
Cuda support: True : 8 devices
Distributed environment: NO
Num processes: 1
Process index: 0
Local process index: 0
Device: cuda
Mixed precision type: no

```

Started with accelerate launch test.py, torch.cuda.is\_available() returns false:

```auto
Cuda support: False : 0 devices
Distributed environment: NO
Num processes: 1
Process index: 0
Local process index: 0
Device: cpu
Mixed precision type: no

```

The reason seems to be that after creating a subprocess (e.g. in line 397 of accelerate/commands/launch.py), calling torch.cuda.is\_available() from the subprocess is not working. Am I missing something?

---

<div class="post-metadata">

**Author:** ![Trusure](https://avatars.discourse-cdn.com/v4/letter/t/df705f/32.png) [@Trusure](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FTrusure)%5C
**Post date:** [March 23, 2023, 9:50am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fno-gpus-found-in-a-machine-definitely-with-gpus%2F32769%2F3 "2023-03-23T09:50:52Z")

</div>

I haven’t figure it out 😂

---

<div class="post-metadata">

**Author:** ![muellerzr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/muellerzr/32/21583_2.png) [@muellerzr](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmuellerzr)%5C
**Post date:** [March 23, 2023, 11:42am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fno-gpus-found-in-a-machine-definitely-with-gpus%2F32769%2F4 "2023-03-23T11:42:06Z")

</div>

What is `accelerate env`’s output currently?

---

<div class="post-metadata">

**Author:** ![violasara](https://avatars.discourse-cdn.com/v4/letter/v/a9adbd/32.png) [@violasara](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fviolasara)%5C
**Post date:** [March 23, 2023, 11:56am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fno-gpus-found-in-a-machine-definitely-with-gpus%2F32769%2F5 "2023-03-23T11:56:03Z")

</div>

```auto
- `Accelerate` version: 0.17.1
- Platform: Linux-5.4.0-65-generic-x86_64-with-glibc2.31
- Python version: 3.9.16
- Numpy version: 1.23.5
- PyTorch version (GPU?): 1.12.1 (True)
- `Accelerate` default config:
        - compute_environment: LOCAL_MACHINE
        - distributed_type: NO
        - mixed_precision: no
        - use_cpu: False
        - num_processes: 1
        - machine_rank: 0
        - num_machines: 1
        - gpu_ids: [3,4,5]
        - rdzv_backend: static
        - same_network: True
        - main_training_function: main
        - deepspeed_config: {}
        - fsdp_config: {}
        - megatron_lm_config: {}
        - downcast_bf16: no
        - tpu_use_cluster: False
        - tpu_use_sudo: False
        - dynamo_config: {}

```

Sorry, I updated accelerate in the meantime. It is line 574 now in launch.py

---

<div class="post-metadata">

**Author:** ![violasara](https://avatars.discourse-cdn.com/v4/letter/v/a9adbd/32.png) [@violasara](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fviolasara)%5C
**Post date:** [March 24, 2023, 12:38pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fno-gpus-found-in-a-machine-definitely-with-gpus%2F32769%2F6 "2023-03-24T12:38:24Z")

</div>

Hi, do you have any suggestion? Any help would be appreciated

---

<div class="post-metadata">

**Author:** ![muellerzr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/muellerzr/32/21583_2.png) [@muellerzr](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmuellerzr)%5C
**Post date:** [March 24, 2023, 1:52pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fno-gpus-found-in-a-machine-definitely-with-gpus%2F32769%2F7 "2023-03-24T13:52:38Z")

</div>

Can you open an issue on our github for this and I can take a look? Please ensure to follow all the bits we ask for on there 🙂 [Sign in to GitHub · GitHub](https://github.com/huggingface/accelerate/issues/new?assignees=&labels=&template=bug-report.yml)

---

<div class="post-metadata">

**Author:** ![violasara](https://avatars.discourse-cdn.com/v4/letter/v/a9adbd/32.png) [@violasara](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fviolasara)%5C
**Post date:** [March 28, 2023, 5:08pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fno-gpus-found-in-a-machine-definitely-with-gpus%2F32769%2F8 "2023-03-28T17:08:31Z")

</div>

I did my best: 🙂 [https://github.com/huggingface/accelerate/issues/1260](https://github.com/huggingface/accelerate/issues/1260)

---

<div class="post-metadata">

**Author:** ![1TuanPham](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/1tuanpham/32/22498_2.png) [@1TuanPham](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2F1TuanPham)%5C
**Post date:** [December 27, 2023, 7:26pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fno-gpus-found-in-a-machine-definitely-with-gpus%2F32769%2F9 "2023-12-27T19:26:25Z")

</div>

Apparently if there’s a mismatch in the cuda version, accelerate env will output False for a setup with gpu, so just a head up for everyone
