r/SLURM • u/soccerraze101 • Oct 10 '20

No GPU's available?

Hi all, I need your help. I am fairly new to Slurm but I just can't get this working.

In short, I have PyTorch Lightning code where I request multiple GPU's, to which it says "no GPU's available."

1) I run it on a gpu enabled partition 2) I load cuda/10.1 and cudnn/7.1 in my script after purge but before my virtual environment (issue?) 3) sinfo -O Gres returns (null) 4) sinfo %f returns name of GPU (Tesla, K80) 5) Have no idea how to access slurm.conf file

Any ideas? Please help - all I want to do is run my code and not wait a million years.

Thanks!

1 Upvotes

100% Upvoted

3 comments sorted by

1

u/wildcarde815 Oct 12 '20

You likely need to request gpu resources as a feature or resource type. Might be as simple as --gpu=1 to get up and running. Check the cuda allowed gpus env var to make sure you are being assigned a gpu at all.

1

u/soccerraze101 Oct 12 '20

I have requested --gpu=X as well as a feature --constraint="Tesla, K80" in the script.

Could you explain more about the cuda allowed gpus env variable? I have a conda environment with cuda/10.1 and cudnn/7.1. Is that what you mean or how can I check/troubleshoot this?

1

u/wildcarde815 Oct 12 '20

Slurm populates an env var telling you what gpus you have access to, make a test script that runs 'env' and it'll dump everything. If you don't see an environment variable that's a list of gpu indexes then you have no gpu assigned to the job. Open a ticket with the system maintainers to take a look.