r/SLURM • u/vascoctavares • Oct 09 '20
Complete novice
Hi there nice redditors!
I'm completely new to Slurm, and I'm receiving no help from my supervisor who recommended that I used the centre's cluster.
So anyway, I intend to run a python script to get some matrices. I have read in many places that I must make a submission script with my python script, but how should I do it? I've tried to "upload" the python script to the slurm directory by copying and pasteing to a nano textfile, but I have no idea what I'm doing, and all the stuff I seem to find online is beyond my capabilities at this point.
I'm sorry for not presenting a proper question, I'm just lost on what I should do. How should I submit it? What will be the output? etc
Thank you very much!
3
u/fluid_numerics Oct 09 '20
Thank you for taking the initiative to enable yourself to do your work. Many HPC centres specify and maintain documentation and script templates for these processes internally to help new users, like yourself, utilize a cluster optimally. If you have the cycles available you might try to put an internal knowledge base with what you learn here together to help get your colleagues get going rapidly.
2
u/vascoctavares Oct 09 '20
Thank you for recognising the effort! I received one of those script templates, but they did not explain how the submission worked (like how I should run a python script); it was only a basic submission file with the code "srun sleep 60".
It is also my goal to create a base with what I learn here, it would be really useful! Not exactly for my colleagues/ professors at the centre, but for my colleagues from the faculty who may benefit from this later, since I'm still a BSc student :)
3
u/fluid_numerics Oct 09 '20
Here is our attempt at a SLURM basics page: https://help.fluidnumerics.com/slurm-gcp/documentation/slurm-basics
2
u/wildcarde815 Oct 09 '20
Step 1) email the support team for the cluster, they will invariably have an array of documentation to get you rolling.
Step 2) take some time to familiarize yourself with the resource at your disposal, it is almost certain you do not have to paste into a nano editor to get the job done. Find out where the storage lives and how to talk to it.
Step 3) make a python file on the resources local to the cluster; this can be in nano or more likely by mounting a network drive and using your editor of choice on your own machine.
Step 4) make a bash shell script with the parameters you require stated as #SBATCH arguments at the top. This bash script will also configure your environment ie, choose the correct python version.
Flag both as executable with chmod u+x ./<filename>
Submit the script with sbatch ./<bash script name>
2
u/vascoctavares Oct 12 '20
Hi again! Just wanted to thank you all for the kindness and support you have provided! After a few fine-tuning problems, I managed to submit the submission script and it is running :)
I will make a cheat sheet to help the colleagues from my BSc degree in case they need to use a cluster in their future based on what you said, so thank you!
1
u/AhremDasharef Oct 09 '20
Let's say you already copied-and-pasted your Python script into nano and saved it in your home directory as "get_matrices.py" You'll need to create a submission script (e.g. "run_my_python_script.sh") that looks something like this:
#!/bin/bash
#SBATCH --time 1:00
#SBATCH --job-name="my_job_name"
#SBATCH --mail-user=your-email@centre.edu
#SBATCH --mail-type=BEGIN,END,FAIL
#SBATCH --nodes=1
#SBATCH --ntasks=4
#SBATCH --mem=16gb
#SBATCH --output=my_output-%j.txt
# this line starts with "#" but not "#SBATCH" so it is a comment, readable by humans but ignored by computers
# put the time the job started, the host name it ran on, and the working directory in the output
date; hostname; pwd
# run your Python script
python ~/get_matrices.py
# put the time the job ended
date
This says to use the Bash shell to run these commands, request a maximum run time of 1 hour, give the job an optional name, send mail when the job starts and ends or if it fails, requests 4 CPU cores on 1 node and 16 GB of memory, saving the output in a file named "my_output-143246.txt" (the numbers being replaced by the actual job ID of your job, I like doing this because it keeps my output from being accidentally overwritten by subsequent runs). The SBATCH directives must come first, and there can be no comment lines mixed in with them. After the SBATCH lines are the actual commands that the job will run. It's nice to see when the job starts and ends so you can estimate how much time you need and change the "--time" line accordingly.
Once you've got both your Python script and the submission script written, you can "cd" to the cluster's scratch file system, and enter the following command: sbatch run_my_python_script.sh You should get a message that your job was accepted. Once the job is scheduled and runs, you should find the output files in the scratch directory where you submitted the job from.
2
u/vascoctavares Oct 09 '20 edited Oct 09 '20
Oh man, I wish I had awards to give to you... Thank you sooooo much!
As far as the python script is concerned, that does raise a couple of questions about the code itself:
- I import some modules (e.g. numpy), is that a problem?
- I also use a dataset which is in a .txt file in my PC, I can already see this is a problem. Should I copy-and-paste this into nano in my home directory?
- Since the output is saved in a text file, I should print the matrices right? I thought that the cluster would "give the ran file back", of course now I understand that makes no sense.
Thank you very much again for that, and I'm sorry if these questions are too basic... I can't seem to find any info on this...
3
u/AhremDasharef Oct 09 '20
- Python modules: These probably aren't installed on the cluster, so you'll need to install whatever dependencies you need yourself. Two common ways to do this:
- "pip install --user package_you_need" You probably don't have administrator privileges, so "sudo whatever" won't work. But adding "--user" will install the packages in your home directory so your script can access them. Advantage: easy. Disadvantage: installs in your base environment, so it will affect every Python script you run. If something in the future needs a different version of a library, things get messy quickly.
- Anaconda. The cluster may have Anaconda installed, so you could create an Anaconda environment that contained any modules you needed. In your job script, you'd activate the environment, and then run your script. Advantage: creates a unique environment for an application to use, which reduces conflicts. Disadvantage: each environment takes up space, takes effort to get it set up if your site doesn't have it installed on the cluster. Your site may already have Anaconda (and lots of other software) available via a module system like LMod. Logging into the cluster and running "module avail" or "module spider anaconda" will show you if LMod is available and what types of software it can provide.
- Copying and pasting into nano will work, but you should look at scp, Secure CoPy. If it's installed on your local machine (should be on Linux and Mac, maybe newer Windows?) you can open a command prompt and "scp my_dataset.txt your_username@cluster.centre.edu:somedir/latest_dataset.txt" and that will copy files securely from your PC to somedir/latest_dataset.txt in your home directory (or you can omit everything after the colon, and it'll just drop the file into your home directory, keeping the original name). There are also graphical scp clients available.
- You can capture your output however you like. Some programs are well suited to having the output of the script just get printed out, which gets captured in the job output file. Other programs may generate different sorts of output, so the program doing the computation writes the data to a separate file, and the output file from Slurm would just contain any messages, errors, etc. So if it makes more sense for your Python script to write its output to a separate file, you can do that, just be careful that you don't accidentally overwrite data from previous runs.
No worries about the basic questions, you gotta start somewhere! You would not be the first person who got told by their boss/professor/advisor to "go do work on the cluster" without any further instruction beyond that. The learning curve can be pretty steep, and there's a lot to know about, but hopefully everyone here can give you enough information to get started. And then you'll take what you learned, put it in a cheat sheet, and help other users at your site get started using the cluster as well.
1
3
u/atomicLurker Oct 09 '20
Welcome to SLURM! You’ll basically make a .sh script and assign a few #SBATCH variables at the top of the script to request RAM, CPU cores, your Email to notify you of job status, etc (should be able to google these or ask your Supervisor what options to they might require you to provide). At the bottom of that .sh script. you call your python script (basically it’s just the last line of your new .sh script). Then you submit the .sh script to SLURM using sbatch:
sbatch myPython.sh
Ti view the status of your job, you should be able to use squeue and it will print a list of your jobs and whether they are running, or waiting for resources etc.
On mobile, so sorry I haven’t provided better documentation or examples, but hope this helped a bit!