Difference between revisions of "Slurm"

From DISI
Jump to: navigation, search
m
(Submit load2d Jobs)
(41 intermediate revisions by 2 users not shown)
Line 1: Line 1:
'''[[Detailed step-by-step instruction:]]'''
+
'''Slurm user-guide'''
 +
=== Submit Jobs with Slurm ===
  
node n-1-17
+
==== SBATCH-MR (beta) ====
 +
It is a slurm-version of qsub-mr for submitting job on Slurm queueing system. Note: this is have not been extensively tested yet. Please contact me if the script is not working out. We are hoping to fully migrate to Slurm from the out-dated SGE system. Any error report would be helpful - Khanh
  
* make sure you have there Centos 7: cat /etc/redhat-release
+
New slurm scripts are located in /nfs/soft/tools/utils/sbatch-slice
* wget https://download.schedmd.com/slurm/slurm-17.02.11.tar.bz2
+
Just simply replace /nfs/soft/tools/utils/qsub-slice/qsub-mr with /nfs/soft/tools/utils/sbatch-slice/sbatch-mr in your script
* yum install readline-devel perl-ExtUtils-MakeMaker.noarch munge-devel pam-devel
+
 
* export VER=17.02.11; rpmbuild -ta slurm-$VER.tar.bz2 --without mysql; mv /root/rpmbuild .
+
To check the status of your job:
 +
By username
 +
  $ squeue -u <username>
 +
By jobid
 +
  $ squeue -j <job_id>
 +
 
 +
==== Submit load2d Jobs ====
 +
 
 +
$ cd <catalog_shortname>
 +
$ source /nfs/exa/work/khtang/ZINC21_load2d/loadenv_zinc21.sh
 +
(development) $ sh /nfs/exa/work/khtang/submit_scripts/sbatch_slice/batch_zinc21.slurm <catalog_shortname>.ism
 +
 
 +
==== Submit DOCK Jobs ====
 +
 
 +
* ANACONDA Installation (Python 2.7)
 +
 
 +
Each user is welcome to download anaconda and install into his/her own folder<br>
 +
https://www.anaconda.com/distribution/<br>
 +
''wget https://repo.anaconda.com/archive/Anaconda2-2019.10-Linux-x86_64.sh''<br>
 +
NB: It is also available for Python3, which is our nearest future<br>
 +
 
 +
simple installation via ''/bin/sh Anaconda2-2019.10-Linux-x86_64.sh''
 +
 
 +
After the installation is completed, you need to install a few packages:
 +
conda install -c free bsddb
 +
conda install -c rdkit rdkit
 +
conda install numpy
 +
 
 +
 
 +
'''Running DOCK-3.7 with Slurm'''
 +
 
 +
Here is a “guinea pig project”, which has been done with DOCK-3.7 locally.<br>
 +
'''GPR40 example:''' /mnt/nfs/home/dudenko/TEST_DOCKING_PROJECT<br>
 +
'''ChEMBL ligands:''' /mnt/nfs/home/dudenko/CHEMBL4422_active_ligands<br>
 +
This test calculation should run smoothly, if not, then there is a problem.<br>
 +
Ultimately, you may need to compare your results with the reference run:
 +
* CHEMBL4422_active_ligands.mol2 - TOP500 scoring poses
 +
* extract_all.sort.uniq.txt - a print-out of scoring details
 +
 
 +
 
 +
 
 +
Slurm queue manager is installed locally at gimel, use it to run this test (and all your future jobs) in parallel.<br>
 +
Do not forget to set DOCKBASE variable: ''export DOCKBASE=/nfs/soft/dock/versions/dock37/DOCK-3.7.3rc1/''
 +
 
 +
'''Useful DOCKING commands to remind:'''
 +
$DOCKBASE/docking/setup/setup_db2_zinc15_file_number.py ./ CHEMBL4422_active_ligands_ CHEMBL4422_active_ligands.sdi 100 count
 +
$DOCKBASE/analysis/extract_all.py -s -20
 +
$DOCKBASE/analysis/getposes.py -l 500 -o CHEMBL4422_active_ligands.mol2
 +
 
 +
 
 +
'''Useful slurm commands (see https://slurm.schedmd.com/quickstart.html):'''
 +
to see what machine resources are offered by the cluster, do ''sinfo -lNe''
 +
to submit a DOCK-3.7 job, run ''$DOCKBASE/docking/submit/submit_slurm_array.csh''
 +
to see what is happening in the queue, run ''squeue''
 +
to see a detailed info for a specific job: ''scontrol show jobid=_JOBID_''
 +
to delete a job from queue, run ''scancel _JOBID_''
 +
 
 +
Should your slurm run correctly, type ''squeue'' and you should see something like this:
 +
 
 +
            JOBID PARTITION    NAME    USER ST      TIME  NODES NODELIST(REASON)
 +
  4187_[637-2091]    gimel array_jo  dudenko PD      0:00      1 (Resources)
 +
          4187_629    gimel array_jo  dudenko  R      0:00      1 n-1-20
 +
          4187_630    gimel array_jo  dudenko  R      0:00      1 n-5-34
 +
          4187_631    gimel array_jo  dudenko  R      0:00      1 n-1-21
 +
          4187_632    gimel array_jo  dudenko  R      0:00      1 n-5-34
 +
          4187_633    gimel array_jo  dudenko  R      0:00      1 n-1-21
 +
          4187_634    gimel array_jo  dudenko  R      0:00      1 n-5-34
 +
          4187_635    gimel array_jo  dudenko  R      0:00      1 n-5-35
 +
          4187_636    gimel array_jo  dudenko  R      0:00      1 n-5-34
 +
          4187_622    gimel array_jo  dudenko  R      0:01      1 n-5-34
 +
          4187_623    gimel array_jo  dudenko  R      0:01      1 n-5-34
 +
          4187_624    gimel array_jo  dudenko  R      0:01      1 n-5-35
 +
          4187_625    gimel array_jo  dudenko  R      0:01      1 n-1-17
 +
 
 +
 
 +
 
 +
As root at gimel, it is possible to modify a particular job, e.g., ''scontrol update jobid=635 TimeLimit=7-00:00:00''
 +
 
 +
=== Slurm Installation Guide ===
 +
'''Detailed step-by-step installation instruction (for sysadmins only)'''
 +
 
 +
==== Setup Node master ====
 +
'''TBA'''
 +
 
 +
==== Setup Compute Nodes ====
 +
 
 +
 
 +
'''Useful links:'''
 +
https://slurm.schedmd.com/quickstart_admin.html
 +
https://wiki.fysik.dtu.dk/niflheim/Slurm_installation
 +
 
 +
'''node n-1-17'''
 +
 
 +
* make sure you have there Centos 7: ''cat /etc/redhat-release''
 +
* ''wget https://download.schedmd.com/slurm/slurm-17.02.11.tar.bz2''
 +
* ''yum install readline-devel perl-ExtUtils-MakeMaker.noarch munge-devel pam-devel''
 +
* ''export VER=17.02.11; rpmbuild -ta slurm-$VER.tar.bz2 --without mysql; mv /root/rpmbuild .''
  
 
installing built packages from rpmbuild:
 
installing built packages from rpmbuild:
* yum install rpmbuild/RPMS/x86_64/slurm-plugins-17.02.11-1.el7.x86_64.rpm
+
* ''yum install rpmbuild/RPMS/x86_64/slurm-plugins-17.02.11-1.el7.x86_64.rpm''
* yum install rpmbuild/RPMS/x86_64/slurm-17.02.11-1.el7.x86_64.rpm
+
* ''yum install rpmbuild/RPMS/x86_64/slurm-17.02.11-1.el7.x86_64.rpm''
* yum install rpmbuild/RPMS/x86_64/slurm-munge-17.02.11-1.el7.x86_64.rpm
+
* ''yum install rpmbuild/RPMS/x86_64/slurm-munge-17.02.11-1.el7.x86_64.rpm''
  
  
Line 17: Line 115:
 
copy over /etc/munge/munge.key from gimel and put locally to /etc/munge. The key should be identical allover the nodes.<br>
 
copy over /etc/munge/munge.key from gimel and put locally to /etc/munge. The key should be identical allover the nodes.<br>
 
Munge is a daemon responsible for secure data exchange between nodes.<br>
 
Munge is a daemon responsible for secure data exchange between nodes.<br>
Set permissions accordingly: chown munge:munge /etc/munge/munge.key; chmod 400 /etc/munge/munge.key<br>
+
Set permissions accordingly: ''chown munge:munge /etc/munge/munge.key; chmod 400 /etc/munge/munge.key''<br>
  
'''starting munge''': systemctl enable munge; systemctl start munge
+
'''starting munge''': ''systemctl enable munge; systemctl start munge''
  
 
'''setting up slurm''':
 
'''setting up slurm''':
Line 25: Line 123:
 
* all UID/GUIDs of slurm user should be identical allover the nodes.<br>
 
* all UID/GUIDs of slurm user should be identical allover the nodes.<br>
 
   Otherwise, one needs to specify a mapping scheme for translating each UID/GUIDs between nodes.<br>
 
   Otherwise, one needs to specify a mapping scheme for translating each UID/GUIDs between nodes.<br>
   To edit slurm UID/GUID, do "vipw" and replace "slurm line" with slurm:x:XXXXX:YYYYY::/nonexistent:/bin/false<br>
+
   To edit slurm UID/GUID, do ''vipw'' and replace "slurm line" with slurm:x:XXXXX:YYYYY::/nonexistent:/bin/false<br>
 
   XXXXX and YYYYY for slurm user can be found at gimel in /etc/passwd<br>
 
   XXXXX and YYYYY for slurm user can be found at gimel in /etc/passwd<br>
 
   NB: don't forget to edit /etc/group as well.<br>
 
   NB: don't forget to edit /etc/group as well.<br>
Line 32: Line 130:
 
   NodeName=n-1-17 NodeAddr=10.20.1.17 CPUs=24 State=UNKNOWN
 
   NodeName=n-1-17 NodeAddr=10.20.1.17 CPUs=24 State=UNKNOWN
 
* append n-1-17 to the partition: PartitionName=gimel Nodes=gimel,n-5-34,n-5-35,n-1-17 Default=YES MaxTime=INFINITE State=UP
 
* append n-1-17 to the partition: PartitionName=gimel Nodes=gimel,n-5-34,n-5-35,n-1-17 Default=YES MaxTime=INFINITE State=UP
 +
* create the following folders:
 +
  ''mkdir -p /var/spool/slurm-llnl /var/run/slurm-llnl /var/log/slurm-llnl''
 +
  ''chown -R slurm:slurm /var/spool/slurm-llnl /var/run/slurm-llnl /var/log/slurm-llnl''
 +
* restarting slurm master node at gimel (Centos 6): ''/etc/init.d/slurm restart''
 +
* enabling and starting slurm computing nodes (Centos 7): ''systemctl enable slurmd; systemctl start slurmd''
 +
 +
And last but not least, asking the firewall to allow communication between master node and computing node n-1-17:
 +
* ''firewall-cmd --permanent --zone=public --add-port=6818/tcp''
 +
* ''firewall-cmd --reload''
 +
 +
 +
To see the current situation of the queue, so ''sinfo -lNe'' and you will see:
 +
Wed May 27 09:49:54 2020
 +
NODELIST  NODES PARTITION      STATE CPUS    S:C:T MEMORY TMP_DISK WEIGHT AVAIL_FE REASON             
 +
gimel          1    gimel*    drained  24    4:6:1      1        0      1  (null) none               
 +
n-1-17        1    gimel*        idle  24  24:1:1      1        0      1  (null) none               
 +
n-5-34        1    gimel*        idle  80  80:1:1      1        0      1  (null) none               
 +
n-5-35        1    gimel*        idle  80  80:1:1      1        0      1  (null) none
 +
 +
 +
To disable a specific node, do ''scontrol update NodeName=n-1-17 State=DRAIN Reason=DRAINED''<br>
 +
To return back to service, do ''scontrol update NodeName=n-1-17 State=IDLE''
 +
 +
 +
p.s. Some users/scripts may require csh/tcsh.<br>
 +
''sudo yum install csh tcsh''
 +
 +
==== Node down after reboot ====
 +
On gimel (master node)
 +
sudo scontrol update NodeName=<node_name> State=RESUME
 +
 +
 +
'''Useful links:'''
 +
https://support.ceci-hpc.be/doc/_contents/QuickStart/SubmittingJobs/SlurmTutorial.html
 +
 +
 +
Back to [[DOCK_3.7]]
 +
 +
[[Category : Slurm]]

Revision as of 15:01, 29 June 2020

Slurm user-guide

Submit Jobs with Slurm

SBATCH-MR (beta)

It is a slurm-version of qsub-mr for submitting job on Slurm queueing system. Note: this is have not been extensively tested yet. Please contact me if the script is not working out. We are hoping to fully migrate to Slurm from the out-dated SGE system. Any error report would be helpful - Khanh

New slurm scripts are located in /nfs/soft/tools/utils/sbatch-slice

Just simply replace /nfs/soft/tools/utils/qsub-slice/qsub-mr with /nfs/soft/tools/utils/sbatch-slice/sbatch-mr in your script

To check the status of your job:

By username
 $ squeue -u <username>
By jobid
 $ squeue -j <job_id>

Submit load2d Jobs

$ cd <catalog_shortname>
$ source /nfs/exa/work/khtang/ZINC21_load2d/loadenv_zinc21.sh
(development) $ sh /nfs/exa/work/khtang/submit_scripts/sbatch_slice/batch_zinc21.slurm <catalog_shortname>.ism

Submit DOCK Jobs

  • ANACONDA Installation (Python 2.7)

Each user is welcome to download anaconda and install into his/her own folder
https://www.anaconda.com/distribution/
wget https://repo.anaconda.com/archive/Anaconda2-2019.10-Linux-x86_64.sh
NB: It is also available for Python3, which is our nearest future

simple installation via /bin/sh Anaconda2-2019.10-Linux-x86_64.sh

After the installation is completed, you need to install a few packages:

conda install -c free bsddb
conda install -c rdkit rdkit
conda install numpy


Running DOCK-3.7 with Slurm

Here is a “guinea pig project”, which has been done with DOCK-3.7 locally.
GPR40 example: /mnt/nfs/home/dudenko/TEST_DOCKING_PROJECT
ChEMBL ligands: /mnt/nfs/home/dudenko/CHEMBL4422_active_ligands
This test calculation should run smoothly, if not, then there is a problem.
Ultimately, you may need to compare your results with the reference run:

  • CHEMBL4422_active_ligands.mol2 - TOP500 scoring poses
  • extract_all.sort.uniq.txt - a print-out of scoring details


Slurm queue manager is installed locally at gimel, use it to run this test (and all your future jobs) in parallel.
Do not forget to set DOCKBASE variable: export DOCKBASE=/nfs/soft/dock/versions/dock37/DOCK-3.7.3rc1/

Useful DOCKING commands to remind:

$DOCKBASE/docking/setup/setup_db2_zinc15_file_number.py ./ CHEMBL4422_active_ligands_ CHEMBL4422_active_ligands.sdi 100 count
$DOCKBASE/analysis/extract_all.py -s -20
$DOCKBASE/analysis/getposes.py -l 500 -o CHEMBL4422_active_ligands.mol2


Useful slurm commands (see https://slurm.schedmd.com/quickstart.html):

to see what machine resources are offered by the cluster, do sinfo -lNe
to submit a DOCK-3.7 job, run $DOCKBASE/docking/submit/submit_slurm_array.csh
to see what is happening in the queue, run squeue
to see a detailed info for a specific job: scontrol show jobid=_JOBID_
to delete a job from queue, run scancel _JOBID_

Should your slurm run correctly, type squeue and you should see something like this:

            JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
  4187_[637-2091]     gimel array_jo  dudenko PD       0:00      1 (Resources)
         4187_629     gimel array_jo  dudenko  R       0:00      1 n-1-20
         4187_630     gimel array_jo  dudenko  R       0:00      1 n-5-34
         4187_631     gimel array_jo  dudenko  R       0:00      1 n-1-21
         4187_632     gimel array_jo  dudenko  R       0:00      1 n-5-34
         4187_633     gimel array_jo  dudenko  R       0:00      1 n-1-21
         4187_634     gimel array_jo  dudenko  R       0:00      1 n-5-34
         4187_635     gimel array_jo  dudenko  R       0:00      1 n-5-35
         4187_636     gimel array_jo  dudenko  R       0:00      1 n-5-34
         4187_622     gimel array_jo  dudenko  R       0:01      1 n-5-34
         4187_623     gimel array_jo  dudenko  R       0:01      1 n-5-34
         4187_624     gimel array_jo  dudenko  R       0:01      1 n-5-35
         4187_625     gimel array_jo  dudenko  R       0:01      1 n-1-17


As root at gimel, it is possible to modify a particular job, e.g., scontrol update jobid=635 TimeLimit=7-00:00:00

Slurm Installation Guide

Detailed step-by-step installation instruction (for sysadmins only)

Setup Node master

TBA

Setup Compute Nodes

Useful links:

https://slurm.schedmd.com/quickstart_admin.html
https://wiki.fysik.dtu.dk/niflheim/Slurm_installation

node n-1-17

  • make sure you have there Centos 7: cat /etc/redhat-release
  • wget https://download.schedmd.com/slurm/slurm-17.02.11.tar.bz2
  • yum install readline-devel perl-ExtUtils-MakeMaker.noarch munge-devel pam-devel
  • export VER=17.02.11; rpmbuild -ta slurm-$VER.tar.bz2 --without mysql; mv /root/rpmbuild .

installing built packages from rpmbuild:

  • yum install rpmbuild/RPMS/x86_64/slurm-plugins-17.02.11-1.el7.x86_64.rpm
  • yum install rpmbuild/RPMS/x86_64/slurm-17.02.11-1.el7.x86_64.rpm
  • yum install rpmbuild/RPMS/x86_64/slurm-munge-17.02.11-1.el7.x86_64.rpm


setting up munge: copy over /etc/munge/munge.key from gimel and put locally to /etc/munge. The key should be identical allover the nodes.
Munge is a daemon responsible for secure data exchange between nodes.
Set permissions accordingly: chown munge:munge /etc/munge/munge.key; chmod 400 /etc/munge/munge.key

starting munge: systemctl enable munge; systemctl start munge

setting up slurm:

  • create a user slurm: adduser slurm.
  • all UID/GUIDs of slurm user should be identical allover the nodes.
 Otherwise, one needs to specify a mapping scheme for translating each UID/GUIDs between nodes.
To edit slurm UID/GUID, do vipw and replace "slurm line" with slurm:x:XXXXX:YYYYY::/nonexistent:/bin/false
XXXXX and YYYYY for slurm user can be found at gimel in /etc/passwd
NB: don't forget to edit /etc/group as well.
  • copy /etc/slurm/slurm.conf from gimel and put locally to /etc/slurm.
  • figure out what CPU/Memory resources you have at n-1-17 (see /proc/cpuinfo) and append the following line:
 NodeName=n-1-17 NodeAddr=10.20.1.17 CPUs=24 State=UNKNOWN
  • append n-1-17 to the partition: PartitionName=gimel Nodes=gimel,n-5-34,n-5-35,n-1-17 Default=YES MaxTime=INFINITE State=UP
  • create the following folders:
 mkdir -p /var/spool/slurm-llnl /var/run/slurm-llnl /var/log/slurm-llnl
 chown -R slurm:slurm /var/spool/slurm-llnl /var/run/slurm-llnl /var/log/slurm-llnl
  • restarting slurm master node at gimel (Centos 6): /etc/init.d/slurm restart
  • enabling and starting slurm computing nodes (Centos 7): systemctl enable slurmd; systemctl start slurmd

And last but not least, asking the firewall to allow communication between master node and computing node n-1-17:

  • firewall-cmd --permanent --zone=public --add-port=6818/tcp
  • firewall-cmd --reload


To see the current situation of the queue, so sinfo -lNe and you will see:

Wed May 27 09:49:54 2020
NODELIST   NODES PARTITION       STATE CPUS    S:C:T MEMORY TMP_DISK WEIGHT AVAIL_FE REASON              
gimel          1    gimel*     drained   24    4:6:1      1        0      1   (null) none                
n-1-17         1    gimel*        idle   24   24:1:1      1        0      1   (null) none                
n-5-34         1    gimel*        idle   80   80:1:1      1        0      1   (null) none                
n-5-35         1    gimel*        idle   80   80:1:1      1        0      1   (null) none


To disable a specific node, do scontrol update NodeName=n-1-17 State=DRAIN Reason=DRAINED
To return back to service, do scontrol update NodeName=n-1-17 State=IDLE


p.s. Some users/scripts may require csh/tcsh.
sudo yum install csh tcsh

Node down after reboot

On gimel (master node)

sudo scontrol update NodeName=<node_name> State=RESUME


Useful links:

https://support.ceci-hpc.be/doc/_contents/QuickStart/SubmittingJobs/SlurmTutorial.html


Back to DOCK_3.7