Difference between revisions of "Slurm"

From DISI
Jump to: navigation, search
m
(Submit load2d Jobs)
(28 intermediate revisions by 2 users not shown)
Line 1: Line 1:
'''Slurm userguide'''
+
'''Slurm user-guide'''
 +
=== Submit Jobs with Slurm ===
  
 +
==== SBATCH-MR (beta) ====
 +
It is a slurm-version of qsub-mr for submitting job on Slurm queueing system. Note: this is have not been extensively tested yet. Please contact me if the script is not working out. We are hoping to fully migrate to Slurm from the out-dated SGE system. Any error report would be helpful - Khanh
  
'''Useful libraries and utilities on master node (gimel)'''
+
New slurm scripts are located in /nfs/soft/tools/utils/sbatch-slice
 +
Just simply replace /nfs/soft/tools/utils/qsub-slice/qsub-mr with /nfs/soft/tools/utils/sbatch-slice/sbatch-mr in your script
  
 +
To check the status of your job:
 +
By username
 +
  $ squeue -u <username>
 +
By jobid
 +
  $ squeue -j <job_id>
  
== ANACONDA Installation (Python 2.7) ==
+
==== Submit load2d Jobs ====
 +
 
 +
$ cd <catalog_shortname>
 +
$ source /nfs/exa/work/khtang/ZINC21_load2d/loadenv_zinc21.sh
 +
(development) $ sh /nfs/exa/work/khtang/submit_scripts/sbatch_slice/batch_zinc21.slurm <catalog_shortname>.ism
 +
 
 +
==== Submit DOCK Jobs ====
 +
 
 +
* ANACONDA Installation (Python 2.7)
  
 
Each user is welcome to download anaconda and install into his/her own folder<br>
 
Each user is welcome to download anaconda and install into his/her own folder<br>
Line 14: Line 31:
 
simple installation via ''/bin/sh Anaconda2-2019.10-Linux-x86_64.sh''
 
simple installation via ''/bin/sh Anaconda2-2019.10-Linux-x86_64.sh''
  
You may need to install a few packages:
+
After the installation is completed, you need to install a few packages:
 
  conda install -c free bsddb
 
  conda install -c free bsddb
 
  conda install -c rdkit rdkit
 
  conda install -c rdkit rdkit
Line 20: Line 37:
  
  
sinfo -lNe
+
'''Running DOCK-3.7 with Slurm'''
scancel
+
squeue
+
export DOCKBASE=/home/docker/DOCK-3.7.3rc1
+
  
 +
Here is a “guinea pig project”, which has been done with DOCK-3.7 locally.<br>
 +
'''GPR40 example:''' /mnt/nfs/home/dudenko/TEST_DOCKING_PROJECT<br>
 +
'''ChEMBL ligands:''' /mnt/nfs/home/dudenko/CHEMBL4422_active_ligands<br>
 +
This test calculation should run smoothly, if not, then there is a problem.<br>
 +
Ultimately, you may need to compare your results with the reference run:
 +
* CHEMBL4422_active_ligands.mol2 - TOP500 scoring poses
 +
* extract_all.sort.uniq.txt - a print-out of scoring details
  
Running DOCK-3.7 with Slurm
 
Here is a “guinea pig project”, which has been done with DOCK-3.7 locally.
 
GPR40 example: /home/docker/Desktop/EXAMPLE_PROJECT_GPR40/TEST
 
This test calculation should run smoothly, if not, then there is a problem.
 
One needs copy over this test folder and try to reproduce CHEMBL4422_active_ligands.sdi
 
  
Slurm queue is installed locally, use it to run this test (and all your future jobs) in parallel.
 
Do not forget to set DOCKBASE: export DOCKBASE=/home/docker/DOCK-3.7.3rc1
 
  
# Useful commands to remind:
+
Slurm queue manager is installed locally at gimel, use it to run this test (and all your future jobs) in parallel.<br>
scp -C -r -P 3333  -i ~/.ssh/id_rsa_SHUO ProjectX_FOLDER shuo@pdl-station.spdns.org:~/
+
Do not forget to set DOCKBASE variable: ''export DOCKBASE=/nfs/soft/dock/versions/dock37/DOCK-3.7.3rc1/''
$DOCKBASE/docking/setup/setup_db2_zinc15_file_number.py ./ CHEMBL4422_active_ligands_ CHEMBL4422_active_ligands.sdi 100 count
+
$DOCKBASE/analysis/extract_all.py -s -10
+
$DOCKBASE/analysis/getposes.py -l 500 -o CHEMBL4422_active_ligands.mol2
+
$DOCKBASE/analysis/enrich.py -i . -l ligands.names.txt -d decoys.names.txt
+
$DOCKBASE/analysis/plots.py -i . -l ligands.names.txt -d decoys.names.txt
+
  
# Useful slurm commands (see https://slurm.schedmd.com/quickstart.html):
+
'''Useful DOCKING commands to remind:'''
To see what machine resources are offered by the cluster, do “sinfo -lNe”
+
$DOCKBASE/docking/setup/setup_db2_zinc15_file_number.py ./ CHEMBL4422_active_ligands_ CHEMBL4422_active_ligands.sdi 100 count
To submit DOCK-3.7 job, run $DOCKBASE/docking/submit/submit_slurm_array.csh
+
$DOCKBASE/analysis/extract_all.py -s -20
To see what is happening in the queue, run “squeue”
+
  $DOCKBASE/analysis/getposes.py -l 500 -o CHEMBL4422_active_ligands.mol2
To delete a job from queue, run “scancel _JOBID_”
+
Should your slurm run correctly, type “squeue” and you should see something like this:
+
 
+
#### BASH command line output to console
+
            JOBID PARTITION    NAME    USER ST      TIME  NODES NODELIST(REASON)
+
      217_[9-100] pdl-stati array_jo  docker PD      0:00      1 (Resources)
+
            217_8 pdl-stati array_jo  docker R      0:00      1 pdl-station
+
            217_5 pdl-stati array_jo  docker  R      0:08      1 pdl-station
+
To delete this job from queue, run “scancel 217”
+
  
  
 +
'''Useful slurm commands (see https://slurm.schedmd.com/quickstart.html):'''
 +
to see what machine resources are offered by the cluster, do ''sinfo -lNe''
 +
to submit a DOCK-3.7 job, run ''$DOCKBASE/docking/submit/submit_slurm_array.csh''
 +
to see what is happening in the queue, run ''squeue''
 +
to see a detailed info for a specific job: ''scontrol show jobid=_JOBID_''
 +
to delete a job from queue, run ''scancel _JOBID_''
  
Running DOCK-3.7 with Slurm
+
Should your slurm run correctly, type ''squeue'' and you should see something like this:
GPR40 example: /home/docker/Desktop/EXAMPLE_PROJECT_GPR40/TEST
+
  
# Useful slurm commands (see https://slurm.schedmd.com/quickstart.html):
 
To see what machine resources are offered by the cluster, do “sinfo -lNe”
 
To submit DOCK-3.7 job, run $DOCKBASE/docking/submit/submit_slurm_array.csh
 
To see what is happening in the queue, run “squeue”
 
To delete a job from queue, run “scancel _JOBID_”
 
Should your slurm run correctly, type “squeue” and you should see something like this:
 
 
#### BASH command line output to console
 
 
             JOBID PARTITION    NAME    USER ST      TIME  NODES NODELIST(REASON)
 
             JOBID PARTITION    NAME    USER ST      TIME  NODES NODELIST(REASON)
      217_[9-100] pdl-stati array_jo   docker PD      0:00      1 (Resources)
+
  4187_[637-2091]     gimel array_jo dudenko PD      0:00      1 (Resources)
            217_8 pdl-stati array_jo   docker R      0:00      1 pdl-station
+
          4187_629    gimel array_jo  dudenko  R      0:00      1 n-1-20
            217_5 pdl-stati array_jo   docker R      0:08     1 pdl-station
+
          4187_630    gimel array_jo dudenko R      0:00      1 n-5-34
So, to delete this job from queue, run “scancel 217”
+
          4187_631    gimel array_jo  dudenko  R      0:00      1 n-1-21
 +
          4187_632    gimel array_jo dudenko R      0:00     1 n-5-34
 +
          4187_633    gimel array_jo  dudenko  R      0:00      1 n-1-21
 +
          4187_634    gimel array_jo  dudenko  R      0:00      1 n-5-34
 +
          4187_635    gimel array_jo  dudenko  R      0:00      1 n-5-35
 +
          4187_636    gimel array_jo  dudenko  R      0:00      1 n-5-34
 +
          4187_622    gimel array_jo  dudenko  R      0:01      1 n-5-34
 +
          4187_623    gimel array_jo  dudenko  R      0:01      1 n-5-34
 +
          4187_624    gimel array_jo  dudenko  R      0:01      1 n-5-35
 +
          4187_625    gimel array_jo  dudenko  R      0:01      1 n-1-17
  
  
  
scontrol show jobid=635
+
As root at gimel, it is possible to modify a particular job, e.g., ''scontrol update jobid=635 TimeLimit=7-00:00:00''
  
as root at gimel: scontrol update jobid=635 TimeLimit=7-00:00:00
+
=== Slurm Installation Guide ===
 +
'''Detailed step-by-step installation instruction (for sysadmins only)'''
  
 +
==== Setup Node master ====
 +
'''TBA'''
  
'''Detailed step-by-step installation instruction'''
+
==== Setup Compute Nodes ====
  
  
Useful link: https://slurm.schedmd.com/quickstart_admin.html
+
'''Useful links:'''
 
+
https://slurm.schedmd.com/quickstart_admin.html
 +
https://wiki.fysik.dtu.dk/niflheim/Slurm_installation
  
 
'''node n-1-17'''
 
'''node n-1-17'''
Line 124: Line 134:
 
   ''chown -R slurm:slurm /var/spool/slurm-llnl /var/run/slurm-llnl /var/log/slurm-llnl''
 
   ''chown -R slurm:slurm /var/spool/slurm-llnl /var/run/slurm-llnl /var/log/slurm-llnl''
 
* restarting slurm master node at gimel (Centos 6): ''/etc/init.d/slurm restart''
 
* restarting slurm master node at gimel (Centos 6): ''/etc/init.d/slurm restart''
* restarting slurm computing nodes (Centos 7): ''systemctl restart slurmd''
+
* enabling and starting slurm computing nodes (Centos 7): ''systemctl enable slurmd; systemctl start slurmd''
  
 
And last but not least, asking the firewall to allow communication between master node and computing node n-1-17:
 
And last but not least, asking the firewall to allow communication between master node and computing node n-1-17:
Line 130: Line 140:
 
* ''firewall-cmd --reload''
 
* ''firewall-cmd --reload''
  
To disable a specific node, do ''scontrol update NodeName=n-1-17 State=DRAIN Reason=DRAINED''
 
To return back to service, do ''scontrol update NodeName=n-1-17 State=IDLE''
 
  
To see the current situation of the queue, so sinfo -lNe and you will see:
+
To see the current situation of the queue, so ''sinfo -lNe'' and you will see:
 
  Wed May 27 09:49:54 2020
 
  Wed May 27 09:49:54 2020
 
  NODELIST  NODES PARTITION      STATE CPUS    S:C:T MEMORY TMP_DISK WEIGHT AVAIL_FE REASON               
 
  NODELIST  NODES PARTITION      STATE CPUS    S:C:T MEMORY TMP_DISK WEIGHT AVAIL_FE REASON               
Line 140: Line 148:
 
  n-5-34        1    gimel*        idle  80  80:1:1      1        0      1  (null) none                 
 
  n-5-34        1    gimel*        idle  80  80:1:1      1        0      1  (null) none                 
 
  n-5-35        1    gimel*        idle  80  80:1:1      1        0      1  (null) none
 
  n-5-35        1    gimel*        idle  80  80:1:1      1        0      1  (null) none
 +
 +
 +
To disable a specific node, do ''scontrol update NodeName=n-1-17 State=DRAIN Reason=DRAINED''<br>
 +
To return back to service, do ''scontrol update NodeName=n-1-17 State=IDLE''
  
  
 
p.s. Some users/scripts may require csh/tcsh.<br>
 
p.s. Some users/scripts may require csh/tcsh.<br>
 
''sudo yum install csh tcsh''
 
''sudo yum install csh tcsh''
 +
 +
==== Node down after reboot ====
 +
On gimel (master node)
 +
sudo scontrol update NodeName=<node_name> State=RESUME
 +
 +
 +
'''Useful links:'''
 +
https://support.ceci-hpc.be/doc/_contents/QuickStart/SubmittingJobs/SlurmTutorial.html
  
  
 
Back to [[DOCK_3.7]]
 
Back to [[DOCK_3.7]]
 +
 +
[[Category : Slurm]]

Revision as of 15:01, 29 June 2020

Slurm user-guide

Submit Jobs with Slurm

SBATCH-MR (beta)

It is a slurm-version of qsub-mr for submitting job on Slurm queueing system. Note: this is have not been extensively tested yet. Please contact me if the script is not working out. We are hoping to fully migrate to Slurm from the out-dated SGE system. Any error report would be helpful - Khanh

New slurm scripts are located in /nfs/soft/tools/utils/sbatch-slice

Just simply replace /nfs/soft/tools/utils/qsub-slice/qsub-mr with /nfs/soft/tools/utils/sbatch-slice/sbatch-mr in your script

To check the status of your job:

By username
 $ squeue -u <username>
By jobid
 $ squeue -j <job_id>

Submit load2d Jobs

$ cd <catalog_shortname>
$ source /nfs/exa/work/khtang/ZINC21_load2d/loadenv_zinc21.sh
(development) $ sh /nfs/exa/work/khtang/submit_scripts/sbatch_slice/batch_zinc21.slurm <catalog_shortname>.ism

Submit DOCK Jobs

  • ANACONDA Installation (Python 2.7)

Each user is welcome to download anaconda and install into his/her own folder
https://www.anaconda.com/distribution/
wget https://repo.anaconda.com/archive/Anaconda2-2019.10-Linux-x86_64.sh
NB: It is also available for Python3, which is our nearest future

simple installation via /bin/sh Anaconda2-2019.10-Linux-x86_64.sh

After the installation is completed, you need to install a few packages:

conda install -c free bsddb
conda install -c rdkit rdkit
conda install numpy


Running DOCK-3.7 with Slurm

Here is a “guinea pig project”, which has been done with DOCK-3.7 locally.
GPR40 example: /mnt/nfs/home/dudenko/TEST_DOCKING_PROJECT
ChEMBL ligands: /mnt/nfs/home/dudenko/CHEMBL4422_active_ligands
This test calculation should run smoothly, if not, then there is a problem.
Ultimately, you may need to compare your results with the reference run:

  • CHEMBL4422_active_ligands.mol2 - TOP500 scoring poses
  • extract_all.sort.uniq.txt - a print-out of scoring details


Slurm queue manager is installed locally at gimel, use it to run this test (and all your future jobs) in parallel.
Do not forget to set DOCKBASE variable: export DOCKBASE=/nfs/soft/dock/versions/dock37/DOCK-3.7.3rc1/

Useful DOCKING commands to remind:

$DOCKBASE/docking/setup/setup_db2_zinc15_file_number.py ./ CHEMBL4422_active_ligands_ CHEMBL4422_active_ligands.sdi 100 count
$DOCKBASE/analysis/extract_all.py -s -20
$DOCKBASE/analysis/getposes.py -l 500 -o CHEMBL4422_active_ligands.mol2


Useful slurm commands (see https://slurm.schedmd.com/quickstart.html):

to see what machine resources are offered by the cluster, do sinfo -lNe
to submit a DOCK-3.7 job, run $DOCKBASE/docking/submit/submit_slurm_array.csh
to see what is happening in the queue, run squeue
to see a detailed info for a specific job: scontrol show jobid=_JOBID_
to delete a job from queue, run scancel _JOBID_

Should your slurm run correctly, type squeue and you should see something like this:

            JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
  4187_[637-2091]     gimel array_jo  dudenko PD       0:00      1 (Resources)
         4187_629     gimel array_jo  dudenko  R       0:00      1 n-1-20
         4187_630     gimel array_jo  dudenko  R       0:00      1 n-5-34
         4187_631     gimel array_jo  dudenko  R       0:00      1 n-1-21
         4187_632     gimel array_jo  dudenko  R       0:00      1 n-5-34
         4187_633     gimel array_jo  dudenko  R       0:00      1 n-1-21
         4187_634     gimel array_jo  dudenko  R       0:00      1 n-5-34
         4187_635     gimel array_jo  dudenko  R       0:00      1 n-5-35
         4187_636     gimel array_jo  dudenko  R       0:00      1 n-5-34
         4187_622     gimel array_jo  dudenko  R       0:01      1 n-5-34
         4187_623     gimel array_jo  dudenko  R       0:01      1 n-5-34
         4187_624     gimel array_jo  dudenko  R       0:01      1 n-5-35
         4187_625     gimel array_jo  dudenko  R       0:01      1 n-1-17


As root at gimel, it is possible to modify a particular job, e.g., scontrol update jobid=635 TimeLimit=7-00:00:00

Slurm Installation Guide

Detailed step-by-step installation instruction (for sysadmins only)

Setup Node master

TBA

Setup Compute Nodes

Useful links:

https://slurm.schedmd.com/quickstart_admin.html
https://wiki.fysik.dtu.dk/niflheim/Slurm_installation

node n-1-17

  • make sure you have there Centos 7: cat /etc/redhat-release
  • wget https://download.schedmd.com/slurm/slurm-17.02.11.tar.bz2
  • yum install readline-devel perl-ExtUtils-MakeMaker.noarch munge-devel pam-devel
  • export VER=17.02.11; rpmbuild -ta slurm-$VER.tar.bz2 --without mysql; mv /root/rpmbuild .

installing built packages from rpmbuild:

  • yum install rpmbuild/RPMS/x86_64/slurm-plugins-17.02.11-1.el7.x86_64.rpm
  • yum install rpmbuild/RPMS/x86_64/slurm-17.02.11-1.el7.x86_64.rpm
  • yum install rpmbuild/RPMS/x86_64/slurm-munge-17.02.11-1.el7.x86_64.rpm


setting up munge: copy over /etc/munge/munge.key from gimel and put locally to /etc/munge. The key should be identical allover the nodes.
Munge is a daemon responsible for secure data exchange between nodes.
Set permissions accordingly: chown munge:munge /etc/munge/munge.key; chmod 400 /etc/munge/munge.key

starting munge: systemctl enable munge; systemctl start munge

setting up slurm:

  • create a user slurm: adduser slurm.
  • all UID/GUIDs of slurm user should be identical allover the nodes.
 Otherwise, one needs to specify a mapping scheme for translating each UID/GUIDs between nodes.
To edit slurm UID/GUID, do vipw and replace "slurm line" with slurm:x:XXXXX:YYYYY::/nonexistent:/bin/false
XXXXX and YYYYY for slurm user can be found at gimel in /etc/passwd
NB: don't forget to edit /etc/group as well.
  • copy /etc/slurm/slurm.conf from gimel and put locally to /etc/slurm.
  • figure out what CPU/Memory resources you have at n-1-17 (see /proc/cpuinfo) and append the following line:
 NodeName=n-1-17 NodeAddr=10.20.1.17 CPUs=24 State=UNKNOWN
  • append n-1-17 to the partition: PartitionName=gimel Nodes=gimel,n-5-34,n-5-35,n-1-17 Default=YES MaxTime=INFINITE State=UP
  • create the following folders:
 mkdir -p /var/spool/slurm-llnl /var/run/slurm-llnl /var/log/slurm-llnl
 chown -R slurm:slurm /var/spool/slurm-llnl /var/run/slurm-llnl /var/log/slurm-llnl
  • restarting slurm master node at gimel (Centos 6): /etc/init.d/slurm restart
  • enabling and starting slurm computing nodes (Centos 7): systemctl enable slurmd; systemctl start slurmd

And last but not least, asking the firewall to allow communication between master node and computing node n-1-17:

  • firewall-cmd --permanent --zone=public --add-port=6818/tcp
  • firewall-cmd --reload


To see the current situation of the queue, so sinfo -lNe and you will see:

Wed May 27 09:49:54 2020
NODELIST   NODES PARTITION       STATE CPUS    S:C:T MEMORY TMP_DISK WEIGHT AVAIL_FE REASON              
gimel          1    gimel*     drained   24    4:6:1      1        0      1   (null) none                
n-1-17         1    gimel*        idle   24   24:1:1      1        0      1   (null) none                
n-5-34         1    gimel*        idle   80   80:1:1      1        0      1   (null) none                
n-5-35         1    gimel*        idle   80   80:1:1      1        0      1   (null) none


To disable a specific node, do scontrol update NodeName=n-1-17 State=DRAIN Reason=DRAINED
To return back to service, do scontrol update NodeName=n-1-17 State=IDLE


p.s. Some users/scripts may require csh/tcsh.
sudo yum install csh tcsh

Node down after reboot

On gimel (master node)

sudo scontrol update NodeName=<node_name> State=RESUME


Useful links:

https://support.ceci-hpc.be/doc/_contents/QuickStart/SubmittingJobs/SlurmTutorial.html


Back to DOCK_3.7