Building The 3D Pipeline ZINC22: Difference between revisions

From DISI
Jump to navigation Jump to search
No edit summary
(29 intermediate revisions by the same user not shown)
Line 14: Line 14:


<ol>
<ol>
<li>clone the appropriate branch of the zinc-3d-build-3 repository (github link) to your working directory</li>
<li>Clone the slurm-2 branch of the zinc-3d-build-3 repository (https://github.com/btingle/zinc-3d-build-3)</li>


<li>Secure copy the software distribution from our cluster at (scp link)</li>
<li>BKS Users: Copy the software distribution from our cluster at /nfs/home/xyz/soft/*.tar.gz to the $HOME/soft directory of the user that will be running the script</li>


<li>Extract the software distribution to the $HOME/soft directory of the user that will be running the script</li>
<li>Others: Copy the software distribution from our cluster, minus corina, which you will need to supply yourself.


<ol type="a">
<ol type="a">
<li>When this script is submitted to a machine for the first time, it will copy and extract the software distribution from $HOME/soft to local storage</li>
<li>When this script is submitted to a machine for the first time, it will copy and install the necessary software from $HOME/soft to local storage</li>


<li>Because of this, it is important that your $HOME be global if you are running the script unmodified</li>
<li>Because of this, it is important that your $HOME be global if you are running the script unmodified</li>
Line 28: Line 28:
</ol>
</ol>


<li>Copy your licenses into the same directory you extracted the software to. Copy your corina distribution (as a tar.gz) into this same folder.</li>
<li>Copy your licenses into your $HOME. Copy your corina distribution (as a tar.gz) into $HOME/soft if not already present.</li>


<ol type="a">
<ol type="a">
<li>This script assumes that the jchem and openeye licenses will be named ".jchem-license.cxl" and ".oe-license.txt" respectively. Corina is assumed to be named "corina.tar.gz"</li>
<li>This script assumes that the jchem and openeye licenses will be named ".jchem-license.cxl" and ".oe-license.txt" respectively. Corina is assumed to be a tarball named "corina.tar.gz" which contains a single directory, "corina" with executables etc.</li>
 
<li>BKS Users: You can copy these licenses /nfs/home/xyz to your own $HOME.</li>
</ol>
</ol>
</ol>
</ol>


== Running The Script ==
== Running The Script ==
FYI: You MUST be in the pipeline bin directory (zinc-3d-build-3) when running the submit-all-jobs script
You should also run this command in a screen, as it needs to persist until all jobs are complete.


General Example
General Example
Line 46: Line 52:
export LINES_PER_BATCH=50000
export LINES_PER_BATCH=50000
export LINES_PER_JOB=50
export LINES_PER_JOB=50
export TEMPDIR=/tmp
export BUILD_DIR=build_dir_$(whoami)
./submit-all-jobs-<slurm/sge>.bash
./submit-all-jobs-<slurm/sge>.bash
</nowiki>
</nowiki>
Line 52: Line 60:


  <nowiki>
  <nowiki>
export INPUT_FILE
export INPUT_FILE=example.smi
export OUTPUT_DEST
export OUTPUT_DEST=/nfs/exb/zinc22/tarballs
export SBATCH_ARGS="--time=00:45:00 --requeue"
export SBATCH_ARGS="--time=02:00:00"
export LINES_PER_BATCH=50000
export LINES_PER_BATCH=50000
export LINES_PER_JOB=50
export LINES_PER_JOB=50
export TEMPDIR=/dev/shm
export TEMPDIR=/dev/shm
export BUILD_DIR=build_dir_$(whoami)
./submit-all-jobs-slurm.bash
./submit-all-jobs-slurm.bash
</nowiki>
</nowiki>
Line 63: Line 72:
Wynton Example
Wynton Example
  <nowiki>
  <nowiki>
export INPUT_FILE
export INPUT_FILE=example.smi
export OUTPUT_DEST
export OUTPUT_DEST=/wynton/group/bks/zinc22
export QSUB_ARGS="-l s_rt=00:28:30 -l h_rt=00:30:00 -r y"
export QSUB_ARGS="-l s_rt=00:28:30 -l h_rt=00:30:00 -r y"
export LINES_PER_BATCH=50000
export LINES_PER_BATCH=50000
export LINES_PER_JOB=50
export LINES_PER_JOB=50
export TEMPDIR=/scratch
export TEMPDIR=/scratch
export BUILD_DIR=build_dir_$(whoami)
./submit-all-jobs-sge.bash
./submit-all-jobs-sge.bash
</nowiki>
</nowiki>
Line 74: Line 84:
Cori Example
Cori Example
  <nowiki>
  <nowiki>
export INPUT_FILE
export INPUT_FILE=example.smi
export OUTPUT_DEST
export OUTPUT_DEST=$SCRATCH/zinc22
export SBATCH_ARGS="--time-min=00:01:00 --time=00:02:00 --requeue -q flex -C haswell"
export SBATCH_ARGS="--cpus-per-task=1 --time=02:00:00 --requeue -q shared -C haswell"
export LINES_PER_BATCH=50000
export LINES_PER_BATCH=20000
export LINES_PER_JOB=50
export LINES_PER_JOB=50
export TEMPDIR=$SCRATCH
export MAX_BATCHES=10
export BUILD_DIR=build_dir_$(whoami)
# $SCRATCH on Cori can be used for debugging, but production batch jobs should take advantage of a burst buffer allocation
# export TEMPDIR=$SCRATCH
./submit-all-jobs-slurm.bash
./submit-all-jobs-slurm.bash
</nowiki>
</nowiki>


== Input And Output ==
== Script Arguments ==


INPUT_FILE
==== INPUT_FILE ====


The input .smi file to be built. This file should contain only two columns of data: (SMILES, NAME) with no header.
The input .smi file to be built. This file should contain only two columns of data: (SMILES, NAME) with no header.


OUTPUT_DEST
==== OUTPUT_DEST ====


The base directory for output to be stored. The script will create a sub-directory here named $INPUT_FILE.batch-3d.d
The base directory for output to be stored. The script will create a sub-directory here named $INPUT_FILE.batch-3d.d
Line 105: Line 118:


Out contains tar.gz output from each job. The tarballs should contain a number of 3d molecule formats for each molecule in the batch, including 1 or more db2.gz files.
Out contains tar.gz output from each job. The tarballs should contain a number of 3d molecule formats for each molecule in the batch, including 1 or more db2.gz files.
==== TEMPDIR ====
The base working directory for the script. By default it is /tmp. If you want your jobs to run a little faster and are not concerned about memory usage, you can set the working directory to /dev/shm.
==== LINES_PER_BATCH ====
How many lines of the source .smi file should be processed per array batch job
==== LINES_PER_JOB ====
How many lines of the batch .smi file should be processed per array task
==== SBATCH_ARGS ====
Additional arguments for the sbatch command. It is recommended to set a --time limit, as build jobs will save progress & terminate if they are still running two minutes before the --time limit.
<s>--requeue allows jobs that reach the time limit to run later, utilizing their saved progress.</s>
Slurm seems to not send signals to our jobs correctly, so this feature should be ignored on slurm clusters. (for now)
==== QSUB_ARGS ====
Additional arguments for the qsub command. Similar to slurm, it is recommended to set a time limit, but you will need to manually specify both s_rt & h_rt. In the example, we set s_rt to be a minute and thirty seconds before h_rt. s_rt is the point where jobs will save progress and terminate, h_rt is when they will be forcibly terminated, even if they've not finished saving.
-r y allows jobs that reach the time limit to run later, utilizing their saved progress.
==== MAX_BATCHES ====
Each batch job will contain LINES_PER_BATCH/LINES_PER_JOB jobs, and there will be a maximum of MAX_BATCHES batches submitted at any given time. By default this value is 30, which corresponds to 30,000 queued jobs at any given time if there are 1000 jobs per batch.
== Repatriation ==
At BKS, we currently store the tarred output of the pipeline @ /nfs/exb/zinc22/tarballs. Currently, we use the following command to repatriate output from other clusters to our cluster:
<nowiki>
### migrate_output.bash
for output in $OUTPUT_DEST/*.batch-3d.d; do
        echo "starting rsync on $output to $MIGRATE_USER@files2.docking.org"
        sshpass -f $PW_FILE rsync -arv $output/out $MIGRATE_USER@files2.docking.org:/nfs/exb/zinc22/tarballs/$(basename $output).out
done
</nowiki>
sshpass is optional here but preferable for convenience's sake. Since files2.docking.org is only visible within the UCSF network, any clusters outside will need to maintain a network tunnel when rsyncing.


== Errors ==
== Errors ==


Sometimes an output tarball will have few or no entries within. Certain molecule types [elaborate] will fail to be built, and often these molecules get bunched together (i.e if the input file is sorted by SMILES). Additionally, a small percentage of all molecules may fail to be processed by corina or amsol. If neither of these explain what is causing your missing entries, check that tarball's corresponding log entry for more info. Mail me @ ben@tingle.org for additional technical support.
Sometimes an output tarball will have few or no entries within. Certain molecule types [elaborate] will fail to be built, and often these molecules get bunched together (i.e if the input file is sorted by SMILES). Additionally, a small percentage of all molecules may fail to be processed by corina or amsol. If neither of these explain what is causing your missing entries, check that tarball's corresponding log entry for more info.


== Additional Notes ==
== Additional Notes ==

Revision as of 18:33, 24 November 2020

Introduction

The 3D pipeline is a collection of scripts and software packages that enable the massively parallel creation of dockable 3D molecules.

Requirements

  • Chemaxon License (email John for license details)
  • Openeye License (email John for license details)
  • Corina License (comes packaged with the executable, so you will need to provide the executable)
  • SGE or SLURM queueing system installed on your cluster
  • A networked file system installed on your cluster

Setup Instructions

  1. Clone the slurm-2 branch of the zinc-3d-build-3 repository (https://github.com/btingle/zinc-3d-build-3)
  2. BKS Users: Copy the software distribution from our cluster at /nfs/home/xyz/soft/*.tar.gz to the $HOME/soft directory of the user that will be running the script
  3. Others: Copy the software distribution from our cluster, minus corina, which you will need to supply yourself.
    1. When this script is submitted to a machine for the first time, it will copy and install the necessary software from $HOME/soft to local storage
    2. Because of this, it is important that your $HOME be global if you are running the script unmodified
    3. If you don't have a global home, or otherwise want your software to reside somewhere else, you can specify an alternative global directory to copy the software from by exporting SOFT_HOME prior to running submit-all-jobs.bash (see below)
  4. Copy your licenses into your $HOME. Copy your corina distribution (as a tar.gz) into $HOME/soft if not already present.
    1. This script assumes that the jchem and openeye licenses will be named ".jchem-license.cxl" and ".oe-license.txt" respectively. Corina is assumed to be a tarball named "corina.tar.gz" which contains a single directory, "corina" with executables etc.
    2. BKS Users: You can copy these licenses /nfs/home/xyz to your own $HOME.

Running The Script

FYI: You MUST be in the pipeline bin directory (zinc-3d-build-3) when running the submit-all-jobs script

You should also run this command in a screen, as it needs to persist until all jobs are complete.

General Example

export INPUT_FILE=$YOUR_INPUT_FILE.smi
export OUTPUT_DEST=$SOME_GLOBAL_DIRECTORY
# if you don't have a global home directory (but some other global directory available), specify an alternate SOFT_HOME
# export SOFT_HOME=$YOUR_SOFT_HOME
export LINES_PER_BATCH=50000
export LINES_PER_JOB=50
export TEMPDIR=/tmp
export BUILD_DIR=build_dir_$(whoami)
./submit-all-jobs-<slurm/sge>.bash

BKS Example

export INPUT_FILE=example.smi
export OUTPUT_DEST=/nfs/exb/zinc22/tarballs
export SBATCH_ARGS="--time=02:00:00"
export LINES_PER_BATCH=50000
export LINES_PER_JOB=50
export TEMPDIR=/dev/shm
export BUILD_DIR=build_dir_$(whoami)
./submit-all-jobs-slurm.bash

Wynton Example

export INPUT_FILE=example.smi
export OUTPUT_DEST=/wynton/group/bks/zinc22
export QSUB_ARGS="-l s_rt=00:28:30 -l h_rt=00:30:00 -r y"
export LINES_PER_BATCH=50000
export LINES_PER_JOB=50
export TEMPDIR=/scratch
export BUILD_DIR=build_dir_$(whoami)
./submit-all-jobs-sge.bash

Cori Example

export INPUT_FILE=example.smi
export OUTPUT_DEST=$SCRATCH/zinc22
export SBATCH_ARGS="--cpus-per-task=1 --time=02:00:00 --requeue -q shared -C haswell"
export LINES_PER_BATCH=20000
export LINES_PER_JOB=50
export MAX_BATCHES=10
export BUILD_DIR=build_dir_$(whoami)
# $SCRATCH on Cori can be used for debugging, but production batch jobs should take advantage of a burst buffer allocation
# export TEMPDIR=$SCRATCH
./submit-all-jobs-slurm.bash

Script Arguments

INPUT_FILE

The input .smi file to be built. This file should contain only two columns of data: (SMILES, NAME) with no header.

OUTPUT_DEST

The base directory for output to be stored. The script will create a sub-directory here named $INPUT_FILE.batch-3d.d

Within this output directory there are 3 sub-directories:

  1. in
  2. log
  3. out

In contains the input file split into fragments and sub-fragments. By default the script first splits the input file into batches of 50000, then splits those batches into sub-batches of 50. Each individual job works on one of these sub-batches. Each array batch job works on one of the batches of 50000. All of the other directories alongside 'in' share the same directory structure.

Log contains log messages from the jobs. If you are re-submitting a file, be aware that log messages from previous runs on this file will be overwritten.

Out contains tar.gz output from each job. The tarballs should contain a number of 3d molecule formats for each molecule in the batch, including 1 or more db2.gz files.

TEMPDIR

The base working directory for the script. By default it is /tmp. If you want your jobs to run a little faster and are not concerned about memory usage, you can set the working directory to /dev/shm.

LINES_PER_BATCH

How many lines of the source .smi file should be processed per array batch job

LINES_PER_JOB

How many lines of the batch .smi file should be processed per array task

SBATCH_ARGS

Additional arguments for the sbatch command. It is recommended to set a --time limit, as build jobs will save progress & terminate if they are still running two minutes before the --time limit.

--requeue allows jobs that reach the time limit to run later, utilizing their saved progress.

Slurm seems to not send signals to our jobs correctly, so this feature should be ignored on slurm clusters. (for now)

QSUB_ARGS

Additional arguments for the qsub command. Similar to slurm, it is recommended to set a time limit, but you will need to manually specify both s_rt & h_rt. In the example, we set s_rt to be a minute and thirty seconds before h_rt. s_rt is the point where jobs will save progress and terminate, h_rt is when they will be forcibly terminated, even if they've not finished saving.

-r y allows jobs that reach the time limit to run later, utilizing their saved progress.

MAX_BATCHES

Each batch job will contain LINES_PER_BATCH/LINES_PER_JOB jobs, and there will be a maximum of MAX_BATCHES batches submitted at any given time. By default this value is 30, which corresponds to 30,000 queued jobs at any given time if there are 1000 jobs per batch.

Repatriation

At BKS, we currently store the tarred output of the pipeline @ /nfs/exb/zinc22/tarballs. Currently, we use the following command to repatriate output from other clusters to our cluster:

### migrate_output.bash

for output in $OUTPUT_DEST/*.batch-3d.d; do
        echo "starting rsync on $output to $MIGRATE_USER@files2.docking.org"
        sshpass -f $PW_FILE rsync -arv $output/out $MIGRATE_USER@files2.docking.org:/nfs/exb/zinc22/tarballs/$(basename $output).out
done

sshpass is optional here but preferable for convenience's sake. Since files2.docking.org is only visible within the UCSF network, any clusters outside will need to maintain a network tunnel when rsyncing.

Errors

Sometimes an output tarball will have few or no entries within. Certain molecule types [elaborate] will fail to be built, and often these molecules get bunched together (i.e if the input file is sorted by SMILES). Additionally, a small percentage of all molecules may fail to be processed by corina or amsol. If neither of these explain what is causing your missing entries, check that tarball's corresponding log entry for more info.

Additional Notes

This script was designed for a 64 bit architecture. You will likely run into some library issues trying to run it on 32 bit machines. If that's all you have you can try to swap out libg2c.so.0* in lib.tar.gz with a 32 bit version, but I cannot help you beyond that.

It is safe to re-run the same file multiple times- the script takes care of making sure not to re-run any jobs that have already completed successfully prior. This is only the case if that file's corresponding batch-3d.d output directory has not been moved or deleted.

For example, if one of your nodes went down and caused a bunch of jobs to fail, it would be safe to re-run ./submit-all-jobs.bash to re-submit those jobs. (assuming there are no jobs for that file currently queued/running)