View source for Running ChemSTEP in the 13 billion space

*Written by Philipp Seemann with the kind assistance of Katie Holland and Joseph Pepe (08/28/2025)
*This tutorial should work on Wynton with the provided scripts and public environments; you might need to catch tiny typos. I apologize.

This is meant to be a simple hands-on step by step guide for ChemSTEP on Wynton in the 13 billion space. For exploring the trillion space please refer to:[[Running ChemSTEP|https://wiki.docking.org/index.php?title=Running_ChemSTEP]]

'''The general workflow is:'''
   |--- Simply DOCK the seeds set (you can use any docking method)
   |
   |--- Run ChemSTEP 
   |
   |--- Build molecules from 1st round of ChemSTEP
   |
   |--- DOCK molecules from 1st round of ChemSTEP
   |
   |--- Run ChemSTEP the 2nd time
   |
   |--- Build molecules from 2nd round of ChemSTEP
   |
   |--- DOCK molecules from 2nd round of ChemSTEP
   |
   |--- Run ChemSTEP the 3rd time
   .
   .
   .
   .
   .
   |--- Repeat until your Recovery rate does not improve

Here is a recommended folder layout after several rounds of ChemSTEP. We will also provide the initial layout here right after this one. This should just give you an overview of what folders we will need to make iteratively, not to confuse our building and docking, and ChemSTEP rounds. (You can change this however you want. After you get the hang of this, you can be creative and find a better layout workflow for you!) This tutorial will lead you through round_0 and round_1. After that, it should become clear what steps you will need to repeat over and over again. This tutorial is written for Wynton. Please use the scripts starting from a dev node.

'''Folder layout:'''
   CHEMSTEP_PROJECT_FOLDER/
   |
   |----run_inital_and_iterative_chemstep/
   |
   |----round_0/
   |
   |----building_1/ 
   |
   |----round_1/ 
   |
   |----building_2/ 
   |
   |----round_2/
   |
   |....

'''Explanation of folders'''
'''run_inital_and_iterative_chemstep/'''

Here, we actually run ChemSTEP with every new round as well. Here will  be copied all scores_round_*.npy and indices_round_*.npy after every new round of docking.

'''round_0/'''

Docking of the seeds set. So this contains your dockfiles, and here we generate the initial indices and score .npys, which will be copied to the run_inital_and_iterative_chemstep/ folder later on

'''building_1/'''

This folder is for building of the first .smi file from run_initial_and_iterative_chemstep/output/complete_info/smi_round_1.smi

'''round_1/'''

docking of the molecules from the building_1 folder. So this contains again your dockfiles with an adjusted INDOCK and a new sdi file. After extracting the scores, it will yield us indices and scores .npys of round_1 for the second round of chemstep. 

'''building_2/'''

After running chemstep from run_inital_and_iterative_chemstep the second time we will have a new .smi file which we will need to build.

'''round_2/'''

If you have read carefully to this point, you will know what comes next. We will dock the molecules from building_2 here and extract the scores. Generate .npys files. Copy those to our run_inital_and_iterative_chemstep folder to generate the next smi_round_*.smi


So this is now all still confusing, but we will start easy. It will make sense as any other Shoichet Lab tutorial. I promise.

'''First round of docking'''

Here are the first folders that you will need to set up for this tutorial

   CHEMSTEP_PROJECT_FOLDER/
   |
   |
   |----round_0/ # docking of the seeds set
   |
   |
   |----run_inital_and_iterative_chemstep/ #here will  be copied all scores_round_*.npy and indices_round_*.npy files

First, we make a traditional DOCKing directory in your work directories (be sure not to be in your home directory because we will dock and build (DISKSPACE!)). The first step for ChemSTEP is basically just docking the 13B_seed_set_built.sdi, so submit as your traditional LSD screen (whatever submission style or script you prefer). This layout follows the above shown layout of folders.  What you need to bring on your own here are your dockfiles (and your way of submission script).

   mkdir CHEMSTEP_PROJECT_FOLDER/
   cd CHEMSTEP_PROJECT_FOLDER/
   mkdir round_0
   cd round_0
   cp -r path/to/your/dockfiles . #check your INDOCK parameters
   mkdir sdi
   cd sdi
   cp /wynton/group/bks/work/bwhall61/mor_chemstep/DOCK/13M/seed/docking/bundle_paths.sdi .
   cd ..
   # cp your favourite submission script or whatever you use for submitting jobs
   # or in case you use SymDOCK with the right executable set in the submission    
   # script

Submit the docking job with your method of preference. 

'''Things to consider for the size of the seeds set'''

Ideally, we want enough molecules in our desired pProp region to be considered as beacons and virtual hits. So it is of great importance that enough molecules score in the desired region to be considered as beacons. On Wynton, there is the 130k seeds set, a 13M set, and somewhere also a 1.3M set. 

   /wynton/group/bks/work/shared/kholland/chemstep_13B/13B_seed_set_built.sdi #130k 
   /wynton/group/bks/work/bwhall61/mor_chemstep/DOCK/13M/seed/docking/bundle_paths.sdi #13M

Our aim is actually to dock a small chunk, which is still big enough to represent the library, and among those molecules we want enough beacons (so probably something between 50-100 molecules in the desired score range) chosen by ChemSTEP

'''When the docking finishes:'''

Source the right environment now:

   source /wynton/group/bks/work/shared/kholland/chemstep_env/bin/activate

We will now extract scores and Molecule IDs, so we run get_scores.py

   #cd into your round_0 directory
   
   cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/get_scores.py .
   python get_scores.py 0
   
   #0 for initial round of chemstep

When get_scores.py runs successfully, we see scores_round_0.txt in our folder. Check your scores.txt file for output. It should look like this:

   MOL12457028547 -29.32
   MOL12457032486 -32.39
   ...

The get_scores.py script expects a certain output folder structure. If you do not see any output in your .txt file, vim into the script and adjust the paths. If you use the copied version, it should be: 
   /output/*/*/OUTDOCK.*

Now we translate scores_round_0.txt into indices_round_0.npy and scores_round_0.npy

   #cd into your round_0 directory
   
   cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/convert_scores_to_npy.py .
   python convert_scores_to_npy.py 0
   
   #0 for initial round of chemstep

now we should find indices_round_0.npy and scores_round_0.npy in our round_0 directory

'''First round of ChemSTEP. EXCITING!!!'''

'''Now we set up ChemSTEP:'''

   #now cd into your CHEMSTEP_PROJECT_FOLDER
   
   mkdir run_initial_and_iterative_chemstep
   cp round_0/*_round_0.npy run_initial_and_iterative_chemstep/
   cd run_initial_and_iterative_chemstep/

Now the two .npy files should be in our run_inital_and_iterative_chemstep folder to run chemstep.
We will now set up our initial submission script, which will be slightly different from the iterative one for the following rounds.

   #cd into your folder for running chemstep - 
   #run_initial_and_iterative_chemstep/ 
   
   cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/launch_chemstep_as_job_initial.sh .
   cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/run_chemstep_initial.py .

We stay in this folder and copy over a params.txt file, and then adjust it to your liking 

   #we are still in run_inital_and_iterative_chemstep
   
   cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/params.txt .

Adjust your params file - with vim if you like.
   vim params.txt

   seed_indices_file: path/to/your/run_initial_and_iterative_chemstep/indices_round_0.npy 
   
   seed_scores_file: path/to/your/run_initial_and_iterative_chemstep/scores_round_0.npy
   
   hit_pprop: 6 #may change depending on library size, here 13B
   
   n_docked_per_round: 1000000 #reasonable for 13B
   
   max_beacons: 100 #reasonable for 13B
   
   max_n_rounds: 250 # recommended
   
Now there should be two scripts and two .npy files, and one params.txt file in your run_inital_and_iterative_chemstep folder.
Now we submit our first round of running ChemSTEP with :

   qsub launch_chemstep_as_job_initial.sh
   
   # this will queue a job
   # the job will split into many jobs, 600, eg which will run for a while
   # the initial job will run for a while after the jobs finish, and should generate 
   # output in /output/complete_info/ ,there should be a .smi file when sucessfully 

'''Two notes here:'''

First, we will never touch the params.txt file again. From my understanding, we need to copy each following scores_round_*.npy and indices_round_*.npy into the same directory (in our case, run_initial_and_iterative_chemstep)

Second, I had issues with getting Wynton to execute python for every submitted job. I hope this is fixed for you as well now.

'''First round of building:'''

When ChemSTEP finishes, we can go to our CHEMSTEP_PROJECT_FOLDER and brace ourselves for building. 

   #cd into your CHEMSTEP_PROJECT_FOLDER
   
   mkdir building_1
   cd building_1
   cp ../run_initial_and_iterative_chemstep/output/complete_info/smi_round_1.smi .
   
   #if there is no smi file, something went wrong

We now source the building environment, prepare the job, submit the job, and wait until the building is completed.

source environment:
   source /wynton/group/bks/soft/DOCK-3.8.5/env.sh

prepare job

   python /wynton/group/bks/soft/DOCK-3.8.5/DOCK3.8/zinc22-3d/submit/submit_building_docker.py --output_folder building_output --bundle_size 1000 --minutes_per_mol 1 --skip_name_check --scheduler sge --container_software apptainer --container_path_or_name /wynton/group/bks/soft/DOCK-3.8.5/building_pipeline.sif smi_round_1.smi
   
   #note: smi_round_*.smi for the following rounds must be adjusted. 

submit job
   qsub building_array_job.sh

When this is finished, check for failed jobs and resubmit. Some will always fail, just try to keep them low.
When the molecules are built, we can proceed with docking them.
Now we generate an .sdi file from our first building round.

   #cd into your building_1 folder
   
   find /wynton/group/bks/work/pseemann/CHEMSTEP_PROJECT_FOLDER/building_1/building_output/ -type f -name "*.tgz" > round1.sdi
   
   #example adjust paths

'''Next round of docking: '''

We proceed with docking the freshly built compounds, so we make a directory called round_1, with dockfiles, and copy our new SDI file here, too

   #cd into your CHEMSTEP_PROJECT_FOLDER
   
   mkdir round_1
   cd round_1
   mkdir sdi
   cd sdi
   cp ../../building_1/*.sdi .
   cd ..
   cp -r ../round_0/dockfiles .
   vim dockfiles/INDOCK 
   
   # adjust the score maximum in the INDOCK to your chosen pprop (recommended)
   # submit with your favourite submission script just as you did for round_0
   # so sh your_way_of_submitting.sh

As before, we will now run get_scores.py

source environment
   source /wynton/group/bks/work/shared/kholland/chemstep_env/bin/activate

Copy scripts and execute them
   #cd into your round_1 directory
   
   cp ../round_0/get_scores.py .
   python get_scores.py 1
   
   # wait until this finishes, always adjust the argument after                 
   # python get_scores.py to not confuse your scores files with other rounds


Copy scripts and execute them
   #still in your round_1 directory
   #copy the right script, IDs have changed, and needed to be adjusted in the convert script
   
   cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/64_char_convert_scores_to_npy.py .
   python 64_char_convert_scores_to_npy.py 1
   
   # Wait until this finishes; always adjust the argument afterward 
   # python 64_char_convert_scores_to_npy.py
   # script was adjusted to fit the output of round_1

'''Next round of ChemSTEP:'''

   #cd into your run_initial_and_iterative_chemstep directory
   
   cp ../round_1/scores_round_1.npy .
   cp ../round_1/indices_round_1.npy .

Copy over the iterative run_chemstep_iteratively.py and launch_chemstep_as_job_iteratively.sh
These are slightly different then the initial ones. But from here, we will only use these scripts. Be sure to always provide the right arguments to the scripts. Be sure to use the right scripts. This is (hopefully) the most confusing step in this ChemSTEP tutorial.

   #still in run_initial_and_iterative_chemstep directory
   
   cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/launch_chemstep_as_job_iteratively.sh .
   cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/run_chemstep_iteratively.py .

'''Submit the 2nd round of ChemSTEP:'''

   qsub launch_chemstep_as_job_iteratively.sh 2 
   
   #Give the number of rounds you are running chemstep if you are using scores_round_1.npy and indices_round_1.npy you provide 2 as an
   #argument, like in the example above.
   #if you run your 3rd round of chemstep and provide indices_round_2.py and 
   #scores_round_2.npy, one would do qsub launch_chemstep_as_job_iteratively.sh 3
   #and so on...
   #If no argument is provided, it will queue but shut down. 
   #You can check this in the chemstep_submission.log file | 
   #So when the job is submitted, it will just quit, when no argument is given, after a few seconds to minutes.

Now we wait until the jobs finish, and brace ourselves for the next time-consuming round of building.

'''Next round of building:'''

   #so in your CHEMSTEP_PROJECT_FOLDER
   
   mkdir building_2
   cd building_2
   cp ../run_initial_and_iterative_chemstep/output/complete_info/smi_round_2.smi .

'''source environment'''
   source /wynton/group/bks/soft/DOCK-3.8.5/env.sh

'''prepare job (always adjust smi_round.smi here)'''

   python /wynton/group/bks/soft/DOCK-3.8.5/DOCK3.8/zinc22-3d/submit/submit_building_docker.py --output_folder building_output --bundle_size 1000 --minutes_per_mol 1 --skip_name_check --scheduler sge --container_software apptainer --container_path_or_name /wynton/group/bks/soft/DOCK-3.8.5/building_pipeline.sif smi_round_2.smi

'''submit the job'''
   qsub building_array_job.sh

Now like shown before. Resubmit failed jobs, make your sdi, and proceed with docking (new folder round_2 recommended).

Now we went full circle. The next step would be to make your next round_2 of the docking directory with the updated .sdi file from the 2nd round of building. With the next round of docking, you would extract scores and IDs, convert them, feed them to ChemSTEP,  and repeat this as often as you don't see an increase in recovery rate. You would always need to update the arguments you pass to the python and submission scripts to match the round of ChemSTEP and building, and extracting etc.

'''Check your beacons'''

In the first rounds, one might want to check the selected beacons that were chosen by ChemSTEP. To do so, go to your run_inital_and_iterative_chemstep/ folder (the folder where you run ChemSTEP). Then make a list of IDs:

   cat chemstep_algo.log | grep "with score" | awk '{print $3}' > list.txt

Copy this file to a new folder. Then source a Python environment.

   source /wynton/group/bks/work/shared/kholland/chemstep_env/bin/activate

Copy over this script and run it

   cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/beacon_to_smiles.py .
   python beacon_to_smiles.py /wynton/group/bks/work/shared/kholland/chemstep_13B/boltz_fplib.pickle list.txt
   #python beacon_to_smiles.py path/to/fingerprint_library list.txt

'''All scripts and files for this tutorial are present in'''

   /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/