http://wiki.docking.org/api.php?action=feedcontributions&feedformat=atom&user=Kholland 2026-04-08T09:14:20Z User contributions MediaWiki 1.39.1 http://wiki.docking.org/index.php?title=Running_ChemSTEP&diff=17120 2026-03-16T16:53:01Z

<p>Kholland: </p> <hr /> <div>last update: dec 16 2025 katie. current ver = 0.3.1.5 (automatic resubmission of failed SGE jobs). <br /> <br /> ChemSTEP (Chemical Space Traversal and Exploration Procedure) is an open-source, transparent acceleration algorithm for molecular docking capable of dealing with virtual libraries of several trillion compounds. This wiki page is a guide for BKS lab members to run ChemSTEP on Wynton HPC, using a drug-like subset of ZINC (22B) or 13.3B library from Enamine REAL. '''for detailed instructions on the 13B space, see 'Running ChemSTEP on the 13B space' wiki page'''. For more general use directions, please refer to [ChemSTEP Read-the-Docs]. <br /> <br /> At a high-level, ChemSTEP is an iterative process that identifies molecules from the larger virtual library to prioritize for docking. First, we identify a random sample of the total library (termed &quot;seed set&quot;, round zero) and dock those molecules to the target of interest. From this seed set, we can calculate total-library pProp values (-log rank percentages) and the number of &quot;virtual hits&quot; in the total library (high-scoring molecules). ChemSTEP will identify a set of maximally diverse molecules that score above the desired pProp threshold (&quot;beacons&quot;) from the seed set. These beacons guide prioritization, where molecules chosen and output by ChemSTEP are close in chemical space to the beacons. Prioritized molecules are then built, docked, and returned to ChemSTEP for a second round of prioritization. This process is iterated until you reach desired virtual hit recovery, or you are no longer recovering virtual hits. <br /> <br /> <br /> = Running ChemSTEP (Auto DOCK and Build) =<br /> <br /> ChemSTEP is configured to run on Wynton with libraries of '''13B''' and '''22B'''. This page covers the full workflow for running ChemSTEP with automatic submission of docking and building jobs.<br /> <br /> __TOC__<br /> <br /> == 1. Source Environment ==<br /> <br /> &lt;pre&gt;<br /> source /wynton/group/bks/work/shared/kholland/chemstep_auto_v02/bin/activate<br /> &lt;/pre&gt;<br /> <br /> == 2. Dock the Seed Set ==<br /> <br /> Copy the &lt;code&gt;.sdi&lt;/code&gt; file for the library you want to use:<br /> <br /> {| class=&quot;wikitable&quot;<br /> ! Library !! Path<br /> |-<br /> | 13B || &lt;code&gt;/wynton/group/bks/work/shared/kholland/chemstep_auto_v02/scripts/libraries/13B/13M_seeds.sdi&lt;/code&gt;<br /> |-<br /> | 22B || &lt;code&gt;/wynton/group/bks/work/shared/kholland/chemstep_auto_v02/scripts/libraries/22B/22M_seeds.sdi&lt;/code&gt;<br /> |}<br /> <br /> Then dock the seed set. See the '''Large-Scale Docking (LSD)''' directions.<br /> <br /> == 3. Gather Scores for the Seed Set ==<br /> <br /> Once docking is complete, run the following from the directory '''one level above''' your docking output (&lt;code&gt;MOLECULES_DIR_TO_BIND&lt;/code&gt;).<br /> <br /> '''22B library:'''<br /> &lt;pre&gt;<br /> python /wynton/group/bks/work/shared/kholland/chemstep_auto_v02/scripts/get_scores.py 0<br /> &lt;/pre&gt;<br /> <br /> '''13B library:'''<br /> &lt;pre&gt;<br /> python /wynton/group/bks/work/shared/kholland/chemstep_auto_v02/scripts/get_scores_13B.py 0 MOL<br /> &lt;/pre&gt;<br /> <br /> {{Note|You must specify the molecule ID prefix for the 13B library (&lt;code&gt;MOL&lt;/code&gt;).}}<br /> <br /> Verify that &lt;code&gt;scores_round_0.txt&lt;/code&gt; was correctly written:<br /> &lt;pre&gt;<br /> wc -l scores_round_0.txt<br /> &lt;/pre&gt;<br /> <br /> == 4. Convert Scores to .npy Files ==<br /> <br /> Convert scores to ChemSTEP-readable &lt;code&gt;.npy&lt;/code&gt; files:<br /> <br /> &lt;pre&gt;<br /> python /wynton/group/bks/work/shared/kholland/chemstep_auto_v02/scripts/convert_scores_to_npy.py 0 &lt;mol_id_prefix&gt;<br /> &lt;/pre&gt;<br /> <br /> The &lt;code&gt;mol_id_prefix&lt;/code&gt; should match the library:<br /> <br /> {| class=&quot;wikitable&quot;<br /> ! Library !! Prefix<br /> |-<br /> | 22B / 72B || &lt;code&gt;CSLB&lt;/code&gt;<br /> |-<br /> | 13B || &lt;code&gt;MOL&lt;/code&gt;<br /> |}<br /> <br /> == 5. Set Up the ChemSTEP Run Directory ==<br /> <br /> Create and enter a new run directory, then copy in the necessary files:<br /> <br /> &lt;pre&gt;<br /> mkdir chemstep_run<br /> cd chemstep_run<br /> chemstep-run-new<br /> &lt;/pre&gt;<br /> <br /> This will populate the directory with &lt;code&gt;params.txt&lt;/code&gt;, &lt;code&gt;run_chemstep.py&lt;/code&gt;, and &lt;code&gt;launch_chemstep_as_job.sh&lt;/code&gt;.<br /> <br /> === Optional: Integrated IFP ===<br /> <br /> If running with integrated IFP for beacon selection, also run:<br /> <br /> &lt;pre&gt;<br /> chemstep-run-ifp<br /> &lt;/pre&gt;<br /> <br /> This copies in the additional files &lt;code&gt;ifp_acceptance_criteria.txt&lt;/code&gt; and &lt;code&gt;interactions.txt&lt;/code&gt;.<br /> <br /> == 6. Edit params.txt ==<br /> <br /> Add the absolute paths to the ChemSTEP-readable score and indices &lt;code&gt;.npy&lt;/code&gt; arrays generated in Step 4.<br /> <br /> &lt;pre&gt;<br /> seed_indices_file: /path/to/your/indices_round_0.npy<br /> seed_scores_file: /path/to/your/scores_round_0.npy<br /> hit_pprop: 5.5<br /> n_docked_per_round: 2000000<br /> bundle_size: 1000<br /> max_beacons: 100<br /> max_n_rounds: 250<br /> &lt;/pre&gt;<br /> <br /> === Parameter Reference ===<br /> <br /> {| class=&quot;wikitable&quot;<br /> ! Parameter !! Description<br /> |-<br /> | &lt;code&gt;hit_pprop&lt;/code&gt; || Defines a &quot;virtual hit.&quot; pProp = −log(rank%) within the total library score distribution. E.g., pProp 4 in 13B space ≈ top 0.01% (~1.3M molecules); pProp 5 ≈ 0.001% (~132K). The seed set should contain at least 10&lt;sup&gt;(pProp+2)&lt;/sup&gt; molecules.<br /> |-<br /> | &lt;code&gt;n_docked_per_round&lt;/code&gt; || Number of molecules prioritized per round. All must be built and docked between rounds. Too many slows throughput and may reduce diversity; too few slows virtual hit recovery.<br /> |-<br /> | &lt;code&gt;max_beacons&lt;/code&gt; || Diverse, well-scoring molecules used to guide prioritization. All molecules above the pProp threshold are candidates. Too many reduces inter-beacon diversity; too few hinders space exploration. Fewer beacons than specified may be assigned if insufficient molecules clear the threshold.<br /> |-<br /> | &lt;code&gt;bundle_size&lt;/code&gt; || In auto docking mode, number of molecules submitted as a single build job.<br /> |-<br /> | &lt;code&gt;max_n_rounds&lt;/code&gt; || No adjustment needed when running ChemSTEP prospectively as described here.<br /> |}<br /> <br /> == 7. Edit run_chemstep.py ==<br /> <br /> '''Note:''' All paths must be '''absolute paths'''.<br /> <br /> === Required Settings ===<br /> <br /> Set &lt;code&gt;lib_path&lt;/code&gt; to the library pickle for your library:<br /> <br /> {| class=&quot;wikitable&quot;<br /> ! Library !! Path<br /> |-<br /> | 13B || &lt;code&gt;/wynton/group/bks/work/shared/kholland/chemstep_auto_v02/scripts/libraries/13B/boltz_fplib.pickle&lt;/code&gt;<br /> |-<br /> | 22B || &lt;code&gt;/wynton/group/bks/work/shared/kholland/chemstep_auto_v02/scripts/libraries/22B/22B_fplib.pickle&lt;/code&gt;<br /> |}<br /> <br /> &lt;pre&gt;<br /> lib_path = '/full/path/to/library.pickle'<br /> &lt;/pre&gt;<br /> <br /> Set &lt;code&gt;dockfiles_path&lt;/code&gt;:<br /> <br /> &lt;pre&gt;<br /> dockfiles_path=&quot;/full/path/to/dockfiles&quot;<br /> &lt;/pre&gt;<br /> <br /> === Optional: minTD Exclusion Zone ===<br /> <br /> Molecules will not be prioritized from within a specified Tanimoto distance of beacons. Comment in the relevant lines and update the value. Consider also setting &lt;code&gt;enforce_n_docked_per_round = True&lt;/code&gt; when using this option:<br /> <br /> &lt;pre&gt;<br /> min_td_search=0.5,<br /> enforce_n_docked_per_round=True,<br /> &lt;/pre&gt;<br /> <br /> === Optional: Integrated IFP ===<br /> <br /> Only selects beacons that satisfy user-defined interaction criteria. Comment in the following lines and update the paths to the necessary files (copied in Step 5 if you ran &lt;code&gt;chemstep-run-ifp&lt;/code&gt;):<br /> <br /> &lt;pre&gt;<br /> use_IFP=True,<br /> ifp_pdb_path='/full/path/to/rec.crg.pdb',<br /> interactions_file='/full/path/to/interactions.txt',<br /> ifp_acceptance_criteria_file='/full/path/to/ifp_acceptance_criteria.txt',<br /> &lt;/pre&gt;<br /> <br /> '''&lt;code&gt;interactions.txt&lt;/code&gt;''' — one interaction per line, comma-separated. Format: &lt;code&gt;interaction_type, residue_name_and_number&lt;/code&gt;. Example:<br /> <br /> &lt;pre&gt;<br /> Hydrogen bond, GLY19<br /> Ionic, ASP149<br /> &lt;/pre&gt;<br /> <br /> Supported interaction types include: Proximal, Hydrogen bond, Ionic, Cation-pi, Hydrophobic, Halogen bond, and others. See LUNA and IFP documentation for the full list.<br /> <br /> '''&lt;code&gt;ifp_acceptance_criteria.txt&lt;/code&gt;''' — defines the number of unsatisfied donors/acceptors/specific interactions required for a molecule to pass IFP and be considered for beacon selection. Example:<br /> <br /> &lt;pre&gt;<br /> #_donors<br /> #_acceptors<br /> #_unstatisfied_donors == 0<br /> #_unstatisfied_acceptors &lt;= 4<br /> Ionic/ASP-149 &gt; 0<br /> &lt;/pre&gt;<br /> <br /> === Example: AmpC on 22B with minTD=0.50, No IFP ===<br /> <br /> &lt;pre&gt;<br /> lib_path = '/wynton/group/bks/work/shared/kholland/chemstep_auto_v02/scripts/libraries/22B/22B_fplib.pickle'<br /> lib = load_library_from_pickle(lib_path)<br /> algo = CSAlgo(lib, 'params.txt', 'output', 16, verbose=True,<br /> scheduler='sge', smi_id_prefix='CSLB',<br /> python_exec=&quot;/wynton/group/bks/work/shared/kholland/chemstep_auto_v02/bin/python&quot;,<br /> dockfiles_path=&quot;/wynton/group/bks/work/kholland/chemstep_ampc_22B/seed_docking/dockfiles&quot;,<br /> min_td_search=0.5,<br /> enforce_n_docked_per_round=True,<br /> #use_IFP=True,<br /> #ifp_pdb_path='/path/to/your/reference/rec.crg.pdb',<br /> #interactions_file='/path/to/your/interactions.txt',<br /> #ifp_acceptance_criteria_file='/path/to/your/ifp_acceptance_criteria.txt',<br /> docking_method=&quot;auto&quot;, track_beacon_orig=True)<br /> &lt;/pre&gt;<br /> <br /> == 8. Launch the Job ==<br /> <br /> Submit the main ChemSTEP job:<br /> <br /> &lt;pre&gt;<br /> qsub launch_chemstep_as_job.sh<br /> &lt;/pre&gt;<br /> <br /> == 9. Monitor Job Status ==<br /> <br /> Check job status with &lt;code&gt;qstat&lt;/code&gt;. The main job will run for up to '''2 weeks''' given no errors. ChemSTEP will launch search, building, and docking jobs in successive rounds.<br /> <br /> '''Note:''' If any building or docking subjobs hang, the main job will not proceed until those are canceled or finished. Monitor job statuses regularly and occasionally verify that docking output files (&lt;code&gt;scores_round_*.txt&lt;/code&gt;) are being populated.<br /> <br /> == 10. View Beacon SMILES and IDs ==<br /> <br /> From the ChemSTEP running directory, run the following in a '''screen session on a dev node''':<br /> <br /> &lt;pre&gt;<br /> python /wynton/group/bks/work/shared/kholland/chemstep_auto_v02/scripts/get_beacon_smiles.py /path/to/library/pickle chemstep_algo.log<br /> &lt;/pre&gt;<br /> <br /> Use the library pickle path from [[#7. Edit run_chemstep.py|Step 7]].<br /> <br /> == 11. Get Poses After Docking ==<br /> <br /> Make a list of &lt;code&gt;test.mol2.gz.0&lt;/code&gt; files from docking:<br /> <br /> &lt;pre&gt;<br /> find /round_*_docking/bundle_paths -maxdepth 2 -name &quot;test.mol2.gz.0&quot; &gt; docked_poses.txt<br /> &lt;/pre&gt;<br /> <br /> Then extract top poses:<br /> <br /> &lt;pre&gt;<br /> python /wynton/group/bks/work/bwhall61/for_beau/top_poses.py \<br /> -t &lt;pProp_threshold&gt; \<br /> -s &lt;num_poses_per_file&gt; \<br /> -dock_results_path docked_poses.txt<br /> &lt;/pre&gt;</div>

Kholland http://wiki.docking.org/index.php?title=Running_ChemSTEP&diff=17056 2026-02-09T20:29:14Z

<p>Kholland: </p> <hr /> <div>last update: dec 16 2025 katie. current ver = 0.3.1.5 (automatic resubmission of failed SGE jobs). <br /> <br /> ChemSTEP (Chemical Space Traversal and Exploration Procedure) is an open-source, transparent acceleration algorithm for molecular docking capable of dealing with virtual libraries of several trillion compounds. This wiki page is a guide for BKS lab members to run ChemSTEP on Wynton HPC, using the current version of ZINC library (96B) or 13.3B library from Enamine REAL. '''for detailed instructions on the 13B space, see 'Running ChemSTEP on the 13B space' wiki page'''. For more general use directions, please refer to [ChemSTEP Read-the-Docs]. <br /> <br /> At a high-level, ChemSTEP is an iterative process that identifies molecules from the larger virtual library to prioritize for docking. First, we identify a random sample of the total library (termed &quot;seed set&quot;, round zero) and dock those molecules to the target of interest. From this seed set, we can calculate total-library pProp values (-log rank percentages) and the number of &quot;virtual hits&quot; in the total library (high-scoring molecules). ChemSTEP will identify a set of maximally diverse molecules that score above the desired pProp threshold (&quot;beacons&quot;) from the seed set. These beacons guide prioritization, where molecules chosen and output by ChemSTEP are close in chemical space to the beacons. Prioritized molecules are then built, docked, and returned to ChemSTEP for a second round of prioritization. This process is iterated until you reach desired virtual hit recovery, or you are no longer recovering virtual hits. <br /> <br /> <br /> Running the ChemSTEP algorithm on Wynton takes place in loosely three steps: (1) submission of the main ChemSTEP job, (2) ChainingLog sub-job array, and (3) gathering of SMILES for prioritization. <br /> <br /> '''1. Main ChemSTEP job:''' reads input files (DOCK scores and line-matched indices NumPy arrays) and assigns beacons. Beacons are a set of N maximally diverse molecules that score above the pProp threshold specified in the parameter file. For the first round of ChemSTEP, the best-scoring molecule from the seed set is assigned as the first beacon. Subsequent beacons would be molecules that score well (above pProp thresh), and are as diverse as possible from already assigned beacons. Greedy max-diversity selection. <br /> <br /> During assignment, ECFP4 Tanimoto distances between the assigned beacon and all potential beacons are calculated. The second beacon is the molecule with the maximum Td from beacon 1. Beacon 3 will be the molecule that has the highest Td from beacon 1 and beacon 2. This continues until the number of beacons specified in the parameter file is reached. In iterative rounds of ChemSTEP, the beacon selection is based on diversity from ALL PREVIOUSLY assigned beacons, including those in earlier rounds. <br /> <br /> <br /> '''2. ChainingLog sub-jobs:''' once beacons are assigned, ChemSTEP will launch a series of sub-jobs that calculate the Tanimoto distances of every molecule remaining in the library to the assigned beacons. Calculated distances to the NEAREST beacon (minimum-minimum Td) are updated in the mintddistrib_*.npy files within the /output directory. <br /> <br /> <br /> '''3. Gathering of SMILES for prioritization:''' When all Td calculations are completed, the algo will select N number of molecules for prioritization, the number of which is specified by the user in the parameter file. Molecules that are ''closest in chemical space'' to beacons are prioritized. I.e. if round size = 1 million, the 1 million molecules with the smallest min-Td to ANY beacon (current or previous rounds) are prioritized. chemstep_algo.log will provide a &quot;max-minTd&quot; for each round, which is the Tanimoto distance of the most dissimilar molecule prioritized to any one beacon per round. The prioritized molecules are output in /output/complete_info as smi_round_{}.smi <br /> <br /> What the user need: DOCKFILES, directories for (1) docking (2) building and (3) running ChemSTEP. '''All iterative rounds of ChemSTEP should be run in the SAME directory.''' <br /> <br /> '''1. Source ChemSTEP virtual environment on Wynton'''<br /> source /wynton/group/bks/work/shared/kholland/chemstep_auto_v02/bin/activate<br /> <br /> <br /> '''2. Copy seed-set SDI file into your docking directory'''<br /> This directory should already contain your dockfiles, with INDOCK parameters set to your liking. In this step, we are copying in a split database index file (SDI) containing paths to bundles of db2 files. This seed set contains 92 million molecules sampled randomly from the total virtual library, currently 96 billion molecules (ZINC).<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_auto/scripts/96M_seeds.wynton.sdi .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_auto/scripts/rebuilt_seeds.wynton.sdi .<br /> <br /> because the seed set is large, the docking must be done in two separate chunks (100k jobs max per docking array), and the scores combined into one file later on. <br /> <br /> if you are interested in running on a library of 13.2B compounds from Enamine REAL, there are several seed set sizes available: 130k, 1.3M, 13M, and 26M. <br /> cp /wynton/group/bks/work/shared/kholland/chemstep_13B/130K_seeds.wynton.sdi .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_13B/1.3M_seeds.wynton.sdi .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_13B/13M_seeds.wynton.sdi .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_13B/26M_seeds.wynton.sdi .<br /> <br /> <br /> '''3. Dock seed set to your receptor of interest using DOCK 3.8''' /docking directions taken from docs.docking.org. This is meant to be done as you would do a normal LSD. <br /> export MOLECULES_DIR_TO_BIND=[outermost folder containing the molecules to dock, just your base docking directory]<br /> export DOCKFILES=[path to your dockfiles]<br /> export INPUT_FOLDER=[the folder containing your .sdi file(s)]<br /> export OUTPUT_FOLDER=[where you want the output ]<br /> <br /> /wynton/group/bks/work/bwhall61/needs_github/super_dock3r.sh<br /> <br /> Wait for docking to complete. Next, you must extract all molecule IDs and corresponding DOCK scores from above. To do so, run the following commands in the base docking directory containing your docking files and output folder, while logged into a dev node (do this in a screen!!!):<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_auto/get_scores.py .<br /> python get_scores.py 0<br /> <br /> <br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/get_scores.py .<br /> python get_scores.py 0<br /> <br /> <br /> If docking the seeds from the ZINC library, please combine your score.txt files from both SDI files. Rename this combined file 'scores_round_0.txt.'<br /> <br /> This script expects a directory named &quot;output*&quot; within the CWD. If your output from docking follows different organizational conventions, vim into get_scores.py and change the path. Successful output of the script will be a file named &quot;scores_round_0.txt&quot;. For iterative rounds, pass increasing numbers into the command line. i.e. When docking the first round of prioritized molecules, pass [1] for scores_round_1.txt. <br /> <br /> '''4. Convert scores and molecule IDS into NumPY arrays for ChemSTEP recognition. Requires ChemSTEP venv'''<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_auto/convert_scores_to_npy.py .<br /> python convert_scores_to_npy.py 0<br /> <br /> <br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/convert_scores_to_npy.py .<br /> python convert_scores_to_npy.py 0<br /> <br /> For python speed purposes, ChemSTEP requires that DOCK scores and IDs be give in the form of a NumPY array. In this step, we are taking our readable scores file and converting them for ChemSTEP recognition. This script expects a txt file with Molecule IDS and DOCK scores (no header). Use the same round number you used in step 3. As run above, this will output two files named scores_round_0.npy and indices_round_0.npy that contain line-matched indices (determined from Mol ID) and their respective docking scores. <br /> <br /> '''5. Enter into or make a directory to run ChemSTEP in. Copy in necessary files for initiating ChemSTEP: params.txt, run_chemstep.py and launch_chemstep_as_job.sh. Copy in your score and indices numpy files as well, which should be in your docking directory.'''<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_auto/scripts/params.txt .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_auto/scripts/run_chemstep.py .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_auto/scripts/launch_chemstep_as_job.sh .<br /> <br /> <br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/params.txt .<br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/run_chemstep.py .<br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/launch_chemstep_as_job.sh .<br /> <br /> '''6. Edit params.txt file''' This ONLY needs to be edited for the initial round of ChemSTEP. The parameters outlined here will be carried through all rounds of ChemSTEP. <br /> seed_indices_file: /absoulte/path/to/your/indices_round_0.npy<br /> seed_scores_file: /absolute/path/to/your/scores_round_0.npy<br /> hit_pprop: 5<br /> bundle_size: 1000<br /> n_docked_per_round: 10000000<br /> max_beacons: 150<br /> max_n_rounds: 250<br /> <br /> Within params.txt, add the absolute paths to the ChemSTEP-readable score and indices NumPY arrays. The rest of the values within the params.txt file are left to the discretion of the user, with some considerations below. <br /> <br /> '''pProp:''' this value will define what is considered a &quot;virtual hit&quot; in your campaign. pProp is defined as the -log(rank %) of a molecule within the total library score-distribution. For example, a pProp of 4 in the 1.1T XReal space is equivalent the top 0.01% of the library, corresponding to the top 110 million molecules. pProp 5 = 0.001% = 11 million virtual hits, etc. From the seed set, ChemSTEP will estimate a DOCK score value in kcal/mol that associated with the lower-limit of your desired pProp zone. Any molecules that score better than the threshold is considered a virtual hit in your campaign. Be mindful of seed set size when choosing desired pProp. Our suggestion is that the size of the seed set should be at least 10^(pProp +2).<br /> <br /> '''n_docked_per_round:''' this number is the desired molecules for prioritization per round. When choosing this value, note that this number of molecules must be built and docked in between each round of ChemSTEP. Prioritizing many molecules will slow building and docking speeds, and coupled with few beacons (below) may lead to decreased diversity. Prioritizing too few molecules may result in slower virtual hit recovery. Round size does not significantly impact the algorithm running time.** <br /> <br /> '''max_beacons:''' this is the number of diverse, well-scoring molecules ChemSTEP will use to guide prioritization per round. All molecules that score better than the assigned pProp threshold may be considered for beacons. By default, ChemSTEP chooses each set of beacons to be as maximally diverse as possible. Choosing too many beacons may result in decreased diversity between beacons, but too few beacons could hinder space exploration. If not enough molecules score above your pProp threshold, ChemSTEP may assign fewer beacons than the assigned value. <br /> <br /> '''max_n_rounds:''' if prospectively running ChemSTEP, as outlined in this wiki, no need to worry about this parameter.<br /> <br /> '''bundle_size:''' if using auto docking mode, the bundle size is used for building. i.e. the number of molecules that get submitted to build as one job. <br /> <br /> <br /> <br /> '''7. Edit run_chemstep.py''' <br /> <br /> There are several configurable parameters that can be passed when first calling the CSAlgo object:<br /> <br /> '''docking_method=''' setting this to &quot;auto&quot; will set up and run iterative rounds of building, docking, and running ChemSTEP. If you choose the auto method, be sure to update the full path to your dockfiles in run_chemstep.py and check that all INDOCK parameters are set to your liking. NOTE: docking method is set to auto if you copy in the scripts as outlined above. Change if desired!!!!!<br /> <br /> <br /> '''min_td_search=''' setting this parameter will establish a minimum-tanimoto distance &quot;zone&quot; around each beacon for prioritization. For example, setting a min_td_search=0.3 will require that each prioritized molecule be greater than 0.3 Td from each beacon. <br /> <br /> more optional parameters to come.<br /> <br /> At this time, we strongly suggest running ChemSTEP as a job array with 8 or 16 CPU slots requested. The number of cores must be specified when calling CSAlgo (in run_chemstep.py) and the SGE wrapper(launch_chemstep_as_job.sh). <br /> <br /> <br /> '''8. Run ChemSTEP''' with the following command: <br /> qsub launch_chemstep_as_job.sh <br /> <br /> <br /> This will launch the main ChemSTEP job to the SGE scheduler. Check your job status with the 'qstat' command. The algorithm will read the score and indices files provided, calculate pProp (for round 1) and assign beacons. As this job runs, it will launch a subsequent job array. These sub-jobs are the Chaining step of ChemSTEP, where each parallel worker calculates the Tanimoto distances of 100 million molecules from the virtual library to the beacons. These distances are then written to the files within the /output directory. When these jobs complete, the main job will read through all Tanimoto distances and pull molecules for prioritization. <br /> <br /> Running ChemSTEP successfully will result in the output of a SMILES file within output/complete_info/. You will also get (1) chemstep_algo.log and (2) chemstep_submission.log in the working directory after job submission. chemstep_algo.log contains the running output of the algorithm in a readable format, with timestamps, including the DOCK score associated with your desired pProp. chemstep_submission.log contains the output of ChemSTEP, as well as any information output by the SGE submission system (python errors, cluster issues, tracebacks if failed). These files will be updated with any information with iterative rounds of ChemSTEP run in the same directory. '''Visually inspect these files after each round of ChemSTEP to ensure the algorithm has picked your desired number of beacons and things ran smoothly.'''<br /> <br /> '''Troubleshooting:''' if no jobs are running and there is no SMI file, check the chemstep_submission.log first. Any traceback error or SGE error should give you some idea of why ChemSTEP failed. Some errors may be due to SGE or Wynton issues. If directed, look at a few .out files within /output/jobs/ . If a ChemSTEP run fails on your FIRST run, delete the output and log files, fix what needs to be fixed, and rerun. Errors in subsequent rounds during the chaining step can potentially corrupt chaining files within the /output directory that are needed for prioritization. At the very least, you will definitely have duplicates of beacons and information written to the log files. At this time, if your run fails during iterative rounds, it's best to start from the beginning. <br /> <br /> Prioritized molecules should be built, docked, and their scores fed back into ChemSTEP. More detailed instructions below for running in MANUAL mode: <br /> <br /> '''9. Build prioritized molecules (DOCK 3.8).''' /taken from docs.docking.org <br /> source /wynton/group/bks/soft/DOCK-3.8.5/env.sh<br /> <br /> python /wynton/group/bks/soft/DOCK-3.8.5/DOCK3.8/zinc22-3d/submit/submit_building_docker.py --output_folder building_output --bundle_size 1000 --minutes_per_mol 5 --skip_name_check --scheduler sge --container_software apptainer --container_path_or_name /wynton/group/bks/soft/DOCK-3.8.5/building_pipeline.sif smi_round_{}.smi<br /> <br /> When building has completed, you must write an SDI file with the complete paths to each built bundle and dock. you can do this with:<br /> find /path/to/your/building_output -name &quot;bundle.db2.tgz&quot; -type f &gt; round_{}.wynton.sdi<br /> <br /> <br /> '''Be sure to change the INDOCK file to save only poses that meet your score pProp score threshold calculated by ChemSTEP!''' Retrieve docking scores as convert to NumPy arrays as outlined above, updating the round number when running get_scores.py and convert_scores_to_npy.py. Copy new score and indices files into the directory you ran ChemSTEP in. If you are following along as a tutorial, you should have scores_round_1.npy and indices_round_1.npy from the previous step (from FIRST round of ChemSTEP prioritization). <br /> <br /> <br /> '''10. Set up for iterative rounds of ChemSTEP'''<br /> <br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/run_chemstep_iterative.py .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/launch_chemstep_iterative.sh .<br /> <br /> <br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/run_chemstep_iterative.py .<br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/launch_chemstep_iterative.sh .<br /> <br /> '''11. Run ChemSTEP'''<br /> qsub launch_chemstep_as_job.sh [round number]<br /> <br /> For the first iterative round, the round number is [2], and should increase by one for every subsequent round of ChemSTEP. The output will be smi_round_****.smi file. Repeat steps 8-10 for as many rounds as needed. The performance is reported in outputy/complete_info/run_summary.df, which contains the number of beacons selected, the number of molecules docked, the number of hits found, the distance threshold for the selected molecules to dock, and the last added beacon's distance to all previous beacons.<br /> <br /> <br /> <br /> <br /> '''Helper scripts:''' a running list of scripts we have been using for analysis and visualization. <br /> <br /> to get SMILES for beacons: <br /> <br /> python /wynton/group/bks/work/bwhall61/needs_github/get_beacon_smi.py --beacon_df_path /path/to/your/chemstep/output/complete_info/beacons.df --n_workers 6 --outfile beacon_smiles.smi<br /> <br /> <br /> ** anecdotally true.</div>

Kholland http://wiki.docking.org/index.php?title=Running_ChemSTEP&diff=17031 2026-01-12T22:43:34Z

<p>Kholland: </p> <hr /> <div>last update: dec 16 2025 katie. current ver = 0.3.1.5 (automatic resubmission of failed SGE jobs). <br /> <br /> ChemSTEP (Chemical Space Traversal and Exploration Procedure) is an open-source, transparent acceleration algorithm for molecular docking capable of dealing with virtual libraries of several trillion compounds. This wiki page is a guide for BKS lab members to run ChemSTEP on Wynton HPC, using the current version of ZINC library (96B) or 13.3B library from Enamine REAL. '''for detailed instructions on the 13B space, see 'Running ChemSTEP on the 13B space' wiki page'''. For more general use directions, please refer to [ChemSTEP Read-the-Docs]. <br /> <br /> At a high-level, ChemSTEP is an iterative process that identifies molecules from the larger virtual library to prioritize for docking. First, we identify a random sample of the total library (termed &quot;seed set&quot;, round zero) and dock those molecules to the target of interest. From this seed set, we can calculate total-library pProp values (-log rank percentages) and the number of &quot;virtual hits&quot; in the total library (high-scoring molecules). ChemSTEP will identify a set of maximally diverse molecules that score above the desired pProp threshold (&quot;beacons&quot;) from the seed set. These beacons guide prioritization, where molecules chosen and output by ChemSTEP are close in chemical space to the beacons. Prioritized molecules are then built, docked, and returned to ChemSTEP for a second round of prioritization. This process is iterated until you reach desired virtual hit recovery, or you are no longer recovering virtual hits. <br /> <br /> <br /> Running the ChemSTEP algorithm on Wynton takes place in loosely three steps: (1) submission of the main ChemSTEP job, (2) ChainingLog sub-job array, and (3) gathering of SMILES for prioritization. <br /> <br /> '''1. Main ChemSTEP job:''' reads input files (DOCK scores and line-matched indices NumPy arrays) and assigns beacons. Beacons are a set of N maximally diverse molecules that score above the pProp threshold specified in the parameter file. For the first round of ChemSTEP, the best-scoring molecule from the seed set is assigned as the first beacon. Subsequent beacons would be molecules that score well (above pProp thresh), and are as diverse as possible from already assigned beacons. Greedy max-diversity selection. <br /> <br /> During assignment, ECFP4 Tanimoto distances between the assigned beacon and all potential beacons are calculated. The second beacon is the molecule with the maximum Td from beacon 1. Beacon 3 will be the molecule that has the highest Td from beacon 1 and beacon 2. This continues until the number of beacons specified in the parameter file is reached. In iterative rounds of ChemSTEP, the beacon selection is based on diversity from ALL PREVIOUSLY assigned beacons, including those in earlier rounds. <br /> <br /> <br /> '''2. ChainingLog sub-jobs:''' once beacons are assigned, ChemSTEP will launch a series of sub-jobs that calculate the Tanimoto distances of every molecule remaining in the library to the assigned beacons. Calculated distances to the NEAREST beacon (minimum-minimum Td) are updated in the mintddistrib_*.npy files within the /output directory. <br /> <br /> <br /> '''3. Gathering of SMILES for prioritization:''' When all Td calculations are completed, the algo will select N number of molecules for prioritization, the number of which is specified by the user in the parameter file. Molecules that are ''closest in chemical space'' to beacons are prioritized. I.e. if round size = 1 million, the 1 million molecules with the smallest min-Td to ANY beacon (current or previous rounds) are prioritized. chemstep_algo.log will provide a &quot;max-minTd&quot; for each round, which is the Tanimoto distance of the most dissimilar molecule prioritized to any one beacon per round. The prioritized molecules are output in /output/complete_info as smi_round_{}.smi <br /> <br /> What the user need: DOCKFILES, directories for (1) docking (2) building and (3) running ChemSTEP. '''All iterative rounds of ChemSTEP should be run in the SAME directory.''' <br /> <br /> '''1. Source ChemSTEP virtual environment on Wynton'''<br /> source /wynton/group/bks/work/shared/kholland/chemstep_auto/bin/activate<br /> <br /> <br /> '''2. Copy seed-set SDI file into your docking directory'''<br /> This directory should already contain your dockfiles, with INDOCK parameters set to your liking. In this step, we are copying in a split database index file (SDI) containing paths to bundles of db2 files. This seed set contains 92 million molecules sampled randomly from the total virtual library, currently 96 billion molecules (ZINC).<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_auto/scripts/96M_seeds.wynton.sdi .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_auto/scripts/rebuilt_seeds.wynton.sdi .<br /> <br /> because the seed set is large, the docking must be done in two separate chunks (100k jobs max per docking array), and the scores combined into one file later on. <br /> <br /> if you are interested in running on a library of 13.2B compounds from Enamine REAL, there are several seed set sizes available: 130k, 1.3M, 13M, and 26M. <br /> cp /wynton/group/bks/work/shared/kholland/chemstep_13B/130K_seeds.wynton.sdi .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_13B/1.3M_seeds.wynton.sdi .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_13B/13M_seeds.wynton.sdi .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_13B/26M_seeds.wynton.sdi .<br /> <br /> <br /> '''3. Dock seed set to your receptor of interest using DOCK 3.8''' /docking directions taken from docs.docking.org. This is meant to be done as you would do a normal LSD. <br /> export MOLECULES_DIR_TO_BIND=[outermost folder containing the molecules to dock, just your base docking directory]<br /> export DOCKFILES=[path to your dockfiles]<br /> export INPUT_FOLDER=[the folder containing your .sdi file(s)]<br /> export OUTPUT_FOLDER=[where you want the output ]<br /> <br /> /wynton/group/bks/work/bwhall61/needs_github/super_dock3r.sh<br /> <br /> Wait for docking to complete. Next, you must extract all molecule IDs and corresponding DOCK scores from above. To do so, run the following commands in the base docking directory containing your docking files and output folder, while logged into a dev node (do this in a screen!!!):<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_auto/get_scores.py .<br /> python get_scores.py 0<br /> <br /> <br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/get_scores.py .<br /> python get_scores.py 0<br /> <br /> <br /> If docking the seeds from the ZINC library, please combine your score.txt files from both SDI files. Rename this combined file 'scores_round_0.txt.'<br /> <br /> This script expects a directory named &quot;output*&quot; within the CWD. If your output from docking follows different organizational conventions, vim into get_scores.py and change the path. Successful output of the script will be a file named &quot;scores_round_0.txt&quot;. For iterative rounds, pass increasing numbers into the command line. i.e. When docking the first round of prioritized molecules, pass [1] for scores_round_1.txt. <br /> <br /> '''4. Convert scores and molecule IDS into NumPY arrays for ChemSTEP recognition. Requires ChemSTEP venv'''<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_auto/convert_scores_to_npy.py .<br /> python convert_scores_to_npy.py 0<br /> <br /> <br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/convert_scores_to_npy.py .<br /> python convert_scores_to_npy.py 0<br /> <br /> For python speed purposes, ChemSTEP requires that DOCK scores and IDs be give in the form of a NumPY array. In this step, we are taking our readable scores file and converting them for ChemSTEP recognition. This script expects a txt file with Molecule IDS and DOCK scores (no header). Use the same round number you used in step 3. As run above, this will output two files named scores_round_0.npy and indices_round_0.npy that contain line-matched indices (determined from Mol ID) and their respective docking scores. <br /> <br /> '''5. Enter into or make a directory to run ChemSTEP in. Copy in necessary files for initiating ChemSTEP: params.txt, run_chemstep.py and launch_chemstep_as_job.sh. Copy in your score and indices numpy files as well, which should be in your docking directory.'''<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_auto/scripts/params.txt .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_auto/scripts/run_chemstep.py .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_auto/scripts/launch_chemstep_as_job.sh .<br /> <br /> <br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/params.txt .<br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/run_chemstep.py .<br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/launch_chemstep_as_job.sh .<br /> <br /> '''6. Edit params.txt file''' This ONLY needs to be edited for the initial round of ChemSTEP. The parameters outlined here will be carried through all rounds of ChemSTEP. <br /> seed_indices_file: /absoulte/path/to/your/indices_round_0.npy<br /> seed_scores_file: /absolute/path/to/your/scores_round_0.npy<br /> hit_pprop: 5<br /> bundle_size: 1000<br /> n_docked_per_round: 10000000<br /> max_beacons: 150<br /> max_n_rounds: 250<br /> <br /> Within params.txt, add the absolute paths to the ChemSTEP-readable score and indices NumPY arrays. The rest of the values within the params.txt file are left to the discretion of the user, with some considerations below. <br /> <br /> '''pProp:''' this value will define what is considered a &quot;virtual hit&quot; in your campaign. pProp is defined as the -log(rank %) of a molecule within the total library score-distribution. For example, a pProp of 4 in the 1.1T XReal space is equivalent the top 0.01% of the library, corresponding to the top 110 million molecules. pProp 5 = 0.001% = 11 million virtual hits, etc. From the seed set, ChemSTEP will estimate a DOCK score value in kcal/mol that associated with the lower-limit of your desired pProp zone. Any molecules that score better than the threshold is considered a virtual hit in your campaign. Be mindful of seed set size when choosing desired pProp. Our suggestion is that the size of the seed set should be at least 10^(pProp +2).<br /> <br /> '''n_docked_per_round:''' this number is the desired molecules for prioritization per round. When choosing this value, note that this number of molecules must be built and docked in between each round of ChemSTEP. Prioritizing many molecules will slow building and docking speeds, and coupled with few beacons (below) may lead to decreased diversity. Prioritizing too few molecules may result in slower virtual hit recovery. Round size does not significantly impact the algorithm running time.** <br /> <br /> '''max_beacons:''' this is the number of diverse, well-scoring molecules ChemSTEP will use to guide prioritization per round. All molecules that score better than the assigned pProp threshold may be considered for beacons. By default, ChemSTEP chooses each set of beacons to be as maximally diverse as possible. Choosing too many beacons may result in decreased diversity between beacons, but too few beacons could hinder space exploration. If not enough molecules score above your pProp threshold, ChemSTEP may assign fewer beacons than the assigned value. <br /> <br /> '''max_n_rounds:''' if prospectively running ChemSTEP, as outlined in this wiki, no need to worry about this parameter.<br /> <br /> '''bundle_size:''' if using auto docking mode, the bundle size is used for building. i.e. the number of molecules that get submitted to build as one job. <br /> <br /> <br /> <br /> '''7. Edit run_chemstep.py''' <br /> <br /> There are several configurable parameters that can be passed when first calling the CSAlgo object:<br /> <br /> '''docking_method=''' setting this to &quot;auto&quot; will set up and run iterative rounds of building, docking, and running ChemSTEP. If you choose the auto method, be sure to update the full path to your dockfiles in run_chemstep.py and check that all INDOCK parameters are set to your liking. NOTE: docking method is set to auto if you copy in the scripts as outlined above. Change if desired!!!!!<br /> <br /> <br /> '''min_td_search=''' setting this parameter will establish a minimum-tanimoto distance &quot;zone&quot; around each beacon for prioritization. For example, setting a min_td_search=0.3 will require that each prioritized molecule be greater than 0.3 Td from each beacon. <br /> <br /> more optional parameters to come.<br /> <br /> At this time, we strongly suggest running ChemSTEP as a job array with 8 or 16 CPU slots requested. The number of cores must be specified when calling CSAlgo (in run_chemstep.py) and the SGE wrapper(launch_chemstep_as_job.sh). <br /> <br /> <br /> '''8. Run ChemSTEP''' with the following command: <br /> qsub launch_chemstep_as_job.sh <br /> <br /> <br /> This will launch the main ChemSTEP job to the SGE scheduler. Check your job status with the 'qstat' command. The algorithm will read the score and indices files provided, calculate pProp (for round 1) and assign beacons. As this job runs, it will launch a subsequent job array. These sub-jobs are the Chaining step of ChemSTEP, where each parallel worker calculates the Tanimoto distances of 100 million molecules from the virtual library to the beacons. These distances are then written to the files within the /output directory. When these jobs complete, the main job will read through all Tanimoto distances and pull molecules for prioritization. <br /> <br /> Running ChemSTEP successfully will result in the output of a SMILES file within output/complete_info/. You will also get (1) chemstep_algo.log and (2) chemstep_submission.log in the working directory after job submission. chemstep_algo.log contains the running output of the algorithm in a readable format, with timestamps, including the DOCK score associated with your desired pProp. chemstep_submission.log contains the output of ChemSTEP, as well as any information output by the SGE submission system (python errors, cluster issues, tracebacks if failed). These files will be updated with any information with iterative rounds of ChemSTEP run in the same directory. '''Visually inspect these files after each round of ChemSTEP to ensure the algorithm has picked your desired number of beacons and things ran smoothly.'''<br /> <br /> '''Troubleshooting:''' if no jobs are running and there is no SMI file, check the chemstep_submission.log first. Any traceback error or SGE error should give you some idea of why ChemSTEP failed. Some errors may be due to SGE or Wynton issues. If directed, look at a few .out files within /output/jobs/ . If a ChemSTEP run fails on your FIRST run, delete the output and log files, fix what needs to be fixed, and rerun. Errors in subsequent rounds during the chaining step can potentially corrupt chaining files within the /output directory that are needed for prioritization. At the very least, you will definitely have duplicates of beacons and information written to the log files. At this time, if your run fails during iterative rounds, it's best to start from the beginning. <br /> <br /> Prioritized molecules should be built, docked, and their scores fed back into ChemSTEP. More detailed instructions below for running in MANUAL mode: <br /> <br /> '''9. Build prioritized molecules (DOCK 3.8).''' /taken from docs.docking.org <br /> source /wynton/group/bks/soft/DOCK-3.8.5/env.sh<br /> <br /> python /wynton/group/bks/soft/DOCK-3.8.5/DOCK3.8/zinc22-3d/submit/submit_building_docker.py --output_folder building_output --bundle_size 1000 --minutes_per_mol 5 --skip_name_check --scheduler sge --container_software apptainer --container_path_or_name /wynton/group/bks/soft/DOCK-3.8.5/building_pipeline.sif smi_round_{}.smi<br /> <br /> When building has completed, you must write an SDI file with the complete paths to each built bundle and dock. you can do this with:<br /> find /path/to/your/building_output -name &quot;bundle.db2.tgz&quot; -type f &gt; round_{}.wynton.sdi<br /> <br /> <br /> '''Be sure to change the INDOCK file to save only poses that meet your score pProp score threshold calculated by ChemSTEP!''' Retrieve docking scores as convert to NumPy arrays as outlined above, updating the round number when running get_scores.py and convert_scores_to_npy.py. Copy new score and indices files into the directory you ran ChemSTEP in. If you are following along as a tutorial, you should have scores_round_1.npy and indices_round_1.npy from the previous step (from FIRST round of ChemSTEP prioritization). <br /> <br /> <br /> '''10. Set up for iterative rounds of ChemSTEP'''<br /> <br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/run_chemstep_iterative.py .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/launch_chemstep_iterative.sh .<br /> <br /> <br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/run_chemstep_iterative.py .<br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/launch_chemstep_iterative.sh .<br /> <br /> '''11. Run ChemSTEP'''<br /> qsub launch_chemstep_as_job.sh [round number]<br /> <br /> For the first iterative round, the round number is [2], and should increase by one for every subsequent round of ChemSTEP. The output will be smi_round_****.smi file. Repeat steps 8-10 for as many rounds as needed. The performance is reported in outputy/complete_info/run_summary.df, which contains the number of beacons selected, the number of molecules docked, the number of hits found, the distance threshold for the selected molecules to dock, and the last added beacon's distance to all previous beacons.<br /> <br /> <br /> <br /> <br /> '''Helper scripts:''' a running list of scripts we have been using for analysis and visualization. <br /> <br /> to get SMILES for beacons: <br /> <br /> python /wynton/group/bks/work/bwhall61/needs_github/get_beacon_smi.py --beacon_df_path /path/to/your/chemstep/output/complete_info/beacons.df --n_workers 6 --outfile beacon_smiles.smi<br /> <br /> <br /> ** anecdotally true.</div>

Kholland http://wiki.docking.org/index.php?title=Running_ChemSTEP&diff=17021 2025-12-16T19:41:19Z

<p>Kholland: </p> <hr /> <div>last update: dec 16 2025 katie. current ver = 0.3.1.5 (automatic resubmission of failed SGE jobs). <br /> <br /> ChemSTEP (Chemical Space Traversal and Exploration Procedure) is an open-source, transparent acceleration algorithm for molecular docking capable of dealing with virtual libraries of several trillion compounds. This wiki page is a guide for BKS lab members to run ChemSTEP on Wynton HPC, using the current version of ZINC library (96B) or 13.3B library from Enamine REAL. '''for detailed instructions on the 13B space, see 'Running ChemSTEP on the 13B space' wiki page'''. For more general use directions, please refer to [ChemSTEP Read-the-Docs]. <br /> <br /> At a high-level, ChemSTEP is an iterative process that identifies molecules from the larger virtual library to prioritize for docking. First, we identify a random sample of the total library (termed &quot;seed set&quot;, round zero) and dock those molecules to the target of interest. From this seed set, we can calculate total-library pProp values (-log rank percentages) and the number of &quot;virtual hits&quot; in the total library (high-scoring molecules). ChemSTEP will identify a set of maximally diverse molecules that score above the desired pProp threshold (&quot;beacons&quot;) from the seed set. These beacons guide prioritization, where molecules chosen and output by ChemSTEP are close in chemical space to the beacons. Prioritized molecules are then built, docked, and returned to ChemSTEP for a second round of prioritization. This process is iterated until you reach desired virtual hit recovery, or you are no longer recovering virtual hits. <br /> <br /> <br /> Running the ChemSTEP algorithm on Wynton takes place in loosely three steps: (1) submission of the main ChemSTEP job, (2) ChainingLog sub-job array, and (3) gathering of SMILES for prioritization. <br /> <br /> '''1. Main ChemSTEP job:''' reads input files (DOCK scores and line-matched indices NumPy arrays) and assigns beacons. Beacons are a set of N maximally diverse molecules that score above the pProp threshold specified in the parameter file. For the first round of ChemSTEP, the best-scoring molecule from the seed set is assigned as the first beacon. Subsequent beacons would be molecules that score well (above pProp thresh), and are as diverse as possible from already assigned beacons. Greedy max-diversity selection. <br /> <br /> During assignment, ECFP4 Tanimoto distances between the assigned beacon and all potential beacons are calculated. The second beacon is the molecule with the maximum Td from beacon 1. Beacon 3 will be the molecule that has the highest Td from beacon 1 and beacon 2. This continues until the number of beacons specified in the parameter file is reached. In iterative rounds of ChemSTEP, the beacon selection is based on diversity from ALL PREVIOUSLY assigned beacons, including those in earlier rounds. <br /> <br /> <br /> '''2. ChainingLog sub-jobs:''' once beacons are assigned, ChemSTEP will launch a series of sub-jobs that calculate the Tanimoto distances of every molecule remaining in the library to the assigned beacons. Calculated distances to the NEAREST beacon (minimum-minimum Td) are updated in the mintddistrib_*.npy files within the /output directory. <br /> <br /> <br /> '''3. Gathering of SMILES for prioritization:''' When all Td calculations are completed, the algo will select N number of molecules for prioritization, the number of which is specified by the user in the parameter file. Molecules that are ''closest in chemical space'' to beacons are prioritized. I.e. if round size = 1 million, the 1 million molecules with the smallest min-Td to ANY beacon (current or previous rounds) are prioritized. chemstep_algo.log will provide a &quot;max-minTd&quot; for each round, which is the Tanimoto distance of the most dissimilar molecule prioritized to any one beacon per round. The prioritized molecules are output in /output/complete_info as smi_round_{}.smi <br /> <br /> What the user need: DOCKFILES, directories for (1) docking (2) building and (3) running ChemSTEP. '''All iterative rounds of ChemSTEP should be run in the SAME directory.''' <br /> <br /> '''1. Source ChemSTEP virtual environment on Wynton'''<br /> source /wynton/group/bks/work/shared/kholland/chemstep_auto/bin/activate<br /> <br /> <br /> '''2. Copy seed-set SDI file into your docking directory'''<br /> This directory should already contain your dockfiles, with INDOCK parameters set to your liking. In this step, we are copying in a split database index file (SDI) containing paths to bundles of db2 files. This seed set contains 92 million molecules sampled randomly from the total virtual library, currently 96 billion molecules (ZINC).<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_auto/scripts/96M_seeds.wynton.sdi .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_auto/scripts/rebuilt_seeds.wynton.sdi .<br /> <br /> because the seed set is large, the docking must be done in two separate chunks (100k jobs max per docking array), and the scores combined into one file later on. <br /> <br /> if you are interested in running on a library of 13.2B compounds from Enamine REAL, there are several seed set sizes available: 130k, 1.3M, 13M, and 26M. <br /> cp /wynton/group/bks/work/shared/kholland/chemstep_13B/130K_seeds.wynton.sdi .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_13B/1.3M_seeds.wynton.sdi .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_13B/13M_seeds.wynton.sdi .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_13B/26M_seeds.wynton.sdi .<br /> <br /> <br /> '''3. Dock seed set to your receptor of interest using DOCK 3.8''' /docking directions taken from docs.docking.org. This is meant to be done as you would do a normal LSD. <br /> export MOLECULES_DIR_TO_BIND=[outermost folder containing the molecules to dock, just your base docking directory]<br /> export DOCKFILES=[path to your dockfiles]<br /> export INPUT_FOLDER=[the folder containing your .sdi file(s)]<br /> export OUTPUT_FOLDER=[where you want the output ]<br /> <br /> /wynton/group/bks/work/bwhall61/needs_github/super_dock3r.sh<br /> <br /> Wait for docking to complete. Next, you must extract all molecule IDs and corresponding DOCK scores from above. To do so, run the following commands in the base docking directory containing your docking files and output folder, while logged into a dev node (do this in a screen!!!):<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_auto/get_scores.py .<br /> python get_scores.py 0<br /> <br /> <br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/get_scores.py .<br /> python get_scores.py 0<br /> <br /> <br /> If docking the seeds from the ZINC library, please combine your score.txt files from both SDI files. Rename this combined file 'scores_round_0.txt.'<br /> <br /> This script expects a directory named &quot;output*&quot; within the CWD. If your output from docking follows different organizational conventions, vim into get_scores.py and change the path. Successful output of the script will be a file named &quot;scores_round_0.txt&quot;. For iterative rounds, pass increasing numbers into the command line. i.e. When docking the first round of prioritized molecules, pass [1] for scores_round_1.txt. <br /> <br /> '''4. Convert scores and molecule IDS into NumPY arrays for ChemSTEP recognition. Requires ChemSTEP venv'''<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_auto/convert_scores_to_npy.py .<br /> python convert_scores_to_npy.py 0<br /> <br /> <br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/convert_scores_to_npy.py .<br /> python convert_scores_to_npy.py 0<br /> <br /> For python speed purposes, ChemSTEP requires that DOCK scores and IDs be give in the form of a NumPY array. In this step, we are taking our readable scores file and converting them for ChemSTEP recognition. This script expects a txt file with Molecule IDS and DOCK scores (no header). Use the same round number you used in step 3. As run above, this will output two files named scores_round_0.npy and indices_round_0.npy that contain line-matched indices (determined from Mol ID) and their respective docking scores. <br /> <br /> '''5. Enter into or make a directory to run ChemSTEP in. Copy in necessary files for initiating ChemSTEP: params.txt, run_chemstep.py and launch_chemstep_as_job.sh. Copy in your score and indices numpy files as well, which should be in your docking directory.'''<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_auto/params.txt .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_auto/run_chemstep.py .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_auto/launch_chemstep_as_job.sh .<br /> <br /> <br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/params.txt .<br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/run_chemstep.py .<br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/launch_chemstep_as_job.sh .<br /> <br /> '''6. Edit params.txt file''' This ONLY needs to be edited for the initial round of ChemSTEP. The parameters outlined here will be carried through all rounds of ChemSTEP. <br /> seed_indices_file: /absoulte/path/to/your/indices_round_0.npy<br /> seed_scores_file: /absolute/path/to/your/scores_round_0.npy<br /> hit_pprop: 5<br /> bundle_size: 1000<br /> n_docked_per_round: 10000000<br /> max_beacons: 150<br /> max_n_rounds: 250<br /> <br /> Within params.txt, add the absolute paths to the ChemSTEP-readable score and indices NumPY arrays. The rest of the values within the params.txt file are left to the discretion of the user, with some considerations below. <br /> <br /> '''pProp:''' this value will define what is considered a &quot;virtual hit&quot; in your campaign. pProp is defined as the -log(rank %) of a molecule within the total library score-distribution. For example, a pProp of 4 in the 1.1T XReal space is equivalent the top 0.01% of the library, corresponding to the top 110 million molecules. pProp 5 = 0.001% = 11 million virtual hits, etc. From the seed set, ChemSTEP will estimate a DOCK score value in kcal/mol that associated with the lower-limit of your desired pProp zone. Any molecules that score better than the threshold is considered a virtual hit in your campaign. Be mindful of seed set size when choosing desired pProp. Our suggestion is that the size of the seed set should be at least 10^(pProp +2).<br /> <br /> '''n_docked_per_round:''' this number is the desired molecules for prioritization per round. When choosing this value, note that this number of molecules must be built and docked in between each round of ChemSTEP. Prioritizing many molecules will slow building and docking speeds, and coupled with few beacons (below) may lead to decreased diversity. Prioritizing too few molecules may result in slower virtual hit recovery. Round size does not significantly impact the algorithm running time.** <br /> <br /> '''max_beacons:''' this is the number of diverse, well-scoring molecules ChemSTEP will use to guide prioritization per round. All molecules that score better than the assigned pProp threshold may be considered for beacons. By default, ChemSTEP chooses each set of beacons to be as maximally diverse as possible. Choosing too many beacons may result in decreased diversity between beacons, but too few beacons could hinder space exploration. If not enough molecules score above your pProp threshold, ChemSTEP may assign fewer beacons than the assigned value. <br /> <br /> '''max_n_rounds:''' if prospectively running ChemSTEP, as outlined in this wiki, no need to worry about this parameter.<br /> <br /> '''bundle_size:''' if using auto docking mode, the bundle size is used for building. i.e. the number of molecules that get submitted to build as one job. <br /> <br /> <br /> <br /> '''7. Edit run_chemstep.py''' <br /> <br /> There are several configurable parameters that can be passed when first calling the CSAlgo object:<br /> <br /> '''docking_method=''' setting this to &quot;auto&quot; will set up and run iterative rounds of building, docking, and running ChemSTEP. If you choose the auto method, be sure to update the full path to your dockfiles in run_chemstep.py and check that all INDOCK parameters are set to your liking. NOTE: docking method is set to auto if you copy in the scripts as outlined above. Change if desired!!!!!<br /> <br /> <br /> '''min_td_search=''' setting this parameter will establish a minimum-tanimoto distance &quot;zone&quot; around each beacon for prioritization. For example, setting a min_td_search=0.3 will require that each prioritized molecule be greater than 0.3 Td from each beacon. <br /> <br /> more optional parameters to come.<br /> <br /> At this time, we strongly suggest running ChemSTEP as a job array with 8 or 16 CPU slots requested. The number of cores must be specified when calling CSAlgo (in run_chemstep.py) and the SGE wrapper(launch_chemstep_as_job.sh). <br /> <br /> <br /> '''8. Run ChemSTEP''' with the following command: <br /> qsub launch_chemstep_as_job.sh <br /> <br /> <br /> This will launch the main ChemSTEP job to the SGE scheduler. Check your job status with the 'qstat' command. The algorithm will read the score and indices files provided, calculate pProp (for round 1) and assign beacons. As this job runs, it will launch a subsequent job array. These sub-jobs are the Chaining step of ChemSTEP, where each parallel worker calculates the Tanimoto distances of 100 million molecules from the virtual library to the beacons. These distances are then written to the files within the /output directory. When these jobs complete, the main job will read through all Tanimoto distances and pull molecules for prioritization. <br /> <br /> Running ChemSTEP successfully will result in the output of a SMILES file within output/complete_info/. You will also get (1) chemstep_algo.log and (2) chemstep_submission.log in the working directory after job submission. chemstep_algo.log contains the running output of the algorithm in a readable format, with timestamps, including the DOCK score associated with your desired pProp. chemstep_submission.log contains the output of ChemSTEP, as well as any information output by the SGE submission system (python errors, cluster issues, tracebacks if failed). These files will be updated with any information with iterative rounds of ChemSTEP run in the same directory. '''Visually inspect these files after each round of ChemSTEP to ensure the algorithm has picked your desired number of beacons and things ran smoothly.'''<br /> <br /> '''Troubleshooting:''' if no jobs are running and there is no SMI file, check the chemstep_submission.log first. Any traceback error or SGE error should give you some idea of why ChemSTEP failed. Some errors may be due to SGE or Wynton issues. If directed, look at a few .out files within /output/jobs/ . If a ChemSTEP run fails on your FIRST run, delete the output and log files, fix what needs to be fixed, and rerun. Errors in subsequent rounds during the chaining step can potentially corrupt chaining files within the /output directory that are needed for prioritization. At the very least, you will definitely have duplicates of beacons and information written to the log files. At this time, if your run fails during iterative rounds, it's best to start from the beginning. <br /> <br /> Prioritized molecules should be built, docked, and their scores fed back into ChemSTEP. More detailed instructions below for running in MANUAL mode: <br /> <br /> '''9. Build prioritized molecules (DOCK 3.8).''' /taken from docs.docking.org <br /> source /wynton/group/bks/soft/DOCK-3.8.5/env.sh<br /> <br /> python /wynton/group/bks/soft/DOCK-3.8.5/DOCK3.8/zinc22-3d/submit/submit_building_docker.py --output_folder building_output --bundle_size 1000 --minutes_per_mol 5 --skip_name_check --scheduler sge --container_software apptainer --container_path_or_name /wynton/group/bks/soft/DOCK-3.8.5/building_pipeline.sif smi_round_{}.smi<br /> <br /> When building has completed, you must write an SDI file with the complete paths to each built bundle and dock. you can do this with:<br /> find /path/to/your/building_output -name &quot;bundle.db2.tgz&quot; -type f &gt; round_{}.wynton.sdi<br /> <br /> <br /> '''Be sure to change the INDOCK file to save only poses that meet your score pProp score threshold calculated by ChemSTEP!''' Retrieve docking scores as convert to NumPy arrays as outlined above, updating the round number when running get_scores.py and convert_scores_to_npy.py. Copy new score and indices files into the directory you ran ChemSTEP in. If you are following along as a tutorial, you should have scores_round_1.npy and indices_round_1.npy from the previous step (from FIRST round of ChemSTEP prioritization). <br /> <br /> <br /> '''10. Set up for iterative rounds of ChemSTEP'''<br /> <br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/run_chemstep_iterative.py .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/launch_chemstep_iterative.sh .<br /> <br /> <br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/run_chemstep_iterative.py .<br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/launch_chemstep_iterative.sh .<br /> <br /> '''11. Run ChemSTEP'''<br /> qsub launch_chemstep_as_job.sh [round number]<br /> <br /> For the first iterative round, the round number is [2], and should increase by one for every subsequent round of ChemSTEP. The output will be smi_round_****.smi file. Repeat steps 8-10 for as many rounds as needed. The performance is reported in outputy/complete_info/run_summary.df, which contains the number of beacons selected, the number of molecules docked, the number of hits found, the distance threshold for the selected molecules to dock, and the last added beacon's distance to all previous beacons.<br /> <br /> <br /> <br /> <br /> '''Helper scripts:''' a running list of scripts we have been using for analysis and visualization. <br /> <br /> to get SMILES for beacons: <br /> <br /> python /wynton/group/bks/work/bwhall61/needs_github/get_beacon_smi.py --beacon_df_path /path/to/your/chemstep/output/complete_info/beacons.df --n_workers 6 --outfile beacon_smiles.smi<br /> <br /> <br /> ** anecdotally true.</div>

Kholland http://wiki.docking.org/index.php?title=Running_ChemSTEP&diff=17020 2025-12-16T19:30:53Z

<p>Kholland: </p> <hr /> <div>last update: dec 16 2025 katie. current ver = 0.3.1.5 (automatic resubmission of failed SGE jobs). <br /> <br /> ChemSTEP (Chemical Space Traversal and Exploration Procedure) is an open-source, transparent acceleration algorithm for molecular docking capable of dealing with virtual libraries of several trillion compounds. This wiki page is a guide for BKS lab members to run ChemSTEP on Wynton HPC, using the current version of ZINC library (96B) or 13.3B library from Enamine REAL. '''for detailed instructions on the 13B space, see 'Running ChemSTEP on the 13B space' wiki page'''. For more general use directions, please refer to [ChemSTEP Read-the-Docs]. <br /> <br /> At a high-level, ChemSTEP is an iterative process that identifies molecules from the larger virtual library to prioritize for docking. First, we identify a random sample of the total library (termed &quot;seed set&quot;, round zero) and dock those molecules to the target of interest. From this seed set, we can calculate total-library pProp values (-log rank percentages) and the number of &quot;virtual hits&quot; in the total library (high-scoring molecules). ChemSTEP will identify a set of maximally diverse molecules that score above the desired pProp threshold (&quot;beacons&quot;) from the seed set. These beacons guide prioritization, where molecules chosen and output by ChemSTEP are close in chemical space to the beacons. Prioritized molecules are then built, docked, and returned to ChemSTEP for a second round of prioritization. This process is iterated until you reach desired virtual hit recovery, or you are no longer recovering virtual hits. <br /> <br /> <br /> Running the ChemSTEP algorithm on Wynton takes place in loosely three steps: (1) submission of the main ChemSTEP job, (2) ChainingLog sub-job array, and (3) gathering of SMILES for prioritization. <br /> <br /> '''1. Main ChemSTEP job:''' reads input files (DOCK scores and line-matched indices NumPy arrays) and assigns beacons. Beacons are a set of N maximally diverse molecules that score above the pProp threshold specified in the parameter file. For the first round of ChemSTEP, the best-scoring molecule from the seed set is assigned as the first beacon. Subsequent beacons would be molecules that score well (above pProp thresh), and are as diverse as possible from already assigned beacons. Greedy max-diversity selection. <br /> <br /> During assignment, ECFP4 Tanimoto distances between the assigned beacon and all potential beacons are calculated. The second beacon is the molecule with the maximum Td from beacon 1. Beacon 3 will be the molecule that has the highest Td from beacon 1 and beacon 2. This continues until the number of beacons specified in the parameter file is reached. In iterative rounds of ChemSTEP, the beacon selection is based on diversity from ALL PREVIOUSLY assigned beacons, including those in earlier rounds. <br /> <br /> <br /> '''2. ChainingLog sub-jobs:''' once beacons are assigned, ChemSTEP will launch a series of sub-jobs that calculate the Tanimoto distances of every molecule remaining in the library to the assigned beacons. Calculated distances to the NEAREST beacon (minimum-minimum Td) are updated in the mintddistrib_*.npy files within the /output directory. <br /> <br /> <br /> '''3. Gathering of SMILES for prioritization:''' When all Td calculations are completed, the algo will select N number of molecules for prioritization, the number of which is specified by the user in the parameter file. Molecules that are ''closest in chemical space'' to beacons are prioritized. I.e. if round size = 1 million, the 1 million molecules with the smallest min-Td to ANY beacon (current or previous rounds) are prioritized. chemstep_algo.log will provide a &quot;max-minTd&quot; for each round, which is the Tanimoto distance of the most dissimilar molecule prioritized to any one beacon per round. The prioritized molecules are output in /output/complete_info as smi_round_{}.smi <br /> <br /> What the user need: DOCKFILES, directories for (1) docking (2) building and (3) running ChemSTEP. '''All iterative rounds of ChemSTEP should be run in the SAME directory.''' <br /> <br /> '''1. Source ChemSTEP virtual environment on Wynton'''<br /> source /wynton/group/bks/work/shared/kholland/chemstep_auto/bin/activate<br /> <br /> <br /> '''2. Copy seed-set SDI file into your docking directory'''<br /> This directory should already contain your dockfiles, with INDOCK parameters set to your liking. In this step, we are copying in a split database index file (SDI) containing paths to bundles of db2 files. This seed set contains 92 million molecules sampled randomly from the total virtual library, currently 96 billion molecules (ZINC).<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_auto/scripts/96M_seeds.wynton.sdi .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_auto/scripts/rebuilt_seeds.wynton.sdi .<br /> <br /> because the seed set is large, the docking must be done in two separate chunks (100k jobs max per docking array), and the scores combined into one file later on. <br /> <br /> if you are interested in running on a library of 13.2B compounds from Enamine REAL, there are several seed set sizes available: 130k, 1.3M, 13M, and 26M. <br /> cp /wynton/group/bks/work/shared/kholland/chemstep_13B/130K_seeds.wynton.sdi .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_13B/1.3M_seeds.wynton.sdi .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_13B/13M_seeds.wynton.sdi .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_13B/26M_seeds.wynton.sdi .<br /> <br /> <br /> '''3. Dock seed set to your receptor of interest using DOCK 3.8''' /docking directions taken from docs.docking.org. This is meant to be done as you would do a normal LSD. <br /> export MOLECULES_DIR_TO_BIND=[outermost folder containing the molecules to dock, just your base docking directory]<br /> export DOCKFILES=[path to your dockfiles]<br /> export INPUT_FOLDER=[the folder containing your .sdi file(s)]<br /> export OUTPUT_FOLDER=[where you want the output ]<br /> <br /> /wynton/group/bks/work/bwhall61/needs_github/super_dock3r.sh<br /> <br /> Wait for docking to complete. Next, you must extract all molecule IDs and corresponding DOCK scores from above. To do so, run the following commands in the base docking directory containing your docking files and output folder, while logged into a dev node (do this in a screen!!!):<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_auto/get_scores.py .<br /> python get_scores.py 0<br /> <br /> <br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/get_scores.py .<br /> python get_scores.py 0<br /> <br /> <br /> If docking the seeds from the ZINC library, please combine your score.txt files from both SDI files. Rename this combined file 'scores_round_0.txt.'<br /> <br /> This script expects a directory named &quot;output*&quot; within the CWD. If your output from docking follows different organizational conventions, vim into get_scores.py and change the path. Successful output of the script will be a file named &quot;scores_round_0.txt&quot;. For iterative rounds, pass increasing numbers into the command line. i.e. When docking the first round of prioritized molecules, pass [1] for scores_round_1.txt. <br /> <br /> '''4. Convert scores and molecule IDS into NumPY arrays for ChemSTEP recognition. Requires ChemSTEP venv'''<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_auto/convert_scores_to_npy.py .<br /> python convert_scores_to_npy.py 0<br /> <br /> <br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/convert_scores_to_npy.py .<br /> python convert_scores_to_npy.py 0<br /> <br /> For python speed purposes, ChemSTEP requires that DOCK scores and IDs be give in the form of a NumPY array. In this step, we are taking our readable scores file and converting them for ChemSTEP recognition. This script expects a txt file with Molecule IDS and DOCK scores (no header). Use the same round number you used in step 3. As run above, this will output two files named scores_round_0.npy and indices_round_0.npy that contain line-matched indices (determined from Mol ID) and their respective docking scores. <br /> <br /> '''5. Enter into or make a directory to run ChemSTEP in. Copy in necessary files for initiating ChemSTEP: params.txt, run_chemstep.py and launch_chemstep_as_job.sh. Copy in your score and indices numpy files as well, which should be in your docking directory.'''<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_auto/params.txt .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_auto/run_chemstep.py .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_auto/launch_chemstep_as_job.sh .<br /> <br /> <br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/params.txt .<br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/run_chemstep.py .<br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/launch_chemstep_as_job.sh .<br /> <br /> '''6. Edit params.txt file''' This ONLY needs to be edited for the initial round of ChemSTEP. The parameters outlined here will be carried through all rounds of ChemSTEP. <br /> seed_indices_file: /absoulte/path/to/your/indices_round_0.npy<br /> seed_scores_file: /absolute/path/to/your/scores_round_0.npy<br /> hit_pprop: 5<br /> bundle_size: 1000<br /> n_docked_per_round: 10000000<br /> max_beacons: 150<br /> max_n_rounds: 250<br /> <br /> Within params.txt, add the absolute paths to the ChemSTEP-readable score and indices NumPY arrays. The rest of the values within the params.txt file are left to the discretion of the user, with some considerations below. <br /> <br /> '''pProp:''' this value will define what is considered a &quot;virtual hit&quot; in your campaign. pProp is defined as the -log(rank %) of a molecule within the total library score-distribution. For example, a pProp of 4 in the 1.1T XReal space is equivalent the top 0.01% of the library, corresponding to the top 110 million molecules. pProp 5 = 0.001% = 11 million virtual hits, etc. From the seed set, ChemSTEP will estimate a DOCK score value in kcal/mol that associated with the lower-limit of your desired pProp zone. Any molecules that score better than the threshold is considered a virtual hit in your campaign. Be mindful of seed set size when choosing desired pProp. Our suggestion is that the size of the seed set should be at least 10^(pProp +2).<br /> <br /> '''n_docked_per_round:''' this number is the desired molecules for prioritization per round. When choosing this value, note that this number of molecules must be built and docked in between each round of ChemSTEP. Prioritizing many molecules will slow building and docking speeds, and coupled with few beacons (below) may lead to decreased diversity. Prioritizing too few molecules may result in slower virtual hit recovery. Round size does not significantly impact the algorithm running time.** <br /> <br /> '''max_beacons:''' this is the number of diverse, well-scoring molecules ChemSTEP will use to guide prioritization per round. All molecules that score better than the assigned pProp threshold may be considered for beacons. By default, ChemSTEP chooses each set of beacons to be as maximally diverse as possible. Choosing too many beacons may result in decreased diversity between beacons, but too few beacons could hinder space exploration. If not enough molecules score above your pProp threshold, ChemSTEP may assign fewer beacons than the assigned value. <br /> <br /> '''max_n_rounds:''' if prospectively running ChemSTEP, as outlined in this wiki, no need to worry about this parameter.<br /> <br /> '''bundle_size:''' if using auto docking mode, the bundle size is used for building. i.e. the number of molecules that get submitted to build as one job. <br /> <br /> <br /> <br /> '''7. Edit run_chemstep.py''' <br /> <br /> There are several configurable parameters that can be passed when first calling the CSAlgo object:<br /> <br /> '''docking_method=''' setting this to &quot;auto&quot; will set up and run iterative rounds of building, docking, and running ChemSTEP. If you choose the auto method, be sure to update the full path to your dockfiles in run_chemstep.py and check that all INDOCK parameters are set to your liking. NOTE: docking method is set to auto if you copy in the scripts as outlined above. Change if desired!!!!!<br /> <br /> <br /> '''min_td_search=''' setting this parameter will establish a minimum-tanimoto distance &quot;zone&quot; around each beacon for prioritization. For example, setting a min_td_search=0.3 will require that each prioritized molecule be greater than 0.3 Td from each beacon. <br /> <br /> more optional parameters to come.<br /> <br /> At this time, we strongly suggest running ChemSTEP as a job array with 8 or 16 CPU slots requested. The number of cores must be specified when calling CSAlgo (in run_chemstep.py) and the SGE wrapper(launch_chemstep_as_job.sh). <br /> <br /> <br /> '''8. Run ChemSTEP''' with the following command: <br /> qsub launch_chemstep_as_job.sh <br /> <br /> <br /> This will launch the main ChemSTEP job to the SGE scheduler. Check your job status with the 'qstat' command. The algorithm will read the score and indices files provided, calculate pProp (for round 1) and assign beacons. As this job runs, it will launch a subsequent job array. These sub-jobs are the Chaining step of ChemSTEP, where each parallel worker calculates the Tanimoto distances of 100 million molecules from the virtual library to the beacons. These distances are then written to the files within the /output directory. When these jobs complete, the main job will read through all Tanimoto distances and pull molecules for prioritization. <br /> <br /> Running ChemSTEP successfully will result in the output of a SMILES file within output/complete_info/. You will also get (1) chemstep_algo.log and (2) chemstep_submission.log in the working directory after job submission. chemstep_algo.log contains the running output of the algorithm in a readable format, with timestamps, including the DOCK score associated with your desired pProp. chemstep_submission.log contains the output of ChemSTEP, as well as any information output by the SGE submission system (python errors, cluster issues, tracebacks if failed). These files will be updated with any information with iterative rounds of ChemSTEP run in the same directory. '''Visually inspect these files after each round of ChemSTEP to ensure the algorithm has picked your desired number of beacons and things ran smoothly.'''<br /> <br /> '''Troubleshooting:''' if no jobs are running and there is no SMI file, check the chemstep_submission.log first. Any traceback error or SGE error should give you some idea of why ChemSTEP failed. Some errors may be due to SGE or Wynton issues. If directed, look at a few .out files within /output/jobs/ . If a ChemSTEP run fails on your FIRST run, delete the output and log files, fix what needs to be fixed, and rerun. Errors in subsequent rounds during the chaining step can potentially corrupt chaining files within the /output directory that are needed for prioritization. At the very least, you will definitely have duplicates of beacons and information written to the log files. At this time, if your run fails during iterative rounds, it's best to start from the beginning. <br /> <br /> Prioritized molecules should be built, docked, and their scores fed back into ChemSTEP. More detailed instructions below for running in MANUAL mode: <br /> <br /> '''9. Build prioritized molecules (DOCK 3.8).''' /taken from docs.docking.org <br /> source /wynton/group/bks/soft/DOCK-3.8.5/env.sh<br /> <br /> python /wynton/group/bks/soft/DOCK-3.8.5/DOCK3.8/zinc22-3d/submit/submit_building_docker.py --output_folder building_output --bundle_size 1000 --minutes_per_mol 5 --skip_name_check --scheduler sge --container_software apptainer --container_path_or_name /wynton/group/bks/soft/DOCK-3.8.5/building_pipeline.sif smi_round_{}.smi<br /> <br /> When building has completed, you must write an SDI file with the complete paths to each built bundle and dock. you can do this with:<br /> find /path/to/your/building_output -name &quot;bundle.db2.tgz&quot; -type f &gt; round_{}.wynton.sdi<br /> <br /> <br /> '''Be sure to change the INDOCK file to save only poses that meet your score pProp score threshold calculated by ChemSTEP!''' Retrieve docking scores as convert to NumPy arrays as outlined above, updating the round number when running get_scores.py and convert_scores_to_npy.py. Copy new score and indices files into the directory you ran ChemSTEP in. If you are following along as a tutorial, you should have scores_round_1.npy and indices_round_1.npy from the previous step (from FIRST round of ChemSTEP prioritization). <br /> <br /> <br /> '''10. Set up for iterative rounds of ChemSTEP'''<br /> <br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/run_chemstep_iterative.py .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/launch_chemstep_iterative.sh .<br /> <br /> <br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/run_chemstep_iterative.py .<br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/launch_chemstep_iterative.sh .<br /> <br /> '''10. Run ChemSTEP'''<br /> qsub launch_chemstep_as_job.sh [round number]<br /> <br /> For the first iterative round, the round number is [2], and should increase by one for every subsequent round of ChemSTEP. The output will be smi_round_****.smi file. Repeat steps 8-10 for as many rounds as needed. The performance is reported in outputy/complete_info/run_summary.df, which contains the number of beacons selected, the number of molecules docked, the number of hits found, the distance threshold for the selected molecules to dock, and the last added beacon's distance to all previous beacons.<br /> <br /> <br /> <br /> <br /> '''Helper scripts:''' a running list of scripts we have been using for analysis and visualization. <br /> <br /> to get SMILES for beacons: <br /> <br /> python /wynton/group/bks/work/bwhall61/needs_github/get_beacon_smi.py --beacon_df_path /path/to/your/chemstep/output/complete_info/beacons.df --n_workers 6 --outfile beacon_smiles.smi<br /> <br /> <br /> ** anecdotally true.</div>

Kholland http://wiki.docking.org/index.php?title=Running_ChemSTEP&diff=17019 2025-12-16T19:27:09Z

<p>Kholland: </p> <hr /> <div>last update: dec 16 2025 katie. current ver = 0.3.1.5 (automatic resubmission of failed SGE jobs). <br /> <br /> ChemSTEP (Chemical Space Traversal and Exploration Procedure) is an open-source, transparent acceleration algorithm for molecular docking capable of dealing with virtual libraries of several trillion compounds. This wiki page is a guide for BKS lab members to run ChemSTEP on Wynton HPC, using the current version of ZINC library (96B) or 13.3B library from Enamine REAL. '''for detailed instructions on the 13B space, see 'Running ChemSTEP on the 13B space' wiki page'''. For more general use directions, please refer to [ChemSTEP Read-the-Docs]. <br /> <br /> At a high-level, ChemSTEP is an iterative process that identifies molecules from the larger virtual library to prioritize for docking. First, we identify a random sample of the total library (termed &quot;seed set&quot;, round zero) and dock those molecules to the target of interest. From this seed set, we can calculate total-library pProp values (-log rank percentages) and the number of &quot;virtual hits&quot; in the total library (high-scoring molecules). ChemSTEP will identify a set of maximally diverse molecules that score above the desired pProp threshold (&quot;beacons&quot;) from the seed set. These beacons guide prioritization, where molecules chosen and output by ChemSTEP are close in chemical space to the beacons. Prioritized molecules are then built, docked, and returned to ChemSTEP for a second round of prioritization. This process is iterated until you reach desired virtual hit recovery, or you are no longer recovering virtual hits. <br /> <br /> <br /> Running the ChemSTEP algorithm on Wynton takes place in loosely three steps: (1) submission of the main ChemSTEP job, (2) ChainingLog sub-job array, and (3) gathering of SMILES for prioritization. <br /> <br /> '''1. Main ChemSTEP job:''' reads input files (DOCK scores and line-matched indices NumPy arrays) and assigns beacons. Beacons are a set of N maximally diverse molecules that score above the pProp threshold specified in the parameter file. For the first round of ChemSTEP, the best-scoring molecule from the seed set is assigned as the first beacon. Subsequent beacons would be molecules that score well (above pProp thresh), and are as diverse as possible from already assigned beacons. Greedy max-diversity selection. <br /> <br /> During assignment, ECFP4 Tanimoto distances between the assigned beacon and all potential beacons are calculated. The second beacon is the molecule with the maximum Td from beacon 1. Beacon 3 will be the molecule that has the highest Td from beacon 1 and beacon 2. This continues until the number of beacons specified in the parameter file is reached. In iterative rounds of ChemSTEP, the beacon selection is based on diversity from ALL PREVIOUSLY assigned beacons, including those in earlier rounds. <br /> <br /> <br /> '''2. ChainingLog sub-jobs:''' once beacons are assigned, ChemSTEP will launch a series of sub-jobs that calculate the Tanimoto distances of every molecule remaining in the library to the assigned beacons. Calculated distances to the NEAREST beacon (minimum-minimum Td) are updated in the mintddistrib_*.npy files within the /output directory. <br /> <br /> <br /> '''3. Gathering of SMILES for prioritization:''' When all Td calculations are completed, the algo will select N number of molecules for prioritization, the number of which is specified by the user in the parameter file. Molecules that are ''closest in chemical space'' to beacons are prioritized. I.e. if round size = 1 million, the 1 million molecules with the smallest min-Td to ANY beacon (current or previous rounds) are prioritized. chemstep_algo.log will provide a &quot;max-minTd&quot; for each round, which is the Tanimoto distance of the most dissimilar molecule prioritized to any one beacon per round. The prioritized molecules are output in /output/complete_info as smi_round_{}.smi <br /> <br /> What the user need: DOCKFILES, directories for (1) docking (2) building and (3) running ChemSTEP. '''All iterative rounds of ChemSTEP should be run in the SAME directory.''' <br /> <br /> '''1. Source ChemSTEP virtual environment on Wynton'''<br /> source /wynton/group/bks/work/shared/kholland/chemstep_auto/bin/activate<br /> <br /> <br /> '''2. Copy seed-set SDI file into your docking directory'''<br /> This directory should already contain your dockfiles, with INDOCK parameters set to your liking. In this step, we are copying in a split database index file (SDI) containing paths to bundles of db2 files. This seed set contains 100 million molecules sampled randomly from the total virtual library, currently 1.1 trillion molecules.<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_auto/scripts/96M_seeds.wynton.sdi .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_auto/scripts/rebuilt_seeds.wynton.sdi .<br /> <br /> because the seed set is large, the docking must be done in two separate chunks (100k jobs max per docking array), and the scores combined into one file later on. <br /> <br /> if you are interested in running on a library of 13.2B compounds from Enamine REAL, there are several seed set sizes available: 130k, 1.3M, 13M, and 26M. <br /> cp /wynton/group/bks/work/shared/kholland/chemstep_13B/130K_seeds.wynton.sdi .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_13B/1.3M_seeds.wynton.sdi .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_13B/13M_seeds.wynton.sdi .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_13B/26M_seeds.wynton.sdi .<br /> <br /> <br /> '''3. Dock seed set to your receptor of interest using DOCK 3.8''' /docking directions taken from docs.docking.org. This is meant to be done as you would do a normal LSD. <br /> export MOLECULES_DIR_TO_BIND=[outermost folder containing the molecules to dock, just your base docking directory]<br /> export DOCKFILES=[path to your dockfiles]<br /> export INPUT_FOLDER=[the folder containing your .sdi file(s)]<br /> export OUTPUT_FOLDER=[where you want the output ]<br /> <br /> /wynton/group/bks/work/bwhall61/needs_github/super_dock3r.sh<br /> <br /> Wait for docking to complete. Next, you must extract all molecule IDs and corresponding DOCK scores from above. To do so, run the following commands in the base docking directory containing your docking files and output folder, while logged into a dev node (do this in a screen!!!):<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_auto/get_scores.py .<br /> python get_scores.py 0<br /> <br /> <br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/get_scores.py .<br /> python get_scores.py 0<br /> <br /> <br /> If docking the seeds from the ZINC library, please combine your score.txt files from both SDI files. Rename this combined file 'scores_round_0.txt.'<br /> <br /> This script expects a directory named &quot;output*&quot; within the CWD. If your output from docking follows different organizational conventions, vim into get_scores.py and change the path. Successful output of the script will be a file named &quot;scores_round_0.txt&quot;. For iterative rounds, pass increasing numbers into the command line. i.e. When docking the first round of prioritized molecules, pass [1] for scores_round_1.txt. <br /> <br /> '''4. Convert scores and molecule IDS into NumPY arrays for ChemSTEP recognition. Requires ChemSTEP venv'''<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_auto/convert_scores_to_npy.py .<br /> python convert_scores_to_npy.py 0<br /> <br /> <br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/convert_scores_to_npy.py .<br /> python convert_scores_to_npy.py 0<br /> <br /> For python speed purposes, ChemSTEP requires that DOCK scores and IDs be give in the form of a NumPY array. In this step, we are taking our readable scores file and converting them for ChemSTEP recognition. This script expects a txt file with Molecule IDS and DOCK scores (no header). Use the same round number you used in step 3. As run above, this will output two files named scores_round_0.npy and indices_round_0.npy that contain line-matched indices (determined from Mol ID) and their respective docking scores. <br /> <br /> '''5. Enter into or make a directory to run ChemSTEP in. Copy in necessary files for initiating ChemSTEP: params.txt, run_chemstep.py and launch_chemstep_as_job.sh. Copy in your score and indices numpy files as well, which should be in your docking directory.'''<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_auto/params.txt .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_auto/run_chemstep.py .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_auto/launch_chemstep_as_job.sh .<br /> <br /> <br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/params.txt .<br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/run_chemstep.py .<br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/launch_chemstep_as_job.sh .<br /> <br /> '''6. Edit params.txt file''' This ONLY needs to be edited for the initial round of ChemSTEP. The parameters outlined here will be carried through all rounds of ChemSTEP. <br /> seed_indices_file: /absoulte/path/to/your/indices_round_0.npy<br /> seed_scores_file: /absolute/path/to/your/scores_round_0.npy<br /> hit_pprop: 5<br /> bundle_size: 1000<br /> n_docked_per_round: 10000000<br /> max_beacons: 150<br /> max_n_rounds: 250<br /> <br /> Within params.txt, add the absolute paths to the ChemSTEP-readable score and indices NumPY arrays. The rest of the values within the params.txt file are left to the discretion of the user, with some considerations below. <br /> <br /> '''pProp:''' this value will define what is considered a &quot;virtual hit&quot; in your campaign. pProp is defined as the -log(rank %) of a molecule within the total library score-distribution. For example, a pProp of 4 in the 1.1T XReal space is equivalent the top 0.01% of the library, corresponding to the top 110 million molecules. pProp 5 = 0.001% = 11 million virtual hits, etc. From the seed set, ChemSTEP will estimate a DOCK score value in kcal/mol that associated with the lower-limit of your desired pProp zone. Any molecules that score better than the threshold is considered a virtual hit in your campaign. Be mindful of seed set size when choosing desired pProp. Our suggestion is that the size of the seed set should be at least 10^(pProp +2).<br /> <br /> '''n_docked_per_round:''' this number is the desired molecules for prioritization per round. When choosing this value, note that this number of molecules must be built and docked in between each round of ChemSTEP. Prioritizing many molecules will slow building and docking speeds, and coupled with few beacons (below) may lead to decreased diversity. Prioritizing too few molecules may result in slower virtual hit recovery. Round size does not significantly impact the algorithm running time.** <br /> <br /> '''max_beacons:''' this is the number of diverse, well-scoring molecules ChemSTEP will use to guide prioritization per round. All molecules that score better than the assigned pProp threshold may be considered for beacons. By default, ChemSTEP chooses each set of beacons to be as maximally diverse as possible. Choosing too many beacons may result in decreased diversity between beacons, but too few beacons could hinder space exploration. If not enough molecules score above your pProp threshold, ChemSTEP may assign fewer beacons than the assigned value. <br /> <br /> '''max_n_rounds:''' if prospectively running ChemSTEP, as outlined in this wiki, no need to worry about this parameter.<br /> <br /> '''bundle_size:''' if using auto docking mode, the bundle size is used for building. i.e. the number of molecules that get submitted to build as one job. <br /> <br /> <br /> <br /> '''7. Edit run_chemstep.py''' <br /> <br /> There are several configurable parameters that can be passed when first calling the CSAlgo object:<br /> <br /> '''docking_method=''' setting this to &quot;auto&quot; will set up and run iterative rounds of building, docking, and running ChemSTEP. If you choose the auto method, be sure to update the full path to your dockfiles in run_chemstep.py and check that all INDOCK parameters are set to your liking. NOTE: docking method is set to auto if you copy in the scripts as outlined above. Change if desired!!!!!<br /> <br /> <br /> '''min_td_search=''' setting this parameter will establish a minimum-tanimoto distance &quot;zone&quot; around each beacon for prioritization. For example, setting a min_td_search=0.3 will require that each prioritized molecule be greater than 0.3 Td from each beacon. <br /> <br /> more optional parameters to come.<br /> <br /> At this time, we strongly suggest running ChemSTEP as a job array with 8 or 16 CPU slots requested. The number of cores must be specified when calling CSAlgo (in run_chemstep.py) and the SGE wrapper(launch_chemstep_as_job.sh). <br /> <br /> <br /> '''8. Run ChemSTEP''' with the following command: <br /> qsub launch_chemstep_as_job.sh <br /> <br /> <br /> This will launch the main ChemSTEP job to the SGE scheduler. Check your job status with the 'qstat' command. The algorithm will read the score and indices files provided, calculate pProp (for round 1) and assign beacons. As this job runs, it will launch a subsequent job array. These sub-jobs are the Chaining step of ChemSTEP, where each parallel worker calculates the Tanimoto distances of 100 million molecules from the virtual library to the beacons. These distances are then written to the files within the /output directory. When these jobs complete, the main job will read through all Tanimoto distances and pull molecules for prioritization. <br /> <br /> Running ChemSTEP successfully will result in the output of a SMILES file within output/complete_info/. You will also get (1) chemstep_algo.log and (2) chemstep_submission.log in the working directory after job submission. chemstep_algo.log contains the running output of the algorithm in a readable format, with timestamps, including the DOCK score associated with your desired pProp. chemstep_submission.log contains the output of ChemSTEP, as well as any information output by the SGE submission system (python errors, cluster issues, tracebacks if failed). These files will be updated with any information with iterative rounds of ChemSTEP run in the same directory. '''Visually inspect these files after each round of ChemSTEP to ensure the algorithm has picked your desired number of beacons and things ran smoothly.'''<br /> <br /> '''Troubleshooting:''' if no jobs are running and there is no SMI file, check the chemstep_submission.log first. Any traceback error or SGE error should give you some idea of why ChemSTEP failed. Some errors may be due to SGE or Wynton issues. If directed, look at a few .out files within /output/jobs/ . If a ChemSTEP run fails on your FIRST run, delete the output and log files, fix what needs to be fixed, and rerun. Errors in subsequent rounds during the chaining step can potentially corrupt chaining files within the /output directory that are needed for prioritization. At the very least, you will definitely have duplicates of beacons and information written to the log files. At this time, if your run fails during iterative rounds, it's best to start from the beginning. <br /> <br /> Prioritized molecules should be built, docked, and their scores fed back into ChemSTEP. More detailed instructions below for running in MANUAL mode: <br /> <br /> '''9. Build prioritized molecules (DOCK 3.8).''' /taken from docs.docking.org <br /> source /wynton/group/bks/soft/DOCK-3.8.5/env.sh<br /> <br /> python /wynton/group/bks/soft/DOCK-3.8.5/DOCK3.8/zinc22-3d/submit/submit_building_docker.py --output_folder building_output --bundle_size 1000 --minutes_per_mol 5 --skip_name_check --scheduler sge --container_software apptainer --container_path_or_name /wynton/group/bks/soft/DOCK-3.8.5/building_pipeline.sif smi_round_{}.smi<br /> <br /> When building has completed, you must write an SDI file with the complete paths to each built bundle and dock. you can do this with:<br /> find /path/to/your/building_output -name &quot;bundle.db2.tgz&quot; -type f &gt; round_{}.wynton.sdi<br /> <br /> <br /> '''Be sure to change the INDOCK file to save only poses that meet your score pProp score threshold calculated by ChemSTEP!''' Retrieve docking scores as convert to NumPy arrays as outlined above, updating the round number when running get_scores.py and convert_scores_to_npy.py. Copy new score and indices files into the directory you ran ChemSTEP in. If you are following along as a tutorial, you should have scores_round_1.npy and indices_round_1.npy from the previous step (from FIRST round of ChemSTEP prioritization). <br /> <br /> <br /> '''10. Set up for iterative rounds of ChemSTEP'''<br /> <br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/run_chemstep_iterative.py .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/launch_chemstep_iterative.sh .<br /> <br /> <br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/run_chemstep_iterative.py .<br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/launch_chemstep_iterative.sh .<br /> <br /> '''10. Run ChemSTEP'''<br /> qsub launch_chemstep_as_job.sh [round number]<br /> <br /> For the first iterative round, the round number is [2], and should increase by one for every subsequent round of ChemSTEP. The output will be smi_round_****.smi file. Repeat steps 8-10 for as many rounds as needed. The performance is reported in outputy/complete_info/run_summary.df, which contains the number of beacons selected, the number of molecules docked, the number of hits found, the distance threshold for the selected molecules to dock, and the last added beacon's distance to all previous beacons.<br /> <br /> <br /> <br /> <br /> '''Helper scripts:''' a running list of scripts we have been using for analysis and visualization. <br /> <br /> to get SMILES for beacons: <br /> <br /> python /wynton/group/bks/work/bwhall61/needs_github/get_beacon_smi.py --beacon_df_path /path/to/your/chemstep/output/complete_info/beacons.df --n_workers 6 --outfile beacon_smiles.smi<br /> <br /> <br /> ** anecdotally true.</div>

Kholland http://wiki.docking.org/index.php?title=Running_ChemSTEP&diff=17018 2025-12-16T19:26:00Z

<p>Kholland: </p> <hr /> <div>last update: dec 16 2025 katie. current ver = 0.3.1.5 (automatic resubmission of failed SGE jobs). <br /> <br /> ChemSTEP (Chemical Space Traversal and Exploration Procedure) is an open-source, transparent acceleration algorithm for molecular docking capable of dealing with virtual libraries of several trillion compounds. This wiki page is a guide for BKS lab members to run ChemSTEP on Wynton HPC, using the current version of ZINC library (96B) or 13.3B library from Enamine REAL. '''for detailed instructions on the 13B space, see 'Running ChemSTEP on the 13B space' wiki page'''. For more general use directions, please refer to [ChemSTEP Read-the-Docs]. <br /> <br /> At a high-level, ChemSTEP is an iterative process that identifies molecules from the larger virtual library to prioritize for docking. First, we identify a random sample of the total library (termed &quot;seed set&quot;, round zero) and dock those molecules to the target of interest. From this seed set, we can calculate total-library pProp values (-log rank percentages) and the number of &quot;virtual hits&quot; in the total library (high-scoring molecules). ChemSTEP will identify a set of maximally diverse molecules that score above the desired pProp threshold (&quot;beacons&quot;) from the seed set. These beacons guide prioritization, where molecules chosen and output by ChemSTEP are close in chemical space to the beacons. Prioritized molecules are then built, docked, and returned to ChemSTEP for a second round of prioritization. This process is iterated until you reach desired virtual hit recovery, or you are no longer recovering virtual hits. <br /> <br /> <br /> Running the ChemSTEP algorithm on Wynton takes place in loosely three steps: (1) submission of the main ChemSTEP job, (2) ChainingLog sub-job array, and (3) gathering of SMILES for prioritization. <br /> <br /> '''1. Main ChemSTEP job:''' reads input files (DOCK scores and line-matched indices NumPy arrays) and assigns beacons. Beacons are a set of N maximally diverse molecules that score above the pProp threshold specified in the parameter file. For the first round of ChemSTEP, the best-scoring molecule from the seed set is assigned as the first beacon. Subsequent beacons would be molecules that score well (above pProp thresh), and are as diverse as possible from already assigned beacons. Greedy max-diversity selection. <br /> <br /> During assignment, ECFP4 Tanimoto distances between the assigned beacon and all potential beacons are calculated. The second beacon is the molecule with the maximum Td from beacon 1. Beacon 3 will be the molecule that has the highest Td from beacon 1 and beacon 2. This continues until the number of beacons specified in the parameter file is reached. In iterative rounds of ChemSTEP, the beacon selection is based on diversity from ALL PREVIOUSLY assigned beacons, including those in earlier rounds. <br /> <br /> <br /> '''2. ChainingLog sub-jobs:''' once beacons are assigned, ChemSTEP will launch a series of sub-jobs that calculate the Tanimoto distances of every molecule remaining in the library to the assigned beacons. Calculated distances to the NEAREST beacon (minimum-minimum Td) are updated in the mintddistrib_*.npy files within the /output directory. <br /> <br /> <br /> '''3. Gathering of SMILES for prioritization:''' When all Td calculations are completed, the algo will select N number of molecules for prioritization, the number of which is specified by the user in the parameter file. Molecules that are ''closest in chemical space'' to beacons are prioritized. I.e. if round size = 1 million, the 1 million molecules with the smallest min-Td to ANY beacon (current or previous rounds) are prioritized. chemstep_algo.log will provide a &quot;max-minTd&quot; for each round, which is the Tanimoto distance of the most dissimilar molecule prioritized to any one beacon per round. The prioritized molecules are output in /output/complete_info as smi_round_{}.smi <br /> <br /> What the user need: DOCKFILES, directories for (1) docking (2) building and (3) running ChemSTEP. '''All iterative rounds of ChemSTEP should be run in the SAME directory.''' <br /> <br /> '''1. Source ChemSTEP virtual environment on Wynton'''<br /> source /wynton/group/bks/work/shared/kholland/chemstep_auto/bin/activate<br /> <br /> <br /> '''2. Copy seed-set SDI file into your docking directory'''<br /> This directory should already contain your dockfiles, with INDOCK parameters set to your liking. In this step, we are copying in a split database index file (SDI) containing paths to bundles of db2 files. This seed set contains 100 million molecules sampled randomly from the total virtual library, currently 1.1 trillion molecules.<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_auto/scripts/96M_seeds.wynton.sdi .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_auto/scripts/rebuilt_seeds.wynton.sdi .<br /> <br /> because the seed set is large, the docking must be done in two separate chunks (100k jobs max per docking array), and the scores combined into one file later on. <br /> <br /> if you are interested in running on a library of 13.2B compounds from Enamine REAL, there are several seed set sizes available: 130k, 1.3M, 13M, and 26M. <br /> cp /wynton/group/bks/work/shared/kholland/chemstep_13B/130K_seeds.wynton.sdi .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_13B/1.3M_seeds.wynton.sdi .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_13B/13M_seeds.wynton.sdi .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_13B/26M_seeds.wynton.sdi .<br /> <br /> <br /> '''3. Dock seed set to your receptor of interest using DOCK 3.8''' /docking directions taken from docs.docking.org. This is meant to be done as you would do a normal LSD. <br /> export MOLECULES_DIR_TO_BIND=[outermost folder containing the molecules to dock, just your base docking directory]<br /> export DOCKFILES=[path to your dockfiles]<br /> export INPUT_FOLDER=[the folder containing your .sdi file(s)]<br /> export OUTPUT_FOLDER=[where you want the output ]<br /> <br /> /wynton/group/bks/work/bwhall61/needs_github/super_dock3r.sh<br /> <br /> Wait for docking to complete. Next, you must extract all molecule IDs and corresponding DOCK scores from above. To do so, run the following commands in the base docking directory containing your docking files and output folder, while logged into a dev node (do this in a screen!!!):<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_auto/get_scores.py .<br /> python get_scores.py 0<br /> <br /> <br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/get_scores.py .<br /> python get_scores.py 0<br /> <br /> <br /> If docking the seeds from the ZINC library, please combine your score.txt files from both SDI files. Rename this combined file 'scores_round_0.txt.'<br /> <br /> This script expects a directory named &quot;output*&quot; within the CWD. If your output from docking follows different organizational conventions, vim into get_scores.py and change the path. Successful output of the script will be a file named &quot;scores_round_0.txt&quot;. For iterative rounds, pass increasing numbers into the command line. i.e. When docking the first round of prioritized molecules, pass [1] for scores_round_1.txt. <br /> <br /> '''4. Convert scores and molecule IDS into NumPY arrays for ChemSTEP recognition. Requires ChemSTEP venv'''<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_auto/convert_scores_to_npy.py .<br /> python convert_scores_to_npy.py 0<br /> <br /> <br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/convert_scores_to_npy.py .<br /> python convert_scores_to_npy.py 0<br /> <br /> For python speed purposes, ChemSTEP requires that DOCK scores and IDs be give in the form of a NumPY array. In this step, we are taking our readable scores file and converting them for ChemSTEP recognition. This script expects a txt file with Molecule IDS and DOCK scores (no header). Use the same round number you used in step 3. As run above, this will output two files named scores_round_0.npy and indices_round_0.npy that contain line-matched indices (determined from Mol ID) and their respective docking scores. <br /> <br /> '''5. Enter into or make a directory to run ChemSTEP in. Copy in necessary files for initiating ChemSTEP: params.txt, run_chemstep.py and launch_chemstep_as_job.sh. Copy in your score and indices numpy files as well, which should be in your docking directory.'''<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_auto/params.txt .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_auto/run_chemstep.py .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_auto/launch_chemstep_as_job.sh .<br /> <br /> <br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/params.txt .<br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/run_chemstep.py .<br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/launch_chemstep_as_job.sh .<br /> <br /> '''6. Edit params.txt file''' This ONLY needs to be edited for the initial round of ChemSTEP. The parameters outlined here will be carried through all rounds of ChemSTEP. <br /> seed_indices_file: /absoulte/path/to/your/indices_round_0.npy<br /> seed_scores_file: /absolute/path/to/your/scores_round_0.npy<br /> hit_pprop: 5<br /> bundle_size: 1000<br /> n_docked_per_round: 10000000<br /> max_beacons: 150<br /> max_n_rounds: 250<br /> <br /> Within params.txt, add the absolute paths to the ChemSTEP-readable score and indices NumPY arrays. The rest of the values within the params.txt file are left to the discretion of the user, with some considerations below. <br /> <br /> '''pProp:''' this value will define what is considered a &quot;virtual hit&quot; in your campaign. pProp is defined as the -log(rank %) of a molecule within the total library score-distribution. For example, a pProp of 4 in the 1.1T XReal space is equivalent the top 0.01% of the library, corresponding to the top 110 million molecules. pProp 5 = 0.001% = 11 million virtual hits, etc. From the seed set, ChemSTEP will estimate a DOCK score value in kcal/mol that associated with the lower-limit of your desired pProp zone. Any molecules that score better than the threshold is considered a virtual hit in your campaign. Be mindful of seed set size when choosing desired pProp. Our suggestion is that the size of the seed set should be at least 10^(pProp +2).<br /> <br /> '''n_docked_per_round:''' this number is the desired molecules for prioritization per round. When choosing this value, note that this number of molecules must be built and docked in between each round of ChemSTEP. Prioritizing many molecules will slow building and docking speeds, and coupled with few beacons (below) may lead to decreased diversity. Prioritizing too few molecules may result in slower virtual hit recovery. Round size does not significantly impact the algorithm running time.** <br /> <br /> '''max_beacons:''' this is the number of diverse, well-scoring molecules ChemSTEP will use to guide prioritization per round. All molecules that score better than the assigned pProp threshold may be considered for beacons. By default, ChemSTEP chooses each set of beacons to be as maximally diverse as possible. Choosing too many beacons may result in decreased diversity between beacons, but too few beacons could hinder space exploration. If not enough molecules score above your pProp threshold, ChemSTEP may assign fewer beacons than the assigned value. <br /> <br /> '''max_n_rounds:''' if prospectively running ChemSTEP, as outlined in this wiki, no need to worry about this parameter.<br /> <br /> '''bundle_size:''' if using auto docking mode, the bundle size is used for building. i.e. the number of molecules that get submitted to build as one job. <br /> <br /> <br /> <br /> '''7. Edit run_chemstep.py''' note that the default parameters when copying over run_chemstep.py invoke the automatic building and docking pipeline. <br /> <br /> There are several configurable parameters that can be passed when first calling the CSAlgo object:<br /> <br /> '''docking_method=''' setting this to &quot;auto&quot; will set up and run iterative rounds of building, docking, and running ChemSTEP. If you choose the auto method, be sure to update the full path to your dockfiles in run_chemstep.py and check that all INDOCK parameters are set to your liking. NOTE: docking method is set to auto if you copy in the scripts as outlined above. Change if desired!!!!!<br /> <br /> <br /> '''min_td_search=''' setting this parameter will establish a minimum-tanimoto distance &quot;zone&quot; around each beacon for prioritization. For example, setting a min_td_search=0.3 will require that each prioritized molecule be greater than 0.3 Td from each beacon. <br /> <br /> more optional parameters to come.<br /> <br /> At this time, we strongly suggest running ChemSTEP as a job array with 16 or 32 CPU slots requested. The number of cores must be specified when calling CSAlgo (in run_chemstep.py) and the SGE wrapper(launch_chemstep_as_job.sh). <br /> <br /> <br /> '''8. Run ChemSTEP''' with the following command: <br /> qsub launch_chemstep_as_job.sh <br /> <br /> <br /> This will launch the main ChemSTEP job to the SGE scheduler. Check your job status with the 'qstat' command. The algorithm will read the score and indices files provided, calculate pProp (for round 1) and assign beacons. As this job runs, it will launch a subsequent job array. These sub-jobs are the Chaining step of ChemSTEP, where each parallel worker calculates the Tanimoto distances of 100 million molecules from the virtual library to the beacons. These distances are then written to the files within the /output directory. When these jobs complete, the main job will read through all Tanimoto distances and pull molecules for prioritization. <br /> <br /> Running ChemSTEP successfully will result in the output of a SMILES file within output/complete_info/. You will also get (1) chemstep_algo.log and (2) chemstep_submission.log in the working directory after job submission. chemstep_algo.log contains the running output of the algorithm in a readable format, with timestamps, including the DOCK score associated with your desired pProp. chemstep_submission.log contains the output of ChemSTEP, as well as any information output by the SGE submission system (python errors, cluster issues, tracebacks if failed). These files will be updated with any information with iterative rounds of ChemSTEP run in the same directory. '''Visually inspect these files after each round of ChemSTEP to ensure the algorithm has picked your desired number of beacons and things ran smoothly.'''<br /> <br /> '''Troubleshooting:''' if no jobs are running and there is no SMI file, check the chemstep_submission.log first. Any traceback error or SGE error should give you some idea of why ChemSTEP failed. Some errors may be due to SGE or Wynton issues. If directed, look at a few .out files within /output/jobs/ . If a ChemSTEP run fails on your FIRST run, delete the output and log files, fix what needs to be fixed, and rerun. Errors in subsequent rounds during the chaining step can potentially corrupt chaining files within the /output directory that are needed for prioritization. At the very least, you will definitely have duplicates of beacons and information written to the log files. At this time, if your run fails during iterative rounds, it's best to start from the beginning. <br /> <br /> Prioritized molecules should be built, docked, and their scores fed back into ChemSTEP. More detailed instructions below for running in MANUAL mode: <br /> <br /> '''9. Build prioritized molecules (DOCK 3.8).''' /taken from docs.docking.org <br /> source /wynton/group/bks/soft/DOCK-3.8.5/env.sh<br /> <br /> python /wynton/group/bks/soft/DOCK-3.8.5/DOCK3.8/zinc22-3d/submit/submit_building_docker.py --output_folder building_output --bundle_size 1000 --minutes_per_mol 5 --skip_name_check --scheduler sge --container_software apptainer --container_path_or_name /wynton/group/bks/soft/DOCK-3.8.5/building_pipeline.sif smi_round_{}.smi<br /> <br /> When building has completed, you must write an SDI file with the complete paths to each built bundle and dock. you can do this with:<br /> find /path/to/your/building_output -name &quot;bundle.db2.tgz&quot; -type f &gt; round_{}.wynton.sdi<br /> <br /> <br /> '''Be sure to change the INDOCK file to save only poses that meet your score pProp score threshold calculated by ChemSTEP!''' Retrieve docking scores as convert to NumPy arrays as outlined above, updating the round number when running get_scores.py and convert_scores_to_npy.py. Copy new score and indices files into the directory you ran ChemSTEP in. If you are following along as a tutorial, you should have scores_round_1.npy and indices_round_1.npy from the previous step (from FIRST round of ChemSTEP prioritization). <br /> <br /> <br /> '''10. Set up for iterative rounds of ChemSTEP'''<br /> <br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/run_chemstep_iterative.py .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/launch_chemstep_iterative.sh .<br /> <br /> <br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/run_chemstep_iterative.py .<br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/launch_chemstep_iterative.sh .<br /> <br /> '''10. Run ChemSTEP'''<br /> qsub launch_chemstep_as_job.sh [round number]<br /> <br /> For the first iterative round, the round number is [2], and should increase by one for every subsequent round of ChemSTEP. The output will be smi_round_****.smi file. Repeat steps 8-10 for as many rounds as needed. The performance is reported in outputy/complete_info/run_summary.df, which contains the number of beacons selected, the number of molecules docked, the number of hits found, the distance threshold for the selected molecules to dock, and the last added beacon's distance to all previous beacons.<br /> <br /> <br /> <br /> <br /> '''Helper scripts:''' a running list of scripts we have been using for analysis and visualization. <br /> <br /> to get SMILES for beacons: <br /> <br /> python /wynton/group/bks/work/bwhall61/needs_github/get_beacon_smi.py --beacon_df_path /path/to/your/chemstep/output/complete_info/beacons.df --n_workers 6 --outfile beacon_smiles.smi<br /> <br /> <br /> ** anecdotally true.</div>

Kholland http://wiki.docking.org/index.php?title=Running_ChemSTEP&diff=17016 2025-12-15T22:06:41Z

<p>Kholland: </p> <hr /> <div>last update: dec 15 2025 katie. current ver = 0.3.1.5 (automatic resubmission of failed SGE jobs). <br /> <br /> ChemSTEP (Chemical Space Traversal and Exploration Procedure) is an open-source, transparent acceleration algorithm for molecular docking capable of dealing with virtual libraries of several trillion compounds. This wiki page is a guide for BKS lab members to run ChemSTEP on Wynton HPC, using the current version of ZINC library (96B) or 13.3B library from Enamine REAL. '''for detailed instructions on the 13B space, see 'Running ChemSTEP on the 13B space' wiki page'''. For more general use directions, please refer to [ChemSTEP Read-the-Docs]. <br /> <br /> At a high-level, ChemSTEP is an iterative process that identifies molecules from the larger virtual library to prioritize for docking. First, we identify a random sample of the total library (termed &quot;seed set&quot;, round zero) and dock those molecules to the target of interest. From this seed set, we can calculate total-library pProp values (-log rank percentages) and the number of &quot;virtual hits&quot; in the total library (high-scoring molecules). ChemSTEP will identify a set of maximally diverse molecules that score above the desired pProp threshold (&quot;beacons&quot;) from the seed set. These beacons guide prioritization, where molecules chosen and output by ChemSTEP are close in chemical space to the beacons. Prioritized molecules are then built, docked, and returned to ChemSTEP for a second round of prioritization. This process is iterated until you reach desired virtual hit recovery, or you are no longer recovering virtual hits. <br /> <br /> <br /> Running the ChemSTEP algorithm on Wynton takes place in loosely three steps: (1) submission of the main ChemSTEP job, (2) ChainingLog sub-job array, and (3) gathering of SMILES for prioritization. <br /> <br /> '''1. Main ChemSTEP job:''' reads input files (DOCK scores and line-matched indices NumPy arrays) and assigns beacons. Beacons are a set of N maximally diverse molecules that score above the pProp threshold specified in the parameter file. For the first round of ChemSTEP, the best-scoring molecule from the seed set is assigned as the first beacon. Subsequent beacons would be molecules that score well (above pProp thresh), and are as diverse as possible from already assigned beacons. Greedy max-diversity selection. <br /> <br /> During assignment, ECFP4 Tanimoto distances between the assigned beacon and all potential beacons are calculated. The second beacon is the molecule with the maximum Td from beacon 1. Beacon 3 will be the molecule that has the highest Td from beacon 1 and beacon 2. This continues until the number of beacons specified in the parameter file is reached. In iterative rounds of ChemSTEP, the beacon selection is based on diversity from ALL PREVIOUSLY assigned beacons, including those in earlier rounds. <br /> <br /> <br /> '''2. ChainingLog sub-jobs:''' once beacons are assigned, ChemSTEP will launch a series of sub-jobs that calculate the Tanimoto distances of every molecule remaining in the library to the assigned beacons. Calculated distances to the NEAREST beacon (minimum-minimum Td) are updated in the mintddistrib_*.npy files within the /output directory. <br /> <br /> <br /> '''3. Gathering of SMILES for prioritization:''' When all Td calculations are completed, the algo will select N number of molecules for prioritization, the number of which is specified by the user in the parameter file. Molecules that are ''closest in chemical space'' to beacons are prioritized. I.e. if round size = 1 million, the 1 million molecules with the smallest min-Td to ANY beacon (current or previous rounds) are prioritized. chemstep_algo.log will provide a &quot;max-minTd&quot; for each round, which is the Tanimoto distance of the most dissimilar molecule prioritized to any one beacon per round. The prioritized molecules are output in /output/complete_info as smi_round_{}.smi <br /> <br /> What the user need: DOCKFILES, directories for (1) docking (2) building and (3) running ChemSTEP. '''All iterative rounds of ChemSTEP should be run in the SAME directory.''' <br /> <br /> '''1. Source ChemSTEP virtual environment on Wynton'''<br /> source /wynton/group/bks/work/shared/kholland/chemstep_auto/bin/activate<br /> <br /> <br /> '''2. Copy seed-set SDI file into your docking directory'''<br /> This directory should already contain your dockfiles, with INDOCK parameters set to your liking. In this step, we are copying in a split database index file (SDI) containing paths to bundles of db2 files. This seed set contains 100 million molecules sampled randomly from the total virtual library, currently 1.1 trillion molecules.<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_auto/scripts/96M_seeds.wynton.sdi .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_auto/scripts/rebuilt_seeds.wynton.sdi .<br /> <br /> because the seed set is large, the docking must be done in two separate chunks (100k jobs max per docking array), and the scores combined into one file later on. <br /> <br /> if you are interested in running on a library of 13.2B compounds from Enamine REAL, there are several seed set sizes available: 130k, 1.3M, 13M, and 26M. <br /> cp /wynton/group/bks/work/shared/kholland/chemstep_13B/130K_seeds.wynton.sdi .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_13B/1.3M_seeds.wynton.sdi .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_13B/13M_seeds.wynton.sdi .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_13B/26M_seeds.wynton.sdi .<br /> <br /> <br /> '''3. Dock seed set to your receptor of interest using DOCK 3.8''' /docking directions taken from docs.docking.org. This is meant to be done as you would do a normal LSD. <br /> export MOLECULES_DIR_TO_BIND=[outermost folder containing the molecules to dock, just your base docking directory]<br /> export DOCKFILES=[path to your dockfiles]<br /> export INPUT_FOLDER=[the folder containing your .sdi file(s)]<br /> export OUTPUT_FOLDER=[where you want the output ]<br /> <br /> /wynton/group/bks/work/bwhall61/needs_github/super_dock3r.sh<br /> <br /> Wait for docking to complete. Next, you must extract all molecule IDs and corresponding DOCK scores from above. To do so, run the following commands in the base docking directory containing your docking files and output folder, while logged into a dev node (do this in a screen!!!):<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_auto/get_scores.py .<br /> python get_scores.py 0<br /> <br /> <br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/get_scores.py .<br /> python get_scores.py 0<br /> <br /> <br /> If docking the seeds from the ZINC library, please combine your score.txt files from both SDI files. Rename this combined file 'scores_round_0.txt.'<br /> <br /> This script expects a directory named &quot;output*&quot; within the CWD. If your output from docking follows different organizational conventions, vim into get_scores.py and change the path. Successful output of the script will be a file named &quot;scores_round_0.txt&quot;. For iterative rounds, pass increasing numbers into the command line. i.e. When docking the first round of prioritized molecules, pass [1] for scores_round_1.txt. <br /> <br /> '''4. Convert scores and molecule IDS into NumPY arrays for ChemSTEP recognition. Requires ChemSTEP venv'''<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_auto/convert_scores_to_npy.py .<br /> python convert_scores_to_npy.py 0<br /> <br /> <br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/convert_scores_to_npy.py .<br /> python convert_scores_to_npy.py 0<br /> <br /> For python speed purposes, ChemSTEP requires that DOCK scores and IDs be give in the form of a NumPY array. In this step, we are taking our readable scores file and converting them for ChemSTEP recognition. This script expects a txt file with Molecule IDS and DOCK scores (no header). Use the same round number you used in step 3. As run above, this will output two files named scores_round_0.npy and indices_round_0.npy that contain line-matched indices (determined from Mol ID) and their respective docking scores. <br /> <br /> '''5. Enter into or make a directory to run ChemSTEP in. Copy in necessary files for initiating ChemSTEP: params.txt, run_chemstep.py and launch_chemstep_as_job.sh. Copy in your score and indices numpy files as well, which should be in your docking directory.'''<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_auto/params.txt .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_auto/run_chemstep.py .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_auto/launch_chemstep_as_job.sh .<br /> <br /> <br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/params.txt .<br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/run_chemstep.py .<br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/launch_chemstep_as_job.sh .<br /> <br /> '''6. Edit params.txt file''' This ONLY needs to be edited for the initial round of ChemSTEP. The parameters outlined here will be carried through all rounds of ChemSTEP. <br /> seed_indices_file: /absoulte/path/to/your/indices_round_0.npy<br /> seed_scores_file: /absolute/path/to/your/scores_round_0.npy<br /> hit_pprop: 5<br /> bundle_size: 1000<br /> n_docked_per_round: 10000000<br /> max_beacons: 150<br /> max_n_rounds: 250<br /> <br /> Within params.txt, add the absolute paths to the ChemSTEP-readable score and indices NumPY arrays. The rest of the values within the params.txt file are left to the discretion of the user, with some considerations below. <br /> <br /> '''pProp:''' this value will define what is considered a &quot;virtual hit&quot; in your campaign. pProp is defined as the -log(rank %) of a molecule within the total library score-distribution. For example, a pProp of 4 in the 1.1T XReal space is equivalent the top 0.01% of the library, corresponding to the top 110 million molecules. pProp 5 = 0.001% = 11 million virtual hits, etc. From the seed set, ChemSTEP will estimate a DOCK score value in kcal/mol that associated with the lower-limit of your desired pProp zone. Any molecules that score better than the threshold is considered a virtual hit in your campaign. Be mindful of seed set size when choosing desired pProp. Our suggestion is that the size of the seed set should be at least 10^(pProp +2).<br /> <br /> '''n_docked_per_round:''' this number is the desired molecules for prioritization per round. When choosing this value, note that this number of molecules must be built and docked in between each round of ChemSTEP. Prioritizing many molecules will slow building and docking speeds, and coupled with few beacons (below) may lead to decreased diversity. Prioritizing too few molecules may result in slower virtual hit recovery. Round size does not significantly impact the algorithm running time.** <br /> <br /> '''max_beacons:''' this is the number of diverse, well-scoring molecules ChemSTEP will use to guide prioritization per round. All molecules that score better than the assigned pProp threshold may be considered for beacons. By default, ChemSTEP chooses each set of beacons to be as maximally diverse as possible. Choosing too many beacons may result in decreased diversity between beacons, but too few beacons could hinder space exploration. If not enough molecules score above your pProp threshold, ChemSTEP may assign fewer beacons than the assigned value. <br /> <br /> '''max_n_rounds:''' if prospectively running ChemSTEP, as outlined in this wiki, no need to worry about this parameter.<br /> <br /> '''bundle_size:''' if using auto docking mode, the bundle size is used for building. i.e. the number of molecules that get submitted to build as one job. <br /> <br /> <br /> <br /> '''7. Edit run_chemstep.py'''<br /> <br /> There are several configurable parameters that can be passed when first calling the CSAlgo object:<br /> <br /> '''docking_method=''' setting this to &quot;auto&quot; will set up and run iterative rounds of building, docking, and running ChemSTEP. If you choose the auto method, be sure to update the full path to your dockfiles in run_chemstep.py and check that all INDOCK parameters are set to your liking. NOTE: docking method is set to auto if you copy in the scripts as outlined above. Change if desired!!!!! <br /> <br /> <br /> '''min_td_search=''' setting this parameter will establich a minimum-tanimoto distance &quot;zone&quot; around each beacon for prioritization. For example, setting a min_td_search=0.3 will require that each prioritized molecule be greater than 0.3 Td from each beacon. <br /> <br /> more optional parameters to come.<br /> <br /> At this time, we strongly suggest running ChemSTEP as a job array with 16 or 32 CPU slots requested. The number of cores must be specified when calling CSAlgo (in run_chemstep.py) and the SGE wrapper(launch_chemstep_as_job.sh). <br /> <br /> <br /> '''8. Run ChemSTEP''' with the following command: <br /> qsub launch_chemstep_as_job.sh <br /> <br /> <br /> This will launch the main ChemSTEP job to the SGE scheduler. Check your job status with the 'qstat' command. The algorithm will read the score and indices files provided, calculate pProp (for round 1) and assign beacons. As this job runs, it will launch a subsequent job array. These sub-jobs are the Chaining step of ChemSTEP, where each parallel worker calculates the Tanimoto distances of 100 million molecules from the virtual library to the beacons. These distances are then written to the files within the /output directory. When these jobs complete, the main job will read through all Tanimoto distances and pull molecules for prioritization. <br /> <br /> Running ChemSTEP successfully will result in the output of a SMILES file within output/complete_info/. You will also get (1) chemstep_algo.log and (2) chemstep_submission.log in the working directory after job submission. chemstep_algo.log contains the running output of the algorithm in a readable format, with timestamps, including the DOCK score associated with your desired pProp. chemstep_submission.log contains the output of ChemSTEP, as well as any information output by the SGE submission system (python errors, cluster issues, tracebacks if failed). These files will be updated with any information with iterative rounds of ChemSTEP run in the same directory. '''Visually inspect these files after each round of ChemSTEP to ensure the algorithm has picked your desired number of beacons and things ran smoothly.'''<br /> <br /> '''Troubleshooting:''' if no jobs are running and there is no SMI file, check the chemstep_submission.log first. Any traceback error or SGE error should give you some idea of why ChemSTEP failed. Some errors may be due to SGE or Wynton issues. If directed, look at a few .out files within /output/jobs/ . If a ChemSTEP run fails on your FIRST run, delete the output and log files, fix what needs to be fixed, and rerun. Errors in subsequent rounds during the chaining step can potentially corrupt chaining files within the /output directory that are needed for prioritization. At the very least, you will definitely have duplicates of beacons and information written to the log files. At this time, if your run fails during iterative rounds, it's best to start from the beginning. <br /> <br /> Prioritized molecules should be built, docked, and their scores fed back into ChemSTEP. More detailed instructions below for running in MANUAL mode: <br /> <br /> '''9. Build prioritized molecules (DOCK 3.8).''' /taken from docs.docking.org <br /> source /wynton/group/bks/soft/DOCK-3.8.5/env.sh<br /> <br /> python /wynton/group/bks/soft/DOCK-3.8.5/DOCK3.8/zinc22-3d/submit/submit_building_docker.py --output_folder building_output --bundle_size 1000 --minutes_per_mol 5 --skip_name_check --scheduler sge --container_software apptainer --container_path_or_name /wynton/group/bks/soft/DOCK-3.8.5/building_pipeline.sif smi_round_{}.smi<br /> <br /> When building has completed, you must write an SDI file with the complete paths to each built bundle and dock. you can do this with:<br /> find /path/to/your/building_output -name &quot;bundle.db2.tgz&quot; -type f &gt; round_{}.wynton.sdi<br /> <br /> <br /> '''Be sure to change the INDOCK file to save only poses that meet your score pProp score threshold calculated by ChemSTEP!''' Retrieve docking scores as convert to NumPy arrays as outlined above, updating the round number when running get_scores.py and convert_scores_to_npy.py. Copy new score and indices files into the directory you ran ChemSTEP in. If you are following along as a tutorial, you should have scores_round_1.npy and indices_round_1.npy from the previous step (from FIRST round of ChemSTEP prioritization). <br /> <br /> <br /> '''10. Set up for iterative rounds of ChemSTEP'''<br /> <br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/run_chemstep_iterative.py .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/launch_chemstep_iterative.sh .<br /> <br /> <br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/run_chemstep_iterative.py .<br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/launch_chemstep_iterative.sh .<br /> <br /> '''10. Run ChemSTEP'''<br /> qsub launch_chemstep_as_job.sh [round number]<br /> <br /> For the first iterative round, the round number is [2], and should increase by one for every subsequent round of ChemSTEP. The output will be smi_round_****.smi file. Repeat steps 8-10 for as many rounds as needed. The performance is reported in outputy/complete_info/run_summary.df, which contains the number of beacons selected, the number of molecules docked, the number of hits found, the distance threshold for the selected molecules to dock, and the last added beacon's distance to all previous beacons.<br /> <br /> <br /> <br /> <br /> '''Helper scripts:''' a running list of scripts we have been using for analysis and visualization. <br /> <br /> to get SMILES for beacons: <br /> <br /> python /wynton/group/bks/work/bwhall61/needs_github/get_beacon_smi.py --beacon_df_path /path/to/your/chemstep/output/complete_info/beacons.df --n_workers 6 --outfile beacon_smiles.smi<br /> <br /> <br /> ** anecdotally true.</div>

Kholland http://wiki.docking.org/index.php?title=Running_ChemSTEP&diff=17015 2025-12-15T22:04:58Z

<p>Kholland: </p> <hr /> <div>last update: dec 15 2025 katie. current ver = 0.3.1.5 (automatic resubmission of failed SGE jobs). <br /> <br /> ChemSTEP (Chemical Space Traversal and Exploration Procedure) is an open-source, transparent acceleration algorithm for molecular docking capable of dealing with virtual libraries of several trillion compounds. This wiki page is a guide for BKS lab members to run ChemSTEP on Wynton HPC, using the current version of ZINC library (96B) or 13.3B library from Enamine REAL. '''for detailed instructions on the 13B space, see 'Running ChemSTEP on the 13B space' wiki page'''. For more general use directions, please refer to [ChemSTEP Read-the-Docs]. <br /> <br /> At a high-level, ChemSTEP is an iterative process that identifies molecules from the larger virtual library to prioritize for docking. First, we identify a random sample of the total library (termed &quot;seed set&quot;, round zero) and dock those molecules to the target of interest. From this seed set, we can calculate total-library pProp values (-log rank percentages) and the number of &quot;virtual hits&quot; in the total library (high-scoring molecules). ChemSTEP will identify a set of maximally diverse molecules that score above the desired pProp threshold (&quot;beacons&quot;) from the seed set. These beacons guide prioritization, where molecules chosen and output by ChemSTEP are close in chemical space to the beacons. Prioritized molecules are then built, docked, and returned to ChemSTEP for a second round of prioritization. This process is iterated until you reach desired virtual hit recovery, or you are no longer recovering virtual hits. <br /> <br /> <br /> Running the ChemSTEP algorithm on Wynton takes place in loosely three steps: (1) submission of the main ChemSTEP job, (2) ChainingLog sub-job array, and (3) gathering of SMILES for prioritization. <br /> <br /> '''1. Main ChemSTEP job:''' reads input files (DOCK scores and line-matched indices NumPy arrays) and assigns beacons. Beacons are a set of N maximally diverse molecules that score above the pProp threshold specified in the parameter file. For the first round of ChemSTEP, the best-scoring molecule from the seed set is assigned as the first beacon. Subsequent beacons would be molecules that score well (above pProp thresh), and are as diverse as possible from already assigned beacons. Greedy max-diversity selection. <br /> <br /> During assignment, ECFP4 Tanimoto distances between the assigned beacon and all potential beacons are calculated. The second beacon is the molecule with the maximum Td from beacon 1. Beacon 3 will be the molecule that has the highest Td from beacon 1 and beacon 2. This continues until the number of beacons specified in the parameter file is reached. In iterative rounds of ChemSTEP, the beacon selection is based on diversity from ALL PREVIOUSLY assigned beacons, including those in earlier rounds. <br /> <br /> <br /> '''2. ChainingLog sub-jobs:''' once beacons are assigned, ChemSTEP will launch a series of sub-jobs that calculate the Tanimoto distances of every molecule remaining in the library to the assigned beacons. Calculated distances to the NEAREST beacon (minimum-minimum Td) are updated in the mintddistrib_*.npy files within the /output directory. <br /> <br /> <br /> '''3. Gathering of SMILES for prioritization:''' When all Td calculations are completed, the algo will select N number of molecules for prioritization, the number of which is specified by the user in the parameter file. Molecules that are ''closest in chemical space'' to beacons are prioritized. I.e. if round size = 1 million, the 1 million molecules with the smallest min-Td to ANY beacon (current or previous rounds) are prioritized. chemstep_algo.log will provide a &quot;max-minTd&quot; for each round, which is the Tanimoto distance of the most dissimilar molecule prioritized to any one beacon per round. The prioritized molecules are output in /output/complete_info as smi_round_{}.smi <br /> <br /> What the user need: DOCKFILES, directories for (1) docking (2) building and (3) running ChemSTEP. '''All iterative rounds of ChemSTEP should be run in the SAME directory.''' <br /> <br /> '''1. Source ChemSTEP virtual environment on Wynton'''<br /> source /wynton/group/bks/work/shared/kholland/chemstep_auto/bin/activate<br /> <br /> <br /> '''2. Copy seed-set SDI file into your docking directory'''<br /> This directory should already contain your dockfiles, with INDOCK parameters set to your liking. In this step, we are copying in a split database index file (SDI) containing paths to bundles of db2 files. This seed set contains 100 million molecules sampled randomly from the total virtual library, currently 1.1 trillion molecules.<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_auto/scripts/96M_seeds.wynton.sdi .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_auto/scripts/rebuilt_seeds.wynton.sdi .<br /> <br /> because the seed set is large, the docking must be done in two separate chunks (100k jobs max per docking array), and the scores combined into one file later on. <br /> <br /> if you are interested in running on a library of 13.2B compounds from Enamine REAL, there are several seed set sizes available: 130k, 1.3M, 13M, and 26M. <br /> cp /wynton/group/bks/work/shared/kholland/chemstep_13B/130K_seeds.wynton.sdi .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_13B/1.3M_seeds.wynton.sdi .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_13B/13M_seeds.wynton.sdi .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_13B/26M_seeds.wynton.sdi .<br /> <br /> <br /> '''3. Dock seed set to your receptor of interest using DOCK 3.8''' /docking directions taken from docs.docking.org. This is meant to be done as you would do a normal LSD. <br /> export MOLECULES_DIR_TO_BIND=[outermost folder containing the molecules to dock, just your base docking directory]<br /> export DOCKFILES=[path to your dockfiles]<br /> export INPUT_FOLDER=[the folder containing your .sdi file(s)]<br /> export OUTPUT_FOLDER=[where you want the output ]<br /> <br /> /wynton/group/bks/work/bwhall61/needs_github/super_dock3r.sh<br /> <br /> Wait for docking to complete. Next, you must extract all molecule IDs and corresponding DOCK scores from above. To do so, run the following commands in the base docking directory containing your docking files and output folder, while logged into a dev node (do this in a screen!!!):<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_auto/get_scores.py .<br /> python get_scores.py 0<br /> <br /> <br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/get_scores.py .<br /> python get_scores.py 0<br /> <br /> <br /> If docking the seeds from the ZINC library, please combine your score.txt files from both SDI files. Rename this combined file 'scores_round_0.txt.'<br /> <br /> This script expects a directory named &quot;output*&quot; within the CWD. If your output from docking follows different organizational conventions, vim into get_scores.py and change the path. Successful output of the script will be a file named &quot;scores_round_0.txt&quot;. For iterative rounds, pass increasing numbers into the command line. i.e. When docking the first round of prioritized molecules, pass [1] for scores_round_1.txt. <br /> <br /> '''4. Convert scores and molecule IDS into NumPY arrays for ChemSTEP recognition. Requires ChemSTEP venv'''<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_auto/convert_scores_to_npy.py .<br /> python convert_scores_to_npy.py 0<br /> <br /> <br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/convert_scores_to_npy.py .<br /> python convert_scores_to_npy.py 0<br /> <br /> For python speed purposes, ChemSTEP requires that DOCK scores and IDs be give in the form of a NumPY array. In this step, we are taking our readable scores file and converting them for ChemSTEP recognition. This script expects a txt file with Molecule IDS and DOCK scores (no header). Use the same round number you used in step 3. As run above, this will output two files named scores_round_0.npy and indices_round_0.npy that contain line-matched indices (determined from Mol ID) and their respective docking scores. <br /> <br /> '''5. Enter into or make a directory to run ChemSTEP in. Copy in necessary files for initiating ChemSTEP: params.txt, run_chemstep.py and launch_chemstep_as_job.sh. Copy in your score and indices numpy files as well, which should be in your docking directory.'''<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_auto/params.txt .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_auto/run_chemstep.py .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_auto/launch_chemstep_as_job.sh .<br /> <br /> <br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/params.txt .<br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/run_chemstep.py .<br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/launch_chemstep_as_job.sh .<br /> <br /> '''6. Edit params.txt file''' This ONLY needs to be edited for the initial round of ChemSTEP. The parameters outlined here will be carried through all rounds of ChemSTEP. <br /> seed_indices_file: /absoulte/path/to/your/indices_round_0.npy<br /> seed_scores_file: /absolute/path/to/your/scores_round_0.npy<br /> hit_pprop: 5<br /> bundle_size: 1000<br /> n_docked_per_round: 10000000<br /> max_beacons: 150<br /> max_n_rounds: 250<br /> <br /> Within params.txt, add the absolute paths to the ChemSTEP-readable score and indices NumPY arrays. The rest of the values within the params.txt file are left to the discretion of the user, with some considerations below. <br /> <br /> '''pProp:''' this value will define what is considered a &quot;virtual hit&quot; in your campaign. pProp is defined as the -log(rank %) of a molecule within the total library score-distribution. For example, a pProp of 4 in the 1.1T XReal space is equivalent the top 0.01% of the library, corresponding to the top 110 million molecules. pProp 5 = 0.001% = 11 million virtual hits, etc. From the seed set, ChemSTEP will estimate a DOCK score value in kcal/mol that associated with the lower-limit of your desired pProp zone. Any molecules that score better than the threshold is considered a virtual hit in your campaign. Be mindful of seed set size when choosing desired pProp. Our suggestion is that the size of the seed set should be at least 10^(pProp +2).<br /> <br /> '''n_docked_per_round:''' this number is the desired molecules for prioritization per round. When choosing this value, note that this number of molecules must be built and docked in between each round of ChemSTEP. Prioritizing many molecules will slow building and docking speeds, and coupled with few beacons (below) may lead to decreased diversity. Prioritizing too few molecules may result in slower virtual hit recovery. Round size does not significantly impact the algorithm running time.** <br /> <br /> '''max_beacons:''' this is the number of diverse, well-scoring molecules ChemSTEP will use to guide prioritization per round. All molecules that score better than the assigned pProp threshold may be considered for beacons. By default, ChemSTEP chooses each set of beacons to be as maximally diverse as possible. Choosing too many beacons may result in decreased diversity between beacons, but too few beacons could hinder space exploration. If not enough molecules score above your pProp threshold, ChemSTEP may assign fewer beacons than the assigned value. <br /> <br /> '''max_n_rounds:''' if prospectively running ChemSTEP, as outlined in this wiki, no need to worry about this parameter.<br /> <br /> '''bundle_size:''' if using auto docking mode, the bundle size is used for building. i.e. the number of molecules that get submitted to build as one job. <br /> <br /> <br /> <br /> '''7. Edit run_chemstep.py'''<br /> <br /> There are several configurable parameters that can be passed when first calling the CSAlgo object:<br /> <br /> '''docking_method=''' setting this to &quot;auto&quot; will set up and run iterative rounds of building, docking, and running ChemSTEP. If you choose the auto method, be sure to update the full path to your dockfiles in run_chemstep.py and check that all INDOCK parameters are set to your liking. <br /> <br /> <br /> '''min_td_search=''' setting this parameter will establich a minimum-tanimoto distance &quot;zone&quot; around each beacon for prioritization. For example, setting a min_td_search=0.3 will require that each prioritized molecule be greater than 0.3 Td from each beacon. <br /> <br /> more optional parameters to come.<br /> <br /> At this time, we strongly suggest running ChemSTEP as a job array with 16 or 32 CPU slots requested. The number of cores must be specified when calling CSAlgo (in run_chemstep.py) and the SGE wrapper(launch_chemstep_as_job.sh). <br /> <br /> <br /> '''8. Run ChemSTEP''' with the following command: <br /> qsub launch_chemstep_as_job.sh <br /> <br /> <br /> This will launch the main ChemSTEP job to the SGE scheduler. Check your job status with the 'qstat' command. The algorithm will read the score and indices files provided, calculate pProp (for round 1) and assign beacons. As this job runs, it will launch a subsequent job array. These sub-jobs are the Chaining step of ChemSTEP, where each parallel worker calculates the Tanimoto distances of 100 million molecules from the virtual library to the beacons. These distances are then written to the files within the /output directory. When these jobs complete, the main job will read through all Tanimoto distances and pull molecules for prioritization. <br /> <br /> Running ChemSTEP successfully will result in the output of a SMILES file within output/complete_info/. You will also get (1) chemstep_algo.log and (2) chemstep_submission.log in the working directory after job submission. chemstep_algo.log contains the running output of the algorithm in a readable format, with timestamps, including the DOCK score associated with your desired pProp. chemstep_submission.log contains the output of ChemSTEP, as well as any information output by the SGE submission system (python errors, cluster issues, tracebacks if failed). These files will be updated with any information with iterative rounds of ChemSTEP run in the same directory. '''Visually inspect these files after each round of ChemSTEP to ensure the algorithm has picked your desired number of beacons and things ran smoothly.'''<br /> <br /> '''Troubleshooting:''' if no jobs are running and there is no SMI file, check the chemstep_submission.log first. Any traceback error or SGE error should give you some idea of why ChemSTEP failed. Some errors may be due to SGE or Wynton issues. If directed, look at a few .out files within /output/jobs/ . If a ChemSTEP run fails on your FIRST run, delete the output and log files, fix what needs to be fixed, and rerun. Errors in subsequent rounds during the chaining step can potentially corrupt chaining files within the /output directory that are needed for prioritization. At the very least, you will definitely have duplicates of beacons and information written to the log files. At this time, if your run fails during iterative rounds, it's best to start from the beginning. <br /> <br /> Prioritized molecules should be built, docked, and their scores fed back into ChemSTEP. More detailed instructions below for running in MANUAL mode: <br /> <br /> '''9. Build prioritized molecules (DOCK 3.8).''' /taken from docs.docking.org <br /> source /wynton/group/bks/soft/DOCK-3.8.5/env.sh<br /> <br /> python /wynton/group/bks/soft/DOCK-3.8.5/DOCK3.8/zinc22-3d/submit/submit_building_docker.py --output_folder building_output --bundle_size 1000 --minutes_per_mol 5 --skip_name_check --scheduler sge --container_software apptainer --container_path_or_name /wynton/group/bks/soft/DOCK-3.8.5/building_pipeline.sif smi_round_{}.smi<br /> <br /> When building has completed, you must write an SDI file with the complete paths to each built bundle and dock. you can do this with:<br /> find /path/to/your/building_output -name &quot;bundle.db2.tgz&quot; -type f &gt; round_{}.wynton.sdi<br /> <br /> <br /> '''Be sure to change the INDOCK file to save only poses that meet your score pProp score threshold calculated by ChemSTEP!''' Retrieve docking scores as convert to NumPy arrays as outlined above, updating the round number when running get_scores.py and convert_scores_to_npy.py. Copy new score and indices files into the directory you ran ChemSTEP in. If you are following along as a tutorial, you should have scores_round_1.npy and indices_round_1.npy from the previous step (from FIRST round of ChemSTEP prioritization). <br /> <br /> <br /> '''10. Set up for iterative rounds of ChemSTEP'''<br /> <br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/run_chemstep_iterative.py .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/launch_chemstep_iterative.sh .<br /> <br /> <br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/run_chemstep_iterative.py .<br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/launch_chemstep_iterative.sh .<br /> <br /> '''10. Run ChemSTEP'''<br /> qsub launch_chemstep_as_job.sh [round number]<br /> <br /> For the first iterative round, the round number is [2], and should increase by one for every subsequent round of ChemSTEP. The output will be smi_round_****.smi file. Repeat steps 8-10 for as many rounds as needed. The performance is reported in outputy/complete_info/run_summary.df, which contains the number of beacons selected, the number of molecules docked, the number of hits found, the distance threshold for the selected molecules to dock, and the last added beacon's distance to all previous beacons.<br /> <br /> <br /> <br /> <br /> '''Helper scripts:''' a running list of scripts we have been using for analysis and visualization. <br /> <br /> to get SMILES for beacons: <br /> <br /> python /wynton/group/bks/work/bwhall61/needs_github/get_beacon_smi.py --beacon_df_path /path/to/your/chemstep/output/complete_info/beacons.df --n_workers 6 --outfile beacon_smiles.smi<br /> <br /> <br /> ** anecdotally true.</div>

Kholland http://wiki.docking.org/index.php?title=Running_ChemSTEP&diff=17014 2025-12-15T21:59:09Z

<p>Kholland: </p> <hr /> <div>last update: dec 15 2025 katie. current ver = 0.3.1.5 (automatic resubmission of failed SGE jobs). <br /> <br /> ChemSTEP (Chemical Space Traversal and Exploration Procedure) is an open-source, transparent acceleration algorithm for molecular docking capable of dealing with virtual libraries of several trillion compounds. This wiki page is a guide for BKS lab members to run ChemSTEP on Wynton HPC, using the current version of ZINC library (96B) or 13.3B library from Enamine REAL. '''for detailed instructions on the 13B space, see 'Running ChemSTEP on the 13B space' wiki page'''. For more general use directions, please refer to [ChemSTEP Read-the-Docs]. <br /> <br /> At a high-level, ChemSTEP is an iterative process that identifies molecules from the larger virtual library to prioritize for docking. First, we identify a random sample of the total library (termed &quot;seed set&quot;, round zero) and dock those molecules to the target of interest. From this seed set, we can calculate total-library pProp values (-log rank percentages) and the number of &quot;virtual hits&quot; in the total library (high-scoring molecules). ChemSTEP will identify a set of maximally diverse molecules that score above the desired pProp threshold (&quot;beacons&quot;) from the seed set. These beacons guide prioritization, where molecules chosen and output by ChemSTEP are close in chemical space to the beacons. Prioritized molecules are then built, docked, and returned to ChemSTEP for a second round of prioritization. This process is iterated until you reach desired virtual hit recovery, or you are no longer recovering virtual hits. <br /> <br /> <br /> Running the ChemSTEP algorithm on Wynton takes place in loosely three steps: (1) submission of the main ChemSTEP job, (2) ChainingLog sub-job array, and (3) gathering of SMILES for prioritization. <br /> <br /> '''1. Main ChemSTEP job:''' reads input files (DOCK scores and line-matched indices NumPy arrays) and assigns beacons. Beacons are a set of N maximally diverse molecules that score above the pProp threshold specified in the parameter file. For the first round of ChemSTEP, the best-scoring molecule from the seed set is assigned as the first beacon. Subsequent beacons would be molecules that score well (above pProp thresh), and are as diverse as possible from already assigned beacons. Greedy max-diversity selection. <br /> <br /> During assignment, ECFP4 Tanimoto distances between the assigned beacon and all potential beacons are calculated. The second beacon is the molecule with the maximum Td from beacon 1. Beacon 3 will be the molecule that has the highest Td from beacon 1 and beacon 2. This continues until the number of beacons specified in the parameter file is reached. In iterative rounds of ChemSTEP, the beacon selection is based on diversity from ALL PREVIOUSLY assigned beacons, including those in earlier rounds. <br /> <br /> <br /> '''2. ChainingLog sub-jobs:''' once beacons are assigned, ChemSTEP will launch a series of sub-jobs that calculate the Tanimoto distances of every molecule remaining in the library to the assigned beacons. Calculated distances to the NEAREST beacon (minimum-minimum Td) are updated in the mintddistrib_*.npy files within the /output directory. <br /> <br /> <br /> '''3. Gathering of SMILES for prioritization:''' When all Td calculations are completed, the algo will select N number of molecules for prioritization, the number of which is specified by the user in the parameter file. Molecules that are ''closest in chemical space'' to beacons are prioritized. I.e. if round size = 1 million, the 1 million molecules with the smallest min-Td to ANY beacon (current or previous rounds) are prioritized. chemstep_algo.log will provide a &quot;max-minTd&quot; for each round, which is the Tanimoto distance of the most dissimilar molecule prioritized to any one beacon per round. The prioritized molecules are output in /output/complete_info as smi_round_{}.smi <br /> <br /> What the user need: DOCKFILES, directories for (1) docking (2) building and (3) running ChemSTEP. '''All iterative rounds of ChemSTEP should be run in the SAME directory.''' <br /> <br /> '''1. Source ChemSTEP virtual environment on Wynton'''<br /> source /wynton/group/bks/work/shared/kholland/chemstep_auto/bin/activate<br /> <br /> <br /> '''2. Copy seed-set SDI file into your docking directory'''<br /> This directory should already contain your dockfiles, with INDOCK parameters set to your liking. In this step, we are copying in a split database index file (SDI) containing paths to bundles of db2 files. This seed set contains 100 million molecules sampled randomly from the total virtual library, currently 1.1 trillion molecules.<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_auto/scripts/96M_seeds.wynton.sdi .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_auto/scripts/rebuilt_seeds.wynton.sdi .<br /> <br /> because the seed set is large, the docking must be done in two separate chunks (100k jobs max per docking array), and the scores combined into one file later on. <br /> <br /> if you are interested in running on a library of 13.2B compounds from Enamine REAL, there are several seed set sizes available: 130k, 1.3M, 13M, and 26M. <br /> cp /wynton/group/bks/work/shared/kholland/chemstep_13B/130K_seeds.wynton.sdi .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_13B/1.3M_seeds.wynton.sdi .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_13B/13M_seeds.wynton.sdi .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_13B/26M_seeds.wynton.sdi .<br /> <br /> <br /> '''3. Dock seed set to your receptor of interest using DOCK 3.8''' /docking directions taken from docs.docking.org. This is meant to be done as you would do a normal LSD. <br /> export MOLECULES_DIR_TO_BIND=[outermost folder containing the molecules to dock, just your base docking directory]<br /> export DOCKFILES=[path to your dockfiles]<br /> export INPUT_FOLDER=[the folder containing your .sdi file(s)]<br /> export OUTPUT_FOLDER=[where you want the output ]<br /> <br /> /wynton/group/bks/work/bwhall61/needs_github/super_dock3r.sh<br /> <br /> Wait for docking to complete. Next, you must extract all molecule IDs and corresponding DOCK scores from above. To do so, run the following commands in the base docking directory containing your docking files and output folder, while logged into a dev node (do this in a screen!!!):<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_auto/get_scores.py .<br /> python get_scores.py 0<br /> <br /> <br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/get_scores.py .<br /> python get_scores.py 0<br /> <br /> <br /> If docking the seeds from the ZINC library, please combine your score.txt files from both SDI files. Rename this combined file 'scores_round_0.txt.'<br /> <br /> This script expects a directory named &quot;output*&quot; within the CWD. If your output from docking follows different organizational conventions, vim into get_scores.py and change the path. Successful output of the script will be a file named &quot;scores_round_0.txt&quot;. For iterative rounds, pass increasing numbers into the command line. i.e. When docking the first round of prioritized molecules, pass [1] for scores_round_1.txt. <br /> <br /> '''4. Convert scores and molecule IDS into NumPY arrays for ChemSTEP recognition. Requires ChemSTEP venv'''<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_auto/convert_scores_to_npy.py .<br /> python convert_scores_to_npy.py 0<br /> <br /> <br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/convert_scores_to_npy.py .<br /> python convert_scores_to_npy.py 0<br /> <br /> For python speed purposes, ChemSTEP requires that DOCK scores and IDs be give in the form of a NumPY array. In this step, we are taking our readable scores file and converting them for ChemSTEP recognition. This script expects a txt file with Molecule IDS and DOCK scores (no header). Use the same round number you used in step 3. As run above, this will output two files named scores_round_0.npy and indices_round_0.npy that contain line-matched indices (determined from Mol ID) and their respective docking scores. <br /> <br /> '''5. Enter into or make a directory to run ChemSTEP in. Copy in necessary files for initiating ChemSTEP: params.txt, run_chemstep.py and launch_chemstep_as_job.sh. Copy in your score and indices numpy files as well, which should be in your docking directory.'''<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_auto/params.txt .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_auto/run_chemstep.py .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_auto/launch_chemstep_as_job.sh .<br /> <br /> <br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/params.txt .<br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/run_chemstep.py .<br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/launch_chemstep_as_job.sh .<br /> <br /> '''6. Edit params.txt file''' This ONLY needs to be edited for the initial round of ChemSTEP. The parameters outlined here will be carried through all rounds of ChemSTEP. <br /> seed_indices_file: /absoulte/path/to/your/indices_round_0.npy<br /> seed_scores_file: /absolute/path/to/your/scores_round_0.npy<br /> hit_pprop: 5<br /> bundle_size: 1000<br /> n_docked_per_round: 10000000<br /> max_beacons: 150<br /> max_n_rounds: 250<br /> <br /> Within params.txt, add the absolute paths to the ChemSTEP-readable score and indices NumPY arrays. The rest of the values within the params.txt file are left to the discretion of the user, with some considerations below. <br /> <br /> '''pProp:''' this value will define what is considered a &quot;virtual hit&quot; in your campaign. pProp is defined as the -log(rank %) of a molecule within the total library score-distribution. For example, a pProp of 4 in the 1.1T XReal space is equivalent the top 0.01% of the library, corresponding to the top 110 million molecules. pProp 5 = 0.001% = 11 million virtual hits, etc. From the seed set, ChemSTEP will estimate a DOCK score value in kcal/mol that associated with the lower-limit of your desired pProp zone. Any molecules that score better than the threshold is considered a virtual hit in your campaign. Be mindful of seed set size when choosing desired pProp. Our suggestion is that the size of the seed set should be at least 10^(pProp +2).<br /> <br /> '''n_docked_per_round:''' this number is the desired molecules for prioritization per round. When choosing this value, note that this number of molecules must be built and docked in between each round of ChemSTEP. Prioritizing many molecules will slow building and docking speeds, and coupled with few beacons (below) may lead to decreased diversity. Prioritizing too few molecules may result in slower virtual hit recovery. Round size does not significantly impact the algorithm running time.** <br /> <br /> '''max_beacons:''' this is the number of diverse, well-scoring molecules ChemSTEP will use to guide prioritization per round. All molecules that score better than the assigned pProp threshold may be considered for beacons. By default, ChemSTEP chooses each set of beacons to be as maximally diverse as possible. Choosing too many beacons may result in decreased diversity between beacons, but too few beacons could hinder space exploration. If not enough molecules score above your pProp threshold, ChemSTEP may assign fewer beacons than the assigned value. <br /> <br /> '''max_n_rounds:''' if prospectively running ChemSTEP, as outlined in this wiki, no need to worry about this parameter.<br /> <br /> '''bundle_size:''' if using auto docking mode, the bundle size is used for building. i.e. the number of molecules that get submitted to build as one job. <br /> <br /> <br /> <br /> '''7. Edit run_chemstep.py'''<br /> <br /> There are several configurable parameters that can be passed when first calling the CSAlgo object:<br /> <br /> '''docking_method='''<br /> <br /> At this time, we strongly suggest running ChemSTEP as a job array with 16 or 32 CPU slots requested. The number of cores must be specified when calling CSAlgo (in run_chemstep.py) and the SGE wrapper(launch_chemstep_as_job.sh). <br /> <br /> <br /> '''7. Run ChemSTEP''' with the following command: <br /> qsub launch_chemstep_as_job.sh <br /> <br /> <br /> This will launch the main ChemSTEP job to the SGE scheduler. Check your job status with the 'qstat' command. The algorithm will read the score and indices files provided, calculate pProp (for round 1) and assign beacons. As this job runs, it will launch a subsequent job array. These sub-jobs are the Chaining step of ChemSTEP, where each parallel worker calculates the Tanimoto distances of 100 million molecules from the virtual library to the beacons. These distances are then written to the files within the /output directory. When these jobs complete, the main job will read through all Tanimoto distances and pull molecules for prioritization. <br /> <br /> Running ChemSTEP successfully will result in the output of a SMILES file within output/complete_info/. You will also get (1) chemstep_algo.log and (2) chemstep_submission.log in the working directory after job submission. chemstep_algo.log contains the running output of the algorithm in a readable format, with timestamps, including the DOCK score associated with your desired pProp. chemstep_submission.log contains the output of ChemSTEP, as well as any information output by the SGE submission system (python errors, cluster issues, tracebacks if failed). These files will be updated with any information with iterative rounds of ChemSTEP run in the same directory. '''Visually inspect these files after each round of ChemSTEP to ensure the algorithm has picked your desired number of beacons and things ran smoothly.'''<br /> <br /> '''Troubleshooting:''' if no jobs are running and there is no SMI file, check the chemstep_submission.log first. Any traceback error or SGE error should give you some idea of why ChemSTEP failed. Some errors may be due to SGE or Wynton issues. If directed, look at a few .out files within /output/jobs/ . If a ChemSTEP run fails on your FIRST run, delete the output and log files, fix what needs to be fixed, and rerun. Errors in subsequent rounds during the chaining step can potentially corrupt chaining files within the /output directory that are needed for prioritization. At the very least, you will definitely have duplicates of beacons and information written to the log files. At this time, if your run fails during iterative rounds, it's best to start from the beginning. <br /> <br /> Prioritized molecules should be built, docked, and their scores fed back into ChemSTEP. More detailed instructions below: <br /> <br /> '''8. Build prioritized molecules (DOCK 3.8).''' /taken from docs.docking.org <br /> source /wynton/group/bks/soft/DOCK-3.8.5/env.sh<br /> <br /> python /wynton/group/bks/soft/DOCK-3.8.5/DOCK3.8/zinc22-3d/submit/submit_building_docker.py --output_folder building_output --bundle_size 1000 --minutes_per_mol 5 --skip_name_check --scheduler sge --container_software apptainer --container_path_or_name /wynton/group/bks/soft/DOCK-3.8.5/building_pipeline.sif smi_round_{}.smi<br /> <br /> When building has completed, you must write an SDI file with the complete paths to each built bundle and dock. you can do this with:<br /> find /path/to/your/building_output -name &quot;bundle.db2.tgz&quot; -type f &gt; round_{}.wynton.sdi<br /> <br /> <br /> '''Be sure to change the INDOCK file to save only poses that meet your score pProp score threshold calculated by ChemSTEP!''' Retrieve docking scores as convert to NumPy arrays as outlined above, updating the round number when running get_scores.py and convert_scores_to_npy.py. Copy new score and indices files into the directory you ran ChemSTEP in. If you are following along as a tutorial, you should have scores_round_1.npy and indices_round_1.npy from the previous step (from FIRST round of ChemSTEP prioritization). <br /> <br /> <br /> '''9. Set up for iterative rounds of ChemSTEP'''<br /> <br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/run_chemstep_iterative.py .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/launch_chemstep_iterative.sh .<br /> <br /> <br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/run_chemstep_iterative.py .<br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/launch_chemstep_iterative.sh .<br /> <br /> '''10. Run ChemSTEP'''<br /> qsub launch_chemstep_as_job.sh [round number]<br /> <br /> For the first iterative round, the round number is [2], and should increase by one for every subsequent round of ChemSTEP. The output will be smi_round_****.smi file. Repeat steps 8-10 for as many rounds as needed. The performance is reported in outputy/complete_info/run_summary.df, which contains the number of beacons selected, the number of molecules docked, the number of hits found, the distance threshold for the selected molecules to dock, and the last added beacon's distance to all previous beacons.<br /> <br /> <br /> '''AUTOMATIC CHEMSTEP RUNNING DIRECTIONS'''<br /> <br /> This version of ChemSTEP will automatically submit building and docking jobs inbetween rounds of ChemSTEP. <br /> <br /> '''1. DOCK the seed set to your receptor as outlined above in steps 1-4'''<br /> <br /> successful output will be a scores_round_0.npy and indices_round_0.npy. <br /> <br /> <br /> '''2. Copy necessary scripts into your working ChemSTEP directory'''<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_auto/scripts/params.txt .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_auto/scripts/run_chemstep.py .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_auto/scripts/launch_chemstep_as_job.sh .<br /> <br /> <br /> '''3. Edit run_chemstep.py to add the path to your dockfiles'''<br /> <br /> the algorithm needs to know where your dockfiles live in order to submit docking jobs. Make sure all INDOCK parameters are set correctly. Use the full path to the dockfiles. <br /> <br /> <br /> '''4. Edit params.txt''' The parameters outlined here will be carried through all rounds of ChemSTEP. <br /> seed_indices_file: /absoulte/path/to/your/indices_round_0.npy<br /> seed_scores_file: /absolute/path/to/your/scores_round_0.npy<br /> hit_pprop: 5<br /> bundle_size: 1000<br /> n_docked_per_round: 10000000<br /> max_beacons: 150<br /> max_n_rounds: 250<br /> <br /> <br /> Within params.txt, add the absolute paths to the ChemSTEP-readable score and indices NumPY arrays. The rest of the values within the params.txt file are left to the discretion of the user. The '''bundle_size:''' parameter is required for auto docking mode of ChemSTEP. This is the number of molecules submitted at a job for building. <br /> <br /> <br /> '''5. Launch ChemSTEP'''<br /> <br /> qsub launch_chemstep_as_job.sh <br /> <br /> <br /> '''Troubleshooting:''' if your job errors out or stops running at any point, check the ''chemstep_submission.log'' file in the working directory. <br /> <br /> <br /> <br /> '''Helper scripts:''' a running list of scripts we have been using for analysis and visualization. <br /> <br /> to get SMILES for beacons: <br /> <br /> python /wynton/group/bks/work/bwhall61/needs_github/get_beacon_smi.py --beacon_df_path /path/to/your/chemstep/output/complete_info/beacons.df --n_workers 6 --outfile beacon_smiles.smi<br /> <br /> <br /> ** anecdotally true.</div>

Kholland http://wiki.docking.org/index.php?title=Running_ChemSTEP&diff=17013 2025-12-15T21:40:39Z

<p>Kholland: </p> <hr /> <div>last update: dec 15 2025 katie. current ver = 0.3.1.5 (automatic resubmission of failed SGE jobs). <br /> <br /> ChemSTEP (Chemical Space Traversal and Exploration Procedure) is an open-source, transparent acceleration algorithm for molecular docking capable of dealing with virtual libraries of several trillion compounds. This wiki page is a guide for BKS lab members to run ChemSTEP on Wynton HPC, using the current version of ZINC library (96B) or 13.3B library from Enamine REAL. '''for detailed instructions on the 13B space, see 'Running ChemSTEP on the 13B space' wiki page'''. For more general use directions, please refer to [ChemSTEP Read-the-Docs]. <br /> <br /> At a high-level, ChemSTEP is an iterative process that identifies molecules from the larger virtual library to prioritize for docking. First, we identify a random sample of the total library (termed &quot;seed set&quot;, round zero) and dock those molecules to the target of interest. From this seed set, we can calculate total-library pProp values (-log rank percentages) and the number of &quot;virtual hits&quot; in the total library (high-scoring molecules). ChemSTEP will identify a set of maximally diverse molecules that score above the desired pProp threshold (&quot;beacons&quot;) from the seed set. These beacons guide prioritization, where molecules chosen and output by ChemSTEP are close in chemical space to the beacons. Prioritized molecules are then built, docked, and returned to ChemSTEP for a second round of prioritization. This process is iterated until you reach desired virtual hit recovery, or you are no longer recovering virtual hits. <br /> <br /> <br /> Running the ChemSTEP algorithm on Wynton takes place in loosely three steps: (1) submission of the main ChemSTEP job, (2) ChainingLog sub-job array, and (3) gathering of SMILES for prioritization. <br /> <br /> '''1. Main ChemSTEP job:''' reads input files (DOCK scores and line-matched indices NumPy arrays) and assigns beacons. Beacons are a set of N maximally diverse molecules that score above the pProp threshold specified in the parameter file. For the first round of ChemSTEP, the best-scoring molecule from the seed set is assigned as the first beacon. Subsequent beacons would be molecules that score well (above pProp thresh), and are as diverse as possible from already assigned beacons. Greedy max-diversity selection. <br /> <br /> During assignment, ECFP4 Tanimoto distances between the assigned beacon and all potential beacons are calculated. The second beacon is the molecule with the maximum Td from beacon 1. Beacon 3 will be the molecule that has the highest Td from beacon 1 and beacon 2. This continues until the number of beacons specified in the parameter file is reached. In iterative rounds of ChemSTEP, the beacon selection is based on diversity from ALL PREVIOUSLY assigned beacons, including those in earlier rounds. <br /> <br /> <br /> '''2. ChainingLog sub-jobs:''' once beacons are assigned, ChemSTEP will launch a series of sub-jobs that calculate the Tanimoto distances of every molecule remaining in the library to the assigned beacons. Calculated distances to the NEAREST beacon (minimum-minimum Td) are updated in the mintddistrib_*.npy files within the /output directory. <br /> <br /> <br /> '''3. Gathering of SMILES for prioritization:''' When all Td calculations are completed, the algo will select N number of molecules for prioritization, the number of which is specified by the user in the parameter file. Molecules that are ''closest in chemical space'' to beacons are prioritized. I.e. if round size = 1 million, the 1 million molecules with the smallest min-Td to ANY beacon (current or previous rounds) are prioritized. chemstep_algo.log will provide a &quot;max-minTd&quot; for each round, which is the Tanimoto distance of the most dissimilar molecule prioritized to any one beacon per round. The prioritized molecules are output in /output/complete_info as smi_round_{}.smi <br /> <br /> What the user need: DOCKFILES, directories for (1) docking (2) building and (3) running ChemSTEP. '''All iterative rounds of ChemSTEP should be run in the SAME directory.''' <br /> <br /> '''1. Source ChemSTEP virtual environment on Wynton'''<br /> source /wynton/group/bks/work/shared/kholland/chemstep_auto/bin/activate<br /> <br /> <br /> '''2. Copy seed-set SDI file into your docking directory'''<br /> This directory should already contain your dockfiles, with INDOCK parameters set to your liking. In this step, we are copying in a split database index file (SDI) containing paths to bundles of db2 files. This seed set contains 100 million molecules sampled randomly from the total virtual library, currently 1.1 trillion molecules.<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_auto/scripts/96M_seeds.wynton.sdi .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_auto/scripts/rebuilt_seeds.wynton.sdi .<br /> <br /> because the seed set is large, the docking must be done in two separate chunks (100k jobs max per docking array), and the scores combined into one file later on. <br /> <br /> if you are interested in running on a library of 13.2B compounds from Enamine REAL, there are several seed set sizes available: 130k, 1.3M, 13M, and 26M. <br /> cp /wynton/group/bks/work/shared/kholland/chemstep_13B/130K_seeds.wynton.sdi .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_13B/1.3M_seeds.wynton.sdi .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_13B/13M_seeds.wynton.sdi .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_13B/26M_seeds.wynton.sdi .<br /> <br /> <br /> '''3. Dock seed set to your receptor of interest using DOCK 3.8''' /docking directions taken from docs.docking.org. This is meant to be done as you would do a normal LSD. <br /> export MOLECULES_DIR_TO_BIND=[outermost folder containing the molecules to dock, just your base docking directory]<br /> export DOCKFILES=[path to your dockfiles]<br /> export INPUT_FOLDER=[the folder containing your .sdi file(s)]<br /> export OUTPUT_FOLDER=[where you want the output ]<br /> <br /> /wynton/group/bks/work/bwhall61/needs_github/super_dock3r.sh<br /> <br /> Wait for docking to complete. Next, you must extract all molecule IDs and corresponding DOCK scores from above. To do so, run the following commands in the base docking directory containing your docking files and output folder, while logged into a dev node (do this in a screen!!!):<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_auto/get_scores.py .<br /> python get_scores.py 0<br /> <br /> <br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/get_scores.py .<br /> python get_scores.py 0<br /> <br /> <br /> If docking the seeds from the ZINC library, please combine your score.txt files from both SDI files. Rename this combined file 'scores_round_0.txt.'<br /> <br /> This script expects a directory named &quot;output*&quot; within the CWD. If your output from docking follows different organizational conventions, vim into get_scores.py and change the path. Successful output of the script will be a file named &quot;scores_round_0.txt&quot;. For iterative rounds, pass increasing numbers into the command line. i.e. When docking the first round of prioritized molecules, pass [1] for scores_round_1.txt. <br /> <br /> '''4. Convert scores and molecule IDS into NumPY arrays for ChemSTEP recognition. Requires ChemSTEP venv'''<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_auto/convert_scores_to_npy.py .<br /> python convert_scores_to_npy.py 0<br /> <br /> <br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/convert_scores_to_npy.py .<br /> python convert_scores_to_npy.py 0<br /> <br /> For python speed purposes, ChemSTEP requires that DOCK scores and IDs be give in the form of a NumPY array. In this step, we are taking our readable scores file and converting them for ChemSTEP recognition. This script expects a txt file with Molecule IDS and DOCK scores (no header). Use the same round number you used in step 3. As run above, this will output two files named scores_round_0.npy and indices_round_0.npy that contain line-matched indices (determined from Mol ID) and their respective docking scores. <br /> <br /> '''5. Enter into or make a directory to run ChemSTEP in. Copy in necessary files for initiating ChemSTEP: params.txt, run_chemstep.py and launch_chemstep_as_job.sh. Copy in your score and indices numpy files as well, which should be in your docking directory.'''<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_auto/params.txt .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_auto/run_chemstep.py .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_auto/launch_chemstep_as_job.sh .<br /> <br /> <br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/params.txt .<br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/run_chemstep.py .<br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/launch_chemstep_as_job.sh .<br /> <br /> '''6. Edit params.txt file''' This ONLY needs to be edited for the initial round of ChemSTEP. The parameters outlined here will be carried through all rounds of ChemSTEP. <br /> seed_indices_file: /absoulte/path/to/your/indices_round_0.npy<br /> seed_scores_file: /absolute/path/to/your/scores_round_0.npy<br /> hit_pprop: 5<br /> bundle_size: 1000<br /> n_docked_per_round: 10000000<br /> max_beacons: 150<br /> max_n_rounds: 250<br /> <br /> Within params.txt, add the absolute paths to the ChemSTEP-readable score and indices NumPY arrays. The rest of the values within the params.txt file are left to the discretion of the user, with some considerations below. <br /> <br /> '''pProp:''' this value will define what is considered a &quot;virtual hit&quot; in your campaign. pProp is defined as the -log(rank %) of a molecule within the total library score-distribution. For example, a pProp of 4 in the 1.1T XReal space is equivalent the top 0.01% of the library, corresponding to the top 110 million molecules. pProp 5 = 0.001% = 11 million virtual hits, etc. From the seed set, ChemSTEP will estimate a DOCK score value in kcal/mol that associated with the lower-limit of your desired pProp zone. Any molecules that score better than the threshold is considered a virtual hit in your campaign. Be mindful of seed set size when choosing desired pProp. Our suggestion is that the size of the seed set should be at least 10^(pProp +2).<br /> <br /> '''n_docked_per_round:''' this number is the desired molecules for prioritization per round. When choosing this value, note that this number of molecules must be built and docked in between each round of ChemSTEP. Prioritizing many molecules will slow building and docking speeds, and coupled with few beacons (below) may lead to decreased diversity. Prioritizing too few molecules may result in slower virtual hit recovery. Round size does not significantly impact the algorithm running time.** <br /> <br /> '''max_beacons:''' this is the number of diverse, well-scoring molecules ChemSTEP will use to guide prioritization per round. All molecules that score better than the assigned pProp threshold may be considered for beacons. By default, ChemSTEP chooses each set of beacons to be as maximally diverse as possible. Choosing too many beacons may result in decreased diversity between beacons, but too few beacons could hinder space exploration. If not enough molecules score above your pProp threshold, ChemSTEP may assign fewer beacons than the assigned value. <br /> <br /> '''max_n_rounds:''' if prospectively running ChemSTEP, as outlined in this wiki, no need to worry about this parameter.<br /> <br /> '''bundle_size:''' if using auto docking mode, the bundle size is used for building. i.e. the number of molecules that get submitted to build as one job. <br /> <br /> <br /> There should be no need to edit run_chemstep.py or the SGE wrapper script. At this time, we strongly suggest running ChemSTEP as a job array with 16 or 32 CPU slots requested. The number of cores must be specified when calling CSAlgo (in run_chemstep.py) and the SGE wrapper(launch_chemstep_as_job.sh). <br /> <br /> <br /> '''7. Run ChemSTEP''' with the following command: <br /> qsub launch_chemstep_as_job.sh <br /> <br /> <br /> This will launch the main ChemSTEP job to the SGE scheduler. Check your job status with the 'qstat' command. The algorithm will read the score and indices files provided, calculate pProp (for round 1) and assign beacons. As this job runs, it will launch a subsequent job array. These sub-jobs are the Chaining step of ChemSTEP, where each parallel worker calculates the Tanimoto distances of 100 million molecules from the virtual library to the beacons. These distances are then written to the files within the /output directory. When these jobs complete, the main job will read through all Tanimoto distances and pull molecules for prioritization. <br /> <br /> Running ChemSTEP successfully will result in the output of a SMILES file within output/complete_info/. You will also get (1) chemstep_algo.log and (2) chemstep_submission.log in the working directory after job submission. chemstep_algo.log contains the running output of the algorithm in a readable format, with timestamps, including the DOCK score associated with your desired pProp. chemstep_submission.log contains the output of ChemSTEP, as well as any information output by the SGE submission system (python errors, cluster issues, tracebacks if failed). These files will be updated with any information with iterative rounds of ChemSTEP run in the same directory. '''Visually inspect these files after each round of ChemSTEP to ensure the algorithm has picked your desired number of beacons and things ran smoothly.'''<br /> <br /> '''Troubleshooting:''' if no jobs are running and there is no SMI file, check the chemstep_submission.log first. Any traceback error or SGE error should give you some idea of why ChemSTEP failed. Some errors may be due to SGE or Wynton issues. If directed, look at a few .out files within /output/jobs/ . If a ChemSTEP run fails on your FIRST run, delete the output and log files, fix what needs to be fixed, and rerun. Errors in subsequent rounds during the chaining step can potentially corrupt chaining files within the /output directory that are needed for prioritization. At the very least, you will definitely have duplicates of beacons and information written to the log files. At this time, if your run fails during iterative rounds, it's best to start from the beginning. <br /> <br /> Prioritized molecules should be built, docked, and their scores fed back into ChemSTEP. More detailed instructions below: <br /> <br /> '''8. Build prioritized molecules (DOCK 3.8).''' /taken from docs.docking.org <br /> source /wynton/group/bks/soft/DOCK-3.8.5/env.sh<br /> <br /> python /wynton/group/bks/soft/DOCK-3.8.5/DOCK3.8/zinc22-3d/submit/submit_building_docker.py --output_folder building_output --bundle_size 1000 --minutes_per_mol 5 --skip_name_check --scheduler sge --container_software apptainer --container_path_or_name /wynton/group/bks/soft/DOCK-3.8.5/building_pipeline.sif smi_round_{}.smi<br /> <br /> When building has completed, you must write an SDI file with the complete paths to each built bundle and dock. you can do this with:<br /> find /path/to/your/building_output -name &quot;bundle.db2.tgz&quot; -type f &gt; round_{}.wynton.sdi<br /> <br /> <br /> '''Be sure to change the INDOCK file to save only poses that meet your score pProp score threshold calculated by ChemSTEP!''' Retrieve docking scores as convert to NumPy arrays as outlined above, updating the round number when running get_scores.py and convert_scores_to_npy.py. Copy new score and indices files into the directory you ran ChemSTEP in. If you are following along as a tutorial, you should have scores_round_1.npy and indices_round_1.npy from the previous step (from FIRST round of ChemSTEP prioritization). <br /> <br /> <br /> '''9. Set up for iterative rounds of ChemSTEP'''<br /> <br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/run_chemstep_iterative.py .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/launch_chemstep_iterative.sh .<br /> <br /> <br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/run_chemstep_iterative.py .<br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/launch_chemstep_iterative.sh .<br /> <br /> '''10. Run ChemSTEP'''<br /> qsub launch_chemstep_as_job.sh [round number]<br /> <br /> For the first iterative round, the round number is [2], and should increase by one for every subsequent round of ChemSTEP. The output will be smi_round_****.smi file. Repeat steps 8-10 for as many rounds as needed. The performance is reported in outputy/complete_info/run_summary.df, which contains the number of beacons selected, the number of molecules docked, the number of hits found, the distance threshold for the selected molecules to dock, and the last added beacon's distance to all previous beacons.<br /> <br /> <br /> '''AUTOMATIC CHEMSTEP RUNNING DIRECTIONS'''<br /> <br /> This version of ChemSTEP will automatically submit building and docking jobs inbetween rounds of ChemSTEP. <br /> <br /> '''1. DOCK the seed set to your receptor as outlined above in steps 1-4'''<br /> <br /> successful output will be a scores_round_0.npy and indices_round_0.npy. <br /> <br /> <br /> '''2. Copy necessary scripts into your working ChemSTEP directory'''<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_auto/scripts/params.txt .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_auto/scripts/run_chemstep.py .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_auto/scripts/launch_chemstep_as_job.sh .<br /> <br /> <br /> '''3. Edit run_chemstep.py to add the path to your dockfiles'''<br /> <br /> the algorithm needs to know where your dockfiles live in order to submit docking jobs. Make sure all INDOCK parameters are set correctly. Use the full path to the dockfiles. <br /> <br /> <br /> '''4. Edit params.txt''' The parameters outlined here will be carried through all rounds of ChemSTEP. <br /> seed_indices_file: /absoulte/path/to/your/indices_round_0.npy<br /> seed_scores_file: /absolute/path/to/your/scores_round_0.npy<br /> hit_pprop: 5<br /> bundle_size: 1000<br /> n_docked_per_round: 10000000<br /> max_beacons: 150<br /> max_n_rounds: 250<br /> <br /> <br /> Within params.txt, add the absolute paths to the ChemSTEP-readable score and indices NumPY arrays. The rest of the values within the params.txt file are left to the discretion of the user. The '''bundle_size:''' parameter is required for auto docking mode of ChemSTEP. This is the number of molecules submitted at a job for building. <br /> <br /> <br /> '''5. Launch ChemSTEP'''<br /> <br /> qsub launch_chemstep_as_job.sh <br /> <br /> <br /> '''Troubleshooting:''' if your job errors out or stops running at any point, check the ''chemstep_submission.log'' file in the working directory. <br /> <br /> <br /> <br /> '''Helper scripts:''' a running list of scripts we have been using for analysis and visualization. <br /> <br /> to get SMILES for beacons: <br /> <br /> python /wynton/group/bks/work/bwhall61/needs_github/get_beacon_smi.py --beacon_df_path /path/to/your/chemstep/output/complete_info/beacons.df --n_workers 6 --outfile beacon_smiles.smi<br /> <br /> <br /> ** anecdotally true.</div>

Kholland http://wiki.docking.org/index.php?title=Running_ChemSTEP&diff=17012 2025-12-15T21:39:05Z

<p>Kholland: </p> <hr /> <div>last update: dec 15 2025 katie. current ver = 0.3.1.5 (automatic resubmission of failed SGE jobs). <br /> <br /> ChemSTEP (Chemical Space Traversal and Exploration Procedure) is an open-source, transparent acceleration algorithm for molecular docking capable of dealing with virtual libraries of several trillion compounds. This wiki page is a guide for BKS lab members to run ChemSTEP on Wynton HPC, using the current version of ZINC library (96B) or 13.3B library from Enamine REAL. '''for detailed instructions on the 13B space, see 'Running ChemSTEP on the 13B space' wiki page'''. For more general use directions, please refer to [ChemSTEP Read-the-Docs]. <br /> <br /> At a high-level, ChemSTEP is an iterative process that identifies molecules from the larger virtual library to prioritize for docking. First, we identify a random sample of the total library (termed &quot;seed set&quot;, round zero) and dock those molecules to the target of interest. From this seed set, we can calculate total-library pProp values (-log rank percentages) and the number of &quot;virtual hits&quot; in the total library (high-scoring molecules). ChemSTEP will identify a set of maximally diverse molecules that score above the desired pProp threshold (&quot;beacons&quot;) from the seed set. These beacons guide prioritization, where molecules chosen and output by ChemSTEP are close in chemical space to the beacons. Prioritized molecules are then built, docked, and returned to ChemSTEP for a second round of prioritization. This process is iterated until you reach desired virtual hit recovery, or you are no longer recovering virtual hits. <br /> <br /> <br /> Running the ChemSTEP algorithm on Wynton takes place in loosely three steps: (1) submission of the main ChemSTEP job, (2) ChainingLog sub-job array, and (3) gathering of SMILES for prioritization. <br /> <br /> '''1. Main ChemSTEP job:''' reads input files (DOCK scores and line-matched indices NumPy arrays) and assigns beacons. Beacons are a set of N maximally diverse molecules that score above the pProp threshold specified in the parameter file. For the first round of ChemSTEP, the best-scoring molecule from the seed set is assigned as the first beacon. Subsequent beacons would be molecules that score well (above pProp thresh), and are as diverse as possible from already assigned beacons. Greedy max-diversity selection. <br /> <br /> During assignment, ECFP4 Tanimoto distances between the assigned beacon and all potential beacons are calculated. The second beacon is the molecule with the maximum Td from beacon 1. Beacon 3 will be the molecule that has the highest Td from beacon 1 and beacon 2. This continues until the number of beacons specified in the parameter file is reached. In iterative rounds of ChemSTEP, the beacon selection is based on diversity from ALL PREVIOUSLY assigned beacons, including those in earlier rounds. <br /> <br /> <br /> '''2. ChainingLog sub-jobs:''' once beacons are assigned, ChemSTEP will launch a series of sub-jobs that calculate the Tanimoto distances of every molecule remaining in the library to the assigned beacons. Calculated distances to the NEAREST beacon (minimum-minimum Td) are updated in the mintddistrib_*.npy files within the /output directory. <br /> <br /> <br /> '''3. Gathering of SMILES for prioritization:''' When all Td calculations are completed, the algo will select N number of molecules for prioritization, the number of which is specified by the user in the parameter file. Molecules that are ''closest in chemical space'' to beacons are prioritized. I.e. if round size = 1 million, the 1 million molecules with the smallest min-Td to ANY beacon (current or previous rounds) are prioritized. chemstep_algo.log will provide a &quot;max-minTd&quot; for each round, which is the Tanimoto distance of the most dissimilar molecule prioritized to any one beacon per round. The prioritized molecules are output in /output/complete_info as smi_round_{}.smi <br /> <br /> What the user need: DOCKFILES, directories for (1) docking (2) building and (3) running ChemSTEP. '''All iterative rounds of ChemSTEP should be run in the SAME directory.''' <br /> <br /> '''1. Source ChemSTEP virtual environment on Wynton'''<br /> source /wynton/group/bks/work/shared/kholland/chemstep_auto/bin/activate<br /> <br /> <br /> '''2. Copy seed-set SDI file into your docking directory'''<br /> This directory should already contain your dockfiles, with INDOCK parameters set to your liking. In this step, we are copying in a split database index file (SDI) containing paths to bundles of db2 files. This seed set contains 100 million molecules sampled randomly from the total virtual library, currently 1.1 trillion molecules.<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_auto/scripts/96M_seeds.wynton.sdi .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_auto/scripts/rebuilt_seeds.wynton.sdi .<br /> <br /> because the seed set is large, the docking must be done in two separate chunks (100k jobs max per docking array), and the scores combined into one file later on. <br /> <br /> if you are interested in running on a library of 13.2B compounds from Enamine REAL, there are several seed set sizes available: 130k, 1.3M, 13M, and 26M. <br /> cp /wynton/group/bks/work/shared/kholland/chemstep_13B/130K_seeds.wynton.sdi .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_13B/1.3M_seeds.wynton.sdi .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_13B/13M_seeds.wynton.sdi .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_13B/26M_seeds.wynton.sdi .<br /> <br /> <br /> '''3. Dock seed set to your receptor of interest using DOCK 3.8''' /docking directions taken from docs.docking.org. This is meant to be done as you would do a normal LSD. <br /> export MOLECULES_DIR_TO_BIND=[outermost folder containing the molecules to dock, just your base docking directory]<br /> export DOCKFILES=[path to your dockfiles]<br /> export INPUT_FOLDER=[the folder containing your .sdi file(s)]<br /> export OUTPUT_FOLDER=[where you want the output ]<br /> <br /> /wynton/group/bks/work/bwhall61/needs_github/super_dock3r.sh<br /> <br /> Wait for docking to complete. Next, you must extract all molecule IDs and corresponding DOCK scores from above. To do so, run the following commands in the base docking directory containing your docking files and output folder, while logged into a dev node (do this in a screen!!!):<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_auto/get_scores.py .<br /> python get_scores.py 0<br /> <br /> <br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/get_scores.py .<br /> python get_scores.py 0<br /> <br /> <br /> If docking the seeds from the ZINC library, please combine your score.txt files from both SDI files. Rename this combined file 'scores_round_0.txt.'<br /> <br /> This script expects a directory named &quot;output*&quot; within the CWD. If your output from docking follows different organizational conventions, vim into get_scores.py and change the path. Successful output of the script will be a file named &quot;scores_round_0.txt&quot;. For iterative rounds, pass increasing numbers into the command line. i.e. When docking the first round of prioritized molecules, pass [1] for scores_round_1.txt. <br /> <br /> '''4. Convert scores and molecule IDS into NumPY arrays for ChemSTEP recognition. Requires ChemSTEP venv'''<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_auto/convert_scores_to_npy.py .<br /> python convert_scores_to_npy.py 0<br /> <br /> <br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/convert_scores_to_npy.py .<br /> python convert_scores_to_npy.py 0<br /> <br /> For python speed purposes, ChemSTEP requires that DOCK scores and IDs be give in the form of a NumPY array. In this step, we are taking our readable scores file and converting them for ChemSTEP recognition. This script expects a txt file with Molecule IDS and DOCK scores (no header). Use the same round number you used in step 3. As run above, this will output two files named scores_round_0.npy and indices_round_0.npy that contain line-matched indices (determined from Mol ID) and their respective docking scores. <br /> <br /> '''5. Enter into or make a directory to run ChemSTEP in. Copy in necessary files for initiating ChemSTEP: params.txt, run_chemstep.py and launch_chemstep_as_job.sh. Copy in your score and indices numpy files as well, which should be in your docking directory.'''<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/params.txt .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/run_chemstep.py .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/launch_chemstep_as_job.sh .<br /> <br /> <br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/params.txt .<br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/run_chemstep.py .<br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/launch_chemstep_as_job.sh .<br /> <br /> '''6. Edit params.txt file''' This ONLY needs to be edited for the initial round of ChemSTEP. The parameters outlined here will be carried through all rounds of ChemSTEP. <br /> seed_indices_file: /absoulte/path/to/your/indices_round_0.npy<br /> seed_scores_file: /absolute/path/to/your/scores_round_0.npy<br /> hit_pprop: 5<br /> n_docked_per_round: 10000000<br /> max_beacons: 150<br /> max_n_rounds: 250<br /> <br /> Within params.txt, add the absolute paths to the ChemSTEP-readable score and indices NumPY arrays. The rest of the values within the params.txt file are left to the discretion of the user, with some considerations below. <br /> <br /> '''pProp:''' this value will define what is considered a &quot;virtual hit&quot; in your campaign. pProp is defined as the -log(rank %) of a molecule within the total library score-distribution. For example, a pProp of 4 in the 1.1T XReal space is equivalent the top 0.01% of the library, corresponding to the top 110 million molecules. pProp 5 = 0.001% = 11 million virtual hits, etc. From the seed set, ChemSTEP will estimate a DOCK score value in kcal/mol that associated with the lower-limit of your desired pProp zone. Any molecules that score better than the threshold is considered a virtual hit in your campaign. Be mindful of seed set size when choosing desired pProp. Our suggestion is that the size of the seed set should be at least 10^(pProp +2).<br /> <br /> '''n_docked_per_round:''' this number is the desired molecules for prioritization per round. When choosing this value, note that this number of molecules must be built and docked in between each round of ChemSTEP. Prioritizing many molecules will slow building and docking speeds, and coupled with few beacons (below) may lead to decreased diversity. Prioritizing too few molecules may result in slower virtual hit recovery. Round size does not significantly impact the algorithm running time.** <br /> <br /> '''max_beacons:''' this is the number of diverse, well-scoring molecules ChemSTEP will use to guide prioritization per round. All molecules that score better than the assigned pProp threshold may be considered for beacons. By default, ChemSTEP chooses each set of beacons to be as maximally diverse as possible. Choosing too many beacons may result in decreased diversity between beacons, but too few beacons could hinder space exploration. If not enough molecules score above your pProp threshold, ChemSTEP may assign fewer beacons than the assigned value. <br /> <br /> '''max_n_rounds:''' if prospectively running ChemSTEP, as outlined in this wiki, no need to worry about this parameter.<br /> <br /> <br /> There should be no need to edit run_chemstep.py or the SGE wrapper script. At this time, we strongly suggest running ChemSTEP as a job array with 16 or 32 CPU slots requested. The number of cores must be specified when calling CSAlgo (in run_chemstep.py) and the SGE wrapper(launch_chemstep_as_job.sh). <br /> <br /> <br /> '''7. Run ChemSTEP''' with the following command: <br /> qsub launch_chemstep_as_job.sh <br /> <br /> <br /> This will launch the main ChemSTEP job to the SGE scheduler. Check your job status with the 'qstat' command. The algorithm will read the score and indices files provided, calculate pProp (for round 1) and assign beacons. As this job runs, it will launch a subsequent job array. These sub-jobs are the Chaining step of ChemSTEP, where each parallel worker calculates the Tanimoto distances of 100 million molecules from the virtual library to the beacons. These distances are then written to the files within the /output directory. When these jobs complete, the main job will read through all Tanimoto distances and pull molecules for prioritization. <br /> <br /> Running ChemSTEP successfully will result in the output of a SMILES file within output/complete_info/. You will also get (1) chemstep_algo.log and (2) chemstep_submission.log in the working directory after job submission. chemstep_algo.log contains the running output of the algorithm in a readable format, with timestamps, including the DOCK score associated with your desired pProp. chemstep_submission.log contains the output of ChemSTEP, as well as any information output by the SGE submission system (python errors, cluster issues, tracebacks if failed). These files will be updated with any information with iterative rounds of ChemSTEP run in the same directory. '''Visually inspect these files after each round of ChemSTEP to ensure the algorithm has picked your desired number of beacons and things ran smoothly.'''<br /> <br /> '''Troubleshooting:''' if no jobs are running and there is no SMI file, check the chemstep_submission.log first. Any traceback error or SGE error should give you some idea of why ChemSTEP failed. Some errors may be due to SGE or Wynton issues. If directed, look at a few .out files within /output/jobs/ . If a ChemSTEP run fails on your FIRST run, delete the output and log files, fix what needs to be fixed, and rerun. Errors in subsequent rounds during the chaining step can potentially corrupt chaining files within the /output directory that are needed for prioritization. At the very least, you will definitely have duplicates of beacons and information written to the log files. At this time, if your run fails during iterative rounds, it's best to start from the beginning. <br /> <br /> Prioritized molecules should be built, docked, and their scores fed back into ChemSTEP. More detailed instructions below: <br /> <br /> '''8. Build prioritized molecules (DOCK 3.8).''' /taken from docs.docking.org <br /> source /wynton/group/bks/soft/DOCK-3.8.5/env.sh<br /> <br /> python /wynton/group/bks/soft/DOCK-3.8.5/DOCK3.8/zinc22-3d/submit/submit_building_docker.py --output_folder building_output --bundle_size 1000 --minutes_per_mol 5 --skip_name_check --scheduler sge --container_software apptainer --container_path_or_name /wynton/group/bks/soft/DOCK-3.8.5/building_pipeline.sif smi_round_{}.smi<br /> <br /> When building has completed, you must write an SDI file with the complete paths to each built bundle and dock. you can do this with:<br /> find /path/to/your/building_output -name &quot;bundle.db2.tgz&quot; -type f &gt; round_{}.wynton.sdi<br /> <br /> <br /> '''Be sure to change the INDOCK file to save only poses that meet your score pProp score threshold calculated by ChemSTEP!''' Retrieve docking scores as convert to NumPy arrays as outlined above, updating the round number when running get_scores.py and convert_scores_to_npy.py. Copy new score and indices files into the directory you ran ChemSTEP in. If you are following along as a tutorial, you should have scores_round_1.npy and indices_round_1.npy from the previous step (from FIRST round of ChemSTEP prioritization). <br /> <br /> <br /> '''9. Set up for iterative rounds of ChemSTEP'''<br /> <br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/run_chemstep_iterative.py .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/launch_chemstep_iterative.sh .<br /> <br /> <br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/run_chemstep_iterative.py .<br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/launch_chemstep_iterative.sh .<br /> <br /> '''10. Run ChemSTEP'''<br /> qsub launch_chemstep_as_job.sh [round number]<br /> <br /> For the first iterative round, the round number is [2], and should increase by one for every subsequent round of ChemSTEP. The output will be smi_round_****.smi file. Repeat steps 8-10 for as many rounds as needed. The performance is reported in outputy/complete_info/run_summary.df, which contains the number of beacons selected, the number of molecules docked, the number of hits found, the distance threshold for the selected molecules to dock, and the last added beacon's distance to all previous beacons.<br /> <br /> <br /> '''AUTOMATIC CHEMSTEP RUNNING DIRECTIONS'''<br /> <br /> This version of ChemSTEP will automatically submit building and docking jobs inbetween rounds of ChemSTEP. <br /> <br /> '''1. DOCK the seed set to your receptor as outlined above in steps 1-4'''<br /> <br /> successful output will be a scores_round_0.npy and indices_round_0.npy. <br /> <br /> <br /> '''2. Copy necessary scripts into your working ChemSTEP directory'''<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_auto/scripts/params.txt .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_auto/scripts/run_chemstep.py .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_auto/scripts/launch_chemstep_as_job.sh .<br /> <br /> <br /> '''3. Edit run_chemstep.py to add the path to your dockfiles'''<br /> <br /> the algorithm needs to know where your dockfiles live in order to submit docking jobs. Make sure all INDOCK parameters are set correctly. Use the full path to the dockfiles. <br /> <br /> <br /> '''4. Edit params.txt''' The parameters outlined here will be carried through all rounds of ChemSTEP. <br /> seed_indices_file: /absoulte/path/to/your/indices_round_0.npy<br /> seed_scores_file: /absolute/path/to/your/scores_round_0.npy<br /> hit_pprop: 5<br /> bundle_size: 1000<br /> n_docked_per_round: 10000000<br /> max_beacons: 150<br /> max_n_rounds: 250<br /> <br /> <br /> Within params.txt, add the absolute paths to the ChemSTEP-readable score and indices NumPY arrays. The rest of the values within the params.txt file are left to the discretion of the user. The '''bundle_size:''' parameter is required for auto docking mode of ChemSTEP. This is the number of molecules submitted at a job for building. <br /> <br /> <br /> '''5. Launch ChemSTEP'''<br /> <br /> qsub launch_chemstep_as_job.sh <br /> <br /> <br /> '''Troubleshooting:''' if your job errors out or stops running at any point, check the ''chemstep_submission.log'' file in the working directory. <br /> <br /> <br /> <br /> '''Helper scripts:''' a running list of scripts we have been using for analysis and visualization. <br /> <br /> to get SMILES for beacons: <br /> <br /> python /wynton/group/bks/work/bwhall61/needs_github/get_beacon_smi.py --beacon_df_path /path/to/your/chemstep/output/complete_info/beacons.df --n_workers 6 --outfile beacon_smiles.smi<br /> <br /> <br /> ** anecdotally true.</div>

Kholland http://wiki.docking.org/index.php?title=Running_ChemSTEP&diff=17011 2025-12-15T21:15:37Z

<p>Kholland: </p> <hr /> <div>last update: dec 11 2025 katie. current ver = 0.3.1.5 (automatic resubmission of failed SGE jobs). <br /> <br /> ChemSTEP (Chemical Space Traversal and Exploration Procedure) is an open-source, transparent acceleration algorithm for molecular docking capable of dealing with virtual libraries of several trillion compounds. This wiki page is a guide for BKS lab members to run ChemSTEP on Wynton HPC, using the current version of InifiSee XReal library (1.1T) or 13.3B library from Enamine REAL. '''for detailed instructions on the 13B space, see 'Running ChemSTEP on the 13B space' wiki page'''. For more general use directions, please refer to [ChemSTEP Read-the-Docs]. <br /> <br /> At a high-level, ChemSTEP is an iterative process that identifies molecules from the larger virtual library to prioritize for docking. First, we identify a random sample of the total library (termed &quot;seed set&quot;, round zero) and dock those molecules to the target of interest. From this seed set, we can calculate total-library pProp values (-log rank percentages) and the number of &quot;virtual hits&quot; in the total library (high-scoring molecules). ChemSTEP will identify a set of maximally diverse molecules that score above the desired pProp threshold (&quot;beacons&quot;) from the seed set. These beacons guide prioritization, where molecules chosen and output by ChemSTEP are close in chemical space to the beacons. Prioritized molecules are then built, docked, and returned to ChemSTEP for a second round of prioritization. This process is iterated until you reach desired virtual hit recovery, or you are no longer recovering virtual hits. <br /> <br /> <br /> Running the ChemSTEP algorithm on Wynton takes place in loosely three steps: (1) submission of the main ChemSTEP job, (2) ChainingLog sub-job array, and (3) gathering of SMILES for prioritization. <br /> <br /> '''1. Main ChemSTEP job:''' reads input files (DOCK scores and line-matched indices NumPy arrays) and assigns beacons. Beacons are a set of N maximally diverse molecules that score above the pProp threshold specified in the parameter file. For the first round of ChemSTEP, the best-scoring molecule from the seed set is assigned as the first beacon. Subsequent beacons would be molecules that score well (above pProp thresh), and are as diverse as possible from already assigned beacons. Greedy max-diversity selection. <br /> <br /> During assignment, ECFP4 Tanimoto distances between the assigned beacon and all potential beacons are calculated. The second beacon is the molecule with the maximum Td from beacon 1. Beacon 3 will be the molecule that has the highest Td from beacon 1 and beacon 2. This continues until the number of beacons specified in the parameter file is reached. In iterative rounds of ChemSTEP, the beacon selection is based on diversity from ALL PREVIOUSLY assigned beacons, including those in earlier rounds. <br /> <br /> <br /> '''2. ChainingLog sub-jobs:''' once beacons are assigned, ChemSTEP will launch a series of sub-jobs that calculate the Tanimoto distances of every molecule remaining in the library to the assigned beacons. Calculated distances to the NEAREST beacon (minimum-minimum Td) are updated in the mintddistrib_*.npy files within the /output directory. <br /> <br /> <br /> '''3. Gathering of SMILES for prioritization:''' When all Td calculations are completed, the algo will select N number of molecules for prioritization, the number of which is specified by the user in the parameter file. Molecules that are ''closest in chemical space'' to beacons are prioritized. I.e. if round size = 1 million, the 1 million molecules with the smallest min-Td to ANY beacon (current or previous rounds) are prioritized. chemstep_algo.log will provide a &quot;max-minTd&quot; for each round, which is the Tanimoto distance of the most dissimilar molecule prioritized to any one beacon per round. The prioritized molecules are output in /output/complete_info as smi_round_{}.smi <br /> <br /> What the user need: DOCKFILES, directories for (1) docking (2) building and (3) running ChemSTEP. '''All iterative rounds of ChemSTEP should be run in the SAME directory.''' <br /> <br /> '''1. Source ChemSTEP virtual environment on Wynton'''<br /> source /wynton/group/bks/work/shared/kholland/chemstep_0.3.1.5/bin/activate<br /> <br /> <br /> '''2. Copy seed-set SDI file into your docking directory'''<br /> This directory should already contain your dockfiles, with INDOCK parameters set to your liking. In this step, we are copying in a split database index file (SDI) containing paths to bundles of db2 files. This seed set contains 100 million molecules sampled randomly from the total virtual library, currently 1.1 trillion molecules.<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/XR_seed_set_v0p0.wynton.sdi .<br /> <br /> <br /> if you are interested in running on a library of 13.2B compounds from Enamine REAL, there are several seed set sizes available: 130k, 1.3M, 13M, and 26M. <br /> cp /wynton/group/bks/work/shared/kholland/chemstep_13B/130K_seeds.wynton.sdi .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_13B/1.3M_seeds.wynton.sdi .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_13B/13M_seeds.wynton.sdi .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_13B/26M_seeds.wynton.sdi .<br /> <br /> <br /> '''3. Dock seed set to your receptor of interest using DOCK 3.8''' /docking directions taken from docs.docking.org. This is meant to be done as you would do a normal LSD. <br /> export MOLECULES_DIR_TO_BIND=[outermost folder containing the molecules to dock, just your base docking directory]<br /> export DOCKFILES=[path to your dockfiles]<br /> export INPUT_FOLDER=[the folder containing your .sdi file(s)]<br /> export OUTPUT_FOLDER=[where you want the output ]<br /> <br /> /wynton/group/bks/work/bwhall61/needs_github/super_dock3r.sh<br /> <br /> Wait for docking to complete. Next, you must extract all molecule IDs and corresponding DOCK scores from above. To do so, run the following commands in the base docking directory containing your docking files and output folder, while logged into a dev node (do this in a screen!!!):<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/get_scores.py .<br /> python get_scores.py 0<br /> <br /> <br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/get_scores.py .<br /> python get_scores.py 0<br /> <br /> <br /> This script expects a directory named &quot;output*&quot; within the CWD. If your output from docking follows different organizational conventions, vim into get_scores.py and change the path. Successful output of the script will be a file named &quot;scores_round_0.txt&quot;. For iterative rounds, pass increasing numbers into the command line. i.e. When docking the first round of prioritized molecules, pass [1] for scores_round_1.txt. <br /> <br /> '''4. Convert scores and molecule IDS into NumPY arrays for ChemSTEP recognition. Requires ChemSTEP venv'''<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/convert_scores_to_npy.py .<br /> python convert_scores_to_npy.py 0<br /> <br /> <br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/convert_scores_to_npy.py .<br /> python convert_scores_to_npy.py 0<br /> <br /> For python speed purposes, ChemSTEP requires that DOCK scores and IDs be give in the form of a NumPY array. In this step, we are taking our readable scores file and converting them for ChemSTEP recognition. This script expects a txt file with Molecule IDS and DOCK scores (no header). Use the same round number you used in step 3. As run above, this will output two files named scores_round_0.npy and indices_round_0.npy that contain line-matched indices (determined from Mol ID) and their respective docking scores. <br /> <br /> '''5. Enter into or make a directory to run ChemSTEP in. Copy in necessary files for initiating ChemSTEP: params.txt, run_chemstep.py and launch_chemstep_as_job.sh. Copy in your score and indices numpy files as well, which should be in your docking directory.'''<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/params.txt .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/run_chemstep.py .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/launch_chemstep_as_job.sh .<br /> <br /> <br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/params.txt .<br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/run_chemstep.py .<br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/launch_chemstep_as_job.sh .<br /> <br /> '''6. Edit params.txt file''' This ONLY needs to be edited for the initial round of ChemSTEP. The parameters outlined here will be carried through all rounds of ChemSTEP. <br /> seed_indices_file: /absoulte/path/to/your/indices_round_0.npy<br /> seed_scores_file: /absolute/path/to/your/scores_round_0.npy<br /> hit_pprop: 5<br /> n_docked_per_round: 10000000<br /> max_beacons: 150<br /> max_n_rounds: 250<br /> <br /> Within params.txt, add the absolute paths to the ChemSTEP-readable score and indices NumPY arrays. The rest of the values within the params.txt file are left to the discretion of the user, with some considerations below. <br /> <br /> '''pProp:''' this value will define what is considered a &quot;virtual hit&quot; in your campaign. pProp is defined as the -log(rank %) of a molecule within the total library score-distribution. For example, a pProp of 4 in the 1.1T XReal space is equivalent the top 0.01% of the library, corresponding to the top 110 million molecules. pProp 5 = 0.001% = 11 million virtual hits, etc. From the seed set, ChemSTEP will estimate a DOCK score value in kcal/mol that associated with the lower-limit of your desired pProp zone. Any molecules that score better than the threshold is considered a virtual hit in your campaign. Be mindful of seed set size when choosing desired pProp. Our suggestion is that the size of the seed set should be at least 10^(pProp +2).<br /> <br /> '''n_docked_per_round:''' this number is the desired molecules for prioritization per round. When choosing this value, note that this number of molecules must be built and docked in between each round of ChemSTEP. Prioritizing many molecules will slow building and docking speeds, and coupled with few beacons (below) may lead to decreased diversity. Prioritizing too few molecules may result in slower virtual hit recovery. Round size does not significantly impact the algorithm running time.** <br /> <br /> '''max_beacons:''' this is the number of diverse, well-scoring molecules ChemSTEP will use to guide prioritization per round. All molecules that score better than the assigned pProp threshold may be considered for beacons. By default, ChemSTEP chooses each set of beacons to be as maximally diverse as possible. Choosing too many beacons may result in decreased diversity between beacons, but too few beacons could hinder space exploration. If not enough molecules score above your pProp threshold, ChemSTEP may assign fewer beacons than the assigned value. <br /> <br /> '''max_n_rounds:''' if prospectively running ChemSTEP, as outlined in this wiki, no need to worry about this parameter.<br /> <br /> <br /> There should be no need to edit run_chemstep.py or the SGE wrapper script. At this time, we strongly suggest running ChemSTEP as a job array with 16 or 32 CPU slots requested. The number of cores must be specified when calling CSAlgo (in run_chemstep.py) and the SGE wrapper(launch_chemstep_as_job.sh). <br /> <br /> <br /> '''7. Run ChemSTEP''' with the following command: <br /> qsub launch_chemstep_as_job.sh <br /> <br /> <br /> This will launch the main ChemSTEP job to the SGE scheduler. Check your job status with the 'qstat' command. The algorithm will read the score and indices files provided, calculate pProp (for round 1) and assign beacons. As this job runs, it will launch a subsequent job array. These sub-jobs are the Chaining step of ChemSTEP, where each parallel worker calculates the Tanimoto distances of 100 million molecules from the virtual library to the beacons. These distances are then written to the files within the /output directory. When these jobs complete, the main job will read through all Tanimoto distances and pull molecules for prioritization. <br /> <br /> Running ChemSTEP successfully will result in the output of a SMILES file within output/complete_info/. You will also get (1) chemstep_algo.log and (2) chemstep_submission.log in the working directory after job submission. chemstep_algo.log contains the running output of the algorithm in a readable format, with timestamps, including the DOCK score associated with your desired pProp. chemstep_submission.log contains the output of ChemSTEP, as well as any information output by the SGE submission system (python errors, cluster issues, tracebacks if failed). These files will be updated with any information with iterative rounds of ChemSTEP run in the same directory. '''Visually inspect these files after each round of ChemSTEP to ensure the algorithm has picked your desired number of beacons and things ran smoothly.'''<br /> <br /> '''Troubleshooting:''' if no jobs are running and there is no SMI file, check the chemstep_submission.log first. Any traceback error or SGE error should give you some idea of why ChemSTEP failed. Some errors may be due to SGE or Wynton issues. If directed, look at a few .out files within /output/jobs/ . If a ChemSTEP run fails on your FIRST run, delete the output and log files, fix what needs to be fixed, and rerun. Errors in subsequent rounds during the chaining step can potentially corrupt chaining files within the /output directory that are needed for prioritization. At the very least, you will definitely have duplicates of beacons and information written to the log files. At this time, if your run fails during iterative rounds, it's best to start from the beginning. <br /> <br /> Prioritized molecules should be built, docked, and their scores fed back into ChemSTEP. More detailed instructions below: <br /> <br /> '''8. Build prioritized molecules (DOCK 3.8).''' /taken from docs.docking.org <br /> source /wynton/group/bks/soft/DOCK-3.8.5/env.sh<br /> <br /> python /wynton/group/bks/soft/DOCK-3.8.5/DOCK3.8/zinc22-3d/submit/submit_building_docker.py --output_folder building_output --bundle_size 1000 --minutes_per_mol 5 --skip_name_check --scheduler sge --container_software apptainer --container_path_or_name /wynton/group/bks/soft/DOCK-3.8.5/building_pipeline.sif smi_round_{}.smi<br /> <br /> When building has completed, you must write an SDI file with the complete paths to each built bundle and dock. you can do this with:<br /> find /path/to/your/building_output -name &quot;bundle.db2.tgz&quot; -type f &gt; round_{}.wynton.sdi<br /> <br /> <br /> '''Be sure to change the INDOCK file to save only poses that meet your score pProp score threshold calculated by ChemSTEP!''' Retrieve docking scores as convert to NumPy arrays as outlined above, updating the round number when running get_scores.py and convert_scores_to_npy.py. Copy new score and indices files into the directory you ran ChemSTEP in. If you are following along as a tutorial, you should have scores_round_1.npy and indices_round_1.npy from the previous step (from FIRST round of ChemSTEP prioritization). <br /> <br /> <br /> '''9. Set up for iterative rounds of ChemSTEP'''<br /> <br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/run_chemstep_iterative.py .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/launch_chemstep_iterative.sh .<br /> <br /> <br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/run_chemstep_iterative.py .<br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/launch_chemstep_iterative.sh .<br /> <br /> '''10. Run ChemSTEP'''<br /> qsub launch_chemstep_as_job.sh [round number]<br /> <br /> For the first iterative round, the round number is [2], and should increase by one for every subsequent round of ChemSTEP. The output will be smi_round_****.smi file. Repeat steps 8-10 for as many rounds as needed. The performance is reported in outputy/complete_info/run_summary.df, which contains the number of beacons selected, the number of molecules docked, the number of hits found, the distance threshold for the selected molecules to dock, and the last added beacon's distance to all previous beacons.<br /> <br /> <br /> '''AUTOMATIC CHEMSTEP RUNNING DIRECTIONS'''<br /> <br /> This version of ChemSTEP will automatically submit building and docking jobs inbetween rounds of ChemSTEP. <br /> <br /> '''1. DOCK the seed set to your receptor as outlined above in steps 1-4'''<br /> <br /> successful output will be a scores_round_0.npy and indices_round_0.npy. <br /> <br /> <br /> '''2. Copy necessary scripts into your working ChemSTEP directory'''<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_auto/scripts/params.txt .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_auto/scripts/run_chemstep.py .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_auto/scripts/launch_chemstep_as_job.sh .<br /> <br /> <br /> '''3. Edit run_chemstep.py to add the path to your dockfiles'''<br /> <br /> the algorithm needs to know where your dockfiles live in order to submit docking jobs. Make sure all INDOCK parameters are set correctly. Use the full path to the dockfiles. <br /> <br /> <br /> '''4. Edit params.txt''' The parameters outlined here will be carried through all rounds of ChemSTEP. <br /> seed_indices_file: /absoulte/path/to/your/indices_round_0.npy<br /> seed_scores_file: /absolute/path/to/your/scores_round_0.npy<br /> hit_pprop: 5<br /> bundle_size: 1000<br /> n_docked_per_round: 10000000<br /> max_beacons: 150<br /> max_n_rounds: 250<br /> <br /> <br /> Within params.txt, add the absolute paths to the ChemSTEP-readable score and indices NumPY arrays. The rest of the values within the params.txt file are left to the discretion of the user. The '''bundle_size:''' parameter is required for auto docking mode of ChemSTEP. This is the number of molecules submitted at a job for building. <br /> <br /> <br /> '''5. Launch ChemSTEP'''<br /> <br /> qsub launch_chemstep_as_job.sh <br /> <br /> <br /> '''Troubleshooting:''' if your job errors out or stops running at any point, check the ''chemstep_submission.log'' file in the working directory. <br /> <br /> <br /> <br /> '''Helper scripts:''' a running list of scripts we have been using for analysis and visualization. <br /> <br /> to get SMILES for beacons: <br /> <br /> python /wynton/group/bks/work/bwhall61/needs_github/get_beacon_smi.py --beacon_df_path /path/to/your/chemstep/output/complete_info/beacons.df --n_workers 6 --outfile beacon_smiles.smi<br /> <br /> <br /> ** anecdotally true.</div>

Kholland http://wiki.docking.org/index.php?title=Running_ChemSTEP&diff=17010 2025-12-15T20:52:45Z

<p>Kholland: </p> <hr /> <div>last update: dec 11 2025 katie. current ver = 0.3.1.5 (automatic resubmission of failed SGE jobs). <br /> <br /> ChemSTEP (Chemical Space Traversal and Exploration Procedure) is an open-source, transparent acceleration algorithm for molecular docking capable of dealing with virtual libraries of several trillion compounds. This wiki page is a guide for BKS lab members to run ChemSTEP on Wynton HPC, using the current version of InifiSee XReal library (1.1T) or 13.3B library from Enamine REAL. '''for detailed instructions on the 13B space, see 'Running ChemSTEP on the 13B space' wiki page'''. For more general use directions, please refer to [ChemSTEP Read-the-Docs]. <br /> <br /> At a high-level, ChemSTEP is an iterative process that identifies molecules from the larger virtual library to prioritize for docking. First, we identify a random sample of the total library (termed &quot;seed set&quot;, round zero) and dock those molecules to the target of interest. From this seed set, we can calculate total-library pProp values (-log rank percentages) and the number of &quot;virtual hits&quot; in the total library (high-scoring molecules). ChemSTEP will identify a set of maximally diverse molecules that score above the desired pProp threshold (&quot;beacons&quot;) from the seed set. These beacons guide prioritization, where molecules chosen and output by ChemSTEP are close in chemical space to the beacons. Prioritized molecules are then built, docked, and returned to ChemSTEP for a second round of prioritization. This process is iterated until you reach desired virtual hit recovery, or you are no longer recovering virtual hits. <br /> <br /> <br /> Running the ChemSTEP algorithm on Wynton takes place in loosely three steps: (1) submission of the main ChemSTEP job, (2) ChainingLog sub-job array, and (3) gathering of SMILES for prioritization. <br /> <br /> '''1. Main ChemSTEP job:''' reads input files (DOCK scores and line-matched indices NumPy arrays) and assigns beacons. Beacons are a set of N maximally diverse molecules that score above the pProp threshold specified in the parameter file. For the first round of ChemSTEP, the best-scoring molecule from the seed set is assigned as the first beacon. Subsequent beacons would be molecules that score well (above pProp thresh), and are as diverse as possible from already assigned beacons. Greedy max-diversity selection. <br /> <br /> During assignment, ECFP4 Tanimoto distances between the assigned beacon and all potential beacons are calculated. The second beacon is the molecule with the maximum Td from beacon 1. Beacon 3 will be the molecule that has the highest Td from beacon 1 and beacon 2. This continues until the number of beacons specified in the parameter file is reached. In iterative rounds of ChemSTEP, the beacon selection is based on diversity from ALL PREVIOUSLY assigned beacons, including those in earlier rounds. <br /> <br /> <br /> '''2. ChainingLog sub-jobs:''' once beacons are assigned, ChemSTEP will launch a series of sub-jobs that calculate the Tanimoto distances of every molecule remaining in the library to the assigned beacons. Calculated distances to the NEAREST beacon (minimum-minimum Td) are updated in the mintddistrib_*.npy files within the /output directory. <br /> <br /> <br /> '''3. Gathering of SMILES for prioritization:''' When all Td calculations are completed, the algo will select N number of molecules for prioritization, the number of which is specified by the user in the parameter file. Molecules that are ''closest in chemical space'' to beacons are prioritized. I.e. if round size = 1 million, the 1 million molecules with the smallest min-Td to ANY beacon (current or previous rounds) are prioritized. chemstep_algo.log will provide a &quot;max-minTd&quot; for each round, which is the Tanimoto distance of the most dissimilar molecule prioritized to any one beacon per round. The prioritized molecules are output in /output/complete_info as smi_round_{}.smi <br /> <br /> What the user need: DOCKFILES, directories for (1) docking (2) building and (3) running ChemSTEP. '''All iterative rounds of ChemSTEP should be run in the SAME directory.''' <br /> <br /> '''1. Source ChemSTEP virtual environment on Wynton'''<br /> source /wynton/group/bks/work/shared/kholland/chemstep_0.3.1.5/bin/activate<br /> <br /> <br /> '''2. Copy seed-set SDI file into your docking directory'''<br /> This directory should already contain your dockfiles, with INDOCK parameters set to your liking. In this step, we are copying in a split database index file (SDI) containing paths to bundles of db2 files. This seed set contains 100 million molecules sampled randomly from the total virtual library, currently 1.1 trillion molecules.<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/XR_seed_set_v0p0.wynton.sdi .<br /> <br /> <br /> if you are interested in running on a library of 13.2B compounds from Enamine REAL, there are several seed set sizes available: 130k, 1.3M, 13M, and 26M. <br /> cp /wynton/group/bks/work/shared/kholland/chemstep_13B/130K_seeds.wynton.sdi .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_13B/1.3M_seeds.wynton.sdi .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_13B/13M_seeds.wynton.sdi .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_13B/26M_seeds.wynton.sdi .<br /> <br /> <br /> '''3. Dock seed set to your receptor of interest using DOCK 3.8''' /docking directions taken from docs.docking.org. This is meant to be done as you would do a normal LSD. <br /> export MOLECULES_DIR_TO_BIND=[outermost folder containing the molecules to dock, just your base docking directory]<br /> export DOCKFILES=[path to your dockfiles]<br /> export INPUT_FOLDER=[the folder containing your .sdi file(s)]<br /> export OUTPUT_FOLDER=[where you want the output ]<br /> <br /> /wynton/group/bks/work/bwhall61/needs_github/super_dock3r.sh<br /> <br /> Wait for docking to complete. Next, you must extract all molecule IDs and corresponding DOCK scores from above. To do so, run the following commands in the base docking directory containing your docking files and output folder, while logged into a dev node (do this in a screen!!!):<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/get_scores.py .<br /> python get_scores.py 0<br /> <br /> <br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/get_scores.py .<br /> python get_scores.py 0<br /> <br /> <br /> This script expects a directory named &quot;output*&quot; within the CWD. If your output from docking follows different organizational conventions, vim into get_scores.py and change the path. Successful output of the script will be a file named &quot;scores_round_0.txt&quot;. For iterative rounds, pass increasing numbers into the command line. i.e. When docking the first round of prioritized molecules, pass [1] for scores_round_1.txt. <br /> <br /> '''4. Convert scores and molecule IDS into NumPY arrays for ChemSTEP recognition. Requires ChemSTEP venv'''<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/convert_scores_to_npy.py .<br /> python convert_scores_to_npy.py 0<br /> <br /> <br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/convert_scores_to_npy.py .<br /> python convert_scores_to_npy.py 0<br /> <br /> For python speed purposes, ChemSTEP requires that DOCK scores and IDs be give in the form of a NumPY array. In this step, we are taking our readable scores file and converting them for ChemSTEP recognition. This script expects a txt file with Molecule IDS and DOCK scores (no header). Use the same round number you used in step 3. As run above, this will output two files named scores_round_0.npy and indices_round_0.npy that contain line-matched indices (determined from Mol ID) and their respective docking scores. <br /> <br /> '''5. Enter into or make a directory to run ChemSTEP in. Copy in necessary files for initiating ChemSTEP: params.txt, run_chemstep.py and launch_chemstep_as_job.sh. Copy in your score and indices numpy files as well, which should be in your docking directory.'''<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/params.txt .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/run_chemstep.py .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/launch_chemstep_as_job.sh .<br /> <br /> <br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/params.txt .<br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/run_chemstep.py .<br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/launch_chemstep_as_job.sh .<br /> <br /> '''6. Edit params.txt file''' This ONLY needs to be edited for the initial round of ChemSTEP. The parameters outlined here will be carried through all rounds of ChemSTEP. <br /> seed_indices_file: /absoulte/path/to/your/indices_round_0.npy<br /> seed_scores_file: /absolute/path/to/your/scores_round_0.npy<br /> hit_pprop: 5<br /> n_docked_per_round: 10000000<br /> max_beacons: 150<br /> max_n_rounds: 250<br /> <br /> Within params.txt, add the absolute paths to the ChemSTEP-readable score and indices NumPY arrays. The rest of the values within the params.txt file are left to the discretion of the user, with some considerations below. <br /> <br /> '''pProp:''' this value will define what is considered a &quot;virtual hit&quot; in your campaign. pProp is defined as the -log(rank %) of a molecule within the total library score-distribution. For example, a pProp of 4 in the 1.1T XReal space is equivalent the top 0.01% of the library, corresponding to the top 110 million molecules. pProp 5 = 0.001% = 11 million virtual hits, etc. From the seed set, ChemSTEP will estimate a DOCK score value in kcal/mol that associated with the lower-limit of your desired pProp zone. Any molecules that score better than the threshold is considered a virtual hit in your campaign. Be mindful of seed set size when choosing desired pProp. Our suggestion is that the size of the seed set should be at least 10^(pProp +2).<br /> <br /> '''n_docked_per_round:''' this number is the desired molecules for prioritization per round. When choosing this value, note that this number of molecules must be built and docked in between each round of ChemSTEP. Prioritizing many molecules will slow building and docking speeds, and coupled with few beacons (below) may lead to decreased diversity. Prioritizing too few molecules may result in slower virtual hit recovery. Round size does not significantly impact the algorithm running time.** <br /> <br /> '''max_beacons:''' this is the number of diverse, well-scoring molecules ChemSTEP will use to guide prioritization per round. All molecules that score better than the assigned pProp threshold may be considered for beacons. By default, ChemSTEP chooses each set of beacons to be as maximally diverse as possible. Choosing too many beacons may result in decreased diversity between beacons, but too few beacons could hinder space exploration. If not enough molecules score above your pProp threshold, ChemSTEP may assign fewer beacons than the assigned value. <br /> <br /> '''max_n_rounds:''' if prospectively running ChemSTEP, as outlined in this wiki, no need to worry about this parameter.<br /> <br /> <br /> There should be no need to edit run_chemstep.py or the SGE wrapper script. At this time, we strongly suggest running ChemSTEP as a job array with 16 or 32 CPU slots requested. The number of cores must be specified when calling CSAlgo (in run_chemstep.py) and the SGE wrapper(launch_chemstep_as_job.sh). <br /> <br /> <br /> '''7. Run ChemSTEP''' with the following command: <br /> qsub launch_chemstep_as_job.sh <br /> <br /> <br /> This will launch the main ChemSTEP job to the SGE scheduler. Check your job status with the 'qstat' command. The algorithm will read the score and indices files provided, calculate pProp (for round 1) and assign beacons. As this job runs, it will launch a subsequent job array. These sub-jobs are the Chaining step of ChemSTEP, where each parallel worker calculates the Tanimoto distances of 100 million molecules from the virtual library to the beacons. These distances are then written to the files within the /output directory. When these jobs complete, the main job will read through all Tanimoto distances and pull molecules for prioritization. <br /> <br /> Running ChemSTEP successfully will result in the output of a SMILES file within output/complete_info/. You will also get (1) chemstep_algo.log and (2) chemstep_submission.log in the working directory after job submission. chemstep_algo.log contains the running output of the algorithm in a readable format, with timestamps, including the DOCK score associated with your desired pProp. chemstep_submission.log contains the output of ChemSTEP, as well as any information output by the SGE submission system (python errors, cluster issues, tracebacks if failed). These files will be updated with any information with iterative rounds of ChemSTEP run in the same directory. '''Visually inspect these files after each round of ChemSTEP to ensure the algorithm has picked your desired number of beacons and things ran smoothly.'''<br /> <br /> '''Troubleshooting:''' if no jobs are running and there is no SMI file, check the chemstep_submission.log first. Any traceback error or SGE error should give you some idea of why ChemSTEP failed. Some errors may be due to SGE or Wynton issues. If directed, look at a few .out files within /output/jobs/ . If a ChemSTEP run fails on your FIRST run, delete the output and log files, fix what needs to be fixed, and rerun. Errors in subsequent rounds during the chaining step can potentially corrupt chaining files within the /output directory that are needed for prioritization. At the very least, you will definitely have duplicates of beacons and information written to the log files. At this time, if your run fails during iterative rounds, it's best to start from the beginning. <br /> <br /> Prioritized molecules should be built, docked, and their scores fed back into ChemSTEP. More detailed instructions below: <br /> <br /> '''8. Build prioritized molecules (DOCK 3.8).''' /taken from docs.docking.org <br /> source /wynton/group/bks/soft/DOCK-3.8.5/env.sh<br /> <br /> python /wynton/group/bks/soft/DOCK-3.8.5/DOCK3.8/zinc22-3d/submit/submit_building_docker.py --output_folder building_output --bundle_size 1000 --minutes_per_mol 5 --skip_name_check --scheduler sge --container_software apptainer --container_path_or_name /wynton/group/bks/soft/DOCK-3.8.5/building_pipeline.sif smi_round_{}.smi<br /> <br /> When building has completed, you must write an SDI file with the complete paths to each built bundle and dock. you can do this with:<br /> find /path/to/your/building_output -name &quot;bundle.db2.tgz&quot; -type f &gt; round_{}.wynton.sdi<br /> <br /> <br /> '''Be sure to change the INDOCK file to save only poses that meet your score pProp score threshold calculated by ChemSTEP!''' Retrieve docking scores as convert to NumPy arrays as outlined above, updating the round number when running get_scores.py and convert_scores_to_npy.py. Copy new score and indices files into the directory you ran ChemSTEP in. If you are following along as a tutorial, you should have scores_round_1.npy and indices_round_1.npy from the previous step (from FIRST round of ChemSTEP prioritization). <br /> <br /> <br /> '''9. Set up for iterative rounds of ChemSTEP'''<br /> <br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/run_chemstep_iterative.py .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/launch_chemstep_iterative.sh .<br /> <br /> <br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/run_chemstep_iterative.py .<br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/launch_chemstep_iterative.sh .<br /> <br /> '''10. Run ChemSTEP'''<br /> qsub launch_chemstep_as_job.sh [round number]<br /> <br /> For the first iterative round, the round number is [2], and should increase by one for every subsequent round of ChemSTEP. The output will be smi_round_****.smi file. Repeat steps 8-10 for as many rounds as needed. The performance is reported in outputy/complete_info/run_summary.df, which contains the number of beacons selected, the number of molecules docked, the number of hits found, the distance threshold for the selected molecules to dock, and the last added beacon's distance to all previous beacons.<br /> <br /> <br /> '''AUTOMATIC CHEMSTEP RUNNING DIRECTIONS'''<br /> <br /> This version of ChemSTEP will automatically submit building and docking jobs inbetween rounds of ChemSTEP. Note, this version does not currently build with strain (to do: add this)<br /> <br /> '''1. DOCK the seed set to your receptor as outlined above in steps 1-4'''<br /> <br /> successful output will be a scores_round_0.npy and indices_round_0.npy. <br /> <br /> <br /> '''2. Copy necessary scripts into your working ChemSTEP directory'''<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_auto/scripts/params.txt .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_auto/scripts/run_chemstep.py .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_auto/scripts/launch_chemstep_as_job.sh .<br /> <br /> <br /> '''3. Edit run_chemstep.py to add the path to your dockfiles'''<br /> <br /> the algorithm needs to know where your dockfiles live in order to submit docking jobs. Make sure all INDOCK parameters are set correctly. Use the full path to the dockfiles. <br /> <br /> <br /> '''4. Edit params.txt''' The parameters outlined here will be carried through all rounds of ChemSTEP. <br /> seed_indices_file: /absoulte/path/to/your/indices_round_0.npy<br /> seed_scores_file: /absolute/path/to/your/scores_round_0.npy<br /> hit_pprop: 5<br /> bundle_size: 1000<br /> n_docked_per_round: 10000000<br /> max_beacons: 150<br /> max_n_rounds: 250<br /> <br /> <br /> Within params.txt, add the absolute paths to the ChemSTEP-readable score and indices NumPY arrays. The rest of the values within the params.txt file are left to the discretion of the user. The '''bundle_size:''' parameter is required for auto docking mode of ChemSTEP. This is the number of molecules submitted at a job for building. <br /> <br /> <br /> '''5. Launch ChemSTEP'''<br /> <br /> qsub launch_chemstep_as_job.sh <br /> <br /> <br /> '''Troubleshooting:''' if your job errors out or stops running at any point, check the ''chemstep_submission.log'' file in the working directory. <br /> <br /> <br /> <br /> '''Helper scripts:''' a running list of scripts we have been using for analysis and visualization. <br /> <br /> to get SMILES for beacons: <br /> <br /> python /wynton/group/bks/work/bwhall61/needs_github/get_beacon_smi.py --beacon_df_path /path/to/your/chemstep/output/complete_info/beacons.df --n_workers 6 --outfile beacon_smiles.smi<br /> <br /> <br /> ** anecdotally true.</div>

Kholland http://wiki.docking.org/index.php?title=Running_ChemSTEP&diff=17009 2025-12-11T19:55:28Z

<p>Kholland: </p> <hr /> <div>last update: dec 11 2025 katie. current ver = 0.3.1.5 (automatic resubmission of failed SGE jobs). <br /> <br /> ChemSTEP (Chemical Space Traversal and Exploration Procedure) is an open-source, transparent acceleration algorithm for molecular docking capable of dealing with virtual libraries of several trillion compounds. This wiki page is a guide for BKS lab members to run ChemSTEP on Wynton HPC, using the current version of InifiSee XReal library (1.1T) or 13.3B library from Enamine REAL. '''for detailed instructions on the 13B space, see 'Running ChemSTEP on the 13B space' wiki page'''. For more general use directions, please refer to [ChemSTEP Read-the-Docs]. <br /> <br /> At a high-level, ChemSTEP is an iterative process that identifies molecules from the larger virtual library to prioritize for docking. First, we identify a random sample of the total library (termed &quot;seed set&quot;, round zero) and dock those molecules to the target of interest. From this seed set, we can calculate total-library pProp values (-log rank percentages) and the number of &quot;virtual hits&quot; in the total library (high-scoring molecules). ChemSTEP will identify a set of maximally diverse molecules that score above the desired pProp threshold (&quot;beacons&quot;) from the seed set. These beacons guide prioritization, where molecules chosen and output by ChemSTEP are close in chemical space to the beacons. Prioritized molecules are then built, docked, and returned to ChemSTEP for a second round of prioritization. This process is iterated until you reach desired virtual hit recovery, or you are no longer recovering virtual hits. <br /> <br /> <br /> Running the ChemSTEP algorithm on Wynton takes place in loosely three steps: (1) submission of the main ChemSTEP job, (2) ChainingLog sub-job array, and (3) gathering of SMILES for prioritization. <br /> <br /> '''1. Main ChemSTEP job:''' reads input files (DOCK scores and line-matched indices NumPy arrays) and assigns beacons. Beacons are a set of N maximally diverse molecules that score above the pProp threshold specified in the parameter file. For the first round of ChemSTEP, the best-scoring molecule from the seed set is assigned as the first beacon. Subsequent beacons would be molecules that score well (above pProp thresh), and are as diverse as possible from already assigned beacons. Greedy max-diversity selection. <br /> <br /> During assignment, ECFP4 Tanimoto distances between the assigned beacon and all potential beacons are calculated. The second beacon is the molecule with the maximum Td from beacon 1. Beacon 3 will be the molecule that has the highest Td from beacon 1 and beacon 2. This continues until the number of beacons specified in the parameter file is reached. In iterative rounds of ChemSTEP, the beacon selection is based on diversity from ALL PREVIOUSLY assigned beacons, including those in earlier rounds. <br /> <br /> <br /> '''2. ChainingLog sub-jobs:''' once beacons are assigned, ChemSTEP will launch a series of sub-jobs that calculate the Tanimoto distances of every molecule remaining in the library to the assigned beacons. Calculated distances to the NEAREST beacon (minimum-minimum Td) are updated in the mintddistrib_*.npy files within the /output directory. <br /> <br /> <br /> '''3. Gathering of SMILES for prioritization:''' When all Td calculations are completed, the algo will select N number of molecules for prioritization, the number of which is specified by the user in the parameter file. Molecules that are ''closest in chemical space'' to beacons are prioritized. I.e. if round size = 1 million, the 1 million molecules with the smallest min-Td to ANY beacon (current or previous rounds) are prioritized. chemstep_algo.log will provide a &quot;max-minTd&quot; for each round, which is the Tanimoto distance of the most dissimilar molecule prioritized to any one beacon per round. The prioritized molecules are output in /output/complete_info as smi_round_{}.smi <br /> <br /> What the user need: DOCKFILES, directories for (1) docking (2) building and (3) running ChemSTEP. '''All iterative rounds of ChemSTEP should be run in the SAME directory.''' <br /> <br /> '''1. Source ChemSTEP virtual environment on Wynton'''<br /> source /wynton/group/bks/work/shared/kholland/chemstep_0.3.1.5/bin/activate<br /> <br /> <br /> '''2. Copy seed-set SDI file into your docking directory'''<br /> This directory should already contain your dockfiles, with INDOCK parameters set to your liking. In this step, we are copying in a split database index file (SDI) containing paths to bundles of db2 files. This seed set contains 100 million molecules sampled randomly from the total virtual library, currently 1.1 trillion molecules.<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/XR_seed_set_v0p0.wynton.sdi .<br /> <br /> <br /> if you are interested in running on a library of 13.2B compounds from Enamine REAL, there are several seed set sizes available: 130k, 1.3M, 13M, and 26M. <br /> cp /wynton/group/bks/work/shared/kholland/chemstep_13B/130K_seeds.wynton.sdi .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_13B/1.3M_seeds.wynton.sdi .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_13B/13M_seeds.wynton.sdi .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_13B/26M_seeds.wynton.sdi .<br /> <br /> <br /> '''3. Dock seed set to your receptor of interest using DOCK 3.8''' /docking directions taken from docs.docking.org. This is meant to be done as you would do a normal LSD. <br /> export MOLECULES_DIR_TO_BIND=[outermost folder containing the molecules to dock, just your base docking directory]<br /> export DOCKFILES=[path to your dockfiles]<br /> export INPUT_FOLDER=[the folder containing your .sdi file(s)]<br /> export OUTPUT_FOLDER=[where you want the output ]<br /> <br /> /wynton/group/bks/work/bwhall61/needs_github/super_dock3r.sh<br /> <br /> Wait for docking to complete. Next, you must extract all molecule IDs and corresponding DOCK scores from above. To do so, run the following commands in the base docking directory containing your docking files and output folder, while logged into a dev node (do this in a screen!!!):<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/get_scores.py .<br /> python get_scores.py 0<br /> <br /> <br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/get_scores.py .<br /> python get_scores.py 0<br /> <br /> <br /> This script expects a directory named &quot;output*&quot; within the CWD. If your output from docking follows different organizational conventions, vim into get_scores.py and change the path. Successful output of the script will be a file named &quot;scores_round_0.txt&quot;. For iterative rounds, pass increasing numbers into the command line. i.e. When docking the first round of prioritized molecules, pass [1] for scores_round_1.txt. <br /> <br /> '''4. Convert scores and molecule IDS into NumPY arrays for ChemSTEP recognition. Requires ChemSTEP venv'''<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/convert_scores_to_npy.py .<br /> python convert_scores_to_npy.py 0<br /> <br /> <br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/convert_scores_to_npy.py .<br /> python convert_scores_to_npy.py 0<br /> <br /> For python speed purposes, ChemSTEP requires that DOCK scores and IDs be give in the form of a NumPY array. In this step, we are taking our readable scores file and converting them for ChemSTEP recognition. This script expects a txt file with Molecule IDS and DOCK scores (no header). Use the same round number you used in step 3. As run above, this will output two files named scores_round_0.npy and indices_round_0.npy that contain line-matched indices (determined from Mol ID) and their respective docking scores. <br /> <br /> '''5. Enter into or make a directory to run ChemSTEP in. Copy in necessary files for initiating ChemSTEP: params.txt, run_chemstep.py and launch_chemstep_as_job.sh. Copy in your score and indices numpy files as well, which should be in your docking directory.'''<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/params.txt .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/run_chemstep.py .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/launch_chemstep_as_job.sh .<br /> <br /> <br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/params.txt .<br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/run_chemstep.py .<br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/launch_chemstep_as_job.sh .<br /> <br /> '''6. Edit params.txt file''' This ONLY needs to be edited for the initial round of ChemSTEP. The parameters outlined here will be carried through all rounds of ChemSTEP. <br /> seed_indices_file: /absoulte/path/to/your/indices_round_0.npy<br /> seed_scores_file: /absolute/path/to/your/scores_round_0.npy<br /> hit_pprop: 5<br /> n_docked_per_round: 10000000<br /> max_beacons: 150<br /> max_n_rounds: 250<br /> <br /> Within params.txt, add the absolute paths to the ChemSTEP-readable score and indices NumPY arrays. The rest of the values within the params.txt file are left to the discretion of the user, with some considerations below. <br /> <br /> '''pProp:''' this value will define what is considered a &quot;virtual hit&quot; in your campaign. pProp is defined as the -log(rank %) of a molecule within the total library score-distribution. For example, a pProp of 4 in the 1.1T XReal space is equivalent the top 0.01% of the library, corresponding to the top 110 million molecules. pProp 5 = 0.001% = 11 million virtual hits, etc. From the seed set, ChemSTEP will estimate a DOCK score value in kcal/mol that associated with the lower-limit of your desired pProp zone. Any molecules that score better than the threshold is considered a virtual hit in your campaign. Be mindful of seed set size when choosing desired pProp. Our suggestion is that the size of the seed set should be at least 10^(pProp +2).<br /> <br /> '''n_docked_per_round:''' this number is the desired molecules for prioritization per round. When choosing this value, note that this number of molecules must be built and docked in between each round of ChemSTEP. Prioritizing many molecules will slow building and docking speeds, and coupled with few beacons (below) may lead to decreased diversity. Prioritizing too few molecules may result in slower virtual hit recovery. Round size does not significantly impact the algorithm running time.** <br /> <br /> '''max_beacons:''' this is the number of diverse, well-scoring molecules ChemSTEP will use to guide prioritization per round. All molecules that score better than the assigned pProp threshold may be considered for beacons. By default, ChemSTEP chooses each set of beacons to be as maximally diverse as possible. Choosing too many beacons may result in decreased diversity between beacons, but too few beacons could hinder space exploration. If not enough molecules score above your pProp threshold, ChemSTEP may assign fewer beacons than the assigned value. <br /> <br /> '''max_n_rounds:''' if prospectively running ChemSTEP, as outlined in this wiki, no need to worry about this parameter.<br /> <br /> <br /> There should be no need to edit run_chemstep.py or the SGE wrapper script. At this time, we strongly suggest running ChemSTEP as a job array with 16 or 32 CPU slots requested. The number of cores must be specified when calling CSAlgo (in run_chemstep.py) and the SGE wrapper(launch_chemstep_as_job.sh). <br /> <br /> <br /> '''7. Run ChemSTEP''' with the following command: <br /> qsub launch_chemstep_as_job.sh <br /> <br /> <br /> This will launch the main ChemSTEP job to the SGE scheduler. Check your job status with the 'qstat' command. The algorithm will read the score and indices files provided, calculate pProp (for round 1) and assign beacons. As this job runs, it will launch a subsequent job array. These sub-jobs are the Chaining step of ChemSTEP, where each parallel worker calculates the Tanimoto distances of 100 million molecules from the virtual library to the beacons. These distances are then written to the files within the /output directory. When these jobs complete, the main job will read through all Tanimoto distances and pull molecules for prioritization. <br /> <br /> Running ChemSTEP successfully will result in the output of a SMILES file within output/complete_info/. You will also get (1) chemstep_algo.log and (2) chemstep_submission.log in the working directory after job submission. chemstep_algo.log contains the running output of the algorithm in a readable format, with timestamps, including the DOCK score associated with your desired pProp. chemstep_submission.log contains the output of ChemSTEP, as well as any information output by the SGE submission system (python errors, cluster issues, tracebacks if failed). These files will be updated with any information with iterative rounds of ChemSTEP run in the same directory. '''Visually inspect these files after each round of ChemSTEP to ensure the algorithm has picked your desired number of beacons and things ran smoothly.'''<br /> <br /> '''Troubleshooting:''' if no jobs are running and there is no SMI file, check the chemstep_submission.log first. Any traceback error or SGE error should give you some idea of why ChemSTEP failed. Some errors may be due to SGE or Wynton issues. If directed, look at a few .out files within /output/jobs/ . If a ChemSTEP run fails on your FIRST run, delete the output and log files, fix what needs to be fixed, and rerun. Errors in subsequent rounds during the chaining step can potentially corrupt chaining files within the /output directory that are needed for prioritization. At the very least, you will definitely have duplicates of beacons and information written to the log files. At this time, if your run fails during iterative rounds, it's best to start from the beginning. <br /> <br /> Prioritized molecules should be built, docked, and their scores fed back into ChemSTEP. More detailed instructions below: <br /> <br /> '''8. Build prioritized molecules (DOCK 3.8).''' /taken from docs.docking.org <br /> source /wynton/group/bks/soft/DOCK-3.8.5/env.sh<br /> <br /> python /wynton/group/bks/soft/DOCK-3.8.5/DOCK3.8/zinc22-3d/submit/submit_building_docker.py --output_folder building_output --bundle_size 1000 --minutes_per_mol 5 --skip_name_check --scheduler sge --container_software apptainer --container_path_or_name /wynton/group/bks/soft/DOCK-3.8.5/building_pipeline.sif smi_round_{}.smi<br /> <br /> When building has completed, you must write an SDI file with the complete paths to each built bundle and dock. you can do this with:<br /> find /path/to/your/building_output -name &quot;bundle.db2.tgz&quot; -type f &gt; round_{}.wynton.sdi<br /> <br /> <br /> '''Be sure to change the INDOCK file to save only poses that meet your score pProp score threshold calculated by ChemSTEP!''' Retrieve docking scores as convert to NumPy arrays as outlined above, updating the round number when running get_scores.py and convert_scores_to_npy.py. Copy new score and indices files into the directory you ran ChemSTEP in. If you are following along as a tutorial, you should have scores_round_1.npy and indices_round_1.npy from the previous step (from FIRST round of ChemSTEP prioritization). <br /> <br /> <br /> '''9. Set up for iterative rounds of ChemSTEP'''<br /> <br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/run_chemstep_iterative.py .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/launch_chemstep_iterative.sh .<br /> <br /> <br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/run_chemstep_iterative.py .<br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/launch_chemstep_iterative.sh .<br /> <br /> '''10. Run ChemSTEP'''<br /> qsub launch_chemstep_as_job.sh [round number]<br /> <br /> For the first iterative round, the round number is [2], and should increase by one for every subsequent round of ChemSTEP. The output will be smi_round_****.smi file. Repeat steps 8-10 for as many rounds as needed. The performance is reported in outputy/complete_info/run_summary.df, which contains the number of beacons selected, the number of molecules docked, the number of hits found, the distance threshold for the selected molecules to dock, and the last added beacon's distance to all previous beacons.<br /> <br /> <br /> '''AUTOMATIC CHEMSTEP RUNNING DIRECTIONS'''<br /> <br /> This version of ChemSTEP will automatically submit building and docking jobs inbetween rounds of ChemSTEP. Note, this version does not currently build with strain (to do: add this)<br /> <br /> '''1. DOCK the seed set to your receptor as outlined above in steps 1-4'''<br /> <br /> successful output will be a scores_round_0.npy and indices_round_0.npy. <br /> <br /> <br /> '''2. Copy necessary scripts into your working ChemSTEP directory'''<br /> cp /wynton/group/bks/work/shared/kholland/auto_chemstep/scripts/params.txt .<br /> cp /wynton/group/bks/work/shared/kholland/auto_chemstep/scripts/run_chemstep.py .<br /> cp /wynton/group/bks/work/shared/kholland/auto_chemstep/scripts/launch_chemstep_as_job.sh .<br /> <br /> <br /> '''3. Edit run_chemstep.py to add the path to your dockfiles'''<br /> <br /> the algorithm needs to know where your dockfiles live in order to submit docking jobs. Make sure all INDOCK parameters are set correctly. Use the full path to the dockfiles. <br /> <br /> <br /> '''4. Edit params.txt''' The parameters outlined here will be carried through all rounds of ChemSTEP. <br /> seed_indices_file: /absoulte/path/to/your/indices_round_0.npy<br /> seed_scores_file: /absolute/path/to/your/scores_round_0.npy<br /> hit_pprop: 5<br /> bundle_size: 1000<br /> n_docked_per_round: 10000000<br /> max_beacons: 150<br /> max_n_rounds: 250<br /> <br /> <br /> Within params.txt, add the absolute paths to the ChemSTEP-readable score and indices NumPY arrays. The rest of the values within the params.txt file are left to the discretion of the user. The '''bundle_size:''' parameter is required for auto docking mode of ChemSTEP. This is the number of molecules submitted at a job for building. <br /> <br /> <br /> '''5. Launch ChemSTEP'''<br /> <br /> qsub launch_chemstep_as_job.sh <br /> <br /> <br /> '''Troubleshooting:''' if your job errors out or stops running at any point, check the ''chemstep_submission.log'' file in the working directory. <br /> <br /> <br /> <br /> '''Helper scripts:''' a running list of scripts we have been using for analysis and visualization. <br /> <br /> to get SMILES for beacons: <br /> <br /> python /wynton/group/bks/work/bwhall61/needs_github/get_beacon_smi.py --beacon_df_path /path/to/your/chemstep/output/complete_info/beacons.df --n_workers 6 --outfile beacon_smiles.smi<br /> <br /> <br /> ** anecdotally true.</div>

Kholland http://wiki.docking.org/index.php?title=Running_ChemSTEP&diff=17008 2025-12-11T19:53:33Z

<p>Kholland: </p> <hr /> <div>last update: dec 11 2025 katie. current ver = 0.3.1.5 (automatic resubmission of failed SGE jobs). <br /> <br /> ChemSTEP (Chemical Space Traversal and Exploration Procedure) is an open-source, transparent acceleration algorithm for molecular docking capable of dealing with virtual libraries of several trillion compounds. This wiki page is a guide for BKS lab members to run ChemSTEP on Wynton HPC, using the current version of InifiSee XReal library (1.1T) or 13.3B library from Enamine REAL. '''for detailed instructions on the 13B space, see 'Running ChemSTEP on the 13B space' wiki page'''. For more general use directions, please refer to [ChemSTEP Read-the-Docs]. <br /> <br /> At a high-level, ChemSTEP is an iterative process that identifies molecules from the larger virtual library to prioritize for docking. First, we identify a random sample of the total library (termed &quot;seed set&quot;, round zero) and dock those molecules to the target of interest. From this seed set, we can calculate total-library pProp values (-log rank percentages) and the number of &quot;virtual hits&quot; in the total library (high-scoring molecules). ChemSTEP will identify a set of maximally diverse molecules that score above the desired pProp threshold (&quot;beacons&quot;) from the seed set. These beacons guide prioritization, where molecules chosen and output by ChemSTEP are close in chemical space to the beacons. Prioritized molecules are then built, docked, and returned to ChemSTEP for a second round of prioritization. This process is iterated until you reach desired virtual hit recovery, or you are no longer recovering virtual hits. <br /> <br /> <br /> Running the ChemSTEP algorithm on Wynton takes place in loosely three steps: (1) submission of the main ChemSTEP job, (2) ChainingLog sub-job array, and (3) gathering of SMILES for prioritization. <br /> <br /> '''1. Main ChemSTEP job:''' reads input files (DOCK scores and line-matched indices NumPy arrays) and assigns beacons. Beacons are a set of N maximally diverse molecules that score above the pProp threshold specified in the parameter file. For the first round of ChemSTEP, the best-scoring molecule from the seed set is assigned as the first beacon. Subsequent beacons would be molecules that score well (above pProp thresh), and are as diverse as possible from already assigned beacons. Greedy max-diversity selection. <br /> <br /> During assignment, ECFP4 Tanimoto distances between the assigned beacon and all potential beacons are calculated. The second beacon is the molecule with the maximum Td from beacon 1. Beacon 3 will be the molecule that has the highest Td from beacon 1 and beacon 2. This continues until the number of beacons specified in the parameter file is reached. In iterative rounds of ChemSTEP, the beacon selection is based on diversity from ALL PREVIOUSLY assigned beacons, including those in earlier rounds. <br /> <br /> <br /> '''2. ChainingLog sub-jobs:''' once beacons are assigned, ChemSTEP will launch a series of sub-jobs that calculate the Tanimoto distances of every molecule remaining in the library to the assigned beacons. Calculated distances to the NEAREST beacon (minimum-minimum Td) are updated in the mintddistrib_*.npy files within the /output directory. <br /> <br /> <br /> '''3. Gathering of SMILES for prioritization:''' When all Td calculations are completed, the algo will select N number of molecules for prioritization, the number of which is specified by the user in the parameter file. Molecules that are ''closest in chemical space'' to beacons are prioritized. I.e. if round size = 1 million, the 1 million molecules with the smallest min-Td to ANY beacon (current or previous rounds) are prioritized. chemstep_algo.log will provide a &quot;max-minTd&quot; for each round, which is the Tanimoto distance of the most dissimilar molecule prioritized to any one beacon per round. The prioritized molecules are output in /output/complete_info as smi_round_{}.smi <br /> <br /> What the user need: DOCKFILES, directories for (1) docking (2) building and (3) running ChemSTEP. '''All iterative rounds of ChemSTEP should be run in the SAME directory.''' <br /> <br /> '''1. Source ChemSTEP virtual environment on Wynton'''<br /> source /wynton/group/bks/work/shared/kholland/chemstep_0.3.1.5/bin/activate<br /> <br /> <br /> '''2. Copy seed-set SDI file into your docking directory'''<br /> This directory should already contain your dockfiles, with INDOCK parameters set to your liking. In this step, we are copying in a split database index file (SDI) containing paths to bundles of db2 files. This seed set contains 100 million molecules sampled randomly from the total virtual library, currently 1.1 trillion molecules.<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/XR_seed_set_v0p0.wynton.sdi .<br /> <br /> <br /> if you are interested in running on a library of 13.2B compounds from Enamine REAL, there are several seed set sizes available: 130k, 1.3M, 13M, and 26M. <br /> cp /wynton/group/bks/work/shared/kholland/chemstep_13B/130K_seeds.wynton.sdi .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_13B/1.3M_seeds.wynton.sdi .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_13B/13M_seeds.wynton.sdi .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_13B/26M_seeds.wynton.sdi .<br /> <br /> <br /> '''3. Dock seed set to your receptor of interest using DOCK 3.8''' /docking directions taken from docs.docking.org. This is meant to be done as you would do a normal LSD. <br /> export MOLECULES_DIR_TO_BIND=[outermost folder containing the molecules to dock, just your base docking directory]<br /> export DOCKFILES=[path to your dockfiles]<br /> export INPUT_FOLDER=[the folder containing your .sdi file(s)]<br /> export OUTPUT_FOLDER=[where you want the output ]<br /> <br /> /wynton/group/bks/work/bwhall61/needs_github/super_dock3r.sh<br /> <br /> Wait for docking to complete. Next, you must extract all molecule IDs and corresponding DOCK scores from above. To do so, run the following commands in the base docking directory containing your docking files and output folder, while logged into a dev node (do this in a screen!!!):<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/get_scores.py .<br /> python get_scores.py 0<br /> <br /> <br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/get_scores.py .<br /> python get_scores.py 0<br /> <br /> <br /> This script expects a directory named &quot;output*&quot; within the CWD. If your output from docking follows different organizational conventions, vim into get_scores.py and change the path. Successful output of the script will be a file named &quot;scores_round_0.txt&quot;. For iterative rounds, pass increasing numbers into the command line. i.e. When docking the first round of prioritized molecules, pass [1] for scores_round_1.txt. <br /> <br /> '''4. Convert scores and molecule IDS into NumPY arrays for ChemSTEP recognition. Requires ChemSTEP venv'''<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/convert_scores_to_npy.py .<br /> python convert_scores_to_npy.py 0<br /> <br /> <br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/convert_scores_to_npy.py .<br /> python convert_scores_to_npy.py 0<br /> <br /> For python speed purposes, ChemSTEP requires that DOCK scores and IDs be give in the form of a NumPY array. In this step, we are taking our readable scores file and converting them for ChemSTEP recognition. This script expects a txt file with Molecule IDS and DOCK scores (no header). Use the same round number you used in step 3. As run above, this will output two files named scores_round_0.npy and indices_round_0.npy that contain line-matched indices (determined from Mol ID) and their respective docking scores. <br /> <br /> '''5. Enter into or make a directory to run ChemSTEP in. Copy in necessary files for initiating ChemSTEP: params.txt, run_chemstep.py and launch_chemstep_as_job.sh. Copy in your score and indices numpy files as well, which should be in your docking directory.'''<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/params.txt .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/run_chemstep.py .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/launch_chemstep_as_job.sh .<br /> <br /> <br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/params.txt .<br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/run_chemstep.py .<br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/launch_chemstep_as_job.sh .<br /> <br /> '''6. Edit params.txt file''' This ONLY needs to be edited for the initial round of ChemSTEP. The parameters outlined here will be carried through all rounds of ChemSTEP. <br /> seed_indices_file: /absoulte/path/to/your/indices_round_0.npy<br /> seed_scores_file: /absolute/path/to/your/scores_round_0.npy<br /> hit_pprop: 5<br /> n_docked_per_round: 10000000<br /> max_beacons: 150<br /> max_n_rounds: 250<br /> <br /> Within params.txt, add the absolute paths to the ChemSTEP-readable score and indices NumPY arrays. The rest of the values within the params.txt file are left to the discretion of the user, with some considerations below. <br /> <br /> '''pProp:''' this value will define what is considered a &quot;virtual hit&quot; in your campaign. pProp is defined as the -log(rank %) of a molecule within the total library score-distribution. For example, a pProp of 4 in the 1.1T XReal space is equivalent the top 0.01% of the library, corresponding to the top 110 million molecules. pProp 5 = 0.001% = 11 million virtual hits, etc. From the seed set, ChemSTEP will estimate a DOCK score value in kcal/mol that associated with the lower-limit of your desired pProp zone. Any molecules that score better than the threshold is considered a virtual hit in your campaign. Be mindful of seed set size when choosing desired pProp. Our suggestion is that the size of the seed set should be at least 10^(pProp +2).<br /> <br /> '''n_docked_per_round:''' this number is the desired molecules for prioritization per round. When choosing this value, note that this number of molecules must be built and docked in between each round of ChemSTEP. Prioritizing many molecules will slow building and docking speeds, and coupled with few beacons (below) may lead to decreased diversity. Prioritizing too few molecules may result in slower virtual hit recovery. Round size does not significantly impact the algorithm running time.** <br /> <br /> '''max_beacons:''' this is the number of diverse, well-scoring molecules ChemSTEP will use to guide prioritization per round. All molecules that score better than the assigned pProp threshold may be considered for beacons. By default, ChemSTEP chooses each set of beacons to be as maximally diverse as possible. Choosing too many beacons may result in decreased diversity between beacons, but too few beacons could hinder space exploration. If not enough molecules score above your pProp threshold, ChemSTEP may assign fewer beacons than the assigned value. <br /> <br /> '''max_n_rounds:''' if prospectively running ChemSTEP, as outlined in this wiki, no need to worry about this parameter.<br /> <br /> <br /> There should be no need to edit run_chemstep.py or the SGE wrapper script. At this time, we strongly suggest running ChemSTEP as a job array with 16 or 32 CPU slots requested. The number of cores must be specified when calling CSAlgo (in run_chemstep.py) and the SGE wrapper(launch_chemstep_as_job.sh). <br /> <br /> <br /> '''7. Run ChemSTEP''' with the following command: <br /> qsub launch_chemstep_as_job.sh <br /> <br /> <br /> This will launch the main ChemSTEP job to the SGE scheduler. Check your job status with the 'qstat' command. The algorithm will read the score and indices files provided, calculate pProp (for round 1) and assign beacons. As this job runs, it will launch a subsequent job array. These sub-jobs are the Chaining step of ChemSTEP, where each parallel worker calculates the Tanimoto distances of 100 million molecules from the virtual library to the beacons. These distances are then written to the files within the /output directory. When these jobs complete, the main job will read through all Tanimoto distances and pull molecules for prioritization. <br /> <br /> Running ChemSTEP successfully will result in the output of a SMILES file within output/complete_info/. You will also get (1) chemstep_algo.log and (2) chemstep_submission.log in the working directory after job submission. chemstep_algo.log contains the running output of the algorithm in a readable format, with timestamps, including the DOCK score associated with your desired pProp. chemstep_submission.log contains the output of ChemSTEP, as well as any information output by the SGE submission system (python errors, cluster issues, tracebacks if failed). These files will be updated with any information with iterative rounds of ChemSTEP run in the same directory. '''Visually inspect these files after each round of ChemSTEP to ensure the algorithm has picked your desired number of beacons and things ran smoothly.'''<br /> <br /> '''Troubleshooting:''' if no jobs are running and there is no SMI file, check the chemstep_submission.log first. Any traceback error or SGE error should give you some idea of why ChemSTEP failed. Some errors may be due to SGE or Wynton issues. If directed, look at a few .out files within /output/jobs/ . If a ChemSTEP run fails on your FIRST run, delete the output and log files, fix what needs to be fixed, and rerun. Errors in subsequent rounds during the chaining step can potentially corrupt chaining files within the /output directory that are needed for prioritization. At the very least, you will definitely have duplicates of beacons and information written to the log files. At this time, if your run fails during iterative rounds, it's best to start from the beginning. <br /> <br /> Prioritized molecules should be built, docked, and their scores fed back into ChemSTEP. More detailed instructions below: <br /> <br /> '''8. Build prioritized molecules (DOCK 3.8).''' /taken from docs.docking.org <br /> source /wynton/group/bks/soft/DOCK-3.8.5/env.sh<br /> <br /> python /wynton/group/bks/soft/DOCK-3.8.5/DOCK3.8/zinc22-3d/submit/submit_building_docker.py --output_folder building_output --bundle_size 1000 --minutes_per_mol 5 --skip_name_check --scheduler sge --container_software apptainer --container_path_or_name /wynton/group/bks/soft/DOCK-3.8.5/building_pipeline.sif smi_round_{}.smi<br /> <br /> When building has completed, you must write an SDI file with the complete paths to each built bundle and dock. you can do this with:<br /> find /path/to/your/building_output -name &quot;bundle.db2.tgz&quot; -type f &gt; round_{}.wynton.sdi<br /> <br /> <br /> '''Be sure to change the INDOCK file to save only poses that meet your score pProp score threshold calculated by ChemSTEP!''' Retrieve docking scores as convert to NumPy arrays as outlined above, updating the round number when running get_scores.py and convert_scores_to_npy.py. Copy new score and indices files into the directory you ran ChemSTEP in. If you are following along as a tutorial, you should have scores_round_1.npy and indices_round_1.npy from the previous step (from FIRST round of ChemSTEP prioritization). <br /> <br /> <br /> '''9. Set up for iterative rounds of ChemSTEP'''<br /> <br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/run_chemstep_iterative.py .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/launch_chemstep_iterative.sh .<br /> <br /> <br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/run_chemstep_iterative.py .<br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/launch_chemstep_iterative.sh .<br /> <br /> '''10. Run ChemSTEP'''<br /> qsub launch_chemstep_as_job.sh [round number]<br /> <br /> For the first iterative round, the round number is [2], and should increase by one for every subsequent round of ChemSTEP. The output will be smi_round_****.smi file. Repeat steps 8-10 for as many rounds as needed. The performance is reported in outputy/complete_info/run_summary.df, which contains the number of beacons selected, the number of molecules docked, the number of hits found, the distance threshold for the selected molecules to dock, and the last added beacon's distance to all previous beacons.<br /> <br /> <br /> ‘’’AUTOMATIC CHEMSTEP RUNNING DIRECTIONS’’’<br /> <br /> This version of ChemSTEP will automatically submit building and docking jobs inbetween rounds of ChemSTEP. Note, this version does not currently build with strain (to do: add this)<br /> <br /> ‘’’1. DOCK the seed set to your receptor as outlined above in steps 1-4’’’ <br /> <br /> successful output will be a scores_round_0.npy and indices_round_0.npy. <br /> <br /> <br /> ‘’’2. Copy necessary scripts into your working ChemSTEP directory’’’<br /> cp /wynton/group/bks/work/shared/kholland/auto_chemstep/scripts/params.txt .<br /> cp /wynton/group/bks/work/shared/kholland/auto_chemstep/scripts/run_chemstep.py .<br /> cp /wynton/group/bks/work/shared/kholland/auto_chemstep/scripts/launch_chemstep_as_job.sh .<br /> <br /> <br /> ‘’’3. Edit run_chemstep.py to add the path to your dockfiles’’’<br /> <br /> the algorithm needs to know where your dockfiles live in order to submit docking jobs. Make sure all INDOCK parameters are set correctly. Use the full path to the dockfiles. <br /> <br /> <br /> ‘’’4. Edit params.txt''' The parameters outlined here will be carried through all rounds of ChemSTEP. <br /> seed_indices_file: /absoulte/path/to/your/indices_round_0.npy<br /> seed_scores_file: /absolute/path/to/your/scores_round_0.npy<br /> hit_pprop: 5<br /> bundle_size: 1000<br /> n_docked_per_round: 10000000<br /> max_beacons: 150<br /> max_n_rounds: 250<br /> <br /> <br /> Within params.txt, add the absolute paths to the ChemSTEP-readable score and indices NumPY arrays. The rest of the values within the params.txt file are left to the discretion of the user. The '''bundle_size:''' parameter is required for auto docking mode of ChemSTEP. This is the number of molecules submitted at a job for building. <br /> <br /> <br /> '''5. Launch ChemSTEP'''<br /> <br /> qsub launch_chemstep_as_job.sh <br /> <br /> <br /> '''Troubleshooting:''' if your job errors out or stops running at any point, check the ''chemstep_submission.log'' file in the working directory. <br /> <br /> <br /> <br /> '''Helper scripts:''' a running list of scripts we have been using for analysis and visualization. <br /> <br /> to get SMILES for beacons: <br /> <br /> python /wynton/group/bks/work/bwhall61/needs_github/get_beacon_smi.py --beacon_df_path /path/to/your/chemstep/output/complete_info/beacons.df --n_workers 6 --outfile beacon_smiles.smi<br /> <br /> <br /> ** anecdotally true.</div>

Kholland http://wiki.docking.org/index.php?title=Running_ChemSTEP&diff=17007 2025-12-11T03:19:25Z

<p>Kholland: </p> <hr /> <div>last update: oct 8 2025 katie. current ver = 0.3.1.5 (automatic resubmission of failed SGE jobs). <br /> <br /> ChemSTEP (Chemical Space Traversal and Exploration Procedure) is an open-source, transparent acceleration algorithm for molecular docking capable of dealing with virtual libraries of several trillion compounds. This wiki page is a guide for BKS lab members to run ChemSTEP on Wynton HPC, using the current version of InifiSee XReal library (1.1T) or 13.3B library from Enamine REAL. '''for detailed instructions on the 13B space, see 'Running ChemSTEP on the 13B space' wiki page'''. For more general use directions, please refer to [ChemSTEP Read-the-Docs]. <br /> <br /> At a high-level, ChemSTEP is an iterative process that identifies molecules from the larger virtual library to prioritize for docking. First, we identify a random sample of the total library (termed &quot;seed set&quot;, round zero) and dock those molecules to the target of interest. From this seed set, we can calculate total-library pProp values (-log rank percentages) and the number of &quot;virtual hits&quot; in the total library (high-scoring molecules). ChemSTEP will identify a set of maximally diverse molecules that score above the desired pProp threshold (&quot;beacons&quot;) from the seed set. These beacons guide prioritization, where molecules chosen and output by ChemSTEP are close in chemical space to the beacons. Prioritized molecules are then built, docked, and returned to ChemSTEP for a second round of prioritization. This process is iterated until you reach desired virtual hit recovery, or you are no longer recovering virtual hits. <br /> <br /> <br /> Running the ChemSTEP algorithm on Wynton takes place in loosely three steps: (1) submission of the main ChemSTEP job, (2) ChainingLog sub-job array, and (3) gathering of SMILES for prioritization. <br /> <br /> '''1. Main ChemSTEP job:''' reads input files (DOCK scores and line-matched indices NumPy arrays) and assigns beacons. Beacons are a set of N maximally diverse molecules that score above the pProp threshold specified in the parameter file. For the first round of ChemSTEP, the best-scoring molecule from the seed set is assigned as the first beacon. Subsequent beacons would be molecules that score well (above pProp thresh), and are as diverse as possible from already assigned beacons. Greedy max-diversity selection. <br /> <br /> During assignment, ECFP4 Tanimoto distances between the assigned beacon and all potential beacons are calculated. The second beacon is the molecule with the maximum Td from beacon 1. Beacon 3 will be the molecule that has the highest Td from beacon 1 and beacon 2. This continues until the number of beacons specified in the parameter file is reached. In iterative rounds of ChemSTEP, the beacon selection is based on diversity from ALL PREVIOUSLY assigned beacons, including those in earlier rounds. <br /> <br /> <br /> '''2. ChainingLog sub-jobs:''' once beacons are assigned, ChemSTEP will launch a series of sub-jobs that calculate the Tanimoto distances of every molecule remaining in the library to the assigned beacons. Calculated distances to the NEAREST beacon (minimum-minimum Td) are updated in the mintddistrib_*.npy files within the /output directory. <br /> <br /> <br /> '''3. Gathering of SMILES for prioritization:''' When all Td calculations are completed, the algo will select N number of molecules for prioritization, the number of which is specified by the user in the parameter file. Molecules that are ''closest in chemical space'' to beacons are prioritized. I.e. if round size = 1 million, the 1 million molecules with the smallest min-Td to ANY beacon (current or previous rounds) are prioritized. chemstep_algo.log will provide a &quot;max-minTd&quot; for each round, which is the Tanimoto distance of the most dissimilar molecule prioritized to any one beacon per round. The prioritized molecules are output in /output/complete_info as smi_round_{}.smi <br /> <br /> What the user need: DOCKFILES, directories for (1) docking (2) building and (3) running ChemSTEP. '''All iterative rounds of ChemSTEP should be run in the SAME directory.''' <br /> <br /> '''1. Source ChemSTEP virtual environment on Wynton'''<br /> source /wynton/group/bks/work/shared/kholland/chemstep_0.3.1.5/bin/activate<br /> <br /> <br /> '''2. Copy seed-set SDI file into your docking directory'''<br /> This directory should already contain your dockfiles, with INDOCK parameters set to your liking. In this step, we are copying in a split database index file (SDI) containing paths to bundles of db2 files. This seed set contains 100 million molecules sampled randomly from the total virtual library, currently 1.1 trillion molecules.<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/XR_seed_set_v0p0.wynton.sdi .<br /> <br /> <br /> if you are interested in running on a library of 13.2B compounds from Enamine REAL, there are several seed set sizes available: 130k, 1.3M, 13M, and 26M. <br /> cp /wynton/group/bks/work/shared/kholland/chemstep_13B/130K_seeds.wynton.sdi .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_13B/1.3M_seeds.wynton.sdi .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_13B/13M_seeds.wynton.sdi .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_13B/26M_seeds.wynton.sdi .<br /> <br /> <br /> '''3. Dock seed set to your receptor of interest using DOCK 3.8''' /docking directions taken from docs.docking.org. This is meant to be done as you would do a normal LSD. <br /> export MOLECULES_DIR_TO_BIND=[outermost folder containing the molecules to dock, just your base docking directory]<br /> export DOCKFILES=[path to your dockfiles]<br /> export INPUT_FOLDER=[the folder containing your .sdi file(s)]<br /> export OUTPUT_FOLDER=[where you want the output ]<br /> <br /> /wynton/group/bks/work/bwhall61/needs_github/super_dock3r.sh<br /> <br /> Wait for docking to complete. Next, you must extract all molecule IDs and corresponding DOCK scores from above. To do so, run the following commands in the base docking directory containing your docking files and output folder, while logged into a dev node (do this in a screen!!!):<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/get_scores.py .<br /> python get_scores.py 0<br /> <br /> <br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/get_scores.py .<br /> python get_scores.py 0<br /> <br /> <br /> This script expects a directory named &quot;output*&quot; within the CWD. If your output from docking follows different organizational conventions, vim into get_scores.py and change the path. Successful output of the script will be a file named &quot;scores_round_0.txt&quot;. For iterative rounds, pass increasing numbers into the command line. i.e. When docking the first round of prioritized molecules, pass [1] for scores_round_1.txt. <br /> <br /> '''4. Convert scores and molecule IDS into NumPY arrays for ChemSTEP recognition. Requires ChemSTEP venv'''<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/convert_scores_to_npy.py .<br /> python convert_scores_to_npy.py 0<br /> <br /> <br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/convert_scores_to_npy.py .<br /> python convert_scores_to_npy.py 0<br /> <br /> For python speed purposes, ChemSTEP requires that DOCK scores and IDs be give in the form of a NumPY array. In this step, we are taking our readable scores file and converting them for ChemSTEP recognition. This script expects a txt file with Molecule IDS and DOCK scores (no header). Use the same round number you used in step 3. As run above, this will output two files named scores_round_0.npy and indices_round_0.npy that contain line-matched indices (determined from Mol ID) and their respective docking scores. <br /> <br /> '''5. Enter into or make a directory to run ChemSTEP in. Copy in necessary files for initiating ChemSTEP: params.txt, run_chemstep.py and launch_chemstep_as_job.sh. Copy in your score and indices numpy files as well, which should be in your docking directory.'''<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/params.txt .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/run_chemstep.py .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/launch_chemstep_as_job.sh .<br /> <br /> <br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/params.txt .<br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/run_chemstep.py .<br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/launch_chemstep_as_job.sh .<br /> <br /> '''6. Edit params.txt file''' This ONLY needs to be edited for the initial round of ChemSTEP. The parameters outlined here will be carried through all rounds of ChemSTEP. <br /> seed_indices_file: /absoulte/path/to/your/indices_round_0.npy<br /> seed_scores_file: /absolute/path/to/your/scores_round_0.npy<br /> hit_pprop: 5<br /> n_docked_per_round: 10000000<br /> max_beacons: 150<br /> max_n_rounds: 250<br /> <br /> Within params.txt, add the absolute paths to the ChemSTEP-readable score and indices NumPY arrays. The rest of the values within the params.txt file are left to the discretion of the user, with some considerations below. <br /> <br /> '''pProp:''' this value will define what is considered a &quot;virtual hit&quot; in your campaign. pProp is defined as the -log(rank %) of a molecule within the total library score-distribution. For example, a pProp of 4 in the 1.1T XReal space is equivalent the top 0.01% of the library, corresponding to the top 110 million molecules. pProp 5 = 0.001% = 11 million virtual hits, etc. From the seed set, ChemSTEP will estimate a DOCK score value in kcal/mol that associated with the lower-limit of your desired pProp zone. Any molecules that score better than the threshold is considered a virtual hit in your campaign. Be mindful of seed set size when choosing desired pProp. Our suggestion is that the size of the seed set should be at least 10^(pProp +2).<br /> <br /> '''n_docked_per_round:''' this number is the desired molecules for prioritization per round. When choosing this value, note that this number of molecules must be built and docked in between each round of ChemSTEP. Prioritizing many molecules will slow building and docking speeds, and coupled with few beacons (below) may lead to decreased diversity. Prioritizing too few molecules may result in slower virtual hit recovery. Round size does not significantly impact the algorithm running time.** <br /> <br /> '''max_beacons:''' this is the number of diverse, well-scoring molecules ChemSTEP will use to guide prioritization per round. All molecules that score better than the assigned pProp threshold may be considered for beacons. By default, ChemSTEP chooses each set of beacons to be as maximally diverse as possible. Choosing too many beacons may result in decreased diversity between beacons, but too few beacons could hinder space exploration. If not enough molecules score above your pProp threshold, ChemSTEP may assign fewer beacons than the assigned value. <br /> <br /> '''max_n_rounds:''' if prospectively running ChemSTEP, as outlined in this wiki, no need to worry about this parameter.<br /> <br /> <br /> There should be no need to edit run_chemstep.py or the SGE wrapper script. At this time, we strongly suggest running ChemSTEP as a job array with 16 or 32 CPU slots requested. The number of cores must be specified when calling CSAlgo (in run_chemstep.py) and the SGE wrapper(launch_chemstep_as_job.sh). <br /> <br /> <br /> '''7. Run ChemSTEP''' with the following command: <br /> qsub launch_chemstep_as_job.sh <br /> <br /> <br /> This will launch the main ChemSTEP job to the SGE scheduler. Check your job status with the 'qstat' command. The algorithm will read the score and indices files provided, calculate pProp (for round 1) and assign beacons. As this job runs, it will launch a subsequent job array. These sub-jobs are the Chaining step of ChemSTEP, where each parallel worker calculates the Tanimoto distances of 100 million molecules from the virtual library to the beacons. These distances are then written to the files within the /output directory. When these jobs complete, the main job will read through all Tanimoto distances and pull molecules for prioritization. <br /> <br /> Running ChemSTEP successfully will result in the output of a SMILES file within output/complete_info/. You will also get (1) chemstep_algo.log and (2) chemstep_submission.log in the working directory after job submission. chemstep_algo.log contains the running output of the algorithm in a readable format, with timestamps, including the DOCK score associated with your desired pProp. chemstep_submission.log contains the output of ChemSTEP, as well as any information output by the SGE submission system (python errors, cluster issues, tracebacks if failed). These files will be updated with any information with iterative rounds of ChemSTEP run in the same directory. '''Visually inspect these files after each round of ChemSTEP to ensure the algorithm has picked your desired number of beacons and things ran smoothly.'''<br /> <br /> '''Troubleshooting:''' if no jobs are running and there is no SMI file, check the chemstep_submission.log first. Any traceback error or SGE error should give you some idea of why ChemSTEP failed. Some errors may be due to SGE or Wynton issues. If directed, look at a few .out files within /output/jobs/ . If a ChemSTEP run fails on your FIRST run, delete the output and log files, fix what needs to be fixed, and rerun. Errors in subsequent rounds during the chaining step can potentially corrupt chaining files within the /output directory that are needed for prioritization. At the very least, you will definitely have duplicates of beacons and information written to the log files. At this time, if your run fails during iterative rounds, it's best to start from the beginning. <br /> <br /> Prioritized molecules should be built, docked, and their scores fed back into ChemSTEP. More detailed instructions below: <br /> <br /> '''8. Build prioritized molecules (DOCK 3.8).''' /taken from docs.docking.org <br /> source /wynton/group/bks/soft/DOCK-3.8.5/env.sh<br /> <br /> python /wynton/group/bks/soft/DOCK-3.8.5/DOCK3.8/zinc22-3d/submit/submit_building_docker.py --output_folder building_output --bundle_size 1000 --minutes_per_mol 5 --skip_name_check --scheduler sge --container_software apptainer --container_path_or_name /wynton/group/bks/soft/DOCK-3.8.5/building_pipeline.sif smi_round_{}.smi<br /> <br /> When building has completed, you must write an SDI file with the complete paths to each built bundle and dock. you can do this with:<br /> find /path/to/your/building_output -name &quot;bundle.db2.tgz&quot; -type f &gt; round_{}.wynton.sdi<br /> <br /> <br /> '''Be sure to change the INDOCK file to save only poses that meet your score pProp score threshold calculated by ChemSTEP!''' Retrieve docking scores as convert to NumPy arrays as outlined above, updating the round number when running get_scores.py and convert_scores_to_npy.py. Copy new score and indices files into the directory you ran ChemSTEP in. If you are following along as a tutorial, you should have scores_round_1.npy and indices_round_1.npy from the previous step (from FIRST round of ChemSTEP prioritization). <br /> <br /> <br /> '''9. Set up for iterative rounds of ChemSTEP'''<br /> <br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/run_chemstep_iterative.py .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/launch_chemstep_iterative.sh .<br /> <br /> <br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/run_chemstep_iterative.py .<br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/launch_chemstep_iterative.sh .<br /> <br /> '''10. Run ChemSTEP'''<br /> qsub launch_chemstep_as_job.sh [round number]<br /> <br /> For the first iterative round, the round number is [2], and should increase by one for every subsequent round of ChemSTEP. The output will be smi_round_****.smi file. Repeat steps 8-10 for as many rounds as needed. The performance is reported in outputy/complete_info/run_summary.df, which contains the number of beacons selected, the number of molecules docked, the number of hits found, the distance threshold for the selected molecules to dock, and the last added beacon's distance to all previous beacons.<br /> <br /> <br /> ‘’’AUTOMATIC CHEMSTEP RUNNING DIRECTIONS’’’<br /> <br /> This version of ChemSTEP will automatically submit building and docking jobs inbetween rounds of ChemSTEP. Note, this version does not currently build with strain (to do: add this)<br /> <br /> ‘’’1. DOCK the seed set to your receptor as outlined above in steps 1-4’’’ <br /> <br /> successful output will be a scores_round_0.npy and indices_round_0.npy. <br /> <br /> <br /> ‘’’2. Copy necessary scripts into your working ChemSTEP directory’’’<br /> cp /wynton/group/bks/work/shared/kholland/auto_chemstep/scripts/params.txt .<br /> cp /wynton/group/bks/work/shared/kholland/auto_chemstep/scripts/run_chemstep.py .<br /> cp /wynton/group/bks/work/shared/kholland/auto_chemstep/scripts/launch_chemstep_as_job.sh .<br /> <br /> <br /> ‘’’3. Edit run_chemstep.py to add the path to your dockfiles’’’<br /> <br /> the algorithm needs to know where your dockfiles live in order to submit docking jobs. Make sure all INDOCK parameters are set correctly. Use the full path to the dockfiles. <br /> <br /> <br /> ‘’’4. Edit params.txt''' The parameters outlined here will be carried through all rounds of ChemSTEP. <br /> seed_indices_file: /absoulte/path/to/your/indices_round_0.npy<br /> seed_scores_file: /absolute/path/to/your/scores_round_0.npy<br /> hit_pprop: 5<br /> bundle_size: 1000<br /> n_docked_per_round: 10000000<br /> max_beacons: 150<br /> max_n_rounds: 250<br /> <br /> <br /> Within params.txt, add the absolute paths to the ChemSTEP-readable score and indices NumPY arrays. The rest of the values within the params.txt file are left to the discretion of the user. The '''bundle_size:''' parameter is required for auto docking mode of ChemSTEP. This is the number of molecules submitted at a job for building. <br /> <br /> <br /> '''5. Launch ChemSTEP'''<br /> <br /> qsub launch_chemstep_as_job.sh <br /> <br /> <br /> '''Troubleshooting:''' if your job errors out or stops running at any point, check the ''chemstep_submission.log'' file in the working directory. <br /> <br /> <br /> <br /> '''Helper scripts:''' a running list of scripts we have been using for analysis and visualization. <br /> <br /> to get SMILES for beacons: <br /> <br /> python /wynton/group/bks/work/bwhall61/needs_github/get_beacon_smi.py --beacon_df_path /path/to/your/chemstep/output/complete_info/beacons.df --n_workers 6 --outfile beacon_smiles.smi<br /> <br /> <br /> ** anecdotally true.</div>

Kholland http://wiki.docking.org/index.php?title=Running_ChemSTEP&diff=16956 2025-10-09T01:39:57Z

<p>Kholland: </p> <hr /> <div>last update: oct 8 2025 katie. current ver = 0.3.1.5 (automatic resubmission of failed SGE jobs). <br /> <br /> ChemSTEP (Chemical Space Traversal and Exploration Procedure) is an open-source, transparent acceleration algorithm for molecular docking capable of dealing with virtual libraries of several trillion compounds. This wiki page is a guide for BKS lab members to run ChemSTEP on Wynton HPC, using the current version of InifiSee XReal library (1.1T) or 13.3B library from Enamine REAL. '''for detailed instructions on the 13B space, see 'Running ChemSTEP on the 13B space' wiki page'''. For more general use directions, please refer to [ChemSTEP Read-the-Docs]. <br /> <br /> At a high-level, ChemSTEP is an iterative process that identifies molecules from the larger virtual library to prioritize for docking. First, we identify a random sample of the total library (termed &quot;seed set&quot;, round zero) and dock those molecules to the target of interest. From this seed set, we can calculate total-library pProp values (-log rank percentages) and the number of &quot;virtual hits&quot; in the total library (high-scoring molecules). ChemSTEP will identify a set of maximally diverse molecules that score above the desired pProp threshold (&quot;beacons&quot;) from the seed set. These beacons guide prioritization, where molecules chosen and output by ChemSTEP are close in chemical space to the beacons. Prioritized molecules are then built, docked, and returned to ChemSTEP for a second round of prioritization. This process is iterated until you reach desired virtual hit recovery, or you are no longer recovering virtual hits. <br /> <br /> <br /> Running the ChemSTEP algorithm on Wynton takes place in loosely three steps: (1) submission of the main ChemSTEP job, (2) ChainingLog sub-job array, and (3) gathering of SMILES for prioritization. <br /> <br /> '''1. Main ChemSTEP job:''' reads input files (DOCK scores and line-matched indices NumPy arrays) and assigns beacons. Beacons are a set of N maximally diverse molecules that score above the pProp threshold specified in the parameter file. For the first round of ChemSTEP, the best-scoring molecule from the seed set is assigned as the first beacon. Subsequent beacons would be molecules that score well (above pProp thresh), and are as diverse as possible from already assigned beacons. Greedy max-diversity selection. <br /> <br /> During assignment, ECFP4 Tanimoto distances between the assigned beacon and all potential beacons are calculated. The second beacon is the molecule with the maximum Td from beacon 1. Beacon 3 will be the molecule that has the highest Td from beacon 1 and beacon 2. This continues until the number of beacons specified in the parameter file is reached. In iterative rounds of ChemSTEP, the beacon selection is based on diversity from ALL PREVIOUSLY assigned beacons, including those in earlier rounds. <br /> <br /> <br /> '''2. ChainingLog sub-jobs:''' once beacons are assigned, ChemSTEP will launch a series of sub-jobs that calculate the Tanimoto distances of every molecule remaining in the library to the assigned beacons. Calculated distances to the NEAREST beacon (minimum-minimum Td) are updated in the mintddistrib_*.npy files within the /output directory. <br /> <br /> <br /> '''3. Gathering of SMILES for prioritization:''' When all Td calculations are completed, the algo will select N number of molecules for prioritization, the number of which is specified by the user in the parameter file. Molecules that are ''closest in chemical space'' to beacons are prioritized. I.e. if round size = 1 million, the 1 million molecules with the smallest min-Td to ANY beacon (current or previous rounds) are prioritized. chemstep_algo.log will provide a &quot;max-minTd&quot; for each round, which is the Tanimoto distance of the most dissimilar molecule prioritized to any one beacon per round. The prioritized molecules are output in /output/complete_info as smi_round_{}.smi <br /> <br /> What the user need: DOCKFILES, directories for (1) docking (2) building and (3) running ChemSTEP. '''All iterative rounds of ChemSTEP should be run in the SAME directory.''' <br /> <br /> '''1. Source ChemSTEP virtual environment on Wynton'''<br /> source /wynton/group/bks/work/shared/kholland/chemstep_0.3.1.5/bin/activate<br /> <br /> <br /> '''2. Copy seed-set SDI file into your docking directory'''<br /> This directory should already contain your dockfiles, with INDOCK parameters set to your liking. In this step, we are copying in a split database index file (SDI) containing paths to bundles of db2 files. This seed set contains 100 million molecules sampled randomly from the total virtual library, currently 1.1 trillion molecules.<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/XR_seed_set_v0p0.wynton.sdi .<br /> <br /> <br /> if you are interested in running on a library of 13.2B compounds from Enamine REAL, there are several seed set sizes available: 130k, 1.3M, 13M, and 26M. <br /> cp /wynton/group/bks/work/shared/kholland/chemstep_13B/130K_seeds.wynton.sdi .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_13B/1.3M_seeds.wynton.sdi .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_13B/13M_seeds.wynton.sdi .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_13B/26M_seeds.wynton.sdi .<br /> <br /> <br /> '''3. Dock seed set to your receptor of interest using DOCK 3.8''' /docking directions taken from docs.docking.org. This is meant to be done as you would do a normal LSD. <br /> export MOLECULES_DIR_TO_BIND=[outermost folder containing the molecules to dock, just your base docking directory]<br /> export DOCKFILES=[path to your dockfiles]<br /> export INPUT_FOLDER=[the folder containing your .sdi file(s)]<br /> export OUTPUT_FOLDER=[where you want the output ]<br /> <br /> /wynton/group/bks/work/bwhall61/needs_github/super_dock3r.sh<br /> <br /> Wait for docking to complete. Next, you must extract all molecule IDs and corresponding DOCK scores from above. To do so, run the following commands in the base docking directory containing your docking files and output folder, while logged into a dev node (do this in a screen!!!):<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/get_scores.py .<br /> python get_scores.py 0<br /> <br /> <br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/get_scores.py .<br /> python get_scores.py 0<br /> <br /> <br /> This script expects a directory named &quot;output*&quot; within the CWD. If your output from docking follows different organizational conventions, vim into get_scores.py and change the path. Successful output of the script will be a file named &quot;scores_round_0.txt&quot;. For iterative rounds, pass increasing numbers into the command line. i.e. When docking the first round of prioritized molecules, pass [1] for scores_round_1.txt. <br /> <br /> '''4. Convert scores and molecule IDS into NumPY arrays for ChemSTEP recognition. Requires ChemSTEP venv'''<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/convert_scores_to_npy.py .<br /> python convert_scores_to_npy.py 0<br /> <br /> <br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/convert_scores_to_npy.py .<br /> python convert_scores_to_npy.py 0<br /> <br /> For python speed purposes, ChemSTEP requires that DOCK scores and IDs be give in the form of a NumPY array. In this step, we are taking our readable scores file and converting them for ChemSTEP recognition. This script expects a txt file with Molecule IDS and DOCK scores. Use the same round number you used in step 3. As run above, this will output two files named scores_round_0.npy and indices_round_0.npy that contain line-matched indices (determined from Mol ID) and their respective docking scores. <br /> <br /> '''5. Enter into or make a directory to run ChemSTEP in. Copy in necessary files for initiating ChemSTEP: params.txt, run_chemstep.py and launch_chemstep_as_job.sh. Copy in your score and indices numpy files as well, which should be in your docking directory.'''<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/params.txt .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/run_chemstep.py .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/launch_chemstep_as_job.sh .<br /> <br /> <br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/params.txt .<br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/run_chemstep.py .<br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/launch_chemstep_as_job.sh .<br /> <br /> '''6. Edit params.txt file''' This ONLY needs to be edited for the initial round of ChemSTEP. The parameters outlined here will be carried through all rounds of ChemSTEP. <br /> seed_indices_file: /absoulte/path/to/your/indices_round_0.npy<br /> seed_scores_file: /absolute/path/to/your/scores_round_0.npy<br /> hit_pprop: 5<br /> n_docked_per_round: 10000000<br /> max_beacons: 150<br /> max_n_rounds: 250<br /> <br /> Within params.txt, add the absolute paths to the ChemSTEP-readable score and indices NumPY arrays. The rest of the values within the params.txt file are left to the discretion of the user, with some considerations below. <br /> <br /> '''pProp:''' this value will define what is considered a &quot;virtual hit&quot; in your campaign. pProp is defined as the -log(rank %) of a molecule within the total library score-distribution. For example, a pProp of 4 in the 1.1T XReal space is equivalent the top 0.01% of the library, corresponding to the top 110 million molecules. pProp 5 = 0.001% = 11 million virtual hits, etc. From the seed set, ChemSTEP will estimate a DOCK score value in kcal/mol that associated with the lower-limit of your desired pProp zone. Any molecules that score better than the threshold is considered a virtual hit in your campaign. Be mindful of seed set size when choosing desired pProp. Our suggestion is that the size of the seed set should be at least 10^(pProp +2).<br /> <br /> '''n_docked_per_round:''' this number is the desired molecules for prioritization per round. When choosing this value, note that this number of molecules must be built and docked in between each round of ChemSTEP. Prioritizing many molecules will slow building and docking speeds, and coupled with few beacons (below) may lead to decreased diversity. Prioritizing too few molecules may result in slower virtual hit recovery. Round size does not significantly impact the algorithm running time.** <br /> <br /> '''max_beacons:''' this is the number of diverse, well-scoring molecules ChemSTEP will use to guide prioritization per round. All molecules that score better than the assigned pProp threshold may be considered for beacons. By default, ChemSTEP chooses each set of beacons to be as maximally diverse as possible. Choosing too many beacons may result in decreased diversity between beacons, but too few beacons could hinder space exploration. If not enough molecules score above your pProp threshold, ChemSTEP may assign fewer beacons than the assigned value. <br /> <br /> '''max_n_rounds:''' if prospectively running ChemSTEP, as outlined in this wiki, no need to worry about this parameter.<br /> <br /> <br /> There should be no need to edit run_chemstep.py or the SGE wrapper script. At this time, we strongly suggest running ChemSTEP as a job array with 16 or 32 CPU slots requested. The number of cores must be specified when calling CSAlgo (in run_chemstep.py) and the SGE wrapper(launch_chemstep_as_job.sh). <br /> <br /> <br /> '''7. Run ChemSTEP''' with the following command: <br /> qsub launch_chemstep_as_job.sh <br /> <br /> <br /> This will launch the main ChemSTEP job to the SGE scheduler. Check your job status with the 'qstat' command. The algorithm will read the score and indices files provided, calculate pProp (for round 1) and assign beacons. As this job runs, it will launch a subsequent job array. These sub-jobs are the Chaining step of ChemSTEP, where each parallel worker calculates the Tanimoto distances of 100 million molecules from the virtual library to the beacons. These distances are then written to the files within the /output directory. When these jobs complete, the main job will read through all Tanimoto distances and pull molecules for prioritization. <br /> <br /> Running ChemSTEP successfully will result in the output of a SMILES file within output/complete_info/. You will also get (1) chemstep_algo.log and (2) chemstep_submission.log in the working directory after job submission. chemstep_algo.log contains the running output of the algorithm in a readable format, with timestamps, including the DOCK score associated with your desired pProp. chemstep_submission.log contains the output of ChemSTEP, as well as any information output by the SGE submission system (python errors, cluster issues, tracebacks if failed). These files will be updated with any information with iterative rounds of ChemSTEP run in the same directory. '''Visually inspect these files after each round of ChemSTEP to ensure the algorithm has picked your desired number of beacons and things ran smoothly.'''<br /> <br /> '''Troubleshooting:''' if no jobs are running and there is no SMI file, check the chemstep_submission.log first. Any traceback error or SGE error should give you some idea of why ChemSTEP failed. Some errors may be due to SGE or Wynton issues. If directed, look at a few .out files within /output/jobs/ . If a ChemSTEP run fails on your FIRST run, delete the output and log files, fix what needs to be fixed, and rerun. Errors in subsequent rounds during the chaining step can potentially corrupt chaining files within the /output directory that are needed for prioritization. At the very least, you will definitely have duplicates of beacons and information written to the log files. At this time, if your run fails during iterative rounds, it's best to start from the beginning. <br /> <br /> Prioritized molecules should be built, docked, and their scores fed back into ChemSTEP. More detailed instructions below: <br /> <br /> '''8. Build prioritized molecules (DOCK 3.8).''' /taken from docs.docking.org <br /> source /wynton/group/bks/soft/DOCK-3.8.5/env.sh<br /> <br /> python /wynton/group/bks/soft/DOCK-3.8.5/DOCK3.8/zinc22-3d/submit/submit_building_docker.py --output_folder building_output --bundle_size 1000 --minutes_per_mol 5 --skip_name_check --scheduler sge --container_software apptainer --container_path_or_name /wynton/group/bks/soft/DOCK-3.8.5/building_pipeline.sif smi_round_{}.smi<br /> <br /> When building has completed, you must write an SDI file with the complete paths to each built bundle and dock. you can do this with:<br /> find /path/to/your/building_output -name &quot;bundle.db2.tgz&quot; -type f &gt; round_{}.wynton.sdi<br /> <br /> <br /> '''Be sure to change the INDOCK file to save only poses that meet your score pProp score threshold calculated by ChemSTEP!''' Retrieve docking scores as convert to NumPy arrays as outlined above, updating the round number when running get_scores.py and convert_scores_to_npy.py. Copy new score and indices files into the directory you ran ChemSTEP in. If you are following along as a tutorial, you should have scores_round_1.npy and indices_round_1.npy from the previous step (from FIRST round of ChemSTEP prioritization). <br /> <br /> <br /> '''9. Set up for iterative rounds of ChemSTEP'''<br /> <br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/run_chemstep_iterative.py .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/launch_chemstep_iterative.sh .<br /> <br /> <br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/run_chemstep_iterative.py .<br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/launch_chemstep_iterative.sh .<br /> <br /> '''10. Run ChemSTEP'''<br /> qsub launch_chemstep_as_job.sh [round number]<br /> <br /> For the first iterative round, the round number is [2], and should increase by one for every subsequent round of ChemSTEP. The output will be smi_round_****.smi file. Repeat steps 8-10 for as many rounds as needed. The performance is reported in outputy/complete_info/run_summary.df, which contains the number of beacons selected, the number of molecules docked, the number of hits found, the distance threshold for the selected molecules to dock, and the last added beacon's distance to all previous beacons.<br /> <br /> <br /> '''Helper scripts:''' a running list of scripts we have been using for analysis and visualization. <br /> <br /> to get SMILES for beacons: <br /> <br /> python /wynton/group/bks/work/bwhall61/needs_github/get_beacon_smi.py --beacon_df_path /path/to/your/chemstep/output/complete_info/beacons.df --n_workers 6 --outfile beacon_smiles.smi<br /> <br /> <br /> ** anecdotally true.</div>

Kholland http://wiki.docking.org/index.php?title=Running_ChemSTEP&diff=16955 2025-10-09T01:35:25Z

<p>Kholland: </p> <hr /> <div>last update: oct 8 2025 katie. current ver = 0.3.1.5 (automatic resubmission of failed SGE jobs). <br /> <br /> ChemSTEP (Chemical Space Traversal and Exploration Procedure) is an open-source, transparent acceleration algorithm for molecular docking capable of dealing with virtual libraries of several trillion compounds. This wiki page is a guide for BKS lab members to run ChemSTEP on Wynton HPC, using the current version of InifiSee XReal library (1.1T) or 13.3B library from Enamine REAL. '''for detailed instructions on the 13B space, see 'Running ChemSTEP on the 13B space' wiki page'''. For more general use directions, please refer to [ChemSTEP Read-the-Docs]. <br /> <br /> At a high-level, ChemSTEP is an iterative process that identifies molecules from the larger virtual library to prioritize for docking. First, we identify a random sample of the total library (termed &quot;seed set&quot;, round zero) and dock those molecules to the target of interest. From this seed set, we can calculate total-library pProp values (-log rank percentages) and the number of &quot;virtual hits&quot; in the total library (high-scoring molecules). ChemSTEP will identify a set of maximally diverse molecules that score above the desired pProp threshold (&quot;beacons&quot;) from the seed set. These beacons guide prioritization, where molecules chosen and output by ChemSTEP are close in chemical space to the beacons. Prioritized molecules are then built, docked, and returned to ChemSTEP for a second round of prioritization. This process is iterated until you reach desired virtual hit recovery, or you are no longer recovering virtual hits. <br /> <br /> <br /> Running the ChemSTEP algorithm on Wynton takes place in loosely three steps: (1) submission of the main ChemSTEP job, (2) ChainingLog sub-job array, and (3) gathering of SMILES for prioritization. <br /> <br /> '''1. Main ChemSTEP job:''' reads input files (DOCK scores and line-matched indices NumPy arrays) and assigns beacons. Beacons are a set of N maximally diverse molecules that score above the pProp threshold specified in the parameter file. For the first round of ChemSTEP, the best-scoring molecule from the seed set is assigned as the first beacon. Subsequent beacons would be molecules that score well (above pProp thresh), and are as diverse as possible from already assigned beacons. Greedy max-diversity selection. <br /> <br /> During assignment, ECFP4 Tanimoto distances between the assigned beacon and all potential beacons are calculated. The second beacon is the molecule with the maximum Td from beacon 1. Beacon 3 will be the molecule that has the highest Td from beacon 1 and beacon 2. This continues until the number of beacons specified in the parameter file is reached. In iterative rounds of ChemSTEP, the beacon selection is based on diversity from ALL PREVIOUSLY assigned beacons, including those in earlier rounds. <br /> <br /> <br /> '''2. ChainingLog sub-jobs:''' once beacons are assigned, ChemSTEP will launch a series of sub-jobs that calculate the Tanimoto distances of every molecule remaining in the library to the assigned beacons. Calculated distances to the NEAREST beacon (minimum-minimum Td) are updated in the mintddistrib_*.npy files within the /output directory. <br /> <br /> <br /> '''3. Gathering of SMILES for prioritization:''' When all Td calculations are completed, the algo will select N number of molecules for prioritization, the number of which is specified by the user in the parameter file. Molecules that are ''closest in chemical space'' to beacons are prioritized. I.e. if round size = 1 million, the 1 million molecules with the smallest min-Td to ANY beacon (current or previous rounds) are prioritized. chemstep_algo.log will provide a &quot;max-minTd&quot; for each round, which is the Tanimoto distance of the most dissimilar molecule prioritized to any one beacon per round. The prioritized molecules are output in /output/complete_info as smi_round_{}.smi <br /> <br /> What the user need: DOCKFILES, directories for (1) docking (2) building and (3) running ChemSTEP. '''All iterative rounds of ChemSTEP should be run in the SAME directory.''' <br /> <br /> '''1. Source ChemSTEP virtual environment on Wynton'''<br /> source /wynton/group/bks/work/shared/kholland/chemstep_0.3.1.5/bin/activate<br /> <br /> <br /> '''2. Copy seed-set SDI file into your docking directory'''<br /> This directory should already contain your dockfiles, with INDOCK parameters set to your liking. In this step, we are copying in a split database index file (SDI) containing paths to bundles of db2 files. This seed set contains 100 million molecules sampled randomly from the total virtual library, currently 1.1 trillion molecules.<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/XR_seed_set_v0p0.wynton.sdi .<br /> <br /> <br /> if you are interested in running on a library of 13.2B compounds from Enamine REAL, there are several seed set sizes available: 130k, 1.3M, 13M, and 26M. <br /> cp /wynton/group/bks/work/shared/kholland/chemstep_13B/130K_seeds.wynton.sdi .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_13B/1.3M_seeds.wynton.sdi .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_13B/13M_seeds.wynton.sdi .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_13B/26M_seeds.wynton.sdi .<br /> <br /> <br /> '''3. Dock seed set to your receptor of interest using DOCK 3.8''' /docking directions taken from docs.docking.org. This is meant to be done as you would do a normal LSD. <br /> export MOLECULES_DIR_TO_BIND=[outermost folder containing the molecules to dock, just your base docking directory]<br /> export DOCKFILES=[path to your dockfiles]<br /> export INPUT_FOLDER=[the folder containing your .sdi file(s)]<br /> export OUTPUT_FOLDER=[where you want the output ]<br /> <br /> /wynton/group/bks/work/bwhall61/needs_github/super_dock3r.sh<br /> <br /> Wait for docking to complete. Next, you must extract all molecule IDs and corresponding DOCK scores from above. To do so, run the following commands in the base docking directory containing your docking files and output folder, while logged into a dev node (do this in a screen!!!):<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/get_scores.py .<br /> python get_scores.py 0<br /> <br /> <br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/get_scores.py .<br /> python get_scores.py 0<br /> <br /> <br /> This script expects a directory named &quot;output*&quot; within the CWD. If your output from docking follows different organizational conventions, vim into get_scores.py and change the path. Successful output of the script will be a file named &quot;scores_round_0.txt&quot;. For iterative rounds, pass increasing numbers into the command line. i.e. When docking the first round of prioritized molecules, pass [1] for scores_round_1.txt. <br /> <br /> '''4. Convert scores and molecule IDS into NumPY arrays for ChemSTEP recognition. Requires ChemSTEP venv'''<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/convert_scores_to_npy.py .<br /> python convert_scores_to_npy.py 0<br /> <br /> <br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/convert_scores_to_npy.py .<br /> python convert_scores_to_npy.py 0<br /> <br /> For python speed purposes, ChemSTEP requires that DOCK scores and IDs be give in the form of a NumPY array. In this step, we are taking our readable scores file and converting them for ChemSTEP recognition. This script expects a txt file with Molecule IDS and DOCK scores. Use the same round number you used in step 3. As run above, this will output two files named scores_round_0.npy and indices_round_0.npy that contain line-matched indices (determined from Mol ID) and their respective docking scores. <br /> <br /> '''5. Enter into or make a directory to run ChemSTEP in. Copy in necessary files for initiating ChemSTEP: params.txt, run_chemstep.py and launch_chemstep_as_job.sh. Copy in your score and indices numpy files as well, which should be in your docking directory.'''<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/params.txt .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/run_chemstep.py .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/launch_chemstep_as_job.sh .<br /> <br /> <br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/params.txt .<br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/run_chemstep.py .<br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/launch_chemstep_as_job.sh .<br /> <br /> '''6. Edit params.txt file''' This ONLY needs to be edited for the initial round of ChemSTEP. The parameters outlined here will be carried through all rounds of ChemSTEP. <br /> seed_indices_file: /absoulte/path/to/your/indices_round_0.npy<br /> seed_scores_file: /absolute/path/to/your/scores_round_0.npy<br /> hit_pprop: 5<br /> n_docked_per_round: 10000000<br /> max_beacons: 150<br /> max_n_rounds: 250<br /> <br /> Within params.txt, add the absolute paths to the ChemSTEP-readable score and indices NumPY arrays. The rest of the values within the params.txt file are left to the discretion of the user, with some considerations below. <br /> <br /> '''pProp:''' this value will define what is considered a &quot;virtual hit&quot; in your campaign. pProp is defined as the -log(rank %) of a molecule within the total library score-distribution. For example, a pProp of 4 in the 1.1T XReal space is equivalent the top 0.01% of the library, corresponding to the top 110 million molecules. pProp 5 = 0.001% = 11 million virtual hits, etc. From the seed set, ChemSTEP will estimate a DOCK score value in kcal/mol that associated with the lower-limit of your desired pProp zone. Any molecules that score better than the threshold is considered a virtual hit in your campaign. Be mindful of seed set size when choosing desired pProp. Our suggestion is that the size of the seed set should be at least 10^(pProp +2).<br /> <br /> '''n_docked_per_round:''' this number is the desired molecules for prioritization per round. When choosing this value, note that this number of molecules must be built and docked in between each round of ChemSTEP. Prioritizing many molecules will slow building and docking speeds, and coupled with few beacons (below) may lead to decreased diversity. Prioritizing too few molecules may result in slower virtual hit recovery. Round size does not significantly impact the algorithm running time.** <br /> <br /> '''max_beacons:''' this is the number of diverse, well-scoring molecules ChemSTEP will use to guide prioritization per round. All molecules that score better than the assigned pProp threshold may be considered for beacons. By default, ChemSTEP chooses each set of beacons to be as maximally diverse as possible. Choosing too many beacons may result in decreased diversity between beacons, but too few beacons could hinder space exploration. If not enough molecules score above your pProp threshold, ChemSTEP may assign fewer beacons than the assigned value. <br /> <br /> '''max_n_rounds:''' if prospectively running ChemSTEP, as outlined in this wiki, no need to worry about this parameter.<br /> <br /> <br /> There should be no need to edit run_chemstep.py or the SGE wrapper script. At this time, we strongly suggest running ChemSTEP as a job array with 16 or 32 CPU slots requested. The number of cores must be specified when calling CSAlgo (in run_chemstep.py) and the SGE wrapper(launch_chemstep_as_job.sh). <br /> <br /> <br /> '''7. Run ChemSTEP''' with the following command: <br /> qsub launch_chemstep_as_job.sh <br /> <br /> <br /> This will launch the main ChemSTEP job to the SGE scheduler. Check your job status with the 'qstat' command. The algorithm will read the score and indices files provided, calculate pProp (for round 1) and assign beacons. As this job runs, it will launch a subsequent job array. These sub-jobs are the Chaining step of ChemSTEP, where each parallel worker calculates the Tanimoto distances of 100 million molecules from the virtual library to the beacons. These distances are then written to the files within the /output directory. When these jobs complete, the main job will read through all Tanimoto distances and pull molecules for prioritization. <br /> <br /> Running ChemSTEP successfully will result in the output of a SMILES file within output/complete_info/. You will also get (1) chemstep_algo.log and (2) chemstep_submission.log in the working directory after job submission. chemstep_algo.log contains the running output of the algorithm in a readable format, with timestamps, including the DOCK score associated with your desired pProp. chemstep_submission.log contains the output of ChemSTEP, as well as any information output by the SGE submission system (python errors, cluster issues, tracebacks if failed). These files will be updated with any information with iterative rounds of ChemSTEP run in the same directory. '''Visually inspect these files after each round of ChemSTEP to ensure the algorithm has picked your desired number of beacons and things ran smoothly.'''<br /> <br /> '''Troubleshooting:''' if no jobs are running and there is no SMI file, check the chemstep_submission.log first. Any traceback error or SGE error should give you some idea of why ChemSTEP failed. Some errors may be due to SGE or Wynton issues. If directed, look at a few .out files within /output/jobs/ . If a ChemSTEP run fails on your FIRST run, delete the output and log files, fix what needs to be fixed, and rerun. Errors in subsequent rounds during the chaining step can potentially corrupt chaining files within the /output directory that are needed for prioritization. At the very least, you will definitely have duplicates of beacons and information written to the log files. At this time, if your run fails during iterative rounds, it's best to start from the beginning. <br /> <br /> Prioritized molecules should be built, docked, and their scores fed back into ChemSTEP. More detailed instructions below: <br /> <br /> '''8. Build prioritized molecules (DOCK 3.8).''' /taken from docs.docking.org <br /> source /wynton/group/bks/soft/DOCK-3.8.5/env.sh<br /> <br /> python /wynton/group/bks/soft/DOCK-3.8.5/DOCK3.8/zinc22-3d/submit/submit_building_docker.py --output_folder building_output --bundle_size 1000 --minutes_per_mol 5 --skip_name_check --scheduler sge --container_software apptainer --container_path_or_name /wynton/group/bks/soft/DOCK-3.8.5/building_pipeline.sif smi_round_{}.smi<br /> <br /> When building has completed, you must write an SDI file with the complete paths to each built bundle and dock. you can do this with:<br /> find /path/to/your/building_output -name &quot;bundle.db2.tgz&quot; -type f &gt; round_{}.wynton.sdi<br /> <br /> <br /> '''Be sure to change the INDOCK file to save only poses that meet your score pProp score threshold calculated by ChemSTEP!''' Retrieve docking scores as convert to NumPy arrays as outlined above, updating the round number when running get_scores.py and convert_scores_to_npy.py. Copy new score and indices files into the directory you ran ChemSTEP in. If you are following along as a tutorial, you should have scores_round_1.npy and indices_round_1.npy from the previous step (from FIRST round of ChemSTEP prioritization). <br /> <br /> <br /> '''9. Set up for iterative rounds of ChemSTEP'''<br /> <br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/run_chemstep_iterative.py .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/launch_chemstep_iterative.sh .<br /> <br /> <br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/run_chemstep_iterative.py .<br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/launch_chemstep_iterative.sh .<br /> <br /> '''10. Run ChemSTEP'''<br /> qsub launch_chemstep_as_job.sh [round number]<br /> <br /> For the first iterative round, the round number is [2], and should increase by one for every subsequent round of ChemSTEP. The output will be smi_round_****.smi file. Repeat steps 8-10 for as many rounds as needed. The performance is reported in outputy/complete_info/run_summary.df, which contains the number of beacons selected, the number of molecules docked, the number of hits found, the distance threshold for the selected molecules to dock, and the last added beacon's distance to all previous beacons.<br /> <br /> <br /> ** anecdotally true.</div>

Kholland http://wiki.docking.org/index.php?title=Running_ChemSTEP&diff=16954 2025-10-09T01:32:03Z

<p>Kholland: </p> <hr /> <div>last update: oct 8 2025 katie. current ver = 0.3.1.5 (automatic resubmission of failed SGE jobs). <br /> <br /> ChemSTEP (Chemical Space Traversal and Exploration Procedure) is an open-source, transparent acceleration algorithm for molecular docking capable of dealing with virtual libraries of several trillion compounds. This wiki page is a guide for BKS lab members to run ChemSTEP on Wynton HPC, using the current version of InifiSee XReal library (1.1T) or 13.3B library from Enamine REAL. '''for detailed instructions on the 13B space, see 'Running ChemSTEP on the 13B space' wiki page'''. For more general use directions, please refer to [ChemSTEP Read-the-Docs]. <br /> <br /> At a high-level, ChemSTEP is an iterative process that identifies molecules from the larger virtual library to prioritize for docking. First, we identify a random sample of the total library (termed &quot;seed set&quot;, round zero) and dock those molecules to the target of interest. From this seed set, we can calculate total-library pProp values (-log rank percentages) and the number of &quot;virtual hits&quot; in the total library (high-scoring molecules). ChemSTEP will identify a set of maximally diverse molecules that score above the desired pProp threshold (&quot;beacons&quot;) from the seed set. These beacons guide prioritization, where molecules chosen and output by ChemSTEP are close in chemical space to the beacons. Prioritized molecules are then built, docked, and returned to ChemSTEP for a second round of prioritization. This process is iterated until you reach desired virtual hit recovery, or you are no longer recovering virtual hits. <br /> <br /> <br /> Running the ChemSTEP algorithm on Wynton takes place in loosely three steps: (1) submission of the main ChemSTEP job, (2) ChainingLog sub-job array, and (3) gathering of SMILES for prioritization. <br /> <br /> '''1. Main ChemSTEP job:''' reads input files (DOCK scores and line-matched indices NumPy arrays) and assigns beacons. Beacons are a set of N maximally diverse molecules that score above the pProp threshold specified in the parameter file. For the first round of ChemSTEP, the best-scoring molecule from the seed set is assigned as the first beacon. Subsequent beacons would be molecules that score well (above pProp thresh), and are as diverse as possible from already assigned beacons. Greedy max-diversity selection. <br /> <br /> During assignment, ECFP4 Tanimoto distances between the assigned beacon and all potential beacons are calculated. The second beacon is the molecule with the maximum Td from beacon 1. Beacon 3 will be the molecule that has the highest Td from beacon 1 and beacon 2. This continues until the number of beacons specified in the parameter file is reached. In iterative rounds of ChemSTEP, the beacon selection is based on diversity from ALL PREVIOUSLY assigned beacons, including those in earlier rounds. <br /> <br /> <br /> '''2. ChainingLog sub-jobs:''' once beacons are assigned, ChemSTEP will launch a series of sub-jobs that calculate the Tanimoto distances of every molecule remaining in the library to the assigned beacons. Calculated distances to the NEAREST beacon (minimum-minimum Td) are updated in the mintddistrib_*.npy files within the /output directory. <br /> <br /> <br /> '''3. Gathering of SMILES for prioritization:''' When all Td calculations are completed, the algo will select N number of molecules for prioritization, the number of which is specified by the user in the parameter file. Molecules that are ''closest in chemical space'' to beacons are prioritized. I.e. if round size = 1 million, the 1 million molecules with the smallest min-Td to ANY beacon (current or previous rounds) are prioritized. chemstep_algo.log will provide a &quot;max-minTd&quot; for each round, which is the Tanimoto distance of the most dissimilar molecule prioritized to any one beacon per round. The prioritized molecules are output in /output/complete_info as smi_round_{}.smi <br /> <br /> What the user need: DOCKFILES, directories for (1) docking (2) building and (3) running ChemSTEP. '''All iterative rounds of ChemSTEP should be run in the SAME directory.''' <br /> <br /> '''1. Source ChemSTEP virtual environment on Wynton'''<br /> source /wynton/group/bks/work/shared/kholland/chemstep_0.3.1.5/bin/activate<br /> <br /> <br /> '''2. Copy seed-set SDI file into your docking directory'''<br /> This directory should already contain your dockfiles, with INDOCK parameters set to your liking. In this step, we are copying in a split database index file (SDI) containing paths to bundles of db2 files. This seed set contains 100 million molecules sampled randomly from the total virtual library, currently 1.1 trillion molecules.<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/XR_seed_set_v0p0.wynton.sdi .<br /> <br /> <br /> if you are interested in running on a library of 13.2B compounds from Enamine REAL, there are several seed set sizes available: 130k, 1.3M, 13M, and 26M. <br /> cp /wynton/group/bks/work/shared/kholland/chemstep_13B/130K_seeds.wynton.sdi .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_13B/1.3M_seeds.wynton.sdi .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_13B/13M_seeds.wynton.sdi .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_13B/26M_seeds.wynton.sdi .<br /> <br /> <br /> '''3. Dock seed set to your receptor of interest using DOCK 3.8''' /docking directions taken from docs.docking.org. This is meant to be done as you would do a normal LSD. <br /> export MOLECULES_DIR_TO_BIND=[outermost folder containing the molecules to dock, just your base docking directory]<br /> export DOCKFILES=[path to your dockfiles]<br /> export INPUT_FOLDER=[the folder containing your .sdi file(s)]<br /> export OUTPUT_FOLDER=[where you want the output ]<br /> <br /> /wynton/group/bks/work/bwhall61/needs_github/super_dock3r.sh<br /> <br /> Wait for docking to complete. Next, you must extract all molecule IDs and corresponding DOCK scores from above. To do so, run the following commands in the base docking directory containing your docking files and output folder, while logged into a dev node (I suggest in a screen):<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/get_scores.py .<br /> python get_scores.py 0<br /> <br /> <br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/get_scores.py .<br /> python get_scores.py 0<br /> <br /> <br /> This script expects a directory named &quot;output*&quot; within the CWD. If your output from docking follows different naming conventions, vim into get_scores.py and change the path. The output will be a file named &quot;scores_round_0.txt&quot;. For iterative rounds, pass increasing numbers into the command line. i.e. When docking the first round of prioritized molecules, pass [1] for scores_round_1.txt. <br /> <br /> '''4. Convert scores and molecule IDS into NumPY arrays for ChemSTEP recognition. Requires ChemSTEP venv'''<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/convert_scores_to_npy.py .<br /> python convert_scores_to_npy.py 0<br /> <br /> <br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/convert_scores_to_npy.py .<br /> python convert_scores_to_npy.py 0<br /> <br /> For python speed purposes, ChemSTEP requires that DOCK scores and IDs be give in the form of a NumPY array. In this step, we are taking our readable scores file and converting them for ChemSTEP recognition. This script expects a txt file with Molecule IDS and DOCK scores. Use the same round number you used in step 3. As run above, this will output two files named scores_round_0.npy and indices_round_0.npy that contain line-matched indices (determined from Mol ID) and their respective docking scores. <br /> <br /> '''5. Enter into or make a directory to run ChemSTEP in. Copy in necessary files for initiating ChemSTEP: params.txt, run_chemstep.py and launch_chemstep_as_job.sh. Copy in your score and indices numpy files as well, which should be in your docking directory.'''<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/params.txt .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/run_chemstep.py .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/launch_chemstep_as_job.sh .<br /> <br /> <br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/params.txt .<br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/run_chemstep.py .<br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/launch_chemstep_as_job.sh .<br /> <br /> '''6. Edit params.txt file''' This ONLY needs to be edited for the initial round of ChemSTEP. The parameters outlined here will be carried through all rounds of ChemSTEP. <br /> seed_indices_file: /absoulte/path/to/your/indices_round_0.npy<br /> seed_scores_file: /absolute/path/to/your/scores_round_0.npy<br /> hit_pprop: 5<br /> n_docked_per_round: 10000000<br /> max_beacons: 150<br /> max_n_rounds: 250<br /> <br /> Within params.txt, add the absolute paths to the ChemSTEP-readable score and indices NumPY arrays. The rest of the values within the params.txt file are left to the discretion of the user, with some considerations below. <br /> <br /> '''pProp:''' this value will define what is considered a &quot;virtual hit&quot; in your campaign. pProp is defined as the -log(rank %) of a molecule within the total library score-distribution. For example, a pProp of 4 in the 1.1T XReal space is equivalent the top 0.01% of the library, corresponding to the top 110 million molecules. pProp 5 = 0.001% = 11 million virtual hits, etc. From the seed set, ChemSTEP will estimate a DOCK score value in kcal/mol that associated with the lower-limit of your desired pProp zone. Any molecules that score better than the threshold is considered a virtual hit in your campaign. Be mindful of seed set size when choosing desired pProp. Our suggestion is that the size of the seed set should be at least 10^(pProp +2).<br /> <br /> '''n_docked_per_round:''' this number is the desired molecules for prioritization per round. When choosing this value, note that this number of molecules must be built and docked in between each round of ChemSTEP. Prioritizing many molecules will slow building and docking speeds, and coupled with few beacons (below) may lead to decreased diversity. Prioritizing too few molecules may result in slower virtual hit recovery. Round size does not significantly impact the algorithm running time.** <br /> <br /> '''max_beacons:''' this is the number of diverse, well-scoring molecules ChemSTEP will use to guide prioritization per round. All molecules that score better than the assigned pProp threshold may be considered for beacons. By default, ChemSTEP chooses each set of beacons to be as maximally diverse as possible. Choosing too many beacons may result in decreased diversity between beacons, but too few beacons could hinder space exploration. If not enough molecules score above your pProp threshold, ChemSTEP may assign fewer beacons than the assigned value. <br /> <br /> '''max_n_rounds:''' if prospectively running ChemSTEP, as outlined in this wiki, no need to worry about this parameter.<br /> <br /> <br /> There should be no need to edit run_chemstep.py or the SGE wrapper script. At this time, we strongly suggest running ChemSTEP as a job array with 16 or 32 CPU slots requested. The number of cores must be specified when calling CSAlgo (in run_chemstep.py) and the SGE wrapper(launch_chemstep_as_job.sh). <br /> <br /> <br /> '''7. Run ChemSTEP''' with the following command: <br /> qsub launch_chemstep_as_job.sh <br /> <br /> <br /> This will launch the main ChemSTEP job to the SGE scheduler. Check your job status with the 'qstat' command. The algorithm will read the score and indices files provided, calculate pProp (for round 1) and assign beacons. As this job runs, it will launch a subsequent job array. These sub-jobs are the Chaining step of ChemSTEP, where each parallel worker calculates the Tanimoto distances of 100 million molecules from the virtual library to the beacons. These distances are then written to the files within the /output directory. When these jobs complete, the main job will read through all Tanimoto distances and pull molecules for prioritization. <br /> <br /> Running ChemSTEP successfully will result in the output of a SMILES file within output/complete_info/. You will also get (1) chemstep_algo.log and (2) chemstep_submission.log in the working directory after job submission. chemstep_algo.log contains the running output of the algorithm in a readable format, with timestamps, including the DOCK score associated with your desired pProp. chemstep_submission.log contains the output of ChemSTEP, as well as any information output by the SGE submission system (python errors, cluster issues, tracebacks if failed). These files will be updated with any information with iterative rounds of ChemSTEP run in the same directory. '''Visually inspect these files after each round of ChemSTEP to ensure the algorithm has picked your desired number of beacons and things ran smoothly.'''<br /> <br /> '''Troubleshooting:''' if no jobs are running and there is no SMI file, check the chemstep_submission.log first. Any traceback error or SGE error should give you some idea of why ChemSTEP failed. Some errors may be due to SGE or Wynton issues. If directed, look at a few .out files within /output/jobs/ . If a ChemSTEP run fails on your FIRST run, delete the output and log files, fix what needs to be fixed, and rerun. Errors in subsequent rounds during the chaining step can potentially corrupt chaining files within the /output directory that are needed for prioritization. At the very least, you will definitely have duplicates of beacons and information written to the log files. At this time, if your run fails during iterative rounds, it's best to start from the beginning. <br /> <br /> Prioritized molecules should be built, docked, and their scores fed back into ChemSTEP. More detailed instructions below: <br /> <br /> '''8. Build prioritized molecules (DOCK 3.8).''' /taken from docs.docking.org <br /> source /wynton/group/bks/soft/DOCK-3.8.5/env.sh<br /> <br /> python /wynton/group/bks/soft/DOCK-3.8.5/DOCK3.8/zinc22-3d/submit/submit_building_docker.py --output_folder building_output --bundle_size 1000 --minutes_per_mol 5 --skip_name_check --scheduler sge --container_software apptainer --container_path_or_name /wynton/group/bks/soft/DOCK-3.8.5/building_pipeline.sif smi_round_{}.smi<br /> <br /> When building has completed, you must write an SDI file with the complete paths to each built bundle and dock. you can do this with:<br /> find /path/to/your/building_output -name &quot;bundle.db2.tgz&quot; -type f &gt; round_{}.wynton.sdi<br /> <br /> <br /> '''Be sure to change the INDOCK file to save only poses that meet your score pProp score threshold calculated by ChemSTEP!''' Retrieve docking scores as convert to NumPy arrays as outlined above, updating the round number when running get_scores.py and convert_scores_to_npy.py. Copy new score and indices files into the directory you ran ChemSTEP in. If you are following along as a tutorial, you should have scores_round_1.npy and indices_round_1.npy from the previous step (from FIRST round of ChemSTEP prioritization). <br /> <br /> <br /> '''9. Set up for iterative rounds of ChemSTEP'''<br /> <br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/run_chemstep_iterative.py .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/launch_chemstep_iterative.sh .<br /> <br /> <br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/run_chemstep_iterative.py .<br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/launch_chemstep_iterative.sh .<br /> <br /> '''10. Run ChemSTEP'''<br /> qsub launch_chemstep_as_job.sh [round number]<br /> <br /> For the first iterative round, the round number is [2], and should increase by one for every subsequent round of ChemSTEP. The output will be smi_round_****.smi file. Repeat steps 8-10 for as many rounds as needed. The performance is reported in outputy/complete_info/run_summary.df, which contains the number of beacons selected, the number of molecules docked, the number of hits found, the distance threshold for the selected molecules to dock, and the last added beacon's distance to all previous beacons.<br /> <br /> <br /> ** anecdotally true.</div>

Kholland http://wiki.docking.org/index.php?title=Running_ChemSTEP&diff=16953 2025-10-09T01:30:42Z

<p>Kholland: </p> <hr /> <div>last update: oct 8 2025 katie. current ver = 0.3.1.5 (automatic resubmission of failed SGE jobs). <br /> <br /> ChemSTEP (Chemical Space Traversal and Exploration Procedure) is an open-source, transparent acceleration algorithm for molecular docking capable of dealing with virtual libraries of several trillion compounds. This wiki page is a guide for BKS lab members to run ChemSTEP on Wynton HPC, using the current version of InifiSee XReal library (1.1T) or 13.3B library from Enamine REAL. '''for detailed instructions on the 13B space, see 'Running ChemSTEP on the 13B space' wiki page'''. For more general use directions, please refer to [ChemSTEP Read-the-Docs]. <br /> <br /> At a high-level, ChemSTEP is an iterative process that identifies molecules from the larger virtual library to prioritize for docking. First, we identify a random sample of the total library (termed &quot;seed set&quot;, round zero) and dock those molecules to the target of interest. From this seed set, we can calculate total-library pProp values (-log rank percentages) and the number of &quot;virtual hits&quot; in the total library (high-scoring molecules). ChemSTEP will identify a set of maximally diverse molecules that score above the desired pProp threshold (&quot;beacons&quot;) from the seed set. These beacons guide prioritization, where molecules chosen and output by ChemSTEP are close in chemical space to the beacons. Prioritized molecules are then built, docked, and returned to ChemSTEP for a second round of prioritization. This process is iterated until you reach desired virtual hit recovery, or you are no longer recovering virtual hits. <br /> <br /> <br /> Running the ChemSTEP algorithm on Wynton takes place in loosely three steps: (1) submission of the main ChemSTEP job, (2) ChainingLog sub-job array, and (3) gathering of SMILES for prioritization. <br /> <br /> '''1. Main ChemSTEP job:''' reads input files (DOCK scores and line-matched indices NumPy arrays) and assigns beacons. Beacons are a set of N maximally diverse molecules that score above the pProp threshold specified in the parameter file. For the first round of ChemSTEP, the best-scoring molecule from the seed set is assigned as the first beacon. Subsequent beacons would be molecules that score well (above pProp thresh), and are as diverse as possible from already assigned beacons. Greedy max-diversity selection. <br /> <br /> During assignment, ECFP4 Tanimoto distances between the assigned beacon and all potential beacons are calculated. The second beacon is the molecule with the maximum Td from beacon 1. Beacon 3 will be the molecule that has the highest Td from beacon 1 and beacon 2. This continues until the number of beacons specified in the parameter file is reached. In iterative rounds of ChemSTEP, the beacon selection is based on diversity from ALL PREVIOUSLY assigned beacons, including those in earlier rounds. <br /> <br /> <br /> '''2. ChainingLog sub-jobs:''' once beacons are assigned, ChemSTEP will launch a series of sub-jobs that calculate the Tanimoto distances of every molecule remaining in the library to the assigned beacons. Calculated distances to the NEAREST beacon (minimum-minimum Td) are updated in the mintddistrib_*.npy files within the /output directory. <br /> <br /> <br /> '''3. Gathering of SMILES for prioritization:''' When all Td calculations are completed, the algo will select N number of molecules for prioritization, the number of which is specified by the user in the parameter file. Molecules that are ''closest in chemical space'' to beacons are prioritized. I.e. if round size = 1 million, the 1 million molecules with the smallest min-Td to ANY beacon (current or previous rounds) are prioritized. chemstep_algo.log will provide a &quot;max-minTd&quot; for each round, which is the Tanimoto distance of the most dissimilar molecule prioritized to any one beacon per round. The prioritized molecules are output in /output/complete_info as smi_round_{}.smi <br /> <br /> What the user need: DOCKFILES, directories for (1) docking (2) building and (3) running ChemSTEP. '''All iterative rounds of ChemSTEP should be run in the SAME directory.''' <br /> <br /> '''1. Source ChemSTEP virtual environment on Wynton'''<br /> source /wynton/group/bks/work/shared/kholland/chemstep_0.3.1.5/bin/activate<br /> <br /> <br /> '''2. Copy seed-set SDI file into your docking directory'''<br /> This directory should already contain your dockfiles, with INDOCK parameters set to your liking. In this step, we are copying in a split database index file (SDI) containing paths to bundles of db2 files. This seed set contains 100 million molecules sampled randomly from the total virtual library, currently 1.1 trillion molecules.<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/XR_seed_set_v0p0.wynton.sdi .<br /> <br /> <br /> if you are interested in running on a library of 13.2B compounds from Enamine REAL, there are several seed set sizes available: 130k, 1.3M, 13M, and 26M. <br /> cp /wynton/group/bks/work/shared/kholland/chemstep_13B/130K_seeds.wynton.sdi .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_13B/1.3M_seeds.wynton.sdi .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_13B/13M_seeds.wynton.sdi .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_13B/26M_seeds.wynton.sdi .<br /> <br /> <br /> '''3. Dock seed set to your receptor of interest using DOCK 3.8''' /docking directions taken from docs.docking.org. This is meant to be done as you would do a normal LSD. <br /> export MOLECULES_DIR_TO_BIND=[outermost folder containing the molecules to dock, just your base docking directory]<br /> export DOCKFILES=[path to your dockfiles]<br /> export INPUT_FOLDER=[the folder containing your .sdi file(s)]<br /> export OUTPUT_FOLDER=[where you want the output ]<br /> <br /> /wynton/group/bks/work/bwhall61/needs_github/super_dock3r.sh<br /> <br /> Wait for docking to complete. Next, you must extract all molecule IDs and corresponding DOCK scores from above. To do so, run the following commands in the base docking directory containing your docking files and output folder, while logged into a dev node (I suggest in a screen):<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/get_scores.py .<br /> python get_scores.py 0<br /> <br /> <br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/get_scores.py<br /> python get_scores.py 0<br /> <br /> <br /> This script expects a directory named &quot;output*&quot; within the CWD. If your output from docking follows different naming conventions, vim into get_scores.py and change the path. The output will be a file named &quot;scores_round_0.txt&quot;. For iterative rounds, pass increasing numbers into the command line. i.e. When docking the first round of prioritized molecules, pass [1] for scores_round_1.txt. <br /> <br /> '''4. Convert scores and molecule IDS into NumPY arrays for ChemSTEP recognition. Requires ChemSTEP venv'''<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/convert_scores_to_npy.py .<br /> python convert_scores_to_npy.py 0<br /> <br /> <br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/convert_scores_to_npy.py<br /> python convert_scores_to_npy.py 0<br /> <br /> For python speed purposes, ChemSTEP requires that DOCK scores and IDs be give in the form of a NumPY array. In this step, we are taking our readable scores file and converting them for ChemSTEP recognition. This script expects a txt file with Molecule IDS and DOCK scores. Use the same round number you used in step 3. As run above, this will output two files named scores_round_0.npy and indices_round_0.npy that contain line-matched indices (determined from Mol ID) and their respective docking scores. <br /> <br /> '''5. Enter into or make a directory to run ChemSTEP in. Copy in necessary files for initiating ChemSTEP: params.txt, run_chemstep.py and launch_chemstep_as_job.sh. Copy in your score and indices numpy files as well, which should be in your docking directory.'''<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/params.txt .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/run_chemstep.py .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/launch_chemstep_as_job.sh .<br /> <br /> <br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/params.txt .<br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/run_chemstep.py .<br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/launch_chemstep_as_job.sh .<br /> <br /> '''6. Edit params.txt file''' This ONLY needs to be edited for the initial round of ChemSTEP. The parameters outlined here will be carried through all rounds of ChemSTEP. <br /> seed_indices_file: /absoulte/path/to/your/indices_round_0.npy<br /> seed_scores_file: /absolute/path/to/your/scores_round_0.npy<br /> hit_pprop: 5<br /> n_docked_per_round: 10000000<br /> max_beacons: 150<br /> max_n_rounds: 250<br /> <br /> Within params.txt, add the absolute paths to the ChemSTEP-readable score and indices NumPY arrays. The rest of the values within the params.txt file are left to the discretion of the user, with some considerations below. <br /> <br /> '''pProp:''' this value will define what is considered a &quot;virtual hit&quot; in your campaign. pProp is defined as the -log(rank %) of a molecule within the total library score-distribution. For example, a pProp of 4 in the 1.1T XReal space is equivalent the top 0.01% of the library, corresponding to the top 110 million molecules. pProp 5 = 0.001% = 11 million virtual hits, etc. From the seed set, ChemSTEP will estimate a DOCK score value in kcal/mol that associated with the lower-limit of your desired pProp zone. Any molecules that score better than the threshold is considered a virtual hit in your campaign. Be mindful of seed set size when choosing desired pProp. Our suggestion is that the size of the seed set should be at least 10^(pProp +2).<br /> <br /> '''n_docked_per_round:''' this number is the desired molecules for prioritization per round. When choosing this value, note that this number of molecules must be built and docked in between each round of ChemSTEP. Prioritizing many molecules will slow building and docking speeds, and coupled with few beacons (below) may lead to decreased diversity. Prioritizing too few molecules may result in slower virtual hit recovery. Round size does not significantly impact the algorithm running time.** <br /> <br /> '''max_beacons:''' this is the number of diverse, well-scoring molecules ChemSTEP will use to guide prioritization per round. All molecules that score better than the assigned pProp threshold may be considered for beacons. By default, ChemSTEP chooses each set of beacons to be as maximally diverse as possible. Choosing too many beacons may result in decreased diversity between beacons, but too few beacons could hinder space exploration. If not enough molecules score above your pProp threshold, ChemSTEP may assign fewer beacons than the assigned value. <br /> <br /> '''max_n_rounds:''' if prospectively running ChemSTEP, as outlined in this wiki, no need to worry about this parameter.<br /> <br /> <br /> There should be no need to edit run_chemstep.py or the SGE wrapper script. ''If using another virtual library, be sure to update the path in run_chemstep.py to point to your FP library.'' At this time, we strongly suggest running ChemSTEP as a job array with 16 CPU slots requested. The number of cores should be specified when calling CSAlgo (in run_chemstep.py) and the SGE wrapper (launch_chemstep_as_job.sh). <br /> <br /> <br /> '''7. Run ChemSTEP''' with the following command: <br /> qsub launch_chemstep_as_job.sh <br /> <br /> <br /> This will launch the main ChemSTEP job to the SGE scheduler. Check your job status with the 'qstat' command. The algorithm will read the score and indices files provided, calculate pProp (for round 1) and assign beacons. As this job runs, it will launch a subsequent job array. These sub-jobs are the Chaining step of ChemSTEP, where each parallel worker calculates the Tanimoto distances of 100 million molecules from the virtual library to the beacons. These distances are then written to the files within the /output directory. When these jobs complete, the main job will read through all Tanimoto distances and pull molecules for prioritization. <br /> <br /> Running ChemSTEP successfully will result in the output of a SMILES file within output/complete_info/. You will also get (1) chemstep_algo.log and (2) chemstep_submission.log in the working directory after job submission. chemstep_algo.log contains the running output of the algorithm in a readable format, with timestamps, including the DOCK score associated with your desired pProp. chemstep_submission.log contains the output of ChemSTEP, as well as any information output by the SGE submission system (python errors, cluster issues, tracebacks if failed). These files will be updated with any information with iterative rounds of ChemSTEP run in the same directory. '''Visually inspect these files after each round of ChemSTEP to ensure the algorithm has picked your desired number of beacons and things ran smoothly.'''<br /> <br /> '''Troubleshooting:''' if no jobs are running and there is no SMI file, check the chemstep_submission.log first. Any traceback error or SGE error should give you some idea of why ChemSTEP failed. Some errors may be due to SGE or Wynton issues. If directed, look at a few .out files within /output/jobs/ . If a ChemSTEP run fails on your FIRST run, delete the output and log files, fix what needs to be fixed, and rerun. Errors in subsequent rounds during the chaining step can potentially corrupt chaining files within the /output directory that are needed for prioritization. At the very least, you will definitely have duplicates of beacons and information written to the log files. At this time, if your run fails during iterative rounds, it's best to start from the beginning. <br /> <br /> Prioritized molecules should be built, docked, and their scores fed back into ChemSTEP. More detailed instructions below: <br /> <br /> '''8. Build prioritized molecules (DOCK 3.8).''' /taken from docs.docking.org <br /> source /wynton/group/bks/soft/DOCK-3.8.5/env.sh<br /> <br /> python /wynton/group/bks/soft/DOCK-3.8.5/DOCK3.8/zinc22-3d/submit/submit_building_docker.py --output_folder building_output --bundle_size 1000 --minutes_per_mol 5 --skip_name_check --scheduler sge --container_software apptainer --container_path_or_name /wynton/group/bks/soft/DOCK-3.8.5/building_pipeline.sif smi_round_{}.smi<br /> <br /> When building has completed, you must write an SDI file with the complete paths to each built bundle and dock. you can do this with:<br /> find /path/to/your/building_output -name &quot;bundle.db2.tgz&quot; -type f &gt; round_{}.wynton.sdi<br /> <br /> <br /> '''Be sure to change the INDOCK file to save only poses that meet your score pProp score threshold calculated by ChemSTEP!''' Retrieve docking scores as convert to NumPy arrays as outlined above, updating the round number when running get_scores.py and convert_scores_to_npy.py. Copy new score and indices files into the directory you ran ChemSTEP in. If you are following along as a tutorial, you should have scores_round_1.npy and indices_round_1.npy from the previous step (from FIRST round of ChemSTEP prioritization). <br /> <br /> <br /> '''9. Set up for iterative rounds of ChemSTEP'''<br /> <br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/run_chemstep_iterative.py .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/launch_chemstep_iterative.sh .<br /> <br /> <br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/run_chemstep_iterative.py .<br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/launch_chemstep_iterative.sh .<br /> <br /> '''10. Run ChemSTEP'''<br /> qsub launch_chemstep_as_job.sh [round number]<br /> <br /> For the first iterative round, the round number is [2], and should increase by one for every subsequent round of ChemSTEP. The output will be smi_round_****.smi file. Repeat steps 8-10 for as many rounds as needed. The performance is reported in outputy/complete_info/run_summary.df, which contains the number of beacons selected, the number of molecules docked, the number of hits found, the distance threshold for the selected molecules to dock, and the last added beacon's distance to all previous beacons.<br /> <br /> <br /> ** anecdotally true.</div>

Kholland http://wiki.docking.org/index.php?title=Running_ChemSTEP&diff=16952 2025-10-09T01:26:55Z

<p>Kholland: </p> <hr /> <div>last update: oct 8 2025 katie. current ver = 0.3.1.5 (automatic resubmission of failed SGE jobs). <br /> <br /> ChemSTEP (Chemical Space Traversal and Exploration Procedure) is an open-source, transparent acceleration algorithm for molecular docking capable of dealing with virtual libraries of several trillion compounds. This wiki page is a guide for BKS lab members to run ChemSTEP on Wynton HPC, using the current version of InifiSee XReal library (1.1T) or 13.3B library from Enamine REAL. '''for detailed instructions on the 13B space, 'Running ChemSTEP on the 13B space' wiki page'''. For more general use directions, please refer to [ChemSTEP Read-the-Docs]. <br /> <br /> At a high-level, ChemSTEP is an iterative process that identifies molecules from the larger virtual library to prioritize for docking. First, we identify a random sample of the total library (termed &quot;seed set&quot;, round zero) and dock those molecules to the target of interest. From this seed set, we can calculate total-library pProp values (-log rank percentages) and the number of &quot;virtual hits&quot; in the total library (high-scoring molecules). ChemSTEP will identify a set of maximally diverse molecules that score above the desired pProp threshold (&quot;beacons&quot;) from the seed set. These beacons guide prioritization, where molecules chosen and output by ChemSTEP are close in chemical space to the beacons. Prioritized molecules are then built, docked, and returned to ChemSTEP for a second round of prioritization. This process is iterated until you reach desired virtual hit recovery, or you are no longer recovering virtual hits. <br /> <br /> <br /> Running the ChemSTEP algorithm on Wynton takes place in loosely three steps: (1) submission of the main ChemSTEP job, (2) ChainingLog sub-job array, and (3) gathering of SMILES for prioritization. <br /> <br /> '''1. Main ChemSTEP job:''' reads input files (DOCK scores and line-matched indices NumPy arrays) and assigns beacons. Beacons are a set of N maximally diverse molecules that score above the pProp threshold specified in the parameter file. For the first round of ChemSTEP, the best-scoring molecule from the seed set is assigned as the first beacon. Subsequent beacons would be molecules that score well (above pProp thresh), and are as diverse as possible from already assigned beacons. Greedy max-diversity selection. <br /> <br /> During assignment, ECFP4 Tanimoto distances between the assigned beacon and all potential beacons are calculated. The second beacon is the molecule with the maximum Td from beacon 1. Beacon 3 will be the molecule that has the highest Td from beacon 1 and beacon 2. This continues until the number of beacons specified in the parameter file is reached. In iterative rounds of ChemSTEP, the beacon selection is based on diversity from ALL PREVIOUSLY assigned beacons, including those in earlier rounds. <br /> <br /> <br /> '''2. ChainingLog sub-jobs:''' once beacons are assigned, ChemSTEP will launch a series of sub-jobs that calculate the Tanimoto distances of every molecule remaining in the library to the assigned beacons. Calculated distances to the NEAREST beacon (minimum-minimum Td) are updated in the mintddistrib_*.npy files within the /output directory. <br /> <br /> <br /> '''3. Gathering of SMILES for prioritization:''' When all Td calculations are completed, the algo will select N number of molecules for prioritization, the number of which is specified by the user in the parameter file. Molecules that are ''closest in chemical space'' to beacons are prioritized. I.e. if round size = 1 million, the 1 million molecules with the smallest min-Td to ANY beacon (current or previous rounds) are prioritized. chemstep_algo.log will provide a &quot;max-minTd&quot; for each round, which is the Tanimoto distance of the most dissimilar molecule prioritized to any one beacon per round. The prioritized molecules are output in /output/complete_info as smi_round_{}.smi <br /> <br /> What the user need: DOCKFILES, directories for (1) docking (2) building and (3) running ChemSTEP. '''All iterative rounds of ChemSTEP should be run in the SAME directory.''' <br /> <br /> '''1. Source ChemSTEP virtual environment on Wynton'''<br /> source /wynton/group/bks/work/shared/kholland/chemstep_0.3.1.5/bin/activate<br /> <br /> <br /> '''2. Copy seed-set SDI file into your docking directory'''<br /> This directory should already contain your dockfiles, with INDOCK parameters set to your liking. In this step, we are copying in a split database index file (SDI) containing paths to bundles of db2 files. This seed set contains 100 million molecules sampled randomly from the total virtual library, currently 1.1 trillion molecules.<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/XR_seed_set_v0p0.wynton.sdi .<br /> <br /> <br /> if you are interested in running on a library of 13.2B compounds from Enamine REAL, there are several seed set sizes available: 130k, 1.3M, 13M, and 26M. <br /> cp /wynton/group/bks/work/shared/kholland/chemstep_13B/130K_seeds.wynton.sdi .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_13B/1.3M_seeds.wynton.sdi .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_13B/13M_seeds.wynton.sdi .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_13B/26M_seeds.wynton.sdi .<br /> <br /> <br /> '''3. Dock seed set to your receptor of interest using DOCK 3.8''' /docking directions taken from docs.docking.org. This is meant to be done as you would do a normal LSD. <br /> export MOLECULES_DIR_TO_BIND=[outermost folder containing the molecules to dock, just your base docking directory]<br /> export DOCKFILES=[path to your dockfiles]<br /> export INPUT_FOLDER=[the folder containing your .sdi file(s)]<br /> export OUTPUT_FOLDER=[where you want the output ]<br /> <br /> /wynton/group/bks/work/bwhall61/needs_github/super_dock3r.sh<br /> <br /> Wait for docking to complete. Next, you must extract all molecule IDs and corresponding DOCK scores from above. To do so, run the following commands in the base docking directory containing your docking files and output folder, while logged into a dev node (I suggest in a screen):<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/get_scores.py .<br /> python get_scores.py 0<br /> <br /> <br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/get_scores.py<br /> python get_scores.py 0<br /> <br /> <br /> This script expects a directory named &quot;output*&quot; within the CWD. If your output from docking follows different naming conventions, vim into get_scores.py and change the path. The output will be a file named &quot;scores_round_0.txt&quot;. For iterative rounds, pass increasing numbers into the command line. i.e. When docking the first round of prioritized molecules, pass [1] for scores_round_1.txt. <br /> <br /> '''4. Convert scores and molecule IDS into NumPY arrays for ChemSTEP recognition. Requires ChemSTEP venv'''<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/convert_scores_to_npy.py .<br /> python convert_scores_to_npy.py 0<br /> <br /> <br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/convert_scores_to_npy.py<br /> python convert_scores_to_npy.py 0<br /> <br /> For python speed purposes, ChemSTEP requires that DOCK scores and IDs be give in the form of a NumPY array. In this step, we are taking our readable scores file and converting them for ChemSTEP recognition. This script expects a txt file with Molecule IDS and DOCK scores. Use the same round number you used in step 3. As run above, this will output two files named scores_round_0.npy and indices_round_0.npy that contain line-matched indices (determined from Mol ID) and their respective docking scores. <br /> <br /> '''5. Enter into or make a directory to run ChemSTEP in. Copy in necessary files for initiating ChemSTEP: params.txt, run_chemstep.py and launch_chemstep_as_job.sh. Copy in your score and indices numpy files as well, which should be in your docking directory.'''<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/params.txt .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/run_chemstep.py .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/launch_chemstep_as_job.sh .<br /> <br /> <br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/params.txt .<br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/run_chemstep.py .<br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/launch_chemstep_as_job.sh .<br /> <br /> '''6. Edit params.txt file''' This ONLY needs to be edited for the initial round of ChemSTEP. The parameters outlined here will be carried through all rounds of ChemSTEP. <br /> seed_indices_file: /absoulte/path/to/your/indices_round_0.npy<br /> seed_scores_file: /absolute/path/to/your/scores_round_0.npy<br /> hit_pprop: 5<br /> n_docked_per_round: 10000000<br /> max_beacons: 150<br /> max_n_rounds: 250<br /> <br /> Within params.txt, add the absolute paths to the ChemSTEP-readable score and indices NumPY arrays. The rest of the values within the params.txt file are left to the discretion of the user, with some considerations below. <br /> <br /> '''pProp:''' this value will define what is considered a &quot;virtual hit&quot; in your campaign. pProp is defined as the -log(rank %) of a molecule within the total library score-distribution. For example, a pProp of 4 in the 1.1T XReal space is equivalent the top 0.01% of the library, corresponding to the top 110 million molecules. pProp 5 = 0.001% = 11 million virtual hits, etc. From the seed set, ChemSTEP will estimate a DOCK score value in kcal/mol that associated with the lower-limit of your desired pProp zone. Any molecules that score better than the threshold is considered a virtual hit in your campaign. Be mindful of seed set size when choosing desired pProp. Our suggestion is that the size of the seed set should be at least 10^(pProp +2).<br /> <br /> '''n_docked_per_round:''' this number is the desired molecules for prioritization per round. When choosing this value, note that this number of molecules must be built and docked in between each round of ChemSTEP. Prioritizing many molecules will slow building and docking speeds, and coupled with few beacons (below) may lead to decreased diversity. Prioritizing too few molecules may result in slower virtual hit recovery. Round size does not significantly impact the algorithm running time.** <br /> <br /> '''max_beacons:''' this is the number of diverse, well-scoring molecules ChemSTEP will use to guide prioritization per round. All molecules that score better than the assigned pProp threshold may be considered for beacons. By default, ChemSTEP chooses each set of beacons to be as maximally diverse as possible. Choosing too many beacons may result in decreased diversity between beacons, but too few beacons could hinder space exploration. If not enough molecules score above your pProp threshold, ChemSTEP may assign fewer beacons than the assigned value. <br /> <br /> '''max_n_rounds:''' if prospectively running ChemSTEP, as outlined in this wiki, no need to worry about this parameter.<br /> <br /> <br /> There should be no need to edit run_chemstep.py or the SGE wrapper script. ''If using another virtual library, be sure to update the path in run_chemstep.py to point to your FP library.'' At this time, we strongly suggest running ChemSTEP as a job array with 16 CPU slots requested. The number of cores should be specified when calling CSAlgo (in run_chemstep.py) and the SGE wrapper (launch_chemstep_as_job.sh). <br /> <br /> <br /> '''7. Run ChemSTEP''' with the following command: <br /> qsub launch_chemstep_as_job.sh <br /> <br /> <br /> This will launch the main ChemSTEP job to the SGE scheduler. Check your job status with the 'qstat' command. The algorithm will read the score and indices files provided, calculate pProp (for round 1) and assign beacons. As this job runs, it will launch a subsequent job array. These sub-jobs are the Chaining step of ChemSTEP, where each parallel worker calculates the Tanimoto distances of 100 million molecules from the virtual library to the beacons. These distances are then written to the files within the /output directory. When these jobs complete, the main job will read through all Tanimoto distances and pull molecules for prioritization. <br /> <br /> Running ChemSTEP successfully will result in the output of a SMILES file within output/complete_info/. You will also get (1) chemstep_algo.log and (2) chemstep_submission.log in the working directory after job submission. chemstep_algo.log contains the running output of the algorithm in a readable format, with timestamps, including the DOCK score associated with your desired pProp. chemstep_submission.log contains the output of ChemSTEP, as well as any information output by the SGE submission system (python errors, cluster issues, tracebacks if failed). These files will be updated with any information with iterative rounds of ChemSTEP run in the same directory. '''Visually inspect these files after each round of ChemSTEP to ensure the algorithm has picked your desired number of beacons and things ran smoothly.'''<br /> <br /> '''Troubleshooting:''' if no jobs are running and there is no SMI file, check the chemstep_submission.log first. Any traceback error or SGE error should give you some idea of why ChemSTEP failed. Some errors may be due to SGE or Wynton issues. If directed, look at a few .out files within /output/jobs/ . If a ChemSTEP run fails on your FIRST run, delete the output and log files, fix what needs to be fixed, and rerun. Errors in subsequent rounds during the chaining step can potentially corrupt chaining files within the /output directory that are needed for prioritization. At the very least, you will definitely have duplicates of beacons and information written to the log files. At this time, if your run fails during iterative rounds, it's best to start from the beginning. <br /> <br /> Prioritized molecules should be built, docked, and their scores fed back into ChemSTEP. More detailed instructions below: <br /> <br /> '''8. Build prioritized molecules (DOCK 3.8).''' /taken from docs.docking.org <br /> source /wynton/group/bks/soft/DOCK-3.8.5/env.sh<br /> <br /> python /wynton/group/bks/soft/DOCK-3.8.5/DOCK3.8/zinc22-3d/submit/submit_building_docker.py --output_folder building_output --bundle_size 1000 --minutes_per_mol 5 --skip_name_check --scheduler sge --container_software apptainer --container_path_or_name /wynton/group/bks/soft/DOCK-3.8.5/building_pipeline.sif smi_round_{}.smi<br /> <br /> When building has completed, you must write an SDI file with the complete paths to each built bundle and dock. you can do this with:<br /> find /path/to/your/building_output -name &quot;bundle.db2.tgz&quot; -type f &gt; round_{}.wynton.sdi<br /> <br /> <br /> '''Be sure to change the INDOCK file to save only poses that meet your score pProp score threshold calculated by ChemSTEP!''' Retrieve docking scores as convert to NumPy arrays as outlined above, updating the round number when running get_scores.py and convert_scores_to_npy.py. Copy new score and indices files into the directory you ran ChemSTEP in. If you are following along as a tutorial, you should have scores_round_1.npy and indices_round_1.npy from the previous step (from FIRST round of ChemSTEP prioritization). <br /> <br /> <br /> '''9. Set up for iterative rounds of ChemSTEP'''<br /> <br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/run_chemstep_iterative.py .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/launch_chemstep_iterative.sh .<br /> <br /> <br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/run_chemstep_iterative.py .<br /> cp /wynton/group/bks/work/shared/chemstep_13B_scripts_tutorial/launch_chemstep_iterative.sh .<br /> <br /> '''10. Run ChemSTEP'''<br /> qsub launch_chemstep_as_job.sh [round number]<br /> <br /> For the first iterative round, the round number is [2], and should increase by one for every subsequent round of ChemSTEP. The output will be smi_round_****.smi file. Repeat steps 8-10 for as many rounds as needed. The performance is reported in outputy/complete_info/run_summary.df, which contains the number of beacons selected, the number of molecules docked, the number of hits found, the distance threshold for the selected molecules to dock, and the last added beacon's distance to all previous beacons.<br /> <br /> <br /> ** anecdotally true.</div>

Kholland http://wiki.docking.org/index.php?title=Running_ChemSTEP&diff=16951 2025-10-09T00:56:30Z

<p>Kholland: </p> <hr /> <div>last update: oct 8 2025 katie. current ver = 0.3.1.5<br /> <br /> ChemSTEP (Chemical Space Traversal and Exploration Procedure) is an open-source, transparent acceleration algorithm for molecular docking capable of dealing with virtual libraries of several trillion compounds. This wiki page is a guide for BKS lab members to run ChemSTEP on Wynton HPC, using the current version of InifiSee XReal library (1.1T). For more general use directions, please refer to [ChemSTEP Read-the-Docs]. <br /> <br /> At a high-level, ChemSTEP is an iterative process that identifies molecules from the larger virtual library to prioritize for docking. First, we identify a random sample of the total library (termed &quot;seed set&quot;, round zero) and dock those molecules to the target of interest. From this seed set, we can calculate total-library pProp values (-log rank percentages) and the number of &quot;virtual hits&quot; in the total library (high-scoring molecules). ChemSTEP will identify a set of maximally diverse molecules that score above the desired pProp threshold (&quot;beacons&quot;) from the seed set. These beacons guide prioritization, where molecules chosen and output by ChemSTEP are close in chemical space to the beacons. Prioritized molecules are then built, docked, and returned to ChemSTEP for a second round of prioritization. This process is iterated until you reach desired virtual hit recovery, or you are no longer recovering virtual hits. <br /> <br /> <br /> Running the ChemSTEP algorithm on Wynton takes place in loosely three steps: (1) submission of the main ChemSTEP job, (2) ChainingLog sub-job array, and (3) gathering of SMILES for prioritization. <br /> <br /> '''1. Main ChemSTEP job:''' reads input files (DOCK scores and line-matched indices NumPy arrays) and assigns beacons. Beacons are a set of N maximally diverse molecules that score above the pProp threshold specified in the parameter file. For the first round of ChemSTEP, the best-scoring molecule from the seed set is assigned as the first beacon. Subsequent beacons would be molecules that score well (above pProp thresh), and are as diverse as possible from already assigned beacons. Greedy max-diversity selection. <br /> <br /> During assignment, ECFP4 Tanimoto distances between the assigned beacon and all potential beacons are calculated. The second beacon is the molecule with the maximum Td from beacon 1. Beacon 3 will be the molecule that has the highest Td from beacon 1 and beacon 2. This continues until the number of beacons specified in the parameter file is reached. In iterative rounds of ChemSTEP, the beacon selection is based on diversity from ALL PREVIOUSLY assigned beacons, including those in earlier rounds. <br /> <br /> <br /> '''2. ChainingLog sub-jobs:''' once beacons are assigned, ChemSTEP will launch a series of sub-jobs that calculate the Tanimoto distances of every molecule remaining in the library to the assigned beacons. Calculated distances to the NEAREST beacon (minimum-minimum Td) are updated in the mintddistrib_*.npy files within the /output directory. <br /> <br /> <br /> '''3. Gathering of SMILES for prioritization:''' When all Td calculations are completed, the algo will select N number of molecules for prioritization, the number of which is specified by the user in the parameter file. Molecules that are ''closest in chemical space'' to beacons are prioritized. I.e. if round size = 1 million, the 1 million molecules with the smallest min-Td to ANY beacon (current or previous rounds) are prioritized. chemstep_algo.log will provide a &quot;max-minTd&quot; for each round, which is the Tanimoto distance of the most dissimilar molecule prioritized to any one beacon per round. The prioritized molecules are output in /output/complete_info as smi_round_{}.smi <br /> <br /> What the user need: DOCKFILES, directories for (1) docking (2) building and (3) running ChemSTEP. '''All iterative rounds of ChemSTEP should be run in the SAME directory.''' <br /> <br /> '''1. Source ChemSTEP virtual environment on Wynton'''<br /> source /wynton/group/bks/work/shared/kholland/chemstep_0.3.1.5/bin/activate<br /> <br /> <br /> '''2. Copy InfiniSee seed-set SDI file into your docking directory'''<br /> This directory should already contain your dockfiles, with INDOCK parameters set to your liking. In this step, we are copying in a split database index file (SDI) containing paths to bundles of db2 files. This seed set contains 100 million molecules sampled randomly from the total virtual library, currently 1.1 trillion molecules.<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/XR_seed_set_v0p0.wynton.sdi .<br /> <br /> '''3. Dock seed set to your receptor of interest using DOCK 3.8''' /docking directions taken from docs.docking.org. This is meant to be done as you would do a normal LSD. <br /> export MOLECULES_DIR_TO_BIND=[outermost folder containing the molecules to dock, just your base docking directory]<br /> export DOCKFILES=[path to your dockfiles]<br /> export INPUT_FOLDER=[the folder containing your .sdi file(s)]<br /> export OUTPUT_FOLDER=[where you want the output ]<br /> <br /> /wynton/group/bks/work/bwhall61/needs_github/super_dock3r.sh<br /> <br /> Wait for docking to complete. Next, you must extract all molecule IDs and corresponding DOCK scores from above. To do so, run the following commands in the base docking directory containing your docking files and output folder, while logged into a dev node (I suggest in a screen):<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/get_scores.py .<br /> python get_scores.py 0<br /> <br /> <br /> This script expects a directory named &quot;output*&quot; within the CWD. If your output from docking follows different naming conventions, vim into get_scores.py and change the path. The output will be a file named &quot;scores_round_0.txt&quot;. For iterative rounds, pass increasing numbers into the command line. i.e. When docking the first round of prioritized molecules, pass [1] for scores_round_1.txt. <br /> <br /> '''4. Convert scores and molecule IDS into NumPY arrays for ChemSTEP recognition. Requires ChemSTEP venv'''<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/convert_scores_to_npy.py .<br /> python convert_scores_to_npy.py 0<br /> <br /> For python speed purposes, ChemSTEP requires that DOCK scores and IDs be give in the form of a NumPY array. In this step, we are taking our readable scores file and converting them for ChemSTEP recognition. This script expects a txt file with Molecule IDS and DOCK scores. Use the same round number you used in step 3. As run above, this will output two files named scores_round_0.npy and indices_round_0.npy that contain line-matched indices (determined from Mol ID) and their respective docking scores. <br /> <br /> '''5. Enter into or make a directory to run ChemSTEP in. Copy in necessary files for initiating ChemSTEP: params.txt, run_chemstep.py and launch_chemstep_as_job.sh. Copy in your score and indices numpy files as well, which should be in your docking directory.'''<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/params.txt .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/run_chemstep.py .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/launch_chemstep_as_job.sh .<br /> <br /> <br /> '''6. Edit params.txt file''' This ONLY needs to be edited for the initial round of ChemSTEP. The parameters outlined here will be carried through all rounds of ChemSTEP. <br /> seed_indices_file: /absoulte/path/to/your/indices_round_0.npy<br /> seed_scores_file: /absolute/path/to/your/scores_round_0.npy<br /> hit_pprop: 5<br /> n_docked_per_round: 10000000<br /> max_beacons: 150<br /> max_n_rounds: 250<br /> <br /> Within params.txt, add the absolute paths to the ChemSTEP-readable score and indices NumPY arrays. The rest of the values within the params.txt file are left to the discretion of the user, with some considerations below. <br /> <br /> '''pProp:''' this value will define what is considered a &quot;virtual hit&quot; in your campaign. pProp is defined as the -log(rank %) of a molecule within the total library score-distribution. For example, a pProp of 4 in the 1.1T XReal space is equivalent the top 0.01% of the library, corresponding to the top 110 million molecules. pProp 5 = 0.001% = 11 million virtual hits, etc. From the seed set, ChemSTEP will estimate a DOCK score value in kcal/mol that associated with the lower-limit of your desired pProp zone. Any molecules that score better than the threshold is considered a virtual hit in your campaign. Be mindful of seed set size when choosing desired pProp. Our suggestion is that the size of the seed set should be at least 10^(pProp +2).<br /> <br /> '''n_docked_per_round:''' this number is the desired molecules for prioritization per round. When choosing this value, note that this number of molecules must be built and docked in between each round of ChemSTEP. Prioritizing many molecules will slow building and docking speeds, and coupled with few beacons (below) may lead to decreased diversity. Prioritizing too few molecules may result in slower virtual hit recovery. Round size does not significantly impact the algorithm running time.** <br /> <br /> '''max_beacons:''' this is the number of diverse, well-scoring molecules ChemSTEP will use to guide prioritization per round. All molecules that score better than the assigned pProp threshold may be considered for beacons. By default, ChemSTEP chooses each set of beacons to be as maximally diverse as possible. Choosing too many beacons may result in decreased diversity between beacons, but too few beacons could hinder space exploration. If not enough molecules score above your pProp threshold, ChemSTEP may assign fewer beacons than the assigned value. <br /> <br /> ''max_n_rounds:''' if prospectively running ChemSTEP, as outlined in this wiki, no need to worry about this parameter.<br /> <br /> <br /> There should be no need to edit run_chemstep.py or the SGE wrapper script. ''If using another virtual library, be sure to update the path in run_chemstep.py to point to your FP library.'' At this time, we strongly suggest running ChemSTEP as a job array with 64 CPU slots requested, which is specified in the default wrapper. <br /> <br /> <br /> '''7. Run ChemSTEP''' with the following command: <br /> qsub launch_chemstep_as_job.sh <br /> <br /> <br /> This will launch the main ChemSTEP job to the SGE scheduler. Check your job status with the 'qstat' command. The algorithm will read the score and indices files provided, calculate pProp (for round 1) and assign beacons. As this job runs, it will launch a subsequent job array. These sub-jobs are the Chaining step of ChemSTEP, where each parallel worker calculates the Tanimoto distances of 100 million molecules from the virtual library to the beacons. These distances are then written to the files within the /output directory. When these jobs complete, the main job will read through all Tanimoto distances and pull molecules for prioritization. <br /> <br /> Running ChemSTEP successfully will result in the output of a SMILES file within output/complete_info/. You will also get (1) chemstep_algo.log and (2) chemstep_submission.log in the working directory after job submission. chemstep_algo.log contains the running output of the algorithm in a readable format, with timestamps, including the DOCK score associated with your desired pProp. chemstep_submission.log contains the output of ChemSTEP, as well as any information output by the SGE submission system (python errors, cluster issues, tracebacks if failed). These files will be updated with any information with iterative rounds of ChemSTEP run in the same directory. '''Visually inspect these files after each round of ChemSTEP to ensure the algorithm has picked your desired number of beacons and things ran smoothly.'''<br /> <br /> '''Troubleshooting:''' if no jobs are running and there is no SMI file, check the chemstep_submission.log first. Any traceback error or SGE error should give you some idea of why ChemSTEP failed. Some errors may be due to SGE or Wynton issues. If directed, look at a few .out files within /output/jobs/ . If a ChemSTEP run fails on your FIRST run, delete the output and log files, fix what needs to be fixed, and rerun. Errors in subsequent rounds during the chaining step can potentially corrupt chaining files within the /output directory that are needed for prioritization. At the very least, you will definitely have duplicates of beacons and information written to the log files. At this time, if your run fails during iterative rounds, it's best to start from the beginning. <br /> <br /> Prioritized molecules should be built, docked, and their scores fed back into ChemSTEP. More detailed instructions below: <br /> <br /> '''8. Build prioritized molecules (DOCK 3.8).''' /taken from docs.docking.org <br /> source /wynton/group/bks/soft/DOCK-3.8.5/env.sh<br /> <br /> python /wynton/group/bks/soft/DOCK-3.8.5/DOCK3.8/zinc22-3d/submit/submit_building_docker.py --output_folder building_output --bundle_size 1000 --minutes_per_mol 5 --skip_name_check --scheduler sge --container_software apptainer --container_path_or_name /wynton/group/bks/soft/DOCK-3.8.5/building_pipeline.sif smi_round_1.smi<br /> <br /> When building has completed, you must write an SDI file with the complete paths to each built bundle and dock. you can do this with:<br /> find /path/to/your/building_output -name &quot;bundle.db2.tgz&quot; -type f &gt; round_N.wynton.sdi<br /> <br /> <br /> '''Be sure to change the INDOCK file to save only poses that meet your score pProp score threshold calculated by ChemSTEP!''' Retrieve docking scores as convert to NumPy arrays as outlined above, updating the round number when running get_scores.py and convert_scores_to_npy.py. Copy new score and indices files into the directory you ran ChemSTEP in. If you are following along as a tutorial, you should have scores_round_1.npy and indices_round_1.npy from the previous step (from FIRST round of ChemSTEP prioritization). <br /> <br /> <br /> '''9. Set up for iterative rounds of ChemSTEP'''<br /> <br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/run_chemstep_iterative.py .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/launch_chemstep_iterative.sh .<br /> <br /> <br /> '''10. Run ChemSTEP'''<br /> qsub launch_chemstep_as_job.sh [round number]<br /> <br /> For the first iterative round, the round number is [2], and should increase by one for every subsequent round of ChemSTEP. The output will be smi_round_****.smi file. Repeat steps 8-10 for as many rounds as needed. The performance is reported in outputy/complete_info/run_summary.df, which contains the number of beacons selected, the number of molecules docked, the number of hits found, the distance threshold for the selected molecules to dock, and the last added beacon's distance to all previous beacons.<br /> <br /> <br /> ** anecdotally true.</div>

Kholland http://wiki.docking.org/index.php?title=Running_ChemSTEP&diff=16950 2025-10-09T00:51:23Z

<p>Kholland: </p> <hr /> <div>last update: oct 8 2025 katie. current ver = 0.3.1.5<br /> <br /> ChemSTEP (Chemical Space Traversal and Exploration Procedure) is an open-source, transparent acceleration algorithm for molecular docking capable of dealing with virtual libraries of several trillion compounds. This wiki page is a guide for BKS lab members to run ChemSTEP on Wynton HPC, using the current version of InifiSee XReal library (1.1T). For more general use directions, please refer to [ChemSTEP Read-the-Docs]. <br /> <br /> At a high-level, ChemSTEP is an iterative process that identifies molecules from the larger virtual library to prioritize for docking. First, we identify a random sample of the total library (termed &quot;seed set&quot;, round zero) and dock those molecules to the target of interest. From this seed set, we can calculate total-library pProp values (-log rank percentages) and the number of &quot;virtual hits&quot; in the total library (high-scoring molecules). ChemSTEP will identify a set of maximally diverse molecules that score above the desired pProp threshold (&quot;beacons&quot;) from the seed set. These beacons guide prioritization, where molecules chosen and output by ChemSTEP are close in chemical space to the beacons. Prioritized molecules are then built, docked, and returned to ChemSTEP for a second round of prioritization. This process is iterated until you reach desired virtual hit recovery, or you are no longer recovering virtual hits. <br /> <br /> <br /> Running the ChemSTEP algorithm on Wynton takes place in loosely three steps: (1) submission of the main ChemSTEP job, (2) ChainingLog sub-job array, and (3) gathering of SMILES for prioritization. <br /> <br /> '''1. Main ChemSTEP job:''' reads input files (DOCK scores and line-matched indices NumPy arrays) and assigns beacons. Beacons are a set of N maximally diverse molecules that score above the pProp threshold specified in the parameter file. For the first round of ChemSTEP, the best-scoring molecule from the seed set is assigned as the first beacon. Subsequent beacons would be molecules that score well (above pProp thresh), and are as far as possible by Td to all assigned beacons.<br /> <br /> During assignment, ECFP4 Tanimoto distances between the assigned beacon and all potential beacons are calculated. The second beacon is the molecule with the maximum Td from beacon 1. Beacon 3 will be the molecule that has the highest Td from beacon 1 and beacon 2. This continues until the number of beacons specified in the parameter file is reached. In iterative rounds of ChemSTEP, the beacon selection is based on diversity from ALL PREVIOUSLY assigned beacons, including those in earlier rounds. <br /> <br /> <br /> '''2. ChainingLog sub-jobs:''' once beacons are assigned, ChemSTEP will launch a series of sub-jobs that calculate the Tanimoto distances of every molecule remaining in the library to the assigned beacons. Calculated distances to the NEAREST beacon (minimum-minimum Td) are updated in the mintddistrib_*.npy files within the /output directory. <br /> <br /> <br /> '''3. Gathering of SMILES for prioritization:''' When all Td calculations are completed, the algo will give N number of molecules for prioritization, the number of which is specified by the user in the parameter file. Molecules that are ''closest in chemical space'' to beacons are prioritized. I.e. if round size = 1 million, the 1 million molecules with the smallest min-Td to ANY beacon (current or previous rounds) are prioritized. chemstep_algo.log will provide a &quot;max-minTd&quot; for each round, which is the Tanimoto distance of the most dissimilar molecule prioritized to any one beacon per round. <br /> <br /> What the user need: DOCKFILES, directories for (1) docking (2) building and (3) running ChemSTEP. '''All iterative rounds of ChemSTEP should be run in the SAME directory.''' <br /> <br /> '''1. Source ChemSTEP virtual environment on Wynton'''<br /> source /wynton/group/bks/work/shared/kholland/chemstep_0.3.1.5/bin/activate<br /> <br /> <br /> '''2. Copy InfiniSee seed-set SDI file into your docking directory'''<br /> This directory should already contain your dockfiles, with INDOCK parameters set to your liking. In this step, we are copying in a split database index file (SDI) containing paths to bundles of db2 files. This seed set contains 100 million molecules sampled randomly from the total virtual library, currently 1.1 trillion molecules.<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/XR_seed_set_v0p0.wynton.sdi .<br /> <br /> '''3. Dock seed set to your receptor of interest using DOCK 3.8''' /docking directions taken from docs.docking.org. This is meant to be done as you would do a normal LSD. <br /> export MOLECULES_DIR_TO_BIND=[outermost folder containing the molecules to dock, just your base docking directory]<br /> export DOCKFILES=[path to your dockfiles]<br /> export INPUT_FOLDER=[the folder containing your .sdi file(s)]<br /> export OUTPUT_FOLDER=[where you want the output ]<br /> <br /> /wynton/group/bks/work/bwhall61/needs_github/super_dock3r.sh<br /> <br /> Wait for docking to complete. Next, you must extract all molecule IDs and corresponding DOCK scores from above. To do so, run the following commands in the base docking directory containing your docking files and output folder, while logged into a dev node (I suggest in a screen):<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/get_scores.py .<br /> python get_scores.py 0<br /> <br /> <br /> This script expects a directory named &quot;output*&quot; within the CWD. If your output from docking follows different naming conventions, vim into get_scores.py and change the path. The output will be a file named &quot;scores_round_0.txt&quot;. For iterative rounds, pass increasing numbers into the command line. i.e. When docking the first round of prioritized molecules, pass [1] for scores_round_1.txt. <br /> <br /> '''4. Convert scores and molecule IDS into NumPY arrays for ChemSTEP recognition. Requires ChemSTEP venv'''<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/convert_scores_to_npy.py .<br /> python convert_scores_to_npy.py 0<br /> <br /> For python speed purposes, ChemSTEP requires that DOCK scores and IDs be give in the form of a NumPY array. In this step, we are taking our readable scores file and converting them for ChemSTEP recognition. This script expects a txt file with Molecule IDS and DOCK scores. Use the same round number you used in step 3. As run above, this will output two files named scores_round_0.npy and indices_round_0.npy that contain line-matched indices (determined from Mol ID) and their respective docking scores. <br /> <br /> '''5. Enter into or make a directory to run ChemSTEP in. Copy in necessary files for initiating ChemSTEP: params.txt, run_chemstep.py and launch_chemstep_as_job.sh. Copy in your score and indices numpy files as well, which should be in your docking directory.'''<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/params.txt .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/run_chemstep.py .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/launch_chemstep_as_job.sh .<br /> <br /> <br /> '''6. Edit params.txt file''' This ONLY needs to be edited for the initial round of ChemSTEP. The parameters outlined here will be carried through all rounds of ChemSTEP. <br /> seed_indices_file: /absoulte/path/to/your/indices_round_0.npy<br /> seed_scores_file: /absolute/path/to/your/scores_round_0.npy<br /> hit_pprop: 5<br /> n_docked_per_round: 10000000<br /> max_beacons: 150<br /> max_n_rounds: 250<br /> <br /> Within params.txt, add the absolute paths to the ChemSTEP-readable score and indices NumPY arrays. The rest of the values within the params.txt file are left to the discretion of the user, with some considerations below. <br /> <br /> '''pProp:''' this value will define what is considered a &quot;virtual hit&quot; in your campaign. pProp is defined as the -log(rank %) of a molecule within the total library score-distribution. For example, a pProp of 4 in the 1.1T XReal space is equivalent the top 0.01% of the library, corresponding to the top 110 million molecules. pProp 5 = 0.001% = 11 million virtual hits, etc. From the seed set, ChemSTEP will estimate a DOCK score value in kcal/mol that associated with the lower-limit of your desired pProp zone. Any molecules that score better than the threshold is considered a virtual hit in your campaign. Be mindful of seed set size when choosing desired pProp. Our suggestion is that the size of the seed set should be at least 10^(pProp +2).<br /> <br /> '''n_docked_per_round:''' this number is the desired molecules for prioritization per round. When choosing this value, note that this number of molecules must be built and docked in between each round of ChemSTEP. Prioritizing many molecules will slow building and docking speeds, and coupled with few beacons (below) may lead to decreased diversity. Prioritizing too few molecules may result in slower virtual hit recovery. Round size does not significantly impact the algorithm running time.** <br /> <br /> '''max_beacons:''' this is the number of diverse, well-scoring molecules ChemSTEP will use to guide prioritization per round. All molecules that score better than the assigned pProp threshold may be considered for beacons. By default, ChemSTEP chooses each set of beacons to be as maximally diverse as possible. Choosing too many beacons may result in decreased diversity between beacons, but too few beacons could hinder space exploration. If not enough molecules score above your pProp threshold, ChemSTEP may assign fewer beacons than the assigned value. <br /> <br /> ''max_n_rounds:''' if prospectively running ChemSTEP, as outlined in this wiki, no need to worry about this parameter.<br /> <br /> <br /> There should be no need to edit run_chemstep.py or the SGE wrapper script. ''If using another virtual library, be sure to update the path in run_chemstep.py to point to your FP library.'' At this time, we strongly suggest running ChemSTEP as a job array with 64 CPU slots requested, which is specified in the default wrapper. <br /> <br /> <br /> '''7. Run ChemSTEP''' with the following command: <br /> qsub launch_chemstep_as_job.sh <br /> <br /> <br /> This will launch the main ChemSTEP job to the SGE scheduler. Check your job status with the 'qstat' command. The algorithm will read the score and indices files provided, calculate pProp (for round 1) and assign beacons. As this job runs, it will launch a subsequent job array. These sub-jobs are the Chaining step of ChemSTEP, where each parallel worker calculates the Tanimoto distances of 100 million molecules from the virtual library to the beacons. These distances are then written to the files within the /output directory. When these jobs complete, the main job will read through all Tanimoto distances and pull molecules for prioritization. <br /> <br /> Running ChemSTEP successfully will result in the output of a SMILES file within output/complete_info/. You will also get (1) chemstep_algo.log and (2) chemstep_submission.log in the working directory after job submission. chemstep_algo.log contains the running output of the algorithm in a readable format, with timestamps, including the DOCK score associated with your desired pProp. chemstep_submission.log contains the output of ChemSTEP, as well as any information output by the SGE submission system (python errors, cluster issues, tracebacks if failed). These files will be updated with any information with iterative rounds of ChemSTEP run in the same directory. '''Visually inspect these files after each round of ChemSTEP to ensure the algorithm has picked your desired number of beacons and things ran smoothly.'''<br /> <br /> '''Troubleshooting:''' if no jobs are running and there is no SMI file, check the chemstep_submission.log first. Any traceback error or SGE error should give you some idea of why ChemSTEP failed. Some errors may be due to SGE or Wynton issues. If directed, look at a few .out files within /output/jobs/ . If a ChemSTEP run fails on your FIRST run, delete the output and log files, fix what needs to be fixed, and rerun. Errors in subsequent rounds during the chaining step can potentially corrupt chaining files within the /output directory that are needed for prioritization. At the very least, you will definitely have duplicates of beacons and information written to the log files. At this time, if your run fails during iterative rounds, it's best to start from the beginning. <br /> <br /> Prioritized molecules should be built, docked, and their scores fed back into ChemSTEP. More detailed instructions below: <br /> <br /> '''8. Build prioritized molecules (DOCK 3.8).''' /taken from docs.docking.org <br /> source /wynton/group/bks/soft/DOCK-3.8.5/env.sh<br /> <br /> python /wynton/group/bks/soft/DOCK-3.8.5/DOCK3.8/zinc22-3d/submit/submit_building_docker.py --output_folder building_output --bundle_size 1000 --minutes_per_mol 5 --skip_name_check --scheduler sge --container_software apptainer --container_path_or_name /wynton/group/bks/soft/DOCK-3.8.5/building_pipeline.sif smi_round_1.smi<br /> <br /> When building has completed, you must write an SDI file with the complete paths to each built bundle and dock. you can do this with:<br /> find /path/to/your/building_output -name &quot;bundle.db2.tgz&quot; -type f &gt; round_N.wynton.sdi<br /> <br /> <br /> '''Be sure to change the INDOCK file to save only poses that meet your score pProp score threshold calculated by ChemSTEP!''' Retrieve docking scores as convert to NumPy arrays as outlined above, updating the round number when running get_scores.py and convert_scores_to_npy.py. Copy new score and indices files into the directory you ran ChemSTEP in. If you are following along as a tutorial, you should have scores_round_1.npy and indices_round_1.npy from the previous step (from FIRST round of ChemSTEP prioritization). <br /> <br /> <br /> '''9. Set up for iterative rounds of ChemSTEP'''<br /> <br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/run_chemstep_iterative.py .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/launch_chemstep_iterative.sh .<br /> <br /> <br /> '''10. Run ChemSTEP'''<br /> qsub launch_chemstep_as_job.sh [round number]<br /> <br /> For the first iterative round, the round number is [2], and should increase by one for every subsequent round of ChemSTEP. The output will be smi_round_****.smi file. Repeat steps 8-10 for as many rounds as needed. The performance is reported in outputy/complete_info/run_summary.df, which contains the number of beacons selected, the number of molecules docked, the number of hits found, the distance threshold for the selected molecules to dock, and the last added beacon's distance to all previous beacons.<br /> <br /> <br /> ** anecdotally true.</div>

Kholland http://wiki.docking.org/index.php?title=Running_ChemSTEP&diff=16949 2025-10-08T22:24:52Z

<p>Kholland: </p> <hr /> <div>last update: oct 8 2025 katie. current ver = 0.3.1.5<br /> <br /> ChemSTEP (Chemical Space Traversal and Exploration Procedure) is an open-source, transparent acceleration algorithm for molecular docking capable of dealing with virtual libraries of several trillion compounds. This wiki page is a guide for BKS lab members to run ChemSTEP on Wynton HPC, using the current version of InifiSee XReal library (1.1T). For more general use directions, please refer to [ChemSTEP Read-the-Docs]. <br /> <br /> At a high-level, ChemSTEP is an iterative process that identifies molecules from the larger virtual library to prioritize for docking. First, we identify a random sample of the total library (termed &quot;seed set&quot;, round zero) and dock those molecules to the target of interest. From this seed set, we can calculate total-library pProp values (-log rank percentages) and the number of &quot;virtual hits&quot; in the total library (high-scoring molecules). ChemSTEP will identify a set of maximally diverse molecules that score above the desired pProp threshold (&quot;beacons&quot;) from the seed set. These beacons guide prioritization, where molecules chosen and output by ChemSTEP are close in chemical space to the beacons. Prioritized molecules are then built, docked, and returned to ChemSTEP for a second round of prioritization. This process is iterated until you reach desired virtual hit recovery, or you are no longer recovering virtual hits. <br /> <br /> <br /> Running the ChemSTEP algorithm on Wynton takes place in loosely three steps: (1) submission of the main ChemSTEP job, (2) ChainingLog sub-job array, and (3) gathering of SMILES for prioritization. <br /> <br /> '''1. Main ChemSTEP job:''' reads input files (DOCK scores and line-matched indices NumPy arrays) and assigns beacons. Beacons are a set of N maximally diverse molecules that score above the pProp threshold specified in the parameter file. For the first round of ChemSTEP, the best-scoring molecule from the seed set is assigned as the first beacon. Subsequent beacons would be molecules that score well (above pProp thresh), and are as far as possible by Td to all assigned beacons. In iterative rounds of ChemSTEP, the first beacon chosen is based on diversity from ALL PREVIOUSLY assigned beacons, including those in earlier rounds. <br /> <br /> <br /> '''2. ChainingLog sub-jobs:''' once beacons are assigned, ChemSTEP will launch a series of sub-jobs that calculate the Tanimoto distances of every molecule remaining in the library to the assigned beacons. Calculated distances to the NEAREST beacon (minimum-minimum Td) are updated in the mintddistrib_*.npy files within the /output directory. <br /> <br /> <br /> '''3. Gathering of SMILES for prioritization:''' When all Td calculations are completed, the algo will give N number of molecules for prioritization, the number of which is specified by the user in the parameter file. Molecules that are ''closest in chemical space'' to beacons are prioritized. I.e. if round size = 1 million, the 1 million molecules with the smallest min-Td to ANY beacon (current or previous rounds) are prioritized. chemstep_algo.log will provide a &quot;max-minTd&quot; for each round, which is the Tanimoto distance of the most dissimilar molecule prioritized to any one beacon per round. <br /> <br /> What the user need: DOCKFILES, directories for (1) docking (2) building and (3) running ChemSTEP. '''All iterative rounds of ChemSTEP should be run in the SAME directory.''' <br /> <br /> '''1. Source ChemSTEP virtual environment on Wynton'''<br /> source /wynton/group/bks/work/shared/kholland/chemstep_0.3.1.5/bin/activate<br /> <br /> <br /> '''2. Copy InfiniSee seed-set SDI file into your docking directory'''<br /> This directory should already contain your dockfiles, with INDOCK parameters set to your liking. In this step, we are copying in a split database index file (SDI) containing paths to bundles of db2 files. This seed set contains 100 million molecules sampled randomly from the total virtual library, currently 1.1 trillion molecules.<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/XR_seed_set_v0p0.wynton.sdi .<br /> <br /> '''3. Dock seed set to your receptor of interest using DOCK 3.8''' /docking directions taken from docs.docking.org. This is meant to be done as you would do a normal LSD. <br /> export MOLECULES_DIR_TO_BIND=[outermost folder containing the molecules to dock, just your base docking directory]<br /> export DOCKFILES=[path to your dockfiles]<br /> export INPUT_FOLDER=[the folder containing your .sdi file(s)]<br /> export OUTPUT_FOLDER=[where you want the output ]<br /> <br /> /wynton/group/bks/work/bwhall61/needs_github/super_dock3r.sh<br /> <br /> Wait for docking to complete. Next, you must extract all molecule IDs and corresponding DOCK scores from above. To do so, run the following commands in the base docking directory containing your docking files and output folder, while logged into a dev node (I suggest in a screen):<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/get_scores.py .<br /> python get_scores.py 0<br /> <br /> <br /> This script expects a directory named &quot;output*&quot; within the CWD. If your output from docking follows different naming conventions, vim into get_scores.py and change the path. The output will be a file named &quot;scores_round_0.txt&quot;. For iterative rounds, pass increasing numbers into the command line. i.e. When docking the first round of prioritized molecules, pass [1] for scores_round_1.txt. <br /> <br /> '''4. Convert scores and molecule IDS into NumPY arrays for ChemSTEP recognition. Requires ChemSTEP venv'''<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/convert_scores_to_npy.py .<br /> python convert_scores_to_npy.py 0<br /> <br /> For python speed purposes, ChemSTEP requires that DOCK scores and IDs be give in the form of a NumPY array. In this step, we are taking our readable scores file and converting them for ChemSTEP recognition. This script expects a txt file with Molecule IDS and DOCK scores. Use the same round number you used in step 3. As run above, this will output two files named scores_round_0.npy and indices_round_0.npy that contain line-matched indices (determined from Mol ID) and their respective docking scores. <br /> <br /> '''5. Enter into or make a directory to run ChemSTEP in. Copy in necessary files for initiating ChemSTEP: params.txt, run_chemstep.py and launch_chemstep_as_job.sh. Copy in your score and indices numpy files as well, which should be in your docking directory.'''<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/params.txt .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/run_chemstep.py .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/launch_chemstep_as_job.sh .<br /> <br /> <br /> '''6. Edit params.txt file''' This ONLY needs to be edited for the initial round of ChemSTEP. The parameters outlined here will be carried through all rounds of ChemSTEP. <br /> seed_indices_file: /absoulte/path/to/your/indices_round_0.npy<br /> seed_scores_file: /absolute/path/to/your/scores_round_0.npy<br /> hit_pprop: 5<br /> n_docked_per_round: 10000000<br /> max_beacons: 150<br /> max_n_rounds: 250<br /> <br /> Within params.txt, add the absolute paths to the ChemSTEP-readable score and indices NumPY arrays. The rest of the values within the params.txt file are left to the discretion of the user, with some considerations below. <br /> <br /> '''pProp:''' this value will define what is considered a &quot;virtual hit&quot; in your campaign. pProp is defined as the -log(rank %) of a molecule within the total library score-distribution. For example, a pProp of 4 in the 1.1T XReal space is equivalent the top 0.01% of the library, corresponding to the top 110 million molecules. pProp 5 = 0.001% = 11 million virtual hits, etc. From the seed set, ChemSTEP will estimate a DOCK score value in kcal/mol that associated with the lower-limit of your desired pProp zone. Any molecules that score better than the threshold is considered a virtual hit in your campaign. Be mindful of seed set size when choosing desired pProp. Our suggestion is that the size of the seed set should be at least 10^(pProp +2).<br /> <br /> '''n_docked_per_round:''' this number is the desired molecules for prioritization per round. When choosing this value, note that this number of molecules must be built and docked in between each round of ChemSTEP. Prioritizing many molecules will slow building and docking speeds, and coupled with few beacons (below) may lead to decreased diversity. Prioritizing too few molecules may result in slower virtual hit recovery. Round size does not significantly impact the algorithm running time.** <br /> <br /> '''max_beacons:''' this is the number of diverse, well-scoring molecules ChemSTEP will use to guide prioritization per round. All molecules that score better than the assigned pProp threshold may be considered for beacons. By default, ChemSTEP chooses each set of beacons to be as maximally diverse as possible. Choosing too many beacons may result in decreased diversity between beacons, but too few beacons could hinder space exploration. If not enough molecules score above your pProp threshold, ChemSTEP may assign fewer beacons than the assigned value. <br /> <br /> ''max_n_rounds:''' if prospectively running ChemSTEP, as outlined in this wiki, no need to worry about this parameter.<br /> <br /> <br /> There should be no need to edit run_chemstep.py or the SGE wrapper script. ''If using another virtual library, be sure to update the path in run_chemstep.py to point to your FP library.'' At this time, we strongly suggest running ChemSTEP as a job array with 64 CPU slots requested, which is specified in the default wrapper. <br /> <br /> <br /> '''7. Run ChemSTEP''' with the following command: <br /> qsub launch_chemstep_as_job.sh <br /> <br /> <br /> This will launch the main ChemSTEP job to the SGE scheduler. Check your job status with the 'qstat' command. The algorithm will read the score and indices files provided, calculate pProp (for round 1) and assign beacons. As this job runs, it will launch a subsequent job array. These sub-jobs are the Chaining step of ChemSTEP, where each parallel worker calculates the Tanimoto distances of 100 million molecules from the virtual library to the beacons. These distances are then written to the files within the /output directory. When these jobs complete, the main job will read through all Tanimoto distances and pull molecules for prioritization. <br /> <br /> Running ChemSTEP successfully will result in the output of a SMILES file within output/complete_info/. You will also get (1) chemstep_algo.log and (2) chemstep_submission.log in the working directory after job submission. chemstep_algo.log contains the running output of the algorithm in a readable format, with timestamps, including the DOCK score associated with your desired pProp. chemstep_submission.log contains the output of ChemSTEP, as well as any information output by the SGE submission system (python errors, cluster issues, tracebacks if failed). These files will be updated with any information with iterative rounds of ChemSTEP run in the same directory. '''Visually inspect these files after each round of ChemSTEP to ensure the algorithm has picked your desired number of beacons and things ran smoothly.'''<br /> <br /> '''Troubleshooting:''' if no jobs are running and there is no SMI file, check the chemstep_submission.log first. Any traceback error or SGE error should give you some idea of why ChemSTEP failed. Some errors may be due to SGE or Wynton issues. If directed, look at a few .out files within /output/jobs/ . If a ChemSTEP run fails on your FIRST run, delete the output and log files, fix what needs to be fixed, and rerun. Errors in subsequent rounds during the chaining step can potentially corrupt chaining files within the /output directory that are needed for prioritization. At the very least, you will definitely have duplicates of beacons and information written to the log files. At this time, if your run fails during iterative rounds, it's best to start from the beginning. <br /> <br /> Prioritized molecules should be built, docked, and their scores fed back into ChemSTEP. More detailed instructions below: <br /> <br /> '''8. Build prioritized molecules (DOCK 3.8).''' /taken from docs.docking.org <br /> source /wynton/group/bks/soft/DOCK-3.8.5/env.sh<br /> <br /> python /wynton/group/bks/soft/DOCK-3.8.5/DOCK3.8/zinc22-3d/submit/submit_building_docker.py --output_folder building_output --bundle_size 1000 --minutes_per_mol 5 --skip_name_check --scheduler sge --container_software apptainer --container_path_or_name /wynton/group/bks/soft/DOCK-3.8.5/building_pipeline.sif smi_round_1.smi<br /> <br /> When building has completed, you must write an SDI file with the complete paths to each built bundle and dock. you can do this with:<br /> find /path/to/your/building_output -name &quot;bundle.db2.tgz&quot; -type f &gt; round_N.wynton.sdi<br /> <br /> <br /> '''Be sure to change the INDOCK file to save only poses that meet your score pProp score threshold calculated by ChemSTEP!''' Retrieve docking scores as convert to NumPy arrays as outlined above, updating the round number when running get_scores.py and convert_scores_to_npy.py. Copy new score and indices files into the directory you ran ChemSTEP in. If you are following along as a tutorial, you should have scores_round_1.npy and indices_round_1.npy from the previous step (from FIRST round of ChemSTEP prioritization). <br /> <br /> <br /> '''9. Set up for iterative rounds of ChemSTEP'''<br /> <br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/run_chemstep_iterative.py .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/launch_chemstep_iterative.sh .<br /> <br /> <br /> '''10. Run ChemSTEP'''<br /> qsub launch_chemstep_as_job.sh [round number]<br /> <br /> For the first iterative round, the round number is [2], and should increase by one for every subsequent round of ChemSTEP. The output will be smi_round_****.smi file. Repeat steps 8-10 for as many rounds as needed. The performance is reported in outputy/complete_info/run_summary.df, which contains the number of beacons selected, the number of molecules docked, the number of hits found, the distance threshold for the selected molecules to dock, and the last added beacon's distance to all previous beacons.<br /> <br /> <br /> ** anecdotally true.</div>

Kholland http://wiki.docking.org/index.php?title=Running_ChemSTEP&diff=16948 2025-10-08T21:37:29Z

<p>Kholland: </p> <hr /> <div>last update: oct 8 2025 katie. current ver = 0.3.1.5<br /> <br /> ChemSTEP (Chemical Space Traversal and Exploration Procedure) is an open-source, transparent acceleration algorithm for molecular docking capable of dealing with virtual libraries of several trillion compounds. This wiki page is a guide for BKS lab members to run ChemSTEP on Wynton HPC, using the current version of InifiSee XReal library (1.1T). For more general use directions, please refer to [ChemSTEP Read-the-Docs]. <br /> <br /> At a high-level, ChemSTEP is an iterative process that identifies molecules from the larger virtual library to prioritize for docking. First, we identify a random sample of the total library (termed &quot;seed set&quot;, round zero) and dock those molecules to the target of interest. From this seed set, we can calculate total-library pProp values (-log rank percentages) and the number of &quot;virtual hits&quot; in the total library (high-scoring molecules). ChemSTEP will identify a set of maximally diverse molecules that score above the desired pProp threshold (&quot;beacons&quot;) from the seed set. These beacons guide prioritization, where molecules chosen and output by ChemSTEP are close in chemical space to the beacons. Prioritized molecules are then built, docked, and returned to ChemSTEP for a second round of prioritization. This process is iterated until you reach desired virtual hit recovery, or you are no longer recovering virtual hits. <br /> <br /> <br /> Running the ChemSTEP algorithm on Wynton takes place in loosely three steps: (1) submission of the main ChemSTEP job, (2) ChainingLog sub-job array, and (3) gathering of SMILES for prioritization. <br /> <br /> '''1. Main ChemSTEP job:''' reads input files (DOCK scores and line-matched indices NumPy arrays) and assigns beacons. Beacons are a set of N maximally diverse molecules that score above the pProp threshold specified in the parameter file. For the first round of ChemSTEP, the best-scoring molecule from the seed set is assigned as the first beacon. Subsequent beacons would be molecules that score well (above pProp thresh), and are as far as possible by Td to all assigned beacons. In iterative rounds of ChemSTEP, the first beacon chosen is based on diversity from ALL PREVIOUSLY assigned beacons, including those in earlier rounds. <br /> <br /> '''2. ChainingLog sub-jobs:''' once beacons are assigned, ChemSTEP will launch a series of sub-jobs that calculate the Tanimoto distances of every molecule remaining in the library to the assigned beacons. Calculated distances to the NEAREST beacon (minimum-minimum Td) are updated in the mintddistrib_*.npy files within the /output directory. <br /> <br /> '''3. Gathering of SMILES for prioritization:''' When all Td calculations are completed, the algo will give N number of molecules for prioritization, the number of which is specified by the user in the parameter file. Molecules that are ''closest in chemical space'' to beacons are prioritized. I.e. if round size = 1 million, the 1 million molecules with the smallest min-Td to ANY beacon (current or previous rounds) are prioritized. chemstep_algo.log will provide a &quot;max-minTd&quot; for each round, which is the Tanimoto distance of the most dissimilar molecule prioritized to any one beacon per round. <br /> <br /> What the user need: DOCKFILES, directories for (1) docking (2) building and (3) running ChemSTEP. '''All iterative rounds of ChemSTEP should be run in the SAME directory.''' <br /> <br /> '''1. Source ChemSTEP virtual environment on Wynton'''<br /> source /wynton/group/bks/work/shared/kholland/chemstep_0.3.1.5/bin/activate<br /> <br /> <br /> '''2. Copy InfiniSee seed-set SDI file into your docking directory'''<br /> This directory should already contain your dockfiles, with INDOCK parameters set to your liking. In this step, we are copying in a split database index file (SDI) containing paths to bundles of db2 files. This seed set contains 100 million molecules sampled randomly from the total virtual library, currently 1.1 trillion molecules.<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/XR_seed_set_v0p0.wynton.sdi .<br /> <br /> '''3. Dock seed set to your receptor of interest using DOCK 3.8''' /docking directions taken from docs.docking.org. This is meant to be done as you would do a normal LSD. <br /> export MOLECULES_DIR_TO_BIND=[outermost folder containing the molecules to dock, just your base docking directory]<br /> export DOCKFILES=[path to your dockfiles]<br /> export INPUT_FOLDER=[the folder containing your .sdi file(s)]<br /> export OUTPUT_FOLDER=[where you want the output ]<br /> <br /> /wynton/group/bks/work/bwhall61/needs_github/super_dock3r.sh<br /> <br /> Wait for docking to complete. Next, you must extract all molecule IDs and corresponding DOCK scores from above. To do so, run the following commands in the base docking directory containing your docking files and output folder, while logged into a dev node (I suggest in a screen):<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/get_scores.py .<br /> python get_scores.py 0<br /> <br /> <br /> This script expects a directory named &quot;output*&quot; within the CWD. If your output from docking follows different naming conventions, vim into get_scores.py and change the path. The output will be a file named &quot;scores_round_0.txt&quot;. For iterative rounds, pass increasing numbers into the command line. i.e. When docking the first round of prioritized molecules, pass [1] for scores_round_1.txt. <br /> <br /> '''4. Convert scores and molecule IDS into NumPY arrays for ChemSTEP recognition. Requires ChemSTEP venv'''<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/convert_scores_to_npy.py .<br /> python convert_scores_to_npy.py 0<br /> <br /> For python speed purposes, ChemSTEP requires that DOCK scores and IDs be give in the form of a NumPY array. In this step, we are taking our readable scores file and converting them for ChemSTEP recognition. This script expects a txt file with Molecule IDS and DOCK scores. Use the same round number you used in step 3. As run above, this will output two files named scores_round_0.npy and indices_round_0.npy that contain line-matched indices (determined from Mol ID) and their respective docking scores. <br /> <br /> '''5. Enter into or make a directory to run ChemSTEP in. Copy in necessary files for initiating ChemSTEP: params.txt, run_chemstep.py and launch_chemstep_as_job.sh. Copy in your score and indices numpy files as well, which should be in your docking directory.'''<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/params.txt .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/run_chemstep.py .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/launch_chemstep_as_job.sh .<br /> <br /> <br /> '''6. Edit params.txt file''' This ONLY needs to be edited for the initial round of ChemSTEP. The parameters outlined here will be carried through all rounds of ChemSTEP. <br /> seed_indices_file: /absoulte/path/to/your/indices_round_0.npy<br /> seed_scores_file: /absolute/path/to/your/scores_round_0.npy<br /> hit_pprop: 5<br /> n_docked_per_round: 10000000<br /> max_beacons: 150<br /> max_n_rounds: 250<br /> <br /> Within params.txt, add the absolute paths to the ChemSTEP-readable score and indices NumPY arrays. The rest of the values within the params.txt file are left to the discretion of the user, with some considerations below. <br /> <br /> '''pProp:''' this value will define what is considered a &quot;virtual hit&quot; in your campaign. pProp is defined as the -log(rank %) of a molecule within the total library score-distribution. For example, a pProp of 4 in the 1.1T XReal space is equivalent the top 0.01% of the library, corresponding to the top 110 million molecules. pProp 5 = 0.001% = 11 million virtual hits, etc. From the seed set, ChemSTEP will estimate a DOCK score value in kcal/mol that associated with the lower-limit of your desired pProp zone. Any molecules that score better than the threshold is considered a virtual hit in your campaign. Be mindful of seed set size when choosing desired pProp. Our suggestion is that the size of the seed set should be at least 10^(pProp +2).<br /> <br /> '''n_docked_per_round:''' this number is the desired molecules for prioritization per round. When choosing this value, note that this number of molecules must be built and docked in between each round of ChemSTEP. Prioritizing many molecules will slow building and docking speeds, and coupled with few beacons (below) may lead to decreased diversity. Prioritizing too few molecules may result in slower virtual hit recovery. Round size does not significantly impact the algorithm running time.** <br /> <br /> '''max_beacons:''' this is the number of diverse, well-scoring molecules ChemSTEP will use to guide prioritization per round. All molecules that score better than the assigned pProp threshold may be considered for beacons. By default, ChemSTEP chooses each set of beacons to be as maximally diverse as possible. Choosing too many beacons may result in decreased diversity between beacons, but too few beacons could hinder space exploration. If not enough molecules score above your pProp threshold, ChemSTEP may assign fewer beacons than the assigned value. <br /> <br /> ''max_n_rounds:''' if prospectively running ChemSTEP, as outlined in this wiki, no need to worry about this parameter.<br /> <br /> <br /> There should be no need to edit run_chemstep.py or the SGE wrapper script. ''If using another virtual library, be sure to update the path in run_chemstep.py to point to your FP library.'' At this time, we strongly suggest running ChemSTEP as a job array with 64 CPU slots requested, which is specified in the default wrapper. <br /> <br /> <br /> '''7. Run ChemSTEP''' with the following command: <br /> qsub launch_chemstep_as_job.sh <br /> <br /> <br /> This will launch the main ChemSTEP job to the SGE scheduler. Check your job status with the 'qstat' command. The algorithm will read the score and indices files provided, calculate pProp (for round 1) and assign beacons. As this job runs, it will launch a subsequent job array. These sub-jobs are the Chaining step of ChemSTEP, where each parallel worker calculates the Tanimoto distances of 100 million molecules from the virtual library to the beacons. These distances are then written to the files within the /output directory. When these jobs complete, the main job will read through all Tanimoto distances and pull molecules for prioritization. <br /> <br /> Running ChemSTEP successfully will result in the output of a SMILES file within output/complete_info/. You will also get (1) chemstep_algo.log and (2) chemstep_submission.log in the working directory after job submission. chemstep_algo.log contains the running output of the algorithm in a readable format, with timestamps, including the DOCK score associated with your desired pProp. chemstep_submission.log contains the output of ChemSTEP, as well as any information output by the SGE submission system (python errors, cluster issues, tracebacks if failed). These files will be updated with any information with iterative rounds of ChemSTEP run in the same directory. '''Visually inspect these files after each round of ChemSTEP to ensure the algorithm has picked your desired number of beacons and things ran smoothly.'''<br /> <br /> '''Troubleshooting:''' if no jobs are running and there is no SMI file, check the chemstep_submission.log first. Any traceback error or SGE error should give you some idea of why ChemSTEP failed. Some errors may be due to SGE or Wynton issues. If directed, look at a few .out files within /output/jobs/ . If a ChemSTEP run fails on your FIRST run, delete the output and log files, fix what needs to be fixed, and rerun. Errors in subsequent rounds during the chaining step can potentially corrupt chaining files within the /output directory that are needed for prioritization. At the very least, you will definitely have duplicates of beacons and information written to the log files. At this time, if your run fails during iterative rounds, it's best to start from the beginning. <br /> <br /> Prioritized molecules should be built, docked, and their scores fed back into ChemSTEP. More detailed instructions below: <br /> <br /> '''8. Build prioritized molecules (DOCK 3.8).''' /taken from docs.docking.org <br /> source /wynton/group/bks/soft/DOCK-3.8.5/env.sh<br /> <br /> python /wynton/group/bks/soft/DOCK-3.8.5/DOCK3.8/zinc22-3d/submit/submit_building_docker.py --output_folder building_output --bundle_size 1000 --minutes_per_mol 5 --skip_name_check --scheduler sge --container_software apptainer --container_path_or_name /wynton/group/bks/soft/DOCK-3.8.5/building_pipeline.sif smi_round_1.smi<br /> <br /> When building has completed, you must write an SDI file with the complete paths to each built bundle and dock. you can do this with:<br /> find /path/to/your/building_output -name &quot;bundle.db2.tgz&quot; -type f &gt; round_N.wynton.sdi<br /> <br /> <br /> '''Be sure to change the INDOCK file to save only poses that meet your score pProp score threshold calculated by ChemSTEP!''' Retrieve docking scores as convert to NumPy arrays as outlined above, updating the round number when running get_scores.py and convert_scores_to_npy.py. Copy new score and indices files into the directory you ran ChemSTEP in. If you are following along as a tutorial, you should have scores_round_1.npy and indices_round_1.npy from the previous step (from FIRST round of ChemSTEP prioritization). <br /> <br /> <br /> '''9. Set up for iterative rounds of ChemSTEP'''<br /> <br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/run_chemstep_iterative.py .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/launch_chemstep_iterative.sh .<br /> <br /> <br /> '''10. Run ChemSTEP'''<br /> qsub launch_chemstep_as_job.sh [round number]<br /> <br /> For the first iterative round, the round number is [2], and should increase by one for every subsequent round of ChemSTEP. The output will be smi_round_****.smi file. Repeat steps 8-10 for as many rounds as needed. The performance is reported in outputy/complete_info/run_summary.df, which contains the number of beacons selected, the number of molecules docked, the number of hits found, the distance threshold for the selected molecules to dock, and the last added beacon's distance to all previous beacons.<br /> <br /> <br /> ** anecdotally true.</div>

Kholland http://wiki.docking.org/index.php?title=Running_ChemSTEP&diff=16947 2025-10-08T21:32:40Z

<p>Kholland: </p> <hr /> <div>last update: oct 8 2025 katie. current ver = 0.3.1.5<br /> <br /> ChemSTEP (Chemical Space Traversal and Exploration Procedure) is an open-source, transparent acceleration algorithm for molecular docking capable of dealing with virtual libraries of several trillion compounds. This wiki page is a guide for BKS lab members to run ChemSTEP on Wynton HPC, using the current version of InifiSee XReal library (1.1T). For more general use directions, please refer to [ChemSTEP Read-the-Docs]. <br /> <br /> At a high-level, ChemSTEP is an iterative process that identifies molecules from the larger virtual library to prioritize for docking. First, we identify a random sample of the total library (termed &quot;seed set&quot;, round zero) and dock those molecules to the target of interest. From this seed set, we can calculate total-library pProp values (-log rank percentages) and the number of &quot;virtual hits&quot; in the total library (high-scoring molecules). ChemSTEP will identify a set of maximally diverse molecules that score above the desired pProp threshold (&quot;beacons&quot;) from the seed set. These beacons guide prioritization, where molecules chosen and output by ChemSTEP are close in chemical space to the beacons. Prioritized molecules are then built, docked, and returned to ChemSTEP for a second round of prioritization. This process is iterated until you reach desired virtual hit recovery, or you are no longer recovering virtual hits. <br /> <br /> <br /> Running the ChemSTEP algorithm on Wynton takes place in loosely three steps: (1) submission of the main ChemSTEP job, (2) ChainingLog sub-job array, and (3) gathering of SMILES for prioritization. <br /> <br /> 1. Main ChemSTEP job: reads input files (DOCK scores and line-matched indices NumPy arrays) and assigns beacons. Beacons are a set of N maximally diverse molecules that score above the pProp threshold specified in the parameter file. For the first round of ChemSTEP, the best-scoring molecule from the seed set is assigned as the first beacon. Subsequent beacons would be molecules that score well (above pProp thresh), and are as far as possible by Td to all assigned beacons. In iterative rounds of ChemSTEP, the first beacon chosen is based on diversity from ALL PREVIOUSLY assigned beacons, including those in earlier rounds. <br /> <br /> 2. ChainingLog sub-jobs: once beacons are assigned, ChemSTEP will launch a series of sub-jobs that calculate the Tanimoto distances of every molecule remaining in the library to the assigned beacons. Calculated distances to the NEAREST beacon (minimum-minimum Td) are updated in the mintddistrib_*.npy files within the /output directory. <br /> <br /> 3. Gathering of SMILES for prioritization: When all Td calculations are completed, the algo will then choose N number of molecules for prioritization, which is specified by the user in the parameter file. Molecules that are ''closest in chemical space'' to beacons are prioritized. I.e. if round size = 1 million, the 1 million molecules with the smallest min-Td to any beacon are prioritized. chemstep_algo.log will provide a &quot;max-minTd&quot; for each round, which is the Tanimoto distance of the most dissimilar molecule prioritized to any one beacon per round. <br /> <br /> <br /> What the user need: DOCKFILES, directories for (1) docking (2) building and (3) running ChemSTEP. '''All iterative rounds of ChemSTEP should be run in the SAME directory.''' <br /> <br /> '''1. Source ChemSTEP virtual environment on Wynton'''<br /> source /wynton/group/bks/work/shared/kholland/chemstep_0.3.1.5/bin/activate<br /> <br /> <br /> '''2. Copy InfiniSee seed-set SDI file into your docking directory'''<br /> This directory should already contain your dockfiles, with INDOCK parameters set to your liking. In this step, we are copying in a split database index file (SDI) containing paths to bundles of db2 files. This seed set contains 100 million molecules sampled randomly from the total virtual library, currently 1.1 trillion molecules.<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/XR_seed_set_v0p0.wynton.sdi .<br /> <br /> '''3. Dock seed set to your receptor of interest using DOCK 3.8''' /docking directions taken from docs.docking.org. This is meant to be done as you would do a normal LSD. <br /> export MOLECULES_DIR_TO_BIND=[outermost folder containing the molecules to dock, just your base docking directory]<br /> export DOCKFILES=[path to your dockfiles]<br /> export INPUT_FOLDER=[the folder containing your .sdi file(s)]<br /> export OUTPUT_FOLDER=[where you want the output ]<br /> <br /> /wynton/group/bks/work/bwhall61/needs_github/super_dock3r.sh<br /> <br /> Wait for docking to complete. Next, you must extract all molecule IDs and corresponding DOCK scores from above. To do so, run the following commands in the base docking directory containing your docking files and output folder, while logged into a dev node (I suggest in a screen):<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/get_scores.py .<br /> python get_scores.py 0<br /> <br /> <br /> This script expects a directory named &quot;output*&quot; within the CWD. If your output from docking follows different naming conventions, vim into get_scores.py and change the path. The output will be a file named &quot;scores_round_0.txt&quot;. For iterative rounds, pass increasing numbers into the command line. i.e. When docking the first round of prioritized molecules, pass [1] for scores_round_1.txt. <br /> <br /> '''4. Convert scores and molecule IDS into NumPY arrays for ChemSTEP recognition. Requires ChemSTEP venv'''<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/convert_scores_to_npy.py .<br /> python convert_scores_to_npy.py 0<br /> <br /> For python speed purposes, ChemSTEP requires that DOCK scores and IDs be give in the form of a NumPY array. In this step, we are taking our readable scores file and converting them for ChemSTEP recognition. This script expects a txt file with Molecule IDS and DOCK scores. Use the same round number you used in step 3. As run above, this will output two files named scores_round_0.npy and indices_round_0.npy that contain line-matched indices (determined from Mol ID) and their respective docking scores. <br /> <br /> '''5. Enter into or make a directory to run ChemSTEP in. Copy in necessary files for initiating ChemSTEP: params.txt, run_chemstep.py and launch_chemstep_as_job.sh. Copy in your score and indices numpy files as well, which should be in your docking directory.'''<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/params.txt .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/run_chemstep.py .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/launch_chemstep_as_job.sh .<br /> <br /> <br /> '''6. Edit params.txt file''' This ONLY needs to be edited for the initial round of ChemSTEP. The parameters outlined here will be carried through all rounds of ChemSTEP. <br /> seed_indices_file: /absoulte/path/to/your/indices_round_0.npy<br /> seed_scores_file: /absolute/path/to/your/scores_round_0.npy<br /> hit_pprop: 5<br /> n_docked_per_round: 10000000<br /> max_beacons: 150<br /> max_n_rounds: 250<br /> <br /> Within params.txt, add the absolute paths to the ChemSTEP-readable score and indices NumPY arrays. The rest of the values within the params.txt file are left to the discretion of the user, with some considerations below. <br /> <br /> '''pProp:''' this value will define what is considered a &quot;virtual hit&quot; in your campaign. pProp is defined as the -log(rank %) of a molecule within the total library score-distribution. For example, a pProp of 4 in the 1.1T XReal space is equivalent the top 0.01% of the library, corresponding to the top 110 million molecules. pProp 5 = 0.001% = 11 million virtual hits, etc. From the seed set, ChemSTEP will estimate a DOCK score value in kcal/mol that associated with the lower-limit of your desired pProp zone. Any molecules that score better than the threshold is considered a virtual hit in your campaign. Be mindful of seed set size when choosing desired pProp. Our suggestion is that the size of the seed set should be at least 10^(pProp +2).<br /> <br /> '''n_docked_per_round:''' this number is the desired molecules for prioritization per round. When choosing this value, note that this number of molecules must be built and docked in between each round of ChemSTEP. Prioritizing many molecules will slow building and docking speeds, and coupled with few beacons (below) may lead to decreased diversity. Prioritizing too few molecules may result in slower virtual hit recovery. Round size does not significantly impact the algorithm running time.** <br /> <br /> '''max_beacons:''' this is the number of diverse, well-scoring molecules ChemSTEP will use to guide prioritization per round. All molecules that score better than the assigned pProp threshold may be considered for beacons. By default, ChemSTEP chooses each set of beacons to be as maximally diverse as possible. Choosing too many beacons may result in decreased diversity between beacons, but too few beacons could hinder space exploration. If not enough molecules score above your pProp threshold, ChemSTEP may assign fewer beacons than the assigned value. <br /> <br /> ''max_n_rounds:''' if prospectively running ChemSTEP, as outlined in this wiki, no need to worry about this parameter.<br /> <br /> <br /> There should be no need to edit run_chemstep.py or the SGE wrapper script. ''If using another virtual library, be sure to update the path in run_chemstep.py to point to your FP library.'' At this time, we strongly suggest running ChemSTEP as a job array with 64 CPU slots requested, which is specified in the default wrapper. <br /> <br /> <br /> '''7. Run ChemSTEP''' with the following command: <br /> qsub launch_chemstep_as_job.sh <br /> <br /> <br /> This will launch the main ChemSTEP job to the SGE scheduler. Check your job status with the 'qstat' command. The algorithm will read the score and indices files provided, calculate pProp (for round 1) and assign beacons. As this job runs, it will launch a subsequent job array. These sub-jobs are the Chaining step of ChemSTEP, where each parallel worker calculates the Tanimoto distances of 100 million molecules from the virtual library to the beacons. These distances are then written to the files within the /output directory. When these jobs complete, the main job will read through all Tanimoto distances and pull molecules for prioritization. <br /> <br /> Running ChemSTEP successfully will result in the output of a SMILES file within output/complete_info/. You will also get (1) chemstep_algo.log and (2) chemstep_submission.log in the working directory after job submission. chemstep_algo.log contains the running output of the algorithm in a readable format, with timestamps, including the DOCK score associated with your desired pProp. chemstep_submission.log contains the output of ChemSTEP, as well as any information output by the SGE submission system (python errors, cluster issues, tracebacks if failed). These files will be updated with any information with iterative rounds of ChemSTEP run in the same directory. '''Visually inspect these files after each round of ChemSTEP to ensure the algorithm has picked your desired number of beacons and things ran smoothly.'''<br /> <br /> '''Troubleshooting:''' if no jobs are running and there is no SMI file, check the chemstep_submission.log first. Any traceback error or SGE error should give you some idea of why ChemSTEP failed. Some errors may be due to SGE or Wynton issues. If directed, look at a few .out files within /output/jobs/ . If a ChemSTEP run fails on your FIRST run, delete the output and log files, fix what needs to be fixed, and rerun. Errors in subsequent rounds during the chaining step can potentially corrupt chaining files within the /output directory that are needed for prioritization. At the very least, you will definitely have duplicates of beacons and information written to the log files. At this time, if your run fails during iterative rounds, it's best to start from the beginning. <br /> <br /> Prioritized molecules should be built, docked, and their scores fed back into ChemSTEP. More detailed instructions below: <br /> <br /> '''8. Build prioritized molecules (DOCK 3.8).''' /taken from docs.docking.org <br /> source /wynton/group/bks/soft/DOCK-3.8.5/env.sh<br /> <br /> python /wynton/group/bks/soft/DOCK-3.8.5/DOCK3.8/zinc22-3d/submit/submit_building_docker.py --output_folder building_output --bundle_size 1000 --minutes_per_mol 5 --skip_name_check --scheduler sge --container_software apptainer --container_path_or_name /wynton/group/bks/soft/DOCK-3.8.5/building_pipeline.sif smi_round_1.smi<br /> <br /> When building has completed, you must write an SDI file with the complete paths to each built bundle and dock. you can do this with:<br /> find /path/to/your/building_output -name &quot;bundle.db2.tgz&quot; -type f &gt; round_N.wynton.sdi<br /> <br /> <br /> '''Be sure to change the INDOCK file to save only poses that meet your score pProp score threshold calculated by ChemSTEP!''' Retrieve docking scores as convert to NumPy arrays as outlined above, updating the round number when running get_scores.py and convert_scores_to_npy.py. Copy new score and indices files into the directory you ran ChemSTEP in. If you are following along as a tutorial, you should have scores_round_1.npy and indices_round_1.npy from the previous step (from FIRST round of ChemSTEP prioritization). <br /> <br /> <br /> '''9. Set up for iterative rounds of ChemSTEP'''<br /> <br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/run_chemstep_iterative.py .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/launch_chemstep_iterative.sh .<br /> <br /> <br /> '''10. Run ChemSTEP'''<br /> qsub launch_chemstep_as_job.sh [round number]<br /> <br /> For the first iterative round, the round number is [2], and should increase by one for every subsequent round of ChemSTEP. The output will be smi_round_****.smi file. Repeat steps 8-10 for as many rounds as needed. The performance is reported in outputy/complete_info/run_summary.df, which contains the number of beacons selected, the number of molecules docked, the number of hits found, the distance threshold for the selected molecules to dock, and the last added beacon's distance to all previous beacons.<br /> <br /> <br /> ** anecdotally true.</div>

Kholland http://wiki.docking.org/index.php?title=Running_ChemSTEP&diff=16946 2025-10-08T21:32:09Z

<p>Kholland: </p> <hr /> <div>last update: oct 8 2025 katie. current ver = 0.3.1.5<br /> <br /> ChemSTEP (Chemical Space Traversal and Exploration Procedure) is an open-source, transparent acceleration algorithm for molecular docking capable of dealing with virtual libraries of several trillion compounds. This wiki page is a guide for BKS lab members to run ChemSTEP on Wynton HPC, using the current version of InifiSee XReal library (1.1T). For more general use directions, please refer to [ChemSTEP Read-the-Docs]. <br /> <br /> At a high-level, ChemSTEP is an iterative process that identifies molecules from the larger virtual library to prioritize for docking. First, we identify a random sample of the total library (termed &quot;seed set&quot;, round zero) and dock those molecules to the target of interest. From this seed set, we can calculate total-library pProp values (-log rank percentages) and the number of &quot;virtual hits&quot; in the total library (high-scoring molecules). ChemSTEP will identify a set of maximally diverse molecules that score above the desired pProp threshold (&quot;beacons&quot;) from the seed set. These beacons guide prioritization, where molecules chosen and output by ChemSTEP are close in chemical space to the beacons. Prioritized molecules are then built, docked, and returned to ChemSTEP for a second round of prioritization. This process is iterated until you reach desired virtual hit recovery, or you are no longer recovering virtual hits. <br /> <br /> <br /> Running the ChemSTEP algorithm on Wynton takes place in loosely three steps: (1) submission of the main ChemSTEP job, (2) ChainingLog sub-job array, and (3) gathering of SMILES for prioritization. <br /> 1. Main ChemSTEP job: reads input files (DOCK scores and line-matched indices NumPy arrays) and assigns beacons. Beacons are a set of N maximally diverse molecules that score above the pProp threshold specified in the parameter file. For the first round of ChemSTEP, the best-scoring molecule from the seed set is assigned as the first beacon. Subsequent beacons would be molecules that score well (above pProp thresh), and are as far as possible by Td to all assigned beacons. In iterative rounds of ChemSTEP, the first beacon chosen is based on diversity from ALL PREVIOUSLY assigned beacons, including those in earlier rounds. <br /> 2. ChainingLog sub-jobs: once beacons are assigned, ChemSTEP will launch a series of sub-jobs that calculate the Tanimoto distances of every molecule remaining in the library to the assigned beacons. Calculated distances to the NEAREST beacon (minimum-minimum Td) are updated in the mintddistrib_*.npy files within the /output directory. <br /> 3. Gathering of SMILES for prioritization: When all Td calculations are completed, the algo will then choose N number of molecules for prioritization, which is specified by the user in the parameter file. Molecules that are ''closest in chemical space'' to beacons are prioritized. I.e. if round size = 1 million, the 1 million molecules with the smallest min-Td to any beacon are prioritized. chemstep_algo.log will provide a &quot;max-minTd&quot; for each round, which is the Tanimoto distance of the most dissimilar molecule prioritized to any one beacon per round. <br /> <br /> <br /> <br /> What the user need: DOCKFILES, directories for (1) docking (2) building and (3) running ChemSTEP. '''All iterative rounds of ChemSTEP should be run in the SAME directory.''' <br /> <br /> '''1. Source ChemSTEP virtual environment on Wynton'''<br /> source /wynton/group/bks/work/shared/kholland/chemstep_0.3.1.5/bin/activate<br /> <br /> <br /> '''2. Copy InfiniSee seed-set SDI file into your docking directory'''<br /> This directory should already contain your dockfiles, with INDOCK parameters set to your liking. In this step, we are copying in a split database index file (SDI) containing paths to bundles of db2 files. This seed set contains 100 million molecules sampled randomly from the total virtual library, currently 1.1 trillion molecules.<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/XR_seed_set_v0p0.wynton.sdi .<br /> <br /> '''3. Dock seed set to your receptor of interest using DOCK 3.8''' /docking directions taken from docs.docking.org. This is meant to be done as you would do a normal LSD. <br /> export MOLECULES_DIR_TO_BIND=[outermost folder containing the molecules to dock, just your base docking directory]<br /> export DOCKFILES=[path to your dockfiles]<br /> export INPUT_FOLDER=[the folder containing your .sdi file(s)]<br /> export OUTPUT_FOLDER=[where you want the output ]<br /> <br /> /wynton/group/bks/work/bwhall61/needs_github/super_dock3r.sh<br /> <br /> Wait for docking to complete. Next, you must extract all molecule IDs and corresponding DOCK scores from above. To do so, run the following commands in the base docking directory containing your docking files and output folder, while logged into a dev node (I suggest in a screen):<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/get_scores.py .<br /> python get_scores.py 0<br /> <br /> <br /> This script expects a directory named &quot;output*&quot; within the CWD. If your output from docking follows different naming conventions, vim into get_scores.py and change the path. The output will be a file named &quot;scores_round_0.txt&quot;. For iterative rounds, pass increasing numbers into the command line. i.e. When docking the first round of prioritized molecules, pass [1] for scores_round_1.txt. <br /> <br /> '''4. Convert scores and molecule IDS into NumPY arrays for ChemSTEP recognition. Requires ChemSTEP venv'''<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/convert_scores_to_npy.py .<br /> python convert_scores_to_npy.py 0<br /> <br /> For python speed purposes, ChemSTEP requires that DOCK scores and IDs be give in the form of a NumPY array. In this step, we are taking our readable scores file and converting them for ChemSTEP recognition. This script expects a txt file with Molecule IDS and DOCK scores. Use the same round number you used in step 3. As run above, this will output two files named scores_round_0.npy and indices_round_0.npy that contain line-matched indices (determined from Mol ID) and their respective docking scores. <br /> <br /> '''5. Enter into or make a directory to run ChemSTEP in. Copy in necessary files for initiating ChemSTEP: params.txt, run_chemstep.py and launch_chemstep_as_job.sh. Copy in your score and indices numpy files as well, which should be in your docking directory.'''<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/params.txt .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/run_chemstep.py .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/launch_chemstep_as_job.sh .<br /> <br /> <br /> '''6. Edit params.txt file''' This ONLY needs to be edited for the initial round of ChemSTEP. The parameters outlined here will be carried through all rounds of ChemSTEP. <br /> seed_indices_file: /absoulte/path/to/your/indices_round_0.npy<br /> seed_scores_file: /absolute/path/to/your/scores_round_0.npy<br /> hit_pprop: 5<br /> n_docked_per_round: 10000000<br /> max_beacons: 150<br /> max_n_rounds: 250<br /> <br /> Within params.txt, add the absolute paths to the ChemSTEP-readable score and indices NumPY arrays. The rest of the values within the params.txt file are left to the discretion of the user, with some considerations below. <br /> <br /> '''pProp:''' this value will define what is considered a &quot;virtual hit&quot; in your campaign. pProp is defined as the -log(rank %) of a molecule within the total library score-distribution. For example, a pProp of 4 in the 1.1T XReal space is equivalent the top 0.01% of the library, corresponding to the top 110 million molecules. pProp 5 = 0.001% = 11 million virtual hits, etc. From the seed set, ChemSTEP will estimate a DOCK score value in kcal/mol that associated with the lower-limit of your desired pProp zone. Any molecules that score better than the threshold is considered a virtual hit in your campaign. Be mindful of seed set size when choosing desired pProp. Our suggestion is that the size of the seed set should be at least 10^(pProp +2).<br /> <br /> '''n_docked_per_round:''' this number is the desired molecules for prioritization per round. When choosing this value, note that this number of molecules must be built and docked in between each round of ChemSTEP. Prioritizing many molecules will slow building and docking speeds, and coupled with few beacons (below) may lead to decreased diversity. Prioritizing too few molecules may result in slower virtual hit recovery. Round size does not significantly impact the algorithm running time.** <br /> <br /> '''max_beacons:''' this is the number of diverse, well-scoring molecules ChemSTEP will use to guide prioritization per round. All molecules that score better than the assigned pProp threshold may be considered for beacons. By default, ChemSTEP chooses each set of beacons to be as maximally diverse as possible. Choosing too many beacons may result in decreased diversity between beacons, but too few beacons could hinder space exploration. If not enough molecules score above your pProp threshold, ChemSTEP may assign fewer beacons than the assigned value. <br /> <br /> ''max_n_rounds:''' if prospectively running ChemSTEP, as outlined in this wiki, no need to worry about this parameter.<br /> <br /> <br /> There should be no need to edit run_chemstep.py or the SGE wrapper script. ''If using another virtual library, be sure to update the path in run_chemstep.py to point to your FP library.'' At this time, we strongly suggest running ChemSTEP as a job array with 64 CPU slots requested, which is specified in the default wrapper. <br /> <br /> <br /> '''7. Run ChemSTEP''' with the following command: <br /> qsub launch_chemstep_as_job.sh <br /> <br /> <br /> This will launch the main ChemSTEP job to the SGE scheduler. Check your job status with the 'qstat' command. The algorithm will read the score and indices files provided, calculate pProp (for round 1) and assign beacons. As this job runs, it will launch a subsequent job array. These sub-jobs are the Chaining step of ChemSTEP, where each parallel worker calculates the Tanimoto distances of 100 million molecules from the virtual library to the beacons. These distances are then written to the files within the /output directory. When these jobs complete, the main job will read through all Tanimoto distances and pull molecules for prioritization. <br /> <br /> Running ChemSTEP successfully will result in the output of a SMILES file within output/complete_info/. You will also get (1) chemstep_algo.log and (2) chemstep_submission.log in the working directory after job submission. chemstep_algo.log contains the running output of the algorithm in a readable format, with timestamps, including the DOCK score associated with your desired pProp. chemstep_submission.log contains the output of ChemSTEP, as well as any information output by the SGE submission system (python errors, cluster issues, tracebacks if failed). These files will be updated with any information with iterative rounds of ChemSTEP run in the same directory. '''Visually inspect these files after each round of ChemSTEP to ensure the algorithm has picked your desired number of beacons and things ran smoothly.'''<br /> <br /> '''Troubleshooting:''' if no jobs are running and there is no SMI file, check the chemstep_submission.log first. Any traceback error or SGE error should give you some idea of why ChemSTEP failed. Some errors may be due to SGE or Wynton issues. If directed, look at a few .out files within /output/jobs/ . If a ChemSTEP run fails on your FIRST run, delete the output and log files, fix what needs to be fixed, and rerun. Errors in subsequent rounds during the chaining step can potentially corrupt chaining files within the /output directory that are needed for prioritization. At the very least, you will definitely have duplicates of beacons and information written to the log files. At this time, if your run fails during iterative rounds, it's best to start from the beginning. <br /> <br /> Prioritized molecules should be built, docked, and their scores fed back into ChemSTEP. More detailed instructions below: <br /> <br /> '''8. Build prioritized molecules (DOCK 3.8).''' /taken from docs.docking.org <br /> source /wynton/group/bks/soft/DOCK-3.8.5/env.sh<br /> <br /> python /wynton/group/bks/soft/DOCK-3.8.5/DOCK3.8/zinc22-3d/submit/submit_building_docker.py --output_folder building_output --bundle_size 1000 --minutes_per_mol 5 --skip_name_check --scheduler sge --container_software apptainer --container_path_or_name /wynton/group/bks/soft/DOCK-3.8.5/building_pipeline.sif smi_round_1.smi<br /> <br /> When building has completed, you must write an SDI file with the complete paths to each built bundle and dock. you can do this with:<br /> find /path/to/your/building_output -name &quot;bundle.db2.tgz&quot; -type f &gt; round_N.wynton.sdi<br /> <br /> <br /> '''Be sure to change the INDOCK file to save only poses that meet your score pProp score threshold calculated by ChemSTEP!''' Retrieve docking scores as convert to NumPy arrays as outlined above, updating the round number when running get_scores.py and convert_scores_to_npy.py. Copy new score and indices files into the directory you ran ChemSTEP in. If you are following along as a tutorial, you should have scores_round_1.npy and indices_round_1.npy from the previous step (from FIRST round of ChemSTEP prioritization). <br /> <br /> <br /> '''9. Set up for iterative rounds of ChemSTEP'''<br /> <br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/run_chemstep_iterative.py .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/launch_chemstep_iterative.sh .<br /> <br /> <br /> '''10. Run ChemSTEP'''<br /> qsub launch_chemstep_as_job.sh [round number]<br /> <br /> For the first iterative round, the round number is [2], and should increase by one for every subsequent round of ChemSTEP. The output will be smi_round_****.smi file. Repeat steps 8-10 for as many rounds as needed. The performance is reported in outputy/complete_info/run_summary.df, which contains the number of beacons selected, the number of molecules docked, the number of hits found, the distance threshold for the selected molecules to dock, and the last added beacon's distance to all previous beacons.<br /> <br /> <br /> ** anecdotally true.</div>

Kholland http://wiki.docking.org/index.php?title=Running_ChemSTEP&diff=16900 2025-09-18T20:15:48Z

<p>Kholland: </p> <hr /> <div>last update: Sept 18 2025 katie. current ver = 0.3.1<br /> <br /> ChemSTEP (Chemical Space Traversal and Exploration Procedure) is an open-source, transparent acceleration algorithm for molecular docking capable of dealing with virtual libraries of several trillion compounds. This wiki page is a guide for BKS lab members to run ChemSTEP on Wynton HPC, using the current version of InifiSee XReal library (1.1T). For more general use directions, please refer to [ChemSTEP Read-the-Docs]. <br /> <br /> At a high-level, ChemSTEP is an iterative process that identifies molecules from the larger virtual library to prioritize for docking. First, we identify a random sample of the total library (termed &quot;seed set&quot;, round zero) and dock those molecules to the target of interest. From this seed set, we can calculate total-library pProp values (-log rank percentages) and the number of &quot;virtual hits&quot; in the total library (high-scoring molecules). ChemSTEP will identify a set of maximally diverse molecules that score above the desired pProp threshold (&quot;beacons&quot;) from the seed set. These beacons guide prioritization, where molecules chosen and output by ChemSTEP are close in chemical space to the beacons. Prioritized molecules are then built, docked, and returned to ChemSTEP for a second round of prioritization. This process is iterated until you reach desired virtual hit recovery, or you are no longer recovering virtual hits. <br /> <br /> <br /> Running the ChemSTEP algorithm on Wynton takes place in loosely three steps: (1) submission of the main ChemSTEP job, (2) ChainingLog sub-job array, and (3) gathering of SMILES for prioritization. <br /> <br /> <br /> What the user need: DOCKFILES, directories for (1) docking (2) building and (3) running ChemSTEP. '''All iterative rounds of ChemSTEP should be run in the SAME directory.''' <br /> <br /> '''1. Source ChemSTEP virtual environment on Wynton'''<br /> source /wynton/group/bks/work/shared/kholland/chemstep_env/bin/activate<br /> <br /> <br /> '''2. Copy InfiniSee seed-set SDI file into your docking directory'''<br /> This directory should already contain your dockfiles, with INDOCK parameters set to your liking. In this step, we are copying in a split database index file (SDI) containing paths to bundles of db2 files. This seed set contains 100 million molecules sampled randomly from the total virtual library, currently 1.1 trillion molecules.<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/XR_seed_set_v0p0.wynton.sdi .<br /> <br /> '''3. Dock seed set to your receptor of interest using DOCK 3.8''' /docking directions taken from docs.docking.org. This is meant to be done as you would do a normal LSD. <br /> export MOLECULES_DIR_TO_BIND=[outermost folder containing the molecules to dock, just your base docking directory]<br /> export DOCKFILES=[path to your dockfiles]<br /> export INPUT_FOLDER=[the folder containing your .sdi file(s)]<br /> export OUTPUT_FOLDER=[where you want the output ]<br /> <br /> /wynton/group/bks/work/bwhall61/needs_github/super_dock3r.sh<br /> <br /> Wait for docking to complete. Next, you must extract all molecule IDs and corresponding DOCK scores from above. To do so, run the following commands in the base docking directory containing your docking files and output folder, while logged into a dev node (I suggest in a screen):<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/get_scores.py .<br /> python get_scores.py 0<br /> <br /> <br /> This script expects a directory named &quot;output*&quot; within the CWD. If your output from docking follows different naming conventions, vim into get_scores.py and change the path. The output will be a file named &quot;scores_round_0.txt&quot;. For iterative rounds, pass increasing numbers into the command line. i.e. When docking the first round of prioritized molecules, pass [1] for scores_round_1.txt. <br /> <br /> '''4. Convert scores and molecule IDS into NumPY arrays for ChemSTEP recognition. Requires ChemSTEP venv'''<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/convert_scores_to_npy.py .<br /> python convert_scores_to_npy.py 0<br /> <br /> For python speed purposes, ChemSTEP requires that DOCK scores and IDs be give in the form of a NumPY array. In this step, we are taking our readable scores file and converting them for ChemSTEP recognition. This script expects a txt file with Molecule IDS and DOCK scores. Use the same round number you used in step 3. As run above, this will output two files named scores_round_0.npy and indices_round_0.npy that contain line-matched indices (determined from Mol ID) and their respective docking scores. <br /> <br /> '''5. Enter into or make a directory to run ChemSTEP in. Copy in necessary files for initiating ChemSTEP: params.txt, run_chemstep.py and launch_chemstep_as_job.sh. Copy in your score and indices numpy files as well, which should be in your docking directory.'''<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/params.txt .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/run_chemstep.py .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/launch_chemstep_as_job.sh .<br /> <br /> <br /> '''6. Edit params.txt file''' This ONLY needs to be edited for the initial round of ChemSTEP. The parameters outlined here will be carried through all rounds of ChemSTEP. <br /> seed_indices_file: /absoulte/path/to/your/indices_round_0.npy<br /> seed_scores_file: /absolute/path/to/your/scores_round_0.npy<br /> hit_pprop: 5<br /> n_docked_per_round: 10000000<br /> max_beacons: 150<br /> max_n_rounds: 250<br /> <br /> Within params.txt, add the absolute paths to the ChemSTEP-readable score and indices NumPY arrays. The rest of the values within the params.txt file are left to the discretion of the user, with some considerations below. <br /> <br /> '''pProp:''' this value will define what is considered a &quot;virtual hit&quot; in your campaign. pProp is defined as the -log(rank %) of a molecule within the total library score-distribution. For example, a pProp of 4 in the 1.1T XReal space is equivalent the top 0.01% of the library, corresponding to the top 110 million molecules. pProp 5 = 0.001% = 11 million virtual hits, etc. From the seed set, ChemSTEP will estimate a DOCK score value in kcal/mol that associated with the lower-limit of your desired pProp zone. Any molecules that score better than the threshold is considered a virtual hit in your campaign. Be mindful of seed set size when choosing desired pProp. Our suggestion is that the size of the seed set should be at least 10^(pProp +2).<br /> <br /> '''n_docked_per_round:''' this number is the desired molecules for prioritization per round. When choosing this value, note that this number of molecules must be built and docked in between each round of ChemSTEP. Prioritizing many molecules will slow building and docking speeds, and coupled with few beacons (below) may lead to decreased diversity. Prioritizing too few molecules may result in slower virtual hit recovery. Round size does not significantly impact the algorithm running time.** <br /> <br /> '''max_beacons:''' this is the number of diverse, well-scoring molecules ChemSTEP will use to guide prioritization per round. All molecules that score better than the assigned pProp threshold are be considered for beacons. By default, ChemSTEP chooses each set of beacons to be as maximally diverse as possible. Choosing too many beacons may result in decreased diversity between beacons, but too few beacons could hinder space exploration. If not enough molecules score above your pProp threshold, ChemSTEP may assign fewer beacons than assigned value. <br /> <br /> ''max_n_rounds:''' if prospectively running ChemSTEP, as outlined in this wiki, no need to worry about this parameter.<br /> <br /> <br /> There should be no need to edit run_chemstep.py or the SGE wrapper script. ''If using another virtual library, be sure to update the path in run_chemstep.py to point to your FP library.'' At this time, we strongly suggest running ChemSTEP as a job array with 64 CPU slots requested, which is specified in the default wrapper. <br /> <br /> <br /> '''7. Run ChemSTEP''' with the following command: <br /> qsub launch_chemstep_as_job.sh <br /> <br /> <br /> This will launch the main ChemSTEP job to the SGE scheduler. Check your job status with the 'qstat' command. The algorithm will read the score and indices files provided, calculate pProp (for round 1) and assign beacons. As this job runs, it will launch a subsequent job array. These sub-jobs are the Chaining step of ChemSTEP, where each parallel worker calculates the Tanimoto distances of 100 million molecules from the virtual library to the beacons. These distances are then written to the files within the /output directory. When these jobs complete, the main job will read through all Tanimoto distances and pull molecules for prioritization. <br /> <br /> Running ChemSTEP successfully will result in the output of a SMILES file within output/complete_info/. You will also get (1) chemstep_algo.log and (2) chemstep_submission.log in the working directory after job submission. chemstep_algo.log contains the running output of the algorithm in a readable format, with timestamps, including the DOCK score associated with your desired pProp. chemstep_submission.log contains the output of ChemSTEP, as well as any information output by the SGE submission system (python errors, cluster issues, tracebacks if failed). These files will be updated with any information with iterative rounds of ChemSTEP run in the same directory. '''Visually inspect these files after each round of ChemSTEP to ensure the algorithm has picked your desired number of beacons and things ran smoothly.'''<br /> <br /> '''Troubleshooting:''' if no jobs are running and there is no SMI file, check the chemstep_submission.log first. Any traceback error or SGE error should give you some idea of why ChemSTEP failed. Some errors may be due to SGE or Wynton issues. If directed, look at a few .out files within /output/jobs/ . If a ChemSTEP run fails on your FIRST run, delete the output and log files, fix what needs to be fixed, and rerun. Errors in subsequent rounds during the chaining step can potentially corrupt chaining files within the /output directory that are needed for prioritization. At the very least, you will definitely have duplicates of beacons and information written to the log files. At this time, if your run fails during iterative rounds, it's best to start from the beginning. <br /> <br /> Prioritized molecules should be built, docked, and their scores fed back into ChemSTEP. More detailed instructions below: <br /> <br /> '''8. Build prioritized molecules (DOCK 3.8).''' /taken from docs.docking.org <br /> source /wynton/group/bks/soft/DOCK-3.8.5/env.sh<br /> <br /> python /wynton/group/bks/soft/DOCK-3.8.5/DOCK3.8/zinc22-3d/submit/submit_building_docker.py --output_folder building_output --bundle_size 1000 --minutes_per_mol 5 --skip_name_check --scheduler sge --container_software apptainer --container_path_or_name /wynton/group/bks/soft/DOCK-3.8.5/building_pipeline.sif smi_round_1.smi<br /> <br /> When building has completed, you must write an SDI file with the complete paths to each built bundle and dock. you can do this with:<br /> find /path/to/your/building_output -name &quot;bundle.db2.tgz&quot; -type f &gt; round_N.wynton.sdi<br /> <br /> <br /> '''Be sure to change the INDOCK file to save only poses that meet your score pProp score threshold calculated by ChemSTEP!''' Retrieve docking scores as convert to NumPy arrays as outlined above, updating the round number when running get_scores.py and convert_scores_to_npy.py. Copy new score and indices files into the directory you ran ChemSTEP in. If you are following along as a tutorial, you should have scores_round_1.npy and indices_round_1.npy from the previous step (from FIRST round of ChemSTEP prioritization). <br /> <br /> <br /> '''9. Set up for iterative rounds of ChemSTEP'''<br /> <br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/run_chemstep_iterative.py .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/launch_chemstep_iterative.sh .<br /> <br /> <br /> '''10. Run ChemSTEP'''<br /> qsub launch_chemstep_as_job.sh [round number]<br /> <br /> For the first iterative round, the round number is [2], and should increase by one for every subsequent round of ChemSTEP. The output will be smi_round_****.smi file. Repeat steps 8-10 for as many rounds as needed. The performance is reported in outputy/complete_info/run_summary.df, which contains the number of beacons selected, the number of molecules docked, the number of hits found, the distance threshold for the selected molecules to dock, and the last added beacon's distance to all previous beacons.<br /> <br /> <br /> ** anecdotally true.</div>

Kholland http://wiki.docking.org/index.php?title=Running_ChemSTEP&diff=16898 2025-09-18T20:14:33Z

<p>Kholland: </p> <hr /> <div>last update: Sept 18 2025 katie. current ver = 0.3.1<br /> <br /> ChemSTEP (Chemical Space Traversal and Exploration Procedure) is an open-source, transparent acceleration algorithm for molecular docking capable of dealing with virtual libraries of several trillion compounds. This wiki page is a guide for BKS lab members to run ChemSTEP on Wynton HPC, using the current version of InifiSee XReal library (1.1T). For more general use directions, please refer to [ChemSTEP Read-the-Docs]. <br /> <br /> At a high-level, ChemSTEP is an iterative process that identifies molecules from the larger virtual library to prioritize for docking. First, we identify a random sample of the total library (termed &quot;seed set&quot;, round zero) and dock those molecules to the target of interest. From this seed set, we can calculate total-library pProp values (-log rank percentages) and the number of &quot;virtual hits&quot; in the total library (high-scoring molecules). ChemSTEP will identify a set of maximally diverse molecules that score above the desired pProp threshold (&quot;beacons&quot;) from the seed set. These beacons guide prioritization, where molecules chosen and output by ChemSTEP are close in chemical space to the beacons. Prioritized molecules are then built, docked, and returned to ChemSTEP for a second round of prioritization. This process is iterated until you reach desired virtual hit recovery, or you are no longer recovering virtual hits. <br /> <br /> <br /> Running the ChemSTEP algorithm on Wynton takes place in loosely three steps: (1) submission of the main ChemSTEP job, (2) ChainingLog sub-job array, and (3) gathering of SMILES for prioritization. <br /> <br /> <br /> What the user need: DOCKFILES, directories for (1) docking (2) building and (3) running ChemSTEP. '''All iterative rounds of ChemSTEP should be run in the SAME directory.''' <br /> <br /> '''1. Source ChemSTEP virtual environment on Wynton'''<br /> source /wynton/group/bks/work/shared/kholland/chemstep_env/bin/activate<br /> <br /> <br /> '''2. Copy InfiniSee seed-set SDI file into your docking directory'''<br /> This directory should already contain your dockfiles, with INDOCK parameters set to your liking. In this step, we are copying in a split database index file (SDI) containing paths to bundles of db2 files. This seed set contains 100 million molecules sampled randomly from the total virtual library, currently 1.1 trillion molecules.<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/XR_seed_set_v0p0.wynton.sdi .<br /> <br /> '''3. Dock seed set to your receptor of interest using DOCK 3.8''' /docking directions taken from docs.docking.org. This is meant to be done as you would do a normal LSD. <br /> export MOLECULES_DIR_TO_BIND=[outermost folder containing the molecules to dock, just your base docking directory]<br /> export DOCKFILES=[path to your dockfiles]<br /> export INPUT_FOLDER=[the folder containing your .sdi file(s)]<br /> export OUTPUT_FOLDER=[where you want the output ]<br /> <br /> /wynton/group/bks/work/bwhall61/needs_github/super_dock3r.sh<br /> <br /> Wait for docking to complete. Next, you must extract all molecule IDs and corresponding DOCK scores from above. To do so, run the following commands in the base docking directory containing your docking files and output folder, while logged into a dev node (I suggest in a screen):<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/get_scores.py .<br /> python get_scores.py 0<br /> <br /> <br /> This script expects a directory named &quot;output*&quot; within the CWD. If your output from docking follows different naming conventions, vim into get_scores.py and change the path. The output will be a file named &quot;scores_round_0.txt&quot;. For iterative rounds, pass increasing numbers into the command line. i.e. When docking the first round of prioritized molecules, pass [1] for scores_round_1.txt. <br /> <br /> '''4. Convert scores and molecule IDS into NumPY arrays for ChemSTEP recognition. Requires ChemSTEP venv'''<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/convert_scores_to_npy.py .<br /> python convert_scores_to_npy.py 0<br /> <br /> For python speed purposes, ChemSTEP requires that DOCK scores and IDs be give in the form of a NumPY array. In this step, we are taking our readable scores file and converting them for ChemSTEP recognition. This script expects a txt file with Molecule IDS and DOCK scores. Use the same round number you used in step 3. As run above, this will output two files named scores_round_0.npy and indices_round_0.npy that contain line-matched indices (determined from Mol ID) and their respective docking scores. <br /> <br /> '''5. Enter into or make a directory to run ChemSTEP in. Copy in necessary files for initiating ChemSTEP: params.txt, run_chemstep.py and launch_chemstep_as_job.sh. Copy in your score and indices numpy files as well, which should be in your docking directory.'''<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/params.txt .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/run_chemstep.py .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/launch_chemstep_as_job.sh .<br /> <br /> <br /> '''6. Edit params.txt file''' This ONLY needs to be edited for the initial round of ChemSTEP. The parameters outlined here will be carried through all rounds of ChemSTEP. <br /> seed_indices_file: /absoulte/path/to/your/indices_round_0.npy<br /> seed_scores_file: /absolute/path/to/your/scores_round_0.npy<br /> hit_pprop: 5<br /> n_docked_per_round: 10000000<br /> max_beacons: 150<br /> max_n_rounds: 250<br /> <br /> Within params.txt, add the absolute paths to the ChemSTEP-readable score and indices NumPY arrays. The rest of the values within the params.txt file are left to the discretion of the user, with some considerations below. <br /> <br /> '''pProp:''' this value will define what is considered a &quot;virtual hit&quot; in your campaign. pProp is defined as the -log(rank %) of a molecule within the total library score-distribution. For example, a pProp of 4 in the 1.1T XReal space is equivalent the top 0.01% of the library, corresponding to the top 110 million molecules. pProp 5 = 0.001% = 11 million virtual hits, etc. From the seed set, ChemSTEP will estimate a DOCK score value in kcal/mol that associated with the lower-limit of your desired pProp zone. Any molecules that score better than the threshold is considered a virtual hit in your campaign. Be mindful of seed set size when choosing desired pProp. Our suggestion is that the size of the seed set should be at least 10^(pProp +2).<br /> <br /> '''n_docked_per_round:''' this number is the desired molecules for prioritization per round. When choosing this value, note that this number of molecules must be built and docked in between each round of ChemSTEP. Prioritizing many molecules will slow building and docking speeds, and coupled with few beacons (below) may lead to decreased diversity. Prioritizing too few molecules may result in slower virtual hit recovery. Round size does not significantly impact the algorithm running time.** <br /> <br /> '''max_beacons:''' this is the number of diverse, well-scoring molecules ChemSTEP will use to guide prioritization per round. All molecules that score better than the assigned pProp threshold are be considered for beacons. By default, ChemSTEP chooses each set of beacons to be as maximally diverse as possible. Choosing too many beacons may result in decreased diversity between beacons, but too few beacons could hinder space exploration. If not enough molecules score above your pProp threshold, ChemSTEP may assign fewer beacons than assigned value. <br /> <br /> ''max_n_rounds:''' if prospectively running ChemSTEP, as outlined in this wiki, no need to worry about this parameter.<br /> <br /> <br /> There should be no need to edit run_chemstep.py or the SGE wrapper script. ''If using another virtual library, be sure to update the path in run_chemstep.py to point to your FP library.'' At this time, we strongly suggest running ChemSTEP as a job array with 64 CPU slots requested, which is specified in the default wrapper. <br /> <br /> <br /> '''7. Run ChemSTEP''' with the following command: <br /> qsub launch_chemstep_as_job.sh <br /> <br /> <br /> This will launch the main ChemSTEP job to the SGE scheduler. Check your job status with the 'qstat' command. The algorithm will read the score and indices files provided, calculate pProp (for round 1) and assign beacons. As this job runs, it will launch a subsequent job array. These sub-jobs are the Chaining step of ChemSTEP, where each parallel worker calculates the Tanimoto distances of 100 million molecules from the virtual library to the beacons. These distances are then written to the files within the /output directory. When these jobs complete, the main job will read through all Tanimoto distances and pull molecules for prioritization. <br /> <br /> Running ChemSTEP successfully will result in the output of a SMILES file within output/complete_info/. You will also get (1) chemstep_algo.log and (2) chemstep_submission.log in the working directory after job submission. chemstep_algo.log contains the running output of the algorithm in a readable format, with timestamps, including the DOCK score associated with your desired pProp. chemstep_submission.log contains the output of ChemSTEP, as well as any information output by the SGE submission system (python errors, cluster issues, tracebacks if failed). These files will be updated with any information with iterative rounds of ChemSTEP run in the same directory. '''Visually inspect these files after each round of ChemSTEP to ensure the algorithm has picked your desired number of beacons and things ran smoothly.'''<br /> <br /> '''Troubleshooting:''' if no jobs are running and there is no SMI file, check the chemstep_submission.log first. Any traceback error or SGE error should give you some idea of why ChemSTEP failed. Some errors may be due to SGE or Wynton issues. If directed, look at a few .out files within /output/jobs/ . If a ChemSTEP run fails on your FIRST run, delete the output and log files, fix what needs to be fixed, and rerun. Errors in subsequent rounds during the chaining step can potentially corrupt chaining files within the /output directory that are needed for prioritization. At the very least, you will definitely have duplicates of beacons and information written to the log files. At this time, if your run fails during iterative rounds, it's best to start from the beginning. <br /> <br /> Prioritized molecules should be built, docked, and their scores fed back into ChemSTEP. More detailed instructions below: <br /> <br /> '''8. Build prioritized molecules (DOCK 3.8).''' /taken from docs.docking.org <br /> source /wynton/group/bks/soft/DOCK-3.8.5/env.sh<br /> <br /> python /wynton/group/bks/soft/DOCK-3.8.5/DOCK3.8/zinc22-3d/submit/submit_building_docker.py --output_folder building_output --bundle_size 1000 --minutes_per_mol 5 --skip_name_check --scheduler sge --container_software apptainer --container_path_or_name /wynton/group/bks/soft/DOCK-3.8.5/building_pipeline.sif smi_round_1.smi<br /> <br /> When building has completed, you must write an SDI file with the complete paths to each built bundle and dock. you can do this with:<br /> find /path/to/your/building_output -name &quot;bundle.db2.tgz&quot; -type f &gt; round_N.wynton.sdi<br /> <br /> <br /> '''Be sure to change the INDOCK file to save only poses that meet your score pProp score threshold calculated by ChemSTEP!''' Retrieve docking scores as convert to NumPy arrays as outlined above, update the round number when running get_scores.py and convert_scores_to_npy.py! Copy new score and indices files into the directory you ran ChemSTEP in. If you are following along as a tutorial, you should have scores_round_1.npy and indices_round_1.npy from the previous step (from FIRST round of ChemSTEP prioritization). <br /> <br /> <br /> '''9. Set up for iterative rounds of ChemSTEP'''<br /> <br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/run_chemstep_iterative.py .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/launch_chemstep_iterative.sh .<br /> <br /> <br /> '''10. Run ChemSTEP'''<br /> qsub launch_chemstep_as_job.sh [round number]<br /> <br /> For the first iterative round, the round number is [2], and should increase by one for every subsequent round of ChemSTEP. The output will be smi_round_****.smi file. Repeat steps 8-10 for as many rounds as needed. The performance is reported in outputy/complete_info/run_summary.df, which contains the number of beacons selected, the number of molecules docked, the number of hits found, the distance threshold for the selected molecules to dock, and the last added beacon's distance to all previous beacons.<br /> <br /> <br /> ** anecdotally true.</div>

Kholland http://wiki.docking.org/index.php?title=Running_ChemSTEP&diff=16897 2025-09-18T20:14:06Z

<p>Kholland: </p> <hr /> <div>last update: Sept 18 2025 katie. current ver = 0.3.1<br /> <br /> ChemSTEP (Chemical Space Traversal and Exploration Procedure) is an open-source, transparent acceleration algorithm for molecular docking capable of dealing with virtual libraries of several trillion compounds. This wiki page is a guide for BKS lab members to run ChemSTEP on Wynton HPC, using the current version of InifiSee XReal library (1.1T). For more general use directions, please refer to [ChemSTEP Read-the-Docs]. <br /> <br /> At a high-level, ChemSTEP is an iterative process that identifies molecules from the larger virtual library to prioritize for docking. First, we identify a random sample of the total library (termed &quot;seed set&quot;, round zero) and dock those molecules to the target of interest. From this seed set, we can calculate total-library pProp values (-log rank percentages) and the number of &quot;virtual hits&quot; in the total library (high-scoring molecules). ChemSTEP will identify a set of maximally diverse molecules that score above the desired pProp threshold (&quot;beacons&quot;) from the seed set. These beacons guide prioritization, where molecules chosen and output by ChemSTEP are close in chemical space to the beacons. Prioritized molecules are then built, docked, and returned to ChemSTEP for a second round of prioritization. This process is iterated until you reach desired virtual hit recovery, or you are no longer recovering virtual hits. <br /> <br /> <br /> Running the ChemSTEP algorithm on Wynton takes place in loosely three steps: (1) submission of the main ChemSTEP job, (2) ChainingLog sub-job array, and (3) gathering of SMILES for prioritization. <br /> <br /> <br /> What the user need: DOCKFILES, directories for (1) docking (2) building and (3) running ChemSTEP. '''All iterative rounds of ChemSTEP should be run in the SAME directory.''' <br /> <br /> '''1. Source ChemSTEP virtual environment on Wynton'''<br /> source /wynton/group/bks/work/shared/kholland/chemstep_env/bin/activate<br /> <br /> <br /> '''2. Copy InfiniSee seed-set SDI file into your docking directory'''<br /> This directory should already contain your dockfiles, with INDOCK parameters set to your liking. In this step, we are copying in a split database index file (SDI) containing paths to bundles of db2 files. This seed set contains 100 million molecules sampled randomly from the total virtual library, currently 1.1 trillion molecules.<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/XR_seed_set_v0p0.wynton.sdi .<br /> <br /> '''3. Dock seed set to your receptor of interest using DOCK 3.8''' /docking directions taken from docs.docking.org. This is meant to be done as you would do a normal LSD. <br /> export MOLECULES_DIR_TO_BIND=[outermost folder containing the molecules to dock, just your base docking directory]<br /> export DOCKFILES=[path to your dockfiles]<br /> export INPUT_FOLDER=[the folder containing your .sdi file(s)]<br /> export OUTPUT_FOLDER=[where you want the output ]<br /> <br /> /wynton/group/bks/work/bwhall61/needs_github/super_dock3r.sh<br /> <br /> Wait for docking to complete. Next, you must extract all molecule IDs and corresponding DOCK scores from above. To do so, run the following commands in the base docking directory containing your docking files and output folder, while logged into a dev node (I suggest in a screen):<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/get_scores.py .<br /> python get_scores.py 0<br /> <br /> <br /> This script expects a directory named &quot;output*&quot; within the CWD. If your output from docking follows different naming conventions, vim into get_scores.py and change the path. The output will be a file named &quot;scores_round_0.txt&quot;. For iterative rounds, pass increasing numbers into the command line. i.e. When docking the first round of prioritized molecules, pass [1] for scores_round_1.txt. <br /> <br /> '''4. Convert scores and molecule IDS into NumPY arrays for ChemSTEP recognition. Requires ChemSTEP venv'''<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/convert_scores_to_npy.py .<br /> python convert_scores_to_npy.py 0<br /> <br /> For python speed purposes, ChemSTEP requires that DOCK scores and IDs be give in the form of a NumPY array. In this step, we are taking our readable scores file and converting them for ChemSTEP recognition. This script expects a txt file with Molecule IDS and DOCK scores. Use the same round number you used in step 3. As run above, this will output two files named scores_round_0.npy and indices_round_0.npy that contain line-matched indices (determined from Mol ID) and their respective docking scores. <br /> <br /> '''5. Enter into or make a directory to run ChemSTEP in. Copy in necessary files for initiating ChemSTEP: params.txt, run_chemstep.py and launch_chemstep_as_job.sh. Copy in your score and indices numpy files as well, which should be in your docking directory.'''<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/params.txt .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/run_chemstep.py .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/launch_chemstep_as_job.sh .<br /> <br /> <br /> '''6. Edit params.txt file''' This ONLY needs to be edited for the initial round of ChemSTEP. The parameters outlined here will be carried through all rounds of ChemSTEP. <br /> seed_indices_file: /absoulte/path/to/your/indices_round_0.npy<br /> seed_scores_file: /absolute/path/to/your/scores_round_0.npy<br /> hit_pprop: 5<br /> n_docked_per_round: 10000000<br /> max_beacons: 150<br /> max_n_rounds: 250<br /> <br /> Within params.txt, add the absolute paths to the ChemSTEP-readable score and indices NumPY arrays. The rest of the values within the params.txt file are left to the discretion of the user, with some considerations below. <br /> <br /> '''pProp:''' this value will define what is considered a &quot;virtual hit&quot; in your campaign. pProp is defined as the -log(rank %) of a molecule within the total library score-distribution. For example, a pProp of 4 in the 1.1T XReal space is equivalent the top 0.01% of the library, corresponding to the top 110 million molecules. pProp 5 = 0.001% = 11 million virtual hits, etc. From the seed set, ChemSTEP will estimate a DOCK score value in kcal/mol that associated with the lower-limit of your desired pProp zone. Any molecules that score better than the threshold is considered a virtual hit in your campaign. Be mindful of seed set size when choosing desired pProp. Our suggestion is that the size of the seed set should be at least 10^(pProp +2).<br /> <br /> '''n_docked_per_round:''' this number is the desired molecules for prioritization per round. When choosing this value, note that this number of molecules must be built and docked in between each round of ChemSTEP. Prioritizing many molecules will slow building and docking speeds, and coupled with few beacons (below) may lead to decreased diversity. Prioritizing too few molecules may result in slower virtual hit recovery. Round size does not significantly impact the algorithm running time.** <br /> <br /> '''max_beacons:''' this is the number of diverse, well-scoring molecules ChemSTEP will use to guide prioritization per round. All molecules that score better than the assigned pProp threshold are be considered for beacons. By default, ChemSTEP chooses each set of beacons to be as maximally diverse as possible. Choosing too many beacons may result in decreased diversity between beacons, but too few beacons could hinder space exploration. If not enough molecules score above your pProp threshold, ChemSTEP may assign fewer beacons than assigned value. <br /> <br /> ''max_n_rounds:''' if prospectively running ChemSTEP, as outlined in this wiki, no need to worry about this parameter.<br /> <br /> <br /> There should be no need to edit run_chemstep.py or the SGE wrapper script. ''If using another virtual library, be sure to update the path in run_chemstep.py to point to your FP library.'' At this time, we strongly suggest running ChemSTEP as a job array with 64 CPU slots requested, which is specified in the default wrapper. <br /> <br /> <br /> '''7. Run ChemSTEP''' with the following command: <br /> qsub launch_chemstep_as_job.sh <br /> <br /> <br /> This will launch the main ChemSTEP job to the SGE scheduler. Check your job status with the 'qstat' command. The algorithm will read the score and indices files provided, calculate pProp (for round 1) and assign beacons. As this job runs, it will launch a subsequent job array. These sub-jobs are the Chaining step of ChemSTEP, where each parallel worker calculates the Tanimoto distances of 100 million molecules from the virtual library to the beacons. These distances are then written to the files within the /output directory. When these jobs complete, the main job will read through all Tanimoto distances and pull molecules for prioritization. <br /> <br /> Running ChemSTEP successfully will result in the output of a SMILES file within output/complete_info/. You will also get (1) chemstep_algo.log and (2) chemstep_submission.log in the working directory after job submission. chemstep_algo.log contains the running output of the algorithm in a readable format, with timestamps, including the DOCK score associated with your desired pProp. chemstep_submission.log contains the output of ChemSTEP, as well as any information output by the SGE submission system (python errors, cluster issues, tracebacks if failed). These files will be updated with any information with iterative rounds of ChemSTEP run in the same directory. '''Visually inspect these files after each round of ChemSTEP to ensure the algorithm has picked your desired number of beacons and things ran smoothly.'''<br /> <br /> '''Troubleshooting:''' if no jobs are running and there is no SMI file, check the chemstep_submission.log first. Any traceback error or SGE error should give you some idea of why ChemSTEP failed. Some errors may be due to SGE or Wynton issues. If directed, look at a few .out files within /output/jobs/ . If a ChemSTEP run fails on your FIRST run, delete the output and log files, fix what needs to be fixed, and rerun. Errors in subsequent rounds during the chaining step can potentially corrupt chaining files within the /output directory that are needed for prioritization. At the very least, you will definitely have duplicates of beacons and information written to the log files. At this time, if your run fails during iterative rounds, it's best to start from the beginning. <br /> <br /> Prioritized molecules should be built, docked, and their scores fed back into ChemSTEP. More detailed instructions below: <br /> <br /> '''8. Build prioritized molecules (DOCK 3.8).''' /taken from docs.docking.org <br /> source /wynton/group/bks/soft/DOCK-3.8.5/env.sh<br /> <br /> python /wynton/group/bks/soft/DOCK-3.8.5/DOCK3.8/zinc22-3d/submit/submit_building_docker.py --output_folder building_output --bundle_size 1000 --minutes_per_mol 5 --skip_name_check --scheduler sge --container_software apptainer --container_path_or_name /wynton/group/bks/soft/DOCK-3.8.5/building_pipeline.sif smi_round_1.smi<br /> <br /> When building has completed, you must write an SDI file with the complete paths to each built bundle and dock. you can do this with:<br /> find /path/to/your/building_output -name &quot;bundle.db2.tgz&quot; -type f &gt; round_N.wynton.sdi<br /> <br /> '''Be sure to change the INDOCK file to save only poses that meet your score pProp score threshold calculated by ChemSTEP!''' Retrieve docking scores as convert to NumPy arrays as outlined above, update the round number when running get_scores.py and convert_scores_to_npy.py! Copy new score and indices files into the directory you ran ChemSTEP in. If you are following along as a tutorial, you should have scores_round_1.npy and indices_round_1.npy from the previous step (from FIRST round of ChemSTEP prioritization). <br /> <br /> <br /> '''9. Set up for iterative rounds of ChemSTEP'''<br /> <br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/run_chemstep_iterative.py .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/launch_chemstep_iterative.sh .<br /> <br /> <br /> '''10. Run ChemSTEP'''<br /> qsub launch_chemstep_as_job.sh [round number]<br /> <br /> For the first iterative round, the round number is [2], and should increase by one for every subsequent round of ChemSTEP. The output will be smi_round_****.smi file. Repeat steps 8-10 for as many rounds as needed. The performance is reported in outputy/complete_info/run_summary.df, which contains the number of beacons selected, the number of molecules docked, the number of hits found, the distance threshold for the selected molecules to dock, and the last added beacon's distance to all previous beacons.<br /> <br /> <br /> ** anecdotally true.</div>

Kholland http://wiki.docking.org/index.php?title=Running_ChemSTEP&diff=16896 2025-09-18T20:10:17Z

<p>Kholland: </p> <hr /> <div>last update: Sept 11 2025 katie. current ver = 0.3.1<br /> <br /> ChemSTEP (Chemical Space Traversal and Exploration Procedure) is an open-source, transparent acceleration algorithm for molecular docking capable of dealing with virtual libraries of several trillion compounds. This wiki page is a guide for BKS lab members to run ChemSTEP on Wynton HPC, using the current version of InifiSee XReal library (1.1T). For more general use directions, please refer to [ChemSTEP Read-the-Docs]. <br /> <br /> At a high-level, ChemSTEP is an iterative process that identifies molecules from the larger virtual library to prioritize for docking. First, we identify a random sample of the total library (termed &quot;seed set&quot;, round zero) and dock those molecules to the target of interest. From this seed set, we can calculate total-library pProp values (-log rank percentages) and the number of &quot;virtual hits&quot; in the total library (high-scoring molecules). ChemSTEP will identify a set of maximally diverse molecules that score above the desired pProp threshold (&quot;beacons&quot;) from the seed set. These beacons guide prioritization, where molecules chosen and output by ChemSTEP are close in chemical space to the beacons. Prioritized molecules are then built, docked, and returned to ChemSTEP for a second round of prioritization. This process is iterated until you reach desired virtual hit recovery, or you are no longer recovering virtual hits. <br /> <br /> <br /> Running the ChemSTEP algorithm on Wynton takes place in loosely three steps: (1) submission of the main ChemSTEP job, (2) ChainingLog sub-job array, and (3) gathering of SMILES for prioritization. <br /> <br /> <br /> What the user need: DOCKFILES, directories for (1) docking (2) building and (3) running ChemSTEP. '''All iterative rounds of ChemSTEP should be run in the SAME directory.''' <br /> <br /> '''1. Source ChemSTEP virtual environment on Wynton'''<br /> source /wynton/group/bks/work/shared/kholland/chemstep_env/bin/activate<br /> <br /> <br /> '''2. Copy InfiniSee seed-set SDI file into your docking directory'''<br /> This directory should already contain your dockfiles, with INDOCK parameters set to your liking. In this step, we are copying in a split database index file (SDI) containing paths to bundles of db2 files. This seed set contains 100 million molecules sampled randomly from the total virtual library, currently 1.1 trillion molecules.<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/XR_seed_set_v0p0.wynton.sdi .<br /> <br /> '''3. Dock seed set to your receptor of interest using DOCK 3.8''' /docking directions taken from docs.docking.org. This is meant to be done as you would do a normal LSD. <br /> export MOLECULES_DIR_TO_BIND=[outermost folder containing the molecules to dock, just your base docking directory]<br /> export DOCKFILES=[path to your dockfiles]<br /> export INPUT_FOLDER=[the folder containing your .sdi file(s)]<br /> export OUTPUT_FOLDER=[where you want the output ]<br /> <br /> /wynton/group/bks/work/bwhall61/needs_github/super_dock3r.sh<br /> <br /> Wait for docking to complete. Next, you must extract all molecule IDs and corresponding DOCK scores from above. To do so, run the following commands in the base docking directory containing your docking files and output folder, while logged into a dev node (I suggest in a screen):<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/get_scores.py .<br /> python get_scores.py 0<br /> <br /> <br /> This script expects a directory named &quot;output*&quot; within the CWD. If your output from docking follows different naming conventions, vim into get_scores.py and change the path. The output will be a file named &quot;scores_round_0.txt&quot;. For iterative rounds, pass increasing numbers into the command line. i.e. When docking the first round of prioritized molecules, pass [1] for scores_round_1.txt. <br /> <br /> '''4. Convert scores and molecule IDS into NumPY arrays for ChemSTEP recognition. Requires ChemSTEP venv'''<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/convert_scores_to_npy.py .<br /> python convert_scores_to_npy.py 0<br /> <br /> For python speed purposes, ChemSTEP requires that DOCK scores and IDs be give in the form of a NumPY array. In this step, we are taking our readable scores file and converting them so ChemSTEP can read it. This script expects a txt file with Molecule IDS and DOCK scores. Use the same round number you used in step 3. As run above, this will output two files named scores_round_0.npy and indices_round_0.npy that contain line-matched molecule IDs (indices) and their respective docking scores. <br /> <br /> '''5. Enter into or make a directory to run ChemSTEP in. Copy in necessary files for initiating ChemSTEP: params.txt, run_chemstep.py and launch_chemstep_as_job.sh. Copy in your score and indices numpy files as well, which should be in your docking directory.'''<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/params.txt .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/run_chemstep.py .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/launch_chemstep_as_job.sh .<br /> <br /> <br /> '''6. Edit params.txt file''' This ONLY needs to be edited for the initial round of ChemSTEP. The parameters outlined here will be carried through all rounds of ChemSTEP. <br /> seed_indices_file: /absoulte/path/to/your/indices_round_0.npy<br /> seed_scores_file: /absolute/path/to/your/scores_round_0.npy<br /> hit_pprop: 5<br /> n_docked_per_round: 10000000<br /> max_beacons: 150<br /> max_n_rounds: 250<br /> <br /> Within params.txt, add the absolute paths to the ChemSTEP-readable score and indices NumPY arrays. The rest of the values within the params.txt file are left to the discretion of the user, with some considerations below. <br /> <br /> '''pProp:''' this value will define what is considered a &quot;virtual hit&quot; in your campaign. pProp is defined as the -log(rank %) of a molecule within the total library score-distribution. For example, a pProp of 4 in the 1.1T XReal space is equivalent the top 0.01% of the library, corresponding to the top 110 million molecules. pProp 5 = 0.001% = 11 million virtual hits, etc. From the seed set, ChemSTEP will estimate a DOCK score value in kcal/mol that associated with the lower-limit of your desired pProp zone. Any molecules that score better than the threshold is considered a virtual hit in your campaign. Be mindful of seed set size when choosing desired pProp. Our suggestion is that the size of the seed set should be at least 10^(pProp +2).<br /> <br /> '''n_docked_per_round:''' this number is the desired molecules for prioritization per round. When choosing this value, note that this number of molecules must be built and docked in between each round of ChemSTEP. Prioritizing many molecules will slow building and docking speeds, and coupled with few beacons (below) may lead to decreased diversity. Prioritizing too few molecules may result in slower virtual hit recovery. Round size does not significantly impact the algorithm running time.** <br /> <br /> '''max_beacons:''' this is the number of diverse, well-scoring molecules ChemSTEP will use to guide prioritization per round. All molecules that score better than the assigned pProp threshold are be considered for beacons. By default, ChemSTEP chooses each set of beacons to be as maximally diverse as possible. Choosing too many beacons may result in decreased diversity between beacons, but too few beacons could hinder space exploration. If not enough molecules score above your pProp threshold, ChemSTEP may assign fewer beacons than assigned value. <br /> <br /> ''max_n_rounds:''' if prospectively running ChemSTEP, as outlined in this wiki, no need to worry about this parameter.<br /> <br /> <br /> There should be no need to edit run_chemstep.py or the SGE wrapper script. ''If using another virtual library, be sure to update the path in run_chemstep.py to point to your FP library.'' At this time, we strongly suggest running ChemSTEP as a job array with 64 CPU slots requested, which is specified in the default wrapper. <br /> <br /> <br /> '''7. Run ChemSTEP''' with the following command: <br /> qsub launch_chemstep_as_job.sh <br /> <br /> <br /> This will launch the main ChemSTEP job to the SGE scheduler. Check your job status with the 'qstat' command. The algorithm will read the score and indices files provided, calculate pProp (for round 1) and assign beacons. As this job runs, it will launch a subsequent job array. These sub-jobs are the Chaining step of ChemSTEP, where each parallel worker calculates the Tanimoto distances of 100 million molecules from the virtual library to the beacons. These distances are then written to the files within the /output directory. When these jobs complete, the main job will read through all Tanimoto distances and pull molecules for prioritization. <br /> <br /> Running ChemSTEP successfully will result in the output of a SMILES file within output/complete_info/. You will also get (1) chemstep_algo.log and (2) chemstep_submission.log in the working directory after job submission. chemstep_algo.log contains the running output of the algorithm in a readable format, with timestamps, including the DOCK score associated with your desired pProp. chemstep_submission.log contains the output of ChemSTEP, as well as any information output by the SGE submission system (python errors, cluster issues, tracebacks if failed). These files will be updated with any information with iterative rounds of ChemSTEP run in the same directory. '''Visually inspect these files after each round of ChemSTEP to ensure the algorithm has picked your desired number of beacons and things ran smoothly.'''<br /> <br /> '''Troubleshooting:''' if no jobs are running and there is no SMI file, check the chemstep_submission.log first. Any traceback error or SGE error should give you some idea of why ChemSTEP failed. Some errors may be due to SGE or Wynton issues. If directed, look at a few .out files within /output/jobs/ . If a ChemSTEP run fails on your FIRST run, delete the output and log files, fix what needs to be fixed, and rerun. Errors in subsequent rounds during the chaining step can potentially corrupt chaining files within the /output directory that are needed for prioritization. At the very least, you will definitely have duplicates of beacons and information written to the log files. At this time, if your run fails during iterative rounds, it's best to start from the beginning. <br /> <br /> Prioritized molecules should be built, docked, and their scores fed back into ChemSTEP. More detailed instructions below: <br /> <br /> '''8. Build prioritized molecules (DOCK 3.8).''' /taken from docs.docking.org <br /> source /wynton/group/bks/soft/DOCK-3.8.5/env.sh<br /> <br /> python /wynton/group/bks/soft/DOCK-3.8.5/DOCK3.8/zinc22-3d/submit/submit_building_docker.py --output_folder building_output --bundle_size 1000 --minutes_per_mol 5 --skip_name_check --scheduler sge --container_software apptainer --container_path_or_name /wynton/group/bks/soft/DOCK-3.8.5/building_pipeline.sif smi_round_1.smi<br /> <br /> When building has completed, you must write an SDI file with the complete paths to each built bundle and dock. '''Be sure to change the INDOCK file to save only poses that meet your score pProp score threshold calculated by ChemSTEP!''' Retrieve docking scores as convert to NumPy arrays as outlined above, update the round number when running get_scores.py and convert_scores_to_npy.py! Copy new score and indices files into the directory you ran ChemSTEP in. If you are following along as a tutorial, you should have scores_round_1.npy and indices_round_1.npy from the previous step (from FIRST round of ChemSTEP prioritization). <br /> <br /> <br /> '''9. Set up for iterative rounds of ChemSTEP'''<br /> <br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/run_chemstep_iterative.py .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/launch_chemstep_iterative.sh .<br /> <br /> <br /> '''10. Run ChemSTEP'''<br /> qsub launch_chemstep_as_job.sh [round number]<br /> <br /> For the first iterative round, the round number is [2], and should increase by one for every subsequent round of ChemSTEP. The output will be smi_round_****.smi file. Repeat steps 8-10 for as many rounds as needed. The performance is reported in outputy/complete_info/run_summary.df, which contains the number of beacons selected, the number of molecules docked, the number of hits found, the distance threshold for the selected molecules to dock, and the last added beacon's distance to all previous beacons.<br /> <br /> <br /> ** anecdotally true.</div>

Kholland http://wiki.docking.org/index.php?title=Running_ChemSTEP&diff=16885 2025-09-11T22:39:16Z

<p>Kholland: </p> <hr /> <div>last update: Sept 11 2025 katie. current ver = 0.3.1<br /> <br /> ChemSTEP (Chemical Space Traversal and Exploration Procedure) is an open-source, transparent acceleration algorithm for molecular docking capable of dealing with virtual libraries of several trillion compounds. This wiki page is a guide for BKS lab members to run ChemSTEP on Wynton HPC, using the current version of InifiSee XReal library (1.1T). For more general use directions, please refer to [ChemSTEP Read-the-Docs]. <br /> <br /> At a high-level, ChemSTEP is an iterative process that identifies molecules from the larger virtual library to prioritize for docking. First, we identify a random sample of the total library (termed &quot;seed set&quot;, round zero) and dock those molecules to the target of interest. From this seed set, we can calculate total-library pProp values (-log rank percentages) and the number of &quot;virtual hits&quot; in the total library (high-scoring molecules). ChemSTEP will identify a set of maximally diverse molecules that score above the desired pProp threshold (&quot;beacons&quot;) from the seed set. These beacons guide prioritization, where molecules chosen and output by ChemSTEP are close in chemical space to the beacons. Prioritized molecules are then built, docked, and returned to ChemSTEP for a second round of prioritization. This process is iterated until you reach desired virtual hit recovery, or you are no longer recovering virtual hits. <br /> <br /> <br /> Running the ChemSTEP algorithm on Wynton takes place in loosely three steps: (1) submission of the main ChemSTEP job, (2) ChainingLog sub-job array, and (3) gathering of SMILES for prioritization. <br /> <br /> <br /> What the user need: DOCKFILES, directories for (1) docking (2) building and (3) running ChemSTEP. '''All iterative rounds of ChemSTEP should be run in the SAME directory.''' <br /> <br /> '''1. Source ChemSTEP virtual environment on Wynton'''<br /> source /wynton/group/bks/work/shared/kholland/chemstep_env/bin/activate<br /> <br /> <br /> '''2. Copy InfiniSee seed-set SDI file into your docking directory'''<br /> This directory should already contain your dockfiles, with INDOCK parameters set to your liking. In this step, we are copying in a split database index file (SDI) containing paths to bundles of db2 files. This seed set contains 100 million molecules sampled randomly from the total virtual library, currently 1.1 trillion molecules.<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/XR_seed_set_v0p0.wynton.sdi .<br /> <br /> '''3. Dock seed set to your receptor of interest using DOCK 3.8''' /docking directions taken from docs.docking.org. This is meant to be done as you would do a normal LSD. <br /> export MOLECULES_DIR_TO_BIND=[outermost folder containing the molecules to dock, just your base docking directory]<br /> export DOCKFILES=[path to your dockfiles]<br /> export INPUT_FOLDER=[the folder containing your .sdi file(s)]<br /> export OUTPUT_FOLDER=[where you want the output ]<br /> <br /> /wynton/group/bks/work/bwhall61/needs_github/super_dock3r.sh<br /> <br /> Wait for docking to complete. Next, you must extract all molecule IDs and corresponding DOCK scores from above. To do so, run the following commands in the base docking directory containing your docking files and output folder, while logged into a dev node (I suggest in a screen):<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/get_scores.py .<br /> python get_scores.py 0<br /> <br /> <br /> This script expects a directory named &quot;output*&quot; within the CWD. If your output from docking follows different naming conventions, vim into get_scores.py and change the path. The output will be a file named &quot;scores_round_0.txt&quot;. For iterative rounds, pass increasing numbers into the command line. i.e. When docking the first round of prioritized molecules, pass [1] for scores_round_1.txt. <br /> <br /> '''4. Convert scores and molecule IDS into NumPY arrays for ChemSTEP recognition. Requires ChemSTEP venv'''<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/convert_scores_to_npy.py .<br /> python convert_scores_to_npy.py 0<br /> <br /> For python speed purposes, ChemSTEP requires that DOCK scores and IDs be give in the form of a NumPY array. In this step, we are taking our readable scores file and converting them so ChemSTEP can read it. This script expects a txt file with Molecule IDS and DOCK scores. Use the same round number you used in step 3. As run above, this will output two files named scores_round_0.npy and indices_round_0.npy that contain line-matched molecule IDs (indices) and their respective docking scores. <br /> <br /> '''5. Enter into or make a directory to run ChemSTEP in. Copy in necessary files for initiating ChemSTEP: params.txt, run_chemstep.py and launch_chemstep_as_job.sh. Copy in your score and indices numpy files as well, which should be in your docking directory.'''<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/params.txt .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/run_chemstep.py .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/launch_chemstep_as_job.sh .<br /> <br /> <br /> '''6. Edit params.txt file''' This ONLY needs to be edited for the initial round of ChemSTEP. The parameters outlined here will be carried through all rounds of ChemSTEP. <br /> seed_indices_file: /absoulte/path/to/your/indices_round_0.npy<br /> seed_scores_file: /absolute/path/to/your/scores_round_0.npy<br /> hit_pprop: 5<br /> n_docked_per_round: 10000000<br /> max_beacons: 150<br /> max_n_rounds: 250<br /> <br /> Within params.txt, add the absolute paths to the ChemSTEP-readable score and indices NumPY arrays. The rest of the values within the params.txt file are left to the discretion of the user, with some considerations below. <br /> <br /> '''pProp:''' this value will define what is considered a &quot;virtual hit&quot; in your campaign. pProp is defined as the -log(rank %) of a molecule within the total library score-distribution. For example, a pProp of 4 in the 1.1T XReal space is equivalent the top 0.01% of the library, corresponding to the top 110 million molecules. pProp 5 = 0.001% = 11 million virtual hits, etc. From the seed set, ChemSTEP will estimate a DOCK score value in kcal/mol that associated with the lower-limit of your desired pProp zone. Any molecules that score better than the threshold is considered a virtual hit in your campaign.<br /> <br /> '''n_docked_per_round:''' this number is the desired molecules for prioritization per round. When choosing this value, note that this number of molecules must be built and docked in between each round of ChemSTEP. Prioritizing many molecules will slow building and docking speeds, and coupled with few beacons (below) may lead to decreased diversity. Prioritizing too few molecules may result in slower virtual hit recovery. Round size does not significantly impact the algorithm running time.** <br /> <br /> '''max_beacons:''' this is the number of diverse, well-scoring molecules ChemSTEP will use to guide prioritization per round. All molecules that score better than pProp threshold can be considered for beacons. By default, ChemSTEP chooses each set of beacons to be as maximally diverse as possible. Choosing too many beacons may result in decreased diversity between beacons, but too few beacons could hinder space exploration. If not enough molecules score above your pProp threshold, ChemSTEP may assign fewer beacons than assigned value. <br /> <br /> ''max_n_rounds:''' if prospectively running ChemSTEP, as outlined in this wiki, no need to worry about this parameter.<br /> <br /> <br /> There should be no need to edit run_chemstep.py or the SGE wrapper script. ''If using another virtual library, be sure to update the path in run_chemstep.py to point to your FP library.'' At this time, we strongly suggest running ChemSTEP as a job array with 64 CPU slots requested, which is specified in the default wrapper. <br /> <br /> <br /> '''7. Run ChemSTEP''' with the following command: <br /> qsub launch_chemstep_as_job.sh <br /> <br /> <br /> This will launch the main ChemSTEP job to the SGE scheduler. Check your job status with the 'qstat' command. The algorithm will read the score and indices files provided, calculate pProp (for round 1) and assign beacons. As this job runs, it will launch a subsequent job array. These sub-jobs are the Chaining step of ChemSTEP, where each parallel worker calculates the Tanimoto distances of 100 million molecules from the virtual library to the beacons. These distances are then written to the files within the /output directory. When these jobs complete, the main job will read through all Tanimoto distances and pull molecules for prioritization. <br /> <br /> Running ChemSTEP successfully will result in the output of a SMILES file within output/complete_info/. You will also get (1) chemstep_algo.log and (2) chemstep_submission.log in the working directory after job submission. chemstep_algo.log contains the running output of the algorithm in a readable format, with timestamps, including the DOCK score associated with your desired pProp. chemstep_submission.log contains the output of ChemSTEP, as well as any information output by the SGE submission system (python errors, cluster issues, tracebacks if failed). These files will be updated with any information with iterative rounds of ChemSTEP run in the same directory. <br /> <br /> '''Troubleshooting:''' if no jobs are running and there is no SMI file, check the chemstep_submission.log first. Any traceback error or SGE error should give you some idea of why ChemSTEP failed. Some errors may be due to SGE or Wynton issues. If directed, look at a few .out files within /output/jobs/ . If a ChemSTEP run fails on your FIRST run, delete the output and log files, fix what needs to be fixed, and rerun. Errors in subsequent rounds during the chaining step can potentially corrupt chaining files within the /output directory that are needed for prioritization. At the very least, you will definitely have duplicates of beacons and information written to the log files. At this time, if your run fails during iterative rounds, it's best to start from the beginning. <br /> <br /> Prioritized molecules should be built, docked, and their scores fed back into ChemSTEP. More detailed instructions below: <br /> <br /> '''8. Build prioritized molecules (DOCK 3.8).''' /taken from docs.docking.org <br /> source /wynton/group/bks/soft/DOCK-3.8.5/env.sh<br /> <br /> python /wynton/group/bks/soft/DOCK-3.8.5/DOCK3.8/zinc22-3d/submit/submit_building_docker.py --output_folder building_output --bundle_size 1000 --minutes_per_mol 5 --skip_name_check --scheduler sge --container_software apptainer --container_path_or_name /wynton/group/bks/soft/DOCK-3.8.5/building_pipeline.sif smi_round_1.smi<br /> <br /> When building has completed, you must write an SDI file with the complete paths to each built bundle and dock. '''Be sure to change the INDOCK file to save only poses that meet your score pProp score threshold calculated by ChemSTEP!''' Retrieve docking scores as convert to NumPy arrays as outlined above, update the round number when running get_scores.py and convert_scores_to_npy.py! Copy new score and indices files into the directory you ran ChemSTEP in. If you are following along as a tutorial, you should have scores_round_1.npy and indices_round_1.npy from the previous step (from FIRST round of ChemSTEP prioritization). <br /> <br /> <br /> '''9. Set up for iterative rounds of ChemSTEP'''<br /> <br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/run_chemstep_iterative.py .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/launch_chemstep_iterative.sh .<br /> <br /> <br /> '''10. Run ChemSTEP'''<br /> qsub launch_chemstep_as_job.sh [round number]<br /> <br /> For the first iterative round, the round number is [2], and should increase by one for every subsequent round of ChemSTEP. The output will be smi_round_****.smi file. Repeat steps 8-10 for as many rounds as needed. The performance is reported in outputy/complete_info/run_summary.df, which contains the number of beacons selected, the number of molecules docked, the number of hits found, the distance threshold for the selected molecules to dock, and the last added beacon's distance to all previous beacons.<br /> <br /> <br /> ** anecdotally true.</div>

Kholland http://wiki.docking.org/index.php?title=Running_ChemSTEP&diff=16884 2025-09-11T22:37:58Z

<p>Kholland: </p> <hr /> <div>last update: Sept 11 2025 katie. current ver = 0.3.1<br /> <br /> ChemSTEP (Chemical Space Traversal and Exploration Procedure) is an open-source, transparent acceleration algorithm for molecular docking capable of dealing with virtual libraries of several trillion compounds. This wiki page is a guide for BKS lab members to run ChemSTEP on Wynton HPC, using the current version of InifiSee XReal library (1.1T). For more general use directions, please refer to [ChemSTEP Read-the-Docs]. <br /> <br /> At a high-level, ChemSTEP is an iterative process that identifies molecules from the larger virtual library to prioritize for docking. First, we identify a random sample of the total library (termed &quot;seed set&quot;, round zero) and dock those molecules to the target of interest. From this seed set, we can calculate total-library pProp values (-log rank percentages) and the number of &quot;virtual hits&quot; in the total library (high-scoring molecules). ChemSTEP will identify a set of maximally diverse molecules that score above the desired pProp threshold (&quot;beacons&quot;) from the seed set. These beacons guide prioritization, where molecules chosen and output by ChemSTEP are close in chemical space to the beacons. Prioritized molecules are then built, docked, and returned to ChemSTEP for a second round of prioritization. This process is iterated until you reach desired virtual hit recovery, or you are no longer recovering virtual hits. <br /> <br /> <br /> Running the ChemSTEP algorithm on Wynton takes place in loosely three steps: (1) submission of the main ChemSTEP job, (2) ChainingLog sub-job array, and (3) gathering of SMILES for prioritization. <br /> <br /> <br /> What the user need: DOCKFILES, directories for (1) docking (2) building and (3) running ChemSTEP. '''All iterative rounds of ChemSTEP should be run in the SAME directory.''' <br /> <br /> '''1. Source ChemSTEP virtual environment on Wynton'''<br /> source /wynton/group/bks/work/shared/kholland/chemstep_env/bin/activate<br /> <br /> <br /> '''2. Copy InfiniSee seed-set SDI file into your docking directory'''<br /> This directory should already contain your dockfiles, with INDOCK parameters set to your liking. In this step, we are copying in a split database index file (SDI) containing paths to bundles of db2 files. This seed set contains 100 million molecules sampled randomly from the total virtual library, currently 1.1 trillion molecules.<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/XR_seed_set_v0p0.wynton.sdi .<br /> <br /> '''3. Dock seed set to your receptor of interest using DOCK 3.8''' /docking directions taken from docs.docking.org. This is meant to be done as you would do a normal LSD. <br /> export MOLECULES_DIR_TO_BIND=[outermost folder containing the molecules to dock, just your base docking directory]<br /> export DOCKFILES=[path to your dockfiles]<br /> export INPUT_FOLDER=[the folder containing your .sdi file(s)]<br /> export OUTPUT_FOLDER=[where you want the output ]<br /> <br /> /wynton/group/bks/work/bwhall61/needs_github/super_dock3r.sh<br /> <br /> Wait for docking to complete. Next, you must extract all molecule IDs and corresponding DOCK scores from above. To do so, run the following commands in the base docking directory containing your docking files and output folder, while logged into a dev node (I suggest in a screen):<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/get_scores.py .<br /> python get_scores.py 0<br /> <br /> <br /> This script expects a directory named &quot;output*&quot; within the CWD. If your output from docking follows different naming conventions, vim into get_scores.py and change the path. The output will be a file named &quot;scores_round_0.txt&quot;. For iterative rounds, pass increasing numbers into the command line. i.e. When docking the first round of prioritized molecules, pass [1] for scores_round_1.txt. <br /> <br /> '''4. Convert scores and molecule IDS into NumPY arrays for ChemSTEP recognition. Requires ChemSTEP venv'''<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/convert_scores_to_npy.py .<br /> python convert_scores_to_npy.py 0<br /> <br /> For python speed purposes, ChemSTEP requires that DOCK scores and IDs be give in the form of a NumPY array. In this step, we are taking our readable scores file and converting them so ChemSTEP can read it. This script expects a txt file with Molecule IDS and DOCK scores. Use the same round number you used in step 3. As run above, this will output two files named scores_round_0.npy and indices_round_0.npy that contain line-matched molecule IDs (indices) and their respective docking scores. <br /> <br /> '''5. Enter into or make a directory to run ChemSTEP in. Copy in necessary files for initiating ChemSTEP: params.txt, run_chemstep.py and launch_chemstep_as_job.sh. Copy in your score and indices numpy files as well, which should be in your docking directory.'''<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/params.txt .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/run_chemstep.py .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/launch_chemstep_as_job.sh .<br /> <br /> <br /> '''6. Edit params.txt file''' This ONLY needs to be edited for the initial round of ChemSTEP. The parameters outlined here will be carried through all rounds of ChemSTEP. <br /> seed_indices_file: /absoulte/path/to/your/indices_round_0.npy<br /> seed_scores_file: /absolute/path/to/your/scores_round_0.npy<br /> hit_pprop: 5<br /> n_docked_per_round: 10000000<br /> max_beacons: 150<br /> max_n_rounds: 250<br /> <br /> Within params.txt, add the absolute paths to the ChemSTEP-readable score and indices NumPY arrays. The rest of the values within the params.txt file are left to the discretion of the user, with some considerations below. <br /> <br /> '''pProp:''' this value will define what is considered a &quot;virtual hit&quot; in your campaign. pProp is defined as the -log(rank %) of a molecule within the total library score-distribution. For example, a pProp of 4 in the 1.1T XReal space is equivalent the top 0.01% of the library, corresponding to the top 110 million molecules. pProp 5 = 0.001% = 11 million virtual hits, etc. From the seed set, ChemSTEP will estimate a DOCK score value in kcal/mol that associated with the lower-limit of your desired pProp zone. Any molecules that score better than the threshold is considered a virtual hit in your campaign.<br /> <br /> '''n_docked_per_round:''' this number is the desired molecules for prioritization per round. When choosing this value, note that this number of molecules must be built and docked in between each round of ChemSTEP. Prioritizing many molecules will slow building and docking speeds, and coupled with few beacons (below) may lead to decreased diversity. Prioritizing too few molecules may result in slower virtual hit recovery. Round size does not significantly impact the algorithm running time.** <br /> <br /> '''max_beacons:''' this is the number of diverse, well-scoring molecules ChemSTEP will use to guide prioritization per round. All molecules that score better than pProp threshold can be considered for beacons. By default, ChemSTEP chooses each set of beacons to be as maximally diverse as possible. Choosing too many beacons may result in decreased diversity between beacons, but too few beacons could hinder space exploration. If not enough molecules score above your pProp threshold, ChemSTEP may assign fewer beacons than assigned value. <br /> <br /> <br /> There should be no need to edit run_chemstep.py or the SGE wrapper script. ''If using another virtual library, be sure to update the path in run_chemstep.py to point to your FP library.'' At this time, we strongly suggest running ChemSTEP as a job array with 64 CPU slots requested, which is specified in the default wrapper. <br /> <br /> <br /> '''7. Run ChemSTEP''' with the following command: <br /> qsub launch_chemstep_as_job.sh <br /> <br /> <br /> This will launch the main ChemSTEP job to the SGE scheduler. Check your job status with the 'qstat' command. The algorithm will read the score and indices files provided, calculate pProp (for round 1) and assign beacons. As this job runs, it will launch a subsequent job array. These sub-jobs are the Chaining step of ChemSTEP, where each parallel worker calculates the Tanimoto distances of 100 million molecules from the virtual library to the beacons. These distances are then written to the files within the /output directory. When these jobs complete, the main job will read through all Tanimoto distances and pull molecules for prioritization. <br /> <br /> Running ChemSTEP successfully will result in the output of a SMILES file within output/complete_info/. You will also get (1) chemstep_algo.log and (2) chemstep_submission.log in the working directory after job submission. chemstep_algo.log contains the running output of the algorithm in a readable format, with timestamps, including the DOCK score associated with your desired pProp. chemstep_submission.log contains the output of ChemSTEP, as well as any information output by the SGE submission system (python errors, cluster issues, tracebacks if failed). These files will be updated with any information with iterative rounds of ChemSTEP run in the same directory. <br /> <br /> '''Troubleshooting:''' if no jobs are running and there is no SMI file, check the chemstep_submission.log first. Any traceback error or SGE error should give you some idea of why ChemSTEP failed. Some errors may be due to SGE or Wynton issues. If directed, look at a few .out files within /output/jobs/ . If a ChemSTEP run fails on your FIRST run, delete the output and log files, fix what needs to be fixed, and rerun. Errors in subsequent rounds during the chaining step can potentially corrupt chaining files within the /output directory that are needed for prioritization. At the very least, you will definitely have duplicates of beacons and information written to the log files. At this time, if your run fails during iterative rounds, it's best to start from the beginning. <br /> <br /> Prioritized molecules should be built, docked, and their scores fed back into ChemSTEP. More detailed instructions below: <br /> <br /> '''8. Build prioritized molecules (DOCK 3.8).''' /taken from docs.docking.org <br /> source /wynton/group/bks/soft/DOCK-3.8.5/env.sh<br /> <br /> python /wynton/group/bks/soft/DOCK-3.8.5/DOCK3.8/zinc22-3d/submit/submit_building_docker.py --output_folder building_output --bundle_size 1000 --minutes_per_mol 5 --skip_name_check --scheduler sge --container_software apptainer --container_path_or_name /wynton/group/bks/soft/DOCK-3.8.5/building_pipeline.sif smi_round_1.smi<br /> <br /> When building has completed, you must write an SDI file with the complete paths to each built bundle and dock. '''Be sure to change the INDOCK file to save only poses that meet your score pProp score threshold calculated by ChemSTEP!''' Retrieve docking scores as convert to NumPy arrays as outlined above, update the round number when running get_scores.py and convert_scores_to_npy.py! Copy new score and indices files into the directory you ran ChemSTEP in. If you are following along as a tutorial, you should have scores_round_1.npy and indices_round_1.npy from the previous step (from FIRST round of ChemSTEP prioritization). <br /> <br /> <br /> '''9. Set up for iterative rounds of ChemSTEP'''<br /> <br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/run_chemstep_iterative.py .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/launch_chemstep_iterative.sh .<br /> <br /> <br /> '''10. Run ChemSTEP'''<br /> qsub launch_chemstep_as_job.sh [round number]<br /> <br /> For the first iterative round, the round number is [2], and should increase by one for every subsequent round of ChemSTEP. The output will be smi_round_****.smi file. Repeat steps 8-10 for as many rounds as needed. The performance is reported in outputy/complete_info/run_summary.df, which contains the number of beacons selected, the number of molecules docked, the number of hits found, the distance threshold for the selected molecules to dock, and the last added beacon's distance to all previous beacons.<br /> <br /> <br /> ** anecdotally true.</div>

Kholland http://wiki.docking.org/index.php?title=Running_ChemSTEP&diff=16883 2025-09-11T22:36:16Z

<p>Kholland: </p> <hr /> <div>last update: Sept 11 2025 katie. current ver = 0.3.1<br /> <br /> ChemSTEP (Chemical Space Traversal and Exploration Procedure) is an open-source, transparent acceleration algorithm for molecular docking capable of dealing with virtual libraries of several trillion compounds. This wiki page is a guide for BKS lab members to run ChemSTEP on Wynton HPC, using the current version of InifiSee XReal library (1.1T). For more general use directions, please refer to [ChemSTEP Read-the-Docs]. <br /> <br /> At a high-level, ChemSTEP is an iterative process that identifies molecules from the larger virtual library to prioritize for docking. First, we identify a random sample of the total library (termed &quot;seed set&quot;) and dock those molecules to the target of interest. From this seed set, we can calculate total-library pProp values (-log rank percentages) and the number of &quot;virtual hits&quot; in the total library (high-scoring molecules). ChemSTEP will identify a set of maximally diverse molecules that score above the desired pProp threshold (&quot;beacons&quot;) from the seed set. These beacons guide prioritization, where molecules chosen and output by ChemSTEP are close in chemical space to the beacons. Prioritized molecules are then built, docked, and returned to ChemSTEP for another round of prioritization. This process is iterated until you reach desired virtual hit recovery, or you are no longer recovering virtual hits. <br /> <br /> <br /> Running the ChemSTEP algorithm on Wynton takes place in loosely three steps: (1) submission of the main ChemSTEP job, (2) ChainingLog sub-job array, and (3) gathering of SMILES for prioritization. <br /> <br /> <br /> What the user need: DOCKFILES, directories for (1) docking (2) building and (3) running ChemSTEP. '''All iterative rounds of ChemSTEP should be run in the SAME directory.''' <br /> <br /> '''1. Source ChemSTEP virtual environment on Wynton'''<br /> source /wynton/group/bks/work/shared/kholland/chemstep_env/bin/activate<br /> <br /> <br /> '''2. Copy InfiniSee seed-set SDI file into your docking directory'''<br /> This directory should already contain your dockfiles, with INDOCK parameters set to your liking. In this step, we are copying in a split database index file (SDI) containing paths to bundles of db2 files. This seed set contains 100 million molecules sampled randomly from the total virtual library, currently 1.1 trillion molecules.<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/XR_seed_set_v0p0.wynton.sdi .<br /> <br /> '''3. Dock seed set to your receptor of interest using DOCK 3.8''' /docking directions taken from docs.docking.org. This is meant to be done as you would do a normal LSD. <br /> export MOLECULES_DIR_TO_BIND=[outermost folder containing the molecules to dock, just your base docking directory]<br /> export DOCKFILES=[path to your dockfiles]<br /> export INPUT_FOLDER=[the folder containing your .sdi file(s)]<br /> export OUTPUT_FOLDER=[where you want the output ]<br /> <br /> /wynton/group/bks/work/bwhall61/needs_github/super_dock3r.sh<br /> <br /> Wait for docking to complete. Next, you must extract all molecule IDs and corresponding DOCK scores from above. To do so, run the following commands in the base docking directory containing your docking files and output folder, while logged into a dev node (I suggest in a screen):<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/get_scores.py .<br /> python get_scores.py 0<br /> <br /> <br /> This script expects a directory named &quot;output*&quot; within the CWD. If your output from docking follows different naming conventions, vim into get_scores.py and change the path. The output will be a file named &quot;scores_round_0.txt&quot;. For iterative rounds, pass increasing numbers into the command line. i.e. When docking the first round of prioritized molecules, pass [1] for scores_round_1.txt. <br /> <br /> '''4. Convert scores and molecule IDS into NumPY arrays for ChemSTEP recognition. Requires ChemSTEP venv'''<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/convert_scores_to_npy.py .<br /> python convert_scores_to_npy.py 0<br /> <br /> For python speed purposes, ChemSTEP requires that DOCK scores and IDs be give in the form of a NumPY array. In this step, we are taking our readable scores file and converting them so ChemSTEP can read it. This script expects a txt file with Molecule IDS and DOCK scores. Use the same round number you used in step 3. As run above, this will output two files named scores_round_0.npy and indices_round_0.npy that contain line-matched molecule IDs (indices) and their respective docking scores. <br /> <br /> '''5. Enter into or make a directory to run ChemSTEP in. Copy in necessary files for initiating ChemSTEP: params.txt, run_chemstep.py and launch_chemstep_as_job.sh. Copy in your score and indices numpy files as well, which should be in your docking directory.'''<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/params.txt .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/run_chemstep.py .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/launch_chemstep_as_job.sh .<br /> <br /> <br /> '''6. Edit params.txt file''' This ONLY needs to be edited for the initial round of ChemSTEP. The parameters outlined here will be carried through all rounds of ChemSTEP. <br /> seed_indices_file: /absoulte/path/to/your/indices_round_0.npy<br /> seed_scores_file: /absolute/path/to/your/scores_round_0.npy<br /> hit_pprop: 5<br /> n_docked_per_round: 10000000<br /> max_beacons: 150<br /> max_n_rounds: 250<br /> <br /> Within params.txt, add the absolute paths to the ChemSTEP-readable score and indices NumPY arrays. The rest of the values within the params.txt file are left to the discretion of the user, with some considerations below. <br /> <br /> '''pProp:''' this value will define what is considered a &quot;virtual hit&quot; in your campaign. pProp is defined as the -log(rank %) of a molecule within the total library score-distribution. For example, a pProp of 4 in the 1.1T XReal space is equivalent the top 0.01% of the library, corresponding to the top 110 million molecules. pProp 5 = 0.001% = 11 million virtual hits, etc. From the seed set, ChemSTEP will estimate a DOCK score value in kcal/mol that associated with the lower-limit of your desired pProp zone. Any molecules that score better than the threshold is considered a virtual hit in your campaign.<br /> <br /> '''n_docked_per_round:''' this number is the desired molecules for prioritization per round. When choosing this value, note that this number of molecules must be built and docked in between each round of ChemSTEP. Prioritizing many molecules will slow building and docking speeds, and coupled with few beacons (below) may lead to decreased diversity. Prioritizing too few molecules may result in slower virtual hit recovery. Round size does not significantly impact the algorithm running time.** <br /> <br /> '''max_beacons:''' this is the number of diverse, well-scoring molecules ChemSTEP will use to guide prioritization per round. All molecules that score better than pProp threshold can be considered for beacons. By default, ChemSTEP chooses each set of beacons to be as maximally diverse as possible. Choosing too many beacons may result in decreased diversity between beacons, but too few beacons could hinder space exploration. If not enough molecules score above your pProp threshold, ChemSTEP may assign fewer beacons than assigned value. <br /> <br /> <br /> There should be no need to edit run_chemstep.py or the SGE wrapper script. ''If using another virtual library, be sure to update the path in run_chemstep.py to point to your FP library.'' At this time, we strongly suggest running ChemSTEP as a job array with 64 CPU slots requested, which is specified in the default wrapper. <br /> <br /> <br /> '''7. Run ChemSTEP''' with the following command: <br /> qsub launch_chemstep_as_job.sh <br /> <br /> <br /> This will launch the main ChemSTEP job to the SGE scheduler. Check your job status with the 'qstat' command. The algorithm will read the score and indices files provided, calculate pProp (for round 1) and assign beacons. As this job runs, it will launch a subsequent job array. These sub-jobs are the Chaining step of ChemSTEP, where each parallel worker calculates the Tanimoto distances of 100 million molecules from the virtual library to the beacons. These distances are then written to the files within the /output directory. When these jobs complete, the main job will read through all Tanimoto distances and pull molecules for prioritization. <br /> <br /> Running ChemSTEP successfully will result in the output of a SMILES file within output/complete_info/. You will also get (1) chemstep_algo.log and (2) chemstep_submission.log in the working directory after job submission. chemstep_algo.log contains the running output of the algorithm in a readable format, with timestamps, including the DOCK score associated with your desired pProp. chemstep_submission.log contains the output of ChemSTEP, as well as any information output by the SGE submission system (python errors, cluster issues, tracebacks if failed). These files will be updated with any information with iterative rounds of ChemSTEP run in the same directory. <br /> <br /> '''Troubleshooting:''' if no jobs are running and there is no SMI file, check the chemstep_submission.log first. Any traceback error or SGE error should give you some idea of why ChemSTEP failed. Some errors may be due to SGE or Wynton issues. If directed, look at a few .out files within /output/jobs/ . If a ChemSTEP run fails on your FIRST run, delete the output and log files, fix what needs to be fixed, and rerun. Errors in subsequent rounds during the chaining step can potentially corrupt chaining files within the /output directory that are needed for prioritization. At the very least, you will definitely have duplicates of beacons and information written to the log files. At this time, if your run fails during iterative rounds, it's best to start from the beginning. <br /> <br /> Prioritized molecules should be built, docked, and their scores fed back into ChemSTEP. More detailed instructions below: <br /> <br /> '''8. Build prioritized molecules (DOCK 3.8).''' /taken from docs.docking.org <br /> source /wynton/group/bks/soft/DOCK-3.8.5/env.sh<br /> <br /> python /wynton/group/bks/soft/DOCK-3.8.5/DOCK3.8/zinc22-3d/submit/submit_building_docker.py --output_folder building_output --bundle_size 1000 --minutes_per_mol 5 --skip_name_check --scheduler sge --container_software apptainer --container_path_or_name /wynton/group/bks/soft/DOCK-3.8.5/building_pipeline.sif smi_round_1.smi<br /> <br /> When building has completed, you must write an SDI file with the complete paths to each built bundle and dock. '''Be sure to change the INDOCK file to save only poses that meet your score pProp score threshold calculated by ChemSTEP!''' Retrieve docking scores as convert to NumPy arrays as outlined above, update the round number when running get_scores.py and convert_scores_to_npy.py! Copy new score and indices files into the directory you ran ChemSTEP in. If you are following along as a tutorial, you should have scores_round_1.npy and indices_round_1.npy from the previous step (from FIRST round of ChemSTEP prioritization). <br /> <br /> <br /> '''9. Set up for iterative rounds of ChemSTEP'''<br /> <br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/run_chemstep_iterative.py .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/launch_chemstep_iterative.sh .<br /> <br /> <br /> '''10. Run ChemSTEP'''<br /> qsub launch_chemstep_as_job.sh [round number]<br /> <br /> For the first iterative round, the round number is [2], and should increase by one for every subsequent round of ChemSTEP. The output will be smi_round_****.smi file. Repeat steps 8-10 for as many rounds as needed. The performance is reported in outputy/complete_info/run_summary.df, which contains the number of beacons selected, the number of molecules docked, the number of hits found, the distance threshold for the selected molecules to dock, and the last added beacon's distance to all previous beacons.<br /> <br /> <br /> ** anecdotally true.</div>

Kholland http://wiki.docking.org/index.php?title=Running_ChemSTEP&diff=16882 2025-09-11T22:33:04Z

<p>Kholland: </p> <hr /> <div>last update: Sept 11 2025 katie. current ver = 0.3.1<br /> <br /> ChemSTEP (Chemical Space Traversal and Exploration Procedure) is an open-source, transparent acceleration algorithm for molecular docking capable of dealing with virtual libraries of several trillion compounds. This wiki page is a guide for BKS lab members to run ChemSTEP on Wynton HPC, using the current version of InifiSee XReal library (1.1T). For more general use directions, please refer to [ChemSTEP Read-the-Docs]. <br /> <br /> At a high-level, ChemSTEP is an iterative process that identifies molecules from the larger virtual library to prioritize for docking. First, we identify a random sample of the total library (termed &quot;seed set&quot;) and dock those molecules to the target of interest. From this seed set, we can calculate total-library pProp values (-log rank percentages) and the number of &quot;virtual hits&quot; in the total library (high-scoring molecules). ChemSTEP will identify a set of maximally diverse molecules that score above the desired pProp threshold (&quot;beacons&quot;) from the seed set. These beacons guide prioritization, where molecules chosen and output by ChemSTEP are close in chemical space to the beacons. Prioritized molecules are then built, docked, and returned to ChemSTEP for another round of prioritization. This process is iterated until you reach desired virtual hit recovery, or you are no longer recovering virtual hits. <br /> <br /> <br /> Running the ChemSTEP algorithm on Wynton takes place in loosely three steps: (1) submission of the main ChemSTEP job, (2) ChainingLog sub-job array, and (3) gathering of SMILES for prioritization. <br /> <br /> <br /> What the user need: DOCKFILES, directories for (1) docking (2) building and (3) running ChemSTEP. '''All iterative rounds of ChemSTEP should be run in the SAME directory.''' <br /> <br /> '''1. Source ChemSTEP virtual environment on Wynton'''<br /> source /wynton/group/bks/work/shared/kholland/chemstep_env/bin/activate<br /> <br /> <br /> '''2. Copy InfiniSee seed-set SDI file into your docking directory'''<br /> This directory should already contain your dockfiles, with INDOCK parameters set to your liking. In this step, we are copying in a split database index file (SDI) containing paths to bundles of db2 files. This seed set contains 100 million molecules sampled randomly from the total virtual library, currently 1.1 trillion molecules.<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/XR_seed_set_v0p0.wynton.sdi .<br /> <br /> '''3. Dock seed set to your receptor of interest using DOCK 3.8''' /docking directions taken from docs.docking.org. This is meant to be done as you would do a normal LSD. <br /> export MOLECULES_DIR_TO_BIND=[outermost folder containing the molecules to dock, just your base docking directory]<br /> export DOCKFILES=[path to your dockfiles]<br /> export INPUT_FOLDER=[the folder containing your .sdi file(s)]<br /> export OUTPUT_FOLDER=[where you want the output ]<br /> <br /> /wynton/group/bks/work/bwhall61/needs_github/super_dock3r.sh<br /> <br /> Wait for docking to complete. Next, you must extract all molecule IDs and corresponding DOCK scores from above. To do so, run the following commands in the base docking directory containing your docking files and output folder, while logged into a dev node (I suggest in a screen):<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/get_scores.py .<br /> python get_scores.py 0<br /> <br /> <br /> This script expects a directory named &quot;output*&quot; within the CWD. If your output from docking follows different naming conventions, vim into get_scores.py and change the path. The output will be a file named &quot;scores_round_0.txt&quot;. For iterative rounds, pass increasing numbers into the command line. i.e. When docking the first round of prioritized molecules, pass [1] for scores_round_1.txt. <br /> <br /> '''4. Convert scores and molecule IDS into NumPY arrays for ChemSTEP recognition. Requires ChemSTEP venv'''<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/convert_scores_to_npy.py .<br /> python convert_scores_to_npy.py 0<br /> <br /> For python speed purposes, ChemSTEP requires that DOCK scores and IDs be give in the form of a NumPY array. In this step, we are taking our readable scores file and converting them so ChemSTEP can read it. This script expects a txt file with Molecule IDS and DOCK scores. Use the same round number you used in step 3. As run above, this will output two files named scores_round_0.npy and indices_round_0.npy that contain line-matched molecule IDs (indices) and their respective docking scores. <br /> <br /> '''5. Enter into or make a directory to run ChemSTEP in. Copy in necessary files for initiating ChemSTEP: params.txt, run_chemstep.py and launch_chemstep_as_job.sh. Copy in your score and indices numpy files as well, which should be in your docking directory.'''<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/params.txt .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/run_chemstep.py .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/launch_chemstep_as_job.sh .<br /> <br /> <br /> '''6. Edit params.txt file''' This ONLY needs to be edited for the initial round of ChemSTEP. The parameters outlined here will be carried through all rounds of ChemSTEP. <br /> seed_indices_file: /absoulte/path/to/your/indices_round_0.npy<br /> seed_scores_file: /absolute/path/to/your/scores_round_0.npy<br /> hit_pprop: 5<br /> n_docked_per_round: 10000000<br /> max_beacons: 150<br /> max_n_rounds: 250<br /> <br /> Within params.txt, add the absolute paths to the ChemSTEP-readable score and indices NumPY arrays. The rest of the values within the params.txt file are left to the discretion of the user, with some considerations below. <br /> <br /> '''pProp:''' this value will define what is considered a &quot;virtual hit&quot; in your campaign. pProp is defined as the -log(rank %) of a molecule within the total library score-distribution. For example, a pProp of 4 in the 1.1T XReal space is equivalent the top 0.01% of the library, corresponding to the top 110 million molecules. pProp 5 = 0.001% = 11 million virtual hits, etc. From the seed set, ChemSTEP will estimate a DOCK score value in kcal/mol that associated with the lower-limit of your desired pProp zone. Any molecules that score better than the threshold is considered a virtual hit in your campaign.<br /> <br /> '''n_docked_per_round:''' this number is the desired molecules for prioritization per round. When choosing this value, note that this number of molecules must be built and docked in between each round of ChemSTEP. Prioritizing many molecules will slow building and docking speeds, and coupled with few beacons (below) may lead to decreased diversity. Prioritizing too few molecules may result in slower virtual hit recovery. Round size does not significantly impact the algorithm running time.** <br /> <br /> '''max_beacons:''' this is the number of diverse, well-scoring molecules ChemSTEP will use to guide prioritization per round. All molecules that score better than pProp threshold can be considered for beacons. By default, ChemSTEP chooses each set of beacons to be as maximally diverse as possible. Choosing too many beacons may result in decreased diversity between beacons, but too few beacons could hinder space exploration. If not enough molecules score above your pProp threshold, ChemSTEP may assign fewer beacons than assigned value. <br /> <br /> <br /> There should be no need to edit run_chemstep.py or the SGE wrapper script. ''If using another virtual library, be sure to update the path in run_chemstep.py to point to your FP library.'' At this time, we strongly suggest running ChemSTEP as a job array with 64 CPU slots requested, which is specified in the default wrapper. <br /> <br /> <br /> '''7. Run ChemSTEP''' with the following command: <br /> qsub launch_chemstep_as_job.sh <br /> <br /> <br /> This will launch the main ChemSTEP job to the SGE scheduler. Check your job status with the 'qstat' command. The algorithm will read the score and indices files provided, calculate pProp (for round 1) and assign beacons. As this job runs, it will launch a subsequent job array. These sub-jobs are the Chaining step of ChemSTEP, where each parallel worker calculates the Tanimoto distances of 100 million molecules from the virtual library to the beacons. These distances are then written to the files within the /output directory. When these jobs complete, the main job will read through all Tanimoto distances and pull molecules for prioritization. <br /> <br /> Running ChemSTEP successfully will result in the output of a SMILES file within output/complete_info/. You will also get (1) chemstep_algo.log and (2) chemstep_submission.log in the working directory after job submission. chemstep_algo.log contains the running output of the algorithm in a readable format, with timestamps, including the DOCK score associated with your desired pProp. chemstep_submission.log contains the output of ChemSTEP, as well as any information output by the SGE submission system (python errors, cluster issues, tracebacks if failed). These files will be updated with any information with iterative rounds of ChemSTEP run in the same directory. <br /> <br /> '''Troubleshooting:''' if no jobs are running and there is no SMI file, check the chemstep_submission.log first. Any traceback error or SGE error should give you some idea of why ChemSTEP failed. Some errors may be due to SGE or Wynton issues. If directed, look at a few .out files within /output/jobs/ . If a ChemSTEP run fails on your FIRST run, delete the output and log files, fix what needs to be fixed, and rerun. Errors in subsequent rounds during the chaining step can potentially corrupt chaining files within the /output directory that are needed for prioritization. At the very least, you will definitely have duplicates of beacons and information written to the log files. At this time, if your run fails during iterative rounds, it's best to start from the beginning. <br /> <br /> Prioritized molecules should be built, docked, and their scores fed back into ChemSTEP. More detailed instructions below: <br /> <br /> '''8. Build prioritized molecules (DOCK 3.8).''' /taken from docs.docking.org <br /> source /wynton/group/bks/soft/DOCK-3.8.5/env.sh<br /> <br /> python /wynton/group/bks/soft/DOCK-3.8.5/DOCK3.8/zinc22-3d/submit/submit_building_docker.py --output_folder building_output --bundle_size 1000 --minutes_per_mol 5 --skip_name_check --scheduler sge --container_software apptainer --container_path_or_name /wynton/group/bks/soft/DOCK-3.8.5/building_pipeline.sif smi_round_1.smi<br /> <br /> When building has completed, you must write an SDI file with the complete paths to each built bundle and dock. '''Be sure to change the INDOCK file to save only poses that meet your score pProp score threshold (output by ChemSTEP)!''' Retrieve docking scores as convert to NumPy arrays as outlined above, update the round number when running get_scores.py and convert_scores_to_npy.py! Copy new score and indices files into the directory you ran ChemSTEP in. If you are following along as a tutorial, you should have scores_round_1.npy and indices_round_1.npy from the previous step (from FIRST round of ChemSTEP prioritization). <br /> <br /> <br /> '''9. Set up for iterative rounds of ChemSTEP'''<br /> <br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/run_chemstep_iterative.py .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/launch_chemstep_iterative.sh .<br /> <br /> <br /> '''10. Run ChemSTEP'''<br /> qsub launch_chemstep_as_job.sh [round number]<br /> <br /> For the first iterative round, the round number is [2], and should increase by one for every subsequent round of ChemSTEP. The output will be smi_round_****.smi file. Repeat steps 8-10 for as many rounds as needed. The performance is reported in outputy/complete_info/run_summary.df, which contains the number of beacons selected, the number of molecules docked, the number of hits found, the distance threshold for the selected molecules to dock, and the last added beacon's distance to all previous beacons.<br /> <br /> <br /> ** anecdotally true.</div>

Kholland http://wiki.docking.org/index.php?title=Running_ChemSTEP&diff=16881 2025-09-11T22:29:52Z

<p>Kholland: </p> <hr /> <div>last update: Sept 11 2025 katie. current ver = 0.3.1<br /> <br /> ChemSTEP (Chemical Space Traversal and Exploration Procedure) is an open-source, transparent acceleration algorithm for molecular docking capable of dealing with virtual libraries of several trillion compounds. This wiki page is a guide for BKS lab members to run ChemSTEP on Wynton HPC, using the current version of InifiSee XReal library (1.1T). For more general use directions, please refer to [ChemSTEP Read-the-Docs]. <br /> <br /> At a high-level, ChemSTEP is an iterative process that identifies molecules from the larger virtual library to prioritize for docking. First, we identify a random sample of the total library (termed &quot;seed set&quot;) and dock those molecules to the target of interest. From this seed set, we can calculate total-library pProp values (-log rank percentages) and the number of &quot;virtual hits&quot; in the total library (high-scoring molecules). ChemSTEP will identify a set of maximally diverse molecules that score above the desired pProp threshold (&quot;beacons&quot;) from the seed set. These beacons guide prioritization, where molecules chosen and output by ChemSTEP are close in chemical space to the beacons. Prioritized molecules are then built, docked, and returned to ChemSTEP for another round of prioritization. This process is iterated until you reach desired virtual hit recovery, or you are no longer recovering virtual hits. <br /> <br /> <br /> Running the ChemSTEP algorithm on Wynton takes place in loosely three steps: (1) submission of the main ChemSTEP job, (2) ChainingLog sub-job array, and (3) gathering of SMILES for prioritization. <br /> <br /> <br /> What the user need: DOCKFILES, directories for (1) docking (2) building and (3) running ChemSTEP. '''All iterative rounds of ChemSTEP should be run in the SAME directory.''' <br /> <br /> '''1. Source ChemSTEP virtual environment on Wynton'''<br /> source /wynton/group/bks/work/shared/kholland/chemstep_env/bin/activate<br /> <br /> <br /> '''2. Copy InfiniSee seed-set SDI file into your docking directory'''<br /> This directory should already contain your dockfiles, with INDOCK parameters set to your liking. In this step, we are copying in a split database index file (SDI) containing paths to bundles of db2 files. This seed set contains 100 million molecules sampled randomly from the total virtual library, currently 1.1 trillion molecules.<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/XR_seed_set_v0p0.wynton.sdi .<br /> <br /> '''3. Dock seed set to your receptor of interest using DOCK 3.8''' /docking directions taken from docs.docking.org. This is meant to be done as you would do a normal LSD. <br /> export MOLECULES_DIR_TO_BIND=[outermost folder containing the molecules to dock, just your base docking directory]<br /> export DOCKFILES=[path to your dockfiles]<br /> export INPUT_FOLDER=[the folder containing your .sdi file(s)]<br /> export OUTPUT_FOLDER=[where you want the output ]<br /> <br /> /wynton/group/bks/work/bwhall61/needs_github/super_dock3r.sh<br /> <br /> Wait for docking to complete. Next, you must extract all molecule IDs and corresponding DOCK scores from above. To do so, run the following commands in the base docking directory containing your docking files and output folder, while logged into a dev node (I suggest in a screen):<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/get_scores.py .<br /> python get_scores.py 0<br /> <br /> <br /> This script expects a directory named &quot;output*&quot; within the CWD. If your output from docking follows different naming conventions, vim into get_scores.py and change the path. The output will be a file named &quot;scores_round_0.txt&quot;. For iterative rounds, pass increasing numbers into the command line. i.e. When docking the first round of prioritized molecules, pass [1] for scores_round_1.txt. <br /> <br /> '''4. Convert scores and molecule IDS into NumPY arrays for ChemSTEP recognition. Requires ChemSTEP venv'''<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/convert_scores_to_npy.py .<br /> python convert_scores_to_npy.py 0<br /> <br /> For python speed purposes, ChemSTEP requires that DOCK scores and IDs be give in the form of a NumPY array. In this step, we are taking our readable scores file and converting them so ChemSTEP can read it. This script expects a txt file with Molecule IDS and DOCK scores. Use the same round number you used in step 3. As run above, this will output two files named scores_round_0.npy and indices_round_0.npy that contain line-matched molecule IDs (indices) and their respective docking scores. <br /> <br /> '''5. Enter into or make a directory to run ChemSTEP in. Copy in necessary files for initiating ChemSTEP: params.txt, run_chemstep.py and launch_chemstep_as_job.sh. Copy in your score and indices numpy files as well, which should be in your docking directory.'''<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/params.txt .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/run_chemstep.py .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/launch_chemstep_as_job.sh .<br /> <br /> <br /> '''6. Edit params.txt file''' This ONLY needs to be edited for the initial round of ChemSTEP. The parameters outlined here will be carried through all rounds of ChemSTEP. <br /> seed_indices_file: /absoulte/path/to/your/indices_round_0.npy<br /> seed_scores_file: /absolute/path/to/your/scores_round_0.npy<br /> hit_pprop: 5<br /> n_docked_per_round: 10000000<br /> max_beacons: 150<br /> max_n_rounds: 250<br /> <br /> Within params.txt, add the absolute paths to the ChemSTEP-readable score and indices NumPY arrays. The rest of the values within the params.txt file are left to the discretion of the user, with some considerations below. <br /> <br /> '''pProp:''' this value will define what is considered a &quot;virtual hit&quot; in your campaign. pProp is defined as the -log(rank %) of a molecule within the total library score-distribution. For example, a pProp of 4 in the 1.1T XReal space is equivalent the top 0.01% of the library, corresponding to the top 110 million molecules. pProp 5 = 0.001% = 11 million virtual hits, etc. From the seed set, ChemSTEP will estimate a DOCK score value in kcal/mol that associated with the lower-limit of your desired pProp zone. Any molecules that score better than the threshold is considered a virtual hit in your campaign.<br /> <br /> '''n_docked_per_round:''' this number is the desired molecules for prioritization per round. When choosing this value, note that this number of molecules must be built and docked in between each round of ChemSTEP. Prioritizing many molecules will slow building and docking speeds, and coupled with few beacons (below) may lead to decreased diversity. Prioritizing too few molecules may result in slower virtual hit recovery. Round size does not significantly impact the algorithm running time.** <br /> <br /> '''max_beacons:''' this is the number of diverse, well-scoring molecules ChemSTEP will assign to be beacons per round. All molecules that score better than pProp threshold can be considered for beacons. Choosing too many beacons may result in decreased diversity, but too few beacons could hinder space exploration. If not enough molecules score above your pProp threshold, ChemSTEP may assign fewer beacons than assigned value. <br /> <br /> <br /> There should be no need to edit run_chemstep.py or the SGE wrapper script. ''If using another virtual library, be sure to update the path in run_chemstep.py to point to your FP library.'' At this time, we strongly suggest running ChemSTEP as a job array with 64 CPU slots requested, which is specified in the default wrapper. <br /> <br /> <br /> '''7. Run ChemSTEP''' with the following command: <br /> qsub launch_chemstep_as_job.sh <br /> <br /> <br /> This will launch the main ChemSTEP job to the SGE scheduler. Check your job status with the 'qstat' command. The algorithm will read the score and indices files provided, calculate pProp (for round 1) and assign beacons. As this job runs, it will launch a subsequent job array. These sub-jobs are the Chaining step of ChemSTEP, where each parallel worker calculates the Tanimoto distances of 100 million molecules from the virtual library to the beacons. These distances are then written to the files within the /output directory. When these jobs complete, the main job will read through all Tanimoto distances and pull molecules for prioritization. <br /> <br /> Running ChemSTEP successfully will result in the output of a SMILES file within output/complete_info/. You will also get (1) chemstep_algo.log and (2) chemstep_submission.log in the working directory after job submission. chemstep_algo.log contains the running output of the algorithm in a readable format, with timestamps, including the DOCK score associated with your desired pProp. chemstep_submission.log contains the output of ChemSTEP, as well as any information output by the SGE submission system (python errors, cluster issues, tracebacks if failed). These files will be updated with any information with iterative rounds of ChemSTEP run in the same directory. <br /> <br /> '''Troubleshooting:''' if no jobs are running and there is no SMI file, check the chemstep_submission.log first. Any traceback error or SGE error should give you some idea of why ChemSTEP failed. Some errors may be due to SGE or Wynton issues. If directed, look at a few .out files within /output/jobs/ . If a ChemSTEP run fails on your FIRST run, delete the output and log files, fix what needs to be fixed, and rerun. Errors in subsequent rounds during the chaining step can potentially corrupt chaining files within the /output directory that are needed for prioritization. At the very least, you will definitely have duplicates of beacons and information written to the log files. At this time, if your run fails during iterative rounds, it's best to start from the beginning. <br /> <br /> Prioritized molecules should be built, docked, and their scores fed back into ChemSTEP. More detailed instructions below: <br /> <br /> '''8. Build prioritized molecules (DOCK 3.8).''' /taken from docs.docking.org <br /> source /wynton/group/bks/soft/DOCK-3.8.5/env.sh<br /> <br /> python /wynton/group/bks/soft/DOCK-3.8.5/DOCK3.8/zinc22-3d/submit/submit_building_docker.py --output_folder building_output --bundle_size 1000 --minutes_per_mol 5 --skip_name_check --scheduler sge --container_software apptainer --container_path_or_name /wynton/group/bks/soft/DOCK-3.8.5/building_pipeline.sif smi_round_1.smi<br /> <br /> When building has completed, you must write an SDI file with the complete paths to each built bundle and dock. '''Be sure to change the INDOCK file to save only poses that meet your score pProp score threshold (output by ChemSTEP)!''' Retrieve docking scores as convert to NumPy arrays as outlined above, update the round number when running get_scores.py and convert_scores_to_npy.py! Copy new score and indices files into the directory you ran ChemSTEP in. If you are following along as a tutorial, you should have scores_round_1.npy and indices_round_1.npy from the previous step (from FIRST round of ChemSTEP prioritization). <br /> <br /> <br /> '''9. Set up for iterative rounds of ChemSTEP'''<br /> <br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/run_chemstep_iterative.py .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/launch_chemstep_iterative.sh .<br /> <br /> <br /> '''10. Run ChemSTEP'''<br /> qsub launch_chemstep_as_job.sh [round number]<br /> <br /> For the first iterative round, the round number is [2], and should increase by one for every subsequent round of ChemSTEP. The output will be smi_round_****.smi file. Repeat steps 8-10 for as many rounds as needed. The performance is reported in outputy/complete_info/run_summary.df, which contains the number of beacons selected, the number of molecules docked, the number of hits found, the distance threshold for the selected molecules to dock, and the last added beacon's distance to all previous beacons.<br /> <br /> <br /> ** anecdotally true.</div>

Kholland http://wiki.docking.org/index.php?title=Running_ChemSTEP&diff=16880 2025-09-11T22:28:25Z

<p>Kholland: </p> <hr /> <div>last update: Sept 11 2025 katie. current ver = 0.3.1<br /> <br /> ChemSTEP (Chemical Space Traversal and Exploration Procedure) is an open-source, transparent acceleration algorithm for molecular docking capable of dealing with virtual libraries of several trillion compounds. This wiki page is a guide for BKS lab members to run ChemSTEP on Wynton HPC, using the current version of InifiSee XReal library (1.1T). For more general use directions, please refer to [ChemSTEP Read-the-Docs]. <br /> <br /> At a high-level, ChemSTEP is an iterative process that identifies molecules from the larger virtual library to prioritize for docking. First, we identify a random sample of the total library (termed &quot;seed set&quot;) and dock those molecules to the target of interest. From this seed set, we can calculate total-library pProp values (-log rank percentages) and the number of &quot;virtual hits&quot; in the total library (high-scoring molecules). ChemSTEP will identify a set of maximally diverse molecules that score above the desired pProp threshold (&quot;beacons&quot;) from the seed set. These beacons guide prioritization, where molecules chosen and output by ChemSTEP are close in chemical space to the beacons. Prioritized molecules are then built, docked, and returned to ChemSTEP for another round of prioritization. This process is iterated until you reach desired virtual hit recovery, or you are no longer recovering virtual hits. <br /> <br /> <br /> Running the ChemSTEP algorithm on Wynton takes place in loosely three steps: (1) submission of the main ChemSTEP job, (2) ChainingLog sub-job array, and (3) gathering of SMILES for prioritization. <br /> <br /> <br /> What the user need: DOCKFILES, directories for (1) docking (2) building and (3) running ChemSTEP. '''All iterative rounds of ChemSTEP should be run in the SAME directory.''' <br /> <br /> '''1. Source ChemSTEP virtual environment on Wynton'''<br /> source /wynton/group/bks/work/shared/kholland/chemstep_env/bin/activate<br /> <br /> <br /> '''2. Copy InfiniSee seed-set SDI file into your docking directory'''<br /> This directory should already contain your dockfiles, with INDOCK parameters set to your liking. In this step, we are copying in a split database index file (SDI) containing paths to bundles of db2 files. This seed set contains 100 million molecules sampled randomly from the total virtual library, currently 1.1 trillion molecules.<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/XR_seed_set_v0p0.wynton.sdi .<br /> <br /> '''3. Dock seed set to your receptor of interest using DOCK 3.8''' /docking directions taken from docs.docking.org. This is meant to be done as you would do a normal LSD. <br /> export MOLECULES_DIR_TO_BIND=[outermost folder containing the molecules to dock, just your base docking directory]<br /> export DOCKFILES=[path to your dockfiles]<br /> export INPUT_FOLDER=[the folder containing your .sdi file(s)]<br /> export OUTPUT_FOLDER=[where you want the output ]<br /> <br /> /wynton/group/bks/work/bwhall61/needs_github/super_dock3r.sh<br /> <br /> Wait for docking to complete. Next, you must extract all molecule IDs and corresponding DOCK scores from above. To do so, run the following commands in the base docking directory containing your docking files and output folder, while logged into a dev node (I suggest in a screen):<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/get_scores.py .<br /> python get_scores.py 0<br /> <br /> <br /> This script expects a directory named &quot;output*&quot; within the CWD. If your output from docking follows different naming conventions, vim into get_scores.py and change the path. The output will be a file named &quot;scores_round_0.txt&quot;. For iterative rounds, pass increasing numbers into the command line. i.e. When docking the first round of prioritized molecules, pass [1] for scores_round_1.txt. <br /> <br /> '''4. Convert scores and molecule IDS into NumPY arrays for ChemSTEP recognition. Requires ChemSTEP venv'''<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/convert_scores_to_npy.py .<br /> python convert_scores_to_npy.py 0<br /> <br /> For python speed purposes, ChemSTEP requires that DOCK scores and IDs be give in the form of a NumPY array. In this step, we are taking our readable scores file and converting them so ChemSTEP can read it. This script expects a txt file with Molecule IDS and DOCK scores. Use the same number you used in step 3. As run above, this will output two files named scores_round_0.npy and indices_round_0.npy that contain line-matched molecule IDs (indices) and their respective docking scores. <br /> <br /> '''5. Enter into or make a directory to run ChemSTEP in. Copy in necessary files for initiating ChemSTEP: params.txt, run_chemstep.py and launch_chemstep_as_job.sh. Copy in your score and indices numpy files as well, which should be in your docking directory.'''<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/params.txt .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/run_chemstep.py .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/launch_chemstep_as_job.sh .<br /> <br /> <br /> '''6. Edit params.txt file''' This ONLY needs to be edited for the initial round of ChemSTEP. The parameters outlined here will be carried through all rounds of ChemSTEP. <br /> seed_indices_file: /absoulte/path/to/your/indices_round_0.npy<br /> seed_scores_file: /absolute/path/to/your/scores_round_0.npy<br /> hit_pprop: 5<br /> n_docked_per_round: 10000000<br /> max_beacons: 150<br /> max_n_rounds: 250<br /> <br /> Within params.txt, add the absolute paths to the ChemSTEP-readable score and indices NumPY arrays. The rest of the values within the params.txt file are left to the discretion of the user, with some considerations below. <br /> <br /> '''pProp:''' this value will define what is considered a &quot;virtual hit&quot; in your campaign. pProp is defined as the -log(rank %) of a molecule within the total library score-distribution. For example, a pProp of 4 in the 1.1T XReal space is equivalent the top 0.01% of the library, corresponding to the top 110 million molecules. pProp 5 = 0.001% = 11 million virtual hits, etc. From the seed set, ChemSTEP will estimate a DOCK score value in kcal/mol that associated with the lower-limit of your desired pProp zone. Any molecules that score better than the threshold is considered a virtual hit in your campaign.<br /> <br /> '''n_docked_per_round:''' this number is the desired molecules for prioritization per round. When choosing this value, note that this number of molecules must be built and docked in between each round of ChemSTEP. Prioritizing many molecules will slow building and docking speeds, and coupled with few beacons (below) may lead to decreased diversity. Prioritizing too few molecules may result in slower virtual hit recovery. Round size does not significantly impact the algorithm running time.** <br /> <br /> '''max_beacons:''' this is the number of diverse, well-scoring molecules ChemSTEP will assign to be beacons per round. All molecules that score better than pProp threshold can be considered for beacons. Choosing too many beacons may result in decreased diversity, but too few beacons could hinder space exploration. If not enough molecules score above your pProp threshold, ChemSTEP may assign fewer beacons than assigned value. <br /> <br /> <br /> There should be no need to edit run_chemstep.py or the SGE wrapper script. ''If using another virtual library, be sure to update the path in run_chemstep.py to point to your FP library.'' At this time, we strongly suggest running ChemSTEP as a job array with 64 CPU slots requested, which is specified in the default wrapper. <br /> <br /> <br /> '''7. Run ChemSTEP''' with the following command: <br /> qsub launch_chemstep_as_job.sh <br /> <br /> <br /> This will launch the main ChemSTEP job to the SGE scheduler. Check your job status with the 'qstat' command. The algorithm will read the score and indices files provided, calculate pProp (for round 1) and assign beacons. As this job runs, it will launch a subsequent job array. These sub-jobs are the Chaining step of ChemSTEP, where each parallel worker calculates the Tanimoto distances of 100 million molecules from the virtual library to the beacons. These distances are then written to the files within the /output directory. When these jobs complete, the main job will read through all Tanimoto distances and pull molecules for prioritization. <br /> <br /> Running ChemSTEP successfully will result in the output of a SMILES file within output/complete_info/. You will also get (1) chemstep_algo.log and (2) chemstep_submission.log in the working directory after job submission. chemstep_algo.log contains the running output of the algorithm in a readable format, with timestamps, including the DOCK score associated with your desired pProp. chemstep_submission.log contains the output of ChemSTEP, as well as any information output by the SGE submission system (python errors, cluster issues, tracebacks if failed). These files will be updated with any information with iterative rounds of ChemSTEP run in the same directory. <br /> <br /> '''Troubleshooting:''' if no jobs are running and there is no SMI file, check the chemstep_submission.log first. Any traceback error or SGE error should give you some idea of why ChemSTEP failed. Some errors may be due to SGE or Wynton issues. If directed, look at a few .out files within /output/jobs/ . If a ChemSTEP run fails on your FIRST run, delete the output and log files, fix what needs to be fixed, and rerun. Errors in subsequent rounds during the chaining step can potentially corrupt chaining files within the /output directory that are needed for prioritization. At the very least, you will definitely have duplicates of beacons and information written to the log files. At this time, if your run fails during iterative rounds, it's best to start from the beginning. <br /> <br /> Prioritized molecules should be built, docked, and their scores fed back into ChemSTEP. More detailed instructions below: <br /> <br /> '''8. Build prioritized molecules (DOCK 3.8).''' /taken from docs.docking.org <br /> source /wynton/group/bks/soft/DOCK-3.8.5/env.sh<br /> <br /> python /wynton/group/bks/soft/DOCK-3.8.5/DOCK3.8/zinc22-3d/submit/submit_building_docker.py --output_folder building_output --bundle_size 1000 --minutes_per_mol 5 --skip_name_check --scheduler sge --container_software apptainer --container_path_or_name /wynton/group/bks/soft/DOCK-3.8.5/building_pipeline.sif smi_round_1.smi<br /> <br /> When building has completed, you must write an SDI file with the complete paths to each built bundle and dock. '''Be sure to change the INDOCK file to save only poses that meet your score pProp score threshold (output by ChemSTEP)!''' Retrieve docking scores as convert to NumPy arrays as outlined above, update the round number when running get_scores.py and convert_scores_to_npy.py! Copy new score and indices files into the directory you ran ChemSTEP in. If you are following along as a tutorial, you should have scores_round_1.npy and indices_round_1.npy from the previous step (from FIRST round of ChemSTEP prioritization). <br /> <br /> <br /> '''9. Set up for iterative rounds of ChemSTEP'''<br /> <br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/run_chemstep_iterative.py .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/launch_chemstep_iterative.sh .<br /> <br /> <br /> '''10. Run ChemSTEP'''<br /> qsub launch_chemstep_as_job.sh [round number]<br /> <br /> For the first iterative round, the round number is [2], and should increase by one for every subsequent round of ChemSTEP. The output will be smi_round_****.smi file. Repeat steps 8-10 for as many rounds as needed. The performance is reported in outputy/complete_info/run_summary.df, which contains the number of beacons selected, the number of molecules docked, the number of hits found, the distance threshold for the selected molecules to dock, and the last added beacon's distance to all previous beacons.<br /> <br /> <br /> ** anecdotally true.</div>

Kholland http://wiki.docking.org/index.php?title=Running_ChemSTEP&diff=16872 2025-09-08T23:36:57Z

<p>Kholland: </p> <hr /> <div>last update: Sept 3 2025 katie. current ver = 0.3.1<br /> <br /> ChemSTEP (Chemical Space Traversal and Exploration Procedure) is an open-source, transparent acceleration algorithm for molecular docking capable of dealing with virtual libraries of several trillion compounds. This wiki page is a guide for BKS lab members to run ChemSTEP on Wynton HPC, using the current version of InifiSee XReal library (1.1T). For more general use directions, please refer to [ChemSTEP Read-the-Docs]. <br /> <br /> At a high-level, ChemSTEP is an iterative process to run in between rounds of docking. The general procedure is as follows: build molecules, dock molecules, convert scores for ChemSTEP, run ChemSTEP. In this case, the first round of building (the &quot;seed set&quot;) has already been done.<br /> <br /> <br /> What you (the user) need: DOCKFILES, directories for (1) docking (2) building and (3) running ChemSTEP<br /> <br /> '''1. Source ChemSTEP virtual environment on Wynton'''<br /> source /wynton/group/bks/work/shared/kholland/chemstep_env/bin/activate<br /> <br /> <br /> '''2. Copy InfiniSee seed-set SDI file into your docking directory'''<br /> This directory should already contain your dockfiles, with INDOCK parameters set to your liking. In this step, we are copying in a split database index file (SDI) containing paths to bundles of db2 files. This seed set contains 100 million molecules sampled randomly from the total virtual library, currently 1.1 trillion molecules. <br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/XR_seed_set_v0p0.wynton.sdi .<br /> <br /> '''3. Dock seed set to your receptor of interest using DOCK 3.8''' /docking directions taken from docs.docking.org. This is meant to be done as you would do a normal LSD. <br /> export MOLECULES_DIR_TO_BIND=[outermost folder containing the molecules to dock]<br /> export DOCKFILES=[path to your dockfiles]<br /> export INPUT_FOLDER=[the folder containing your .sdi file(s)]<br /> export OUTPUT_FOLDER=[where you want the output ]<br /> <br /> /wynton/group/bks/work/bwhall61/needs_github/super_dock3r.sh<br /> <br /> Wait for docking to complete. Next, you must extract all molecule IDs and corresponding DOCK scores from above. To do so, run the following commands in the base docking directory (containing your docking files and output folder) while logged into a dev node:<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/get_scores.py .<br /> python get_scores.py 0<br /> <br /> <br /> This script expects a directory named &quot;output*&quot; within the CWD. If your output from docking follows different naming conventions, vim into get_scores.py and change the path. The output will be a file named &quot;scores_round_0.txt&quot;. For iterative rounds, pass increasing numbers into the command line. i.e. When docking the first round of prioritized molecules, pass [1] for scores_round_1.txt. <br /> <br /> '''4. Convert scores and molecule IDS into NumPY arrays for ChemSTEP recognition. Requires ChemSTEP venv'''<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/convert_scores_to_npy.py .<br /> python convert_scores_to_npy.py 0<br /> <br /> This script expects a txt file with Molecule IDS and DOCK scores. Use the same number you used in step 3. As run above, this will output two files named scores_round_0.npy and indices_round_0.npy that contain line-matched molecule IDs (indices) and their respective docking scores. <br /> <br /> '''5. Enter into or make a directory to run ChemSTEP in. Copy in necessary files for initiating ChemSTEP: params.txt, run_chemstep.py and launch_chemstep_as_job.sh, your score and indices numpy files.'''<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/params.txt .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/run_chemstep.py .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/launch_chemstep_as_job.sh .<br /> <br /> <br /> '''6. Edit params.txt file''' This ONLY needs to be edited for the initial round of ChemSTEP. The parameters outlined here will be carried through the round of ChemSTEP chaining. <br /> seed_indices_file: /absoulte/path/to/your/indices_round_0.npy<br /> seed_scores_file: /absolute/path/to/your/scores_round_0.npy<br /> hit_pprop: 5<br /> n_docked_per_round: 10000000<br /> max_beacons: 150<br /> max_n_rounds: 250<br /> <br /> Be sure that this file reflects your score and indices files for round zero (the seed set). Define your desired pProp, number of beacons, and number to prioritize per round. There should be no need to edit run_chemstep.py or the SGE wrapper script for the FIRST round of XReal docking. ''If using another virtual library, be sure to update the path in run_chemstep.py to point to your FP library.'' We strongly suggest running ChemSTEP as a job array with 64 CPU slots requested (specified in wrapper). <br /> <br /> '''7. Run ChemSTEP''' with the following command: <br /> qsub launch_chemstep_as_job.sh <br /> <br /> <br /> <br /> Running ChemSTEP successfully will result in the output of a SMILES file within output/complete_info/. You will also get (1) chemstep_algo.log and (2) chemstep_submission.log in the working directory after job submission. chemstep_algo.log contains the running output of the alogorithm in a readable format, with timestamps. chemstep_submission log contains the output of ChemSTEP, as well as any information output by the SGE submission system (python errors, cluster issues, tracebacks if failed). These files will be updated with any information with iterative rounds of ChemSTEP run in the same directory. <br /> <br /> Troubleshooting: if the job is completed and there is no SMI file, check the chemstep_submission.log first. Any traceback error or SGE error should give you some idea of why ChemSTEP failed. Some errors may be due to SGE/wynton issues. If directed, look at a few .out files within /output/jobs/ . If a ChemSTEP run fails on your FIRST run, delete the output and log files, fix what needs to be fixed, and rerun. Errors in subsequent rounds during the chaining step can potentially corrupt chaining files within the /output directory that are needed for prioritization. At the very least, you will definitely have duplicates of beacons and information written to the log files. At this time, if your run fails during iterative rounds, it's best to start from the beginning. <br /> <br /> Prioritized molecules should be built, docked, and their scores fed back into ChemSTEP. More detailed instructions below: <br /> <br /> '''8. Build prioritized molecules (DOCK 3.8).''' /taken from docs.docking.org <br /> source /wynton/group/bks/soft/DOCK-3.8.5/env.sh<br /> <br /> python /wynton/group/bks/soft/DOCK-3.8.5/DOCK3.8/zinc22-3d/submit/submit_building_docker.py --output_folder building_output --bundle_size 1000 --minutes_per_mol 5 --skip_name_check --scheduler sge --container_software apptainer --container_path_or_name /wynton/group/bks/soft/DOCK-3.8.5/building_pipeline.sif smi_round_1.smi<br /> <br /> When building has completed, you must write an SDI file with the complete paths to each built bundle and dock. '''Be sure to change the INDOCK file to save only poses that meet your score pProp score threshold (output by ChemSTEP)!''' Retrieve docking scores as convert to NumPy arrays as outlined above, update the round number when running get_scores.py and convert_scores_to_npy.py! Copy new score and indices files into the directory you ran ChemSTEP in. If you are following along as a tutorial, you should have scores_round_1.npy and indices_round_1.npy from the previous step (from FIRST round of ChemSTEP prioritization). <br /> <br /> <br /> '''9. Set up for iterative rounds of ChemSTEP'''<br /> <br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/run_chemstep_iterative.py .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/launch_chemstep_iterative.sh .<br /> <br /> <br /> '''10. Run ChemSTEP'''<br /> qsub launch_chemstep_as_job.sh [round number]<br /> <br /> For the first iterative round, the round number is [2], and should increase by one for every subsequent round of ChemSTEP. The output will be smi_round_****.smi file. Repeat steps 8-10 for as many rounds as needed. The performance is reported in outputy/complete_info/run_summary.df, which contains the number of beacons selected, the number of molecules docked, the number of hits found, the distance threshold for the selected molecules to dock, and the last added beacon's distance to all previous beacons.</div>

Kholland http://wiki.docking.org/index.php?title=Running_ChemSTEP&diff=16863 2025-09-03T22:54:41Z

<p>Kholland: </p> <hr /> <div>last update: Sept 3 2025 katie. current ver = 0.3.1<br /> <br /> ChemSTEP (Chemical Space Traversal and Exploration Procedure) is an open-source, transparent acceleration algorithm for molecular docking capable of dealing with virtual libraries of several trillion compounds. This wiki page is a guide for BKS lab members to run ChemSTEP on Wynton HPC, using the current version of InifiSee XReal library (1.1T). For more general use directions, please refer to [ChemSTEP Read-the-Docs]. <br /> <br /> At a high-level, ChemSTEP is an iterative process to run in between rounds of docking. The general procedure is as follows: build molecules, dock molecules, convert scores for ChemSTEP, run ChemSTEP. In this case, the first round of building (the &quot;seed set&quot;) has already been done.<br /> <br /> <br /> What you (the user) need: DOCKFILES, directories for (1) docking (2) building and (3) running ChemSTEP<br /> <br /> '''1. Source ChemSTEP virtual environment on Wynton'''<br /> source /wynton/group/bks/work/shared/kholland/chemstep_env/bin/activate<br /> <br /> <br /> '''2. Copy InfiniSee seed-set SDI file into your docking directory'''<br /> This directory should already contain your dockfiles, with INDOCK parameters set to your liking. In this step, we are copying in a split database index file (SDI) containing paths to bundles of db2 files. This seed set contains 100 million molecules sampled randomly from the total virtual library, currently 1.1 trillion molecules. <br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/XR_00_seed_set.wynton.sdi .<br /> <br /> '''3. Dock seed set to your receptor of interest using DOCK 3.8''' /docking directions taken from docs.docking.org. This is meant to be done as you would do a normal LSD. <br /> export MOLECULES_DIR_TO_BIND=[outermost folder containing the molecules to dock]<br /> export DOCKFILES=[path to your dockfiles]<br /> export INPUT_FOLDER=[the folder containing your .sdi file(s)]<br /> export OUTPUT_FOLDER=[where you want the output ]<br /> <br /> /wynton/group/bks/work/bwhall61/needs_github/super_dock3r.sh<br /> <br /> Wait for docking to complete. Next, you must extract all molecule IDs and corresponding DOCK scores from above. To do so, run the following commands in the base docking directory (containing your docking files and output folder) while logged into a dev node:<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/get_scores.py .<br /> python get_scores.py 0<br /> <br /> <br /> This script expects a directory named &quot;output*&quot; within the CWD. If your output from docking follows different naming conventions, vim into get_scores.py and change the path. The output will be a file named &quot;scores_round_0.txt&quot;. For iterative rounds, pass increasing numbers into the command line. i.e. When docking the first round of prioritized molecules, pass [1] for scores_round_1.txt. <br /> <br /> '''4. Convert scores and molecule IDS into NumPY arrays for ChemSTEP recognition. Requires ChemSTEP venv'''<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/convert_scores_to_npy.py .<br /> python convert_scores_to_npy.py 0<br /> <br /> This script expects a txt file with Molecule IDS and DOCK scores. Use the same number you used in step 3. As run above, this will output two files named scores_round_0.npy and indices_round_0.npy that contain line-matched molecule IDs (indices) and their respective docking scores. <br /> <br /> '''5. Enter into or make a directory to run ChemSTEP in. Copy in necessary files for initiating ChemSTEP: params.txt, run_chemstep.py and launch_chemstep_as_job.sh, your score and indices numpy files.'''<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/params.txt .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/run_chemstep.py .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/launch_chemstep_as_job.sh .<br /> <br /> <br /> '''6. Edit params.txt file''' This ONLY needs to be edited for the initial round of ChemSTEP. The parameters outlined here will be carried through the round of ChemSTEP chaining. <br /> seed_indices_file: /absoulte/path/to/your/indices_round_0.npy<br /> seed_scores_file: /absolute/path/to/your/scores_round_0.npy<br /> hit_pprop: 5<br /> n_docked_per_round: 10000000<br /> max_beacons: 150<br /> max_n_rounds: 250<br /> <br /> Be sure that this file reflects your score and indices files for round zero (the seed set). Define your desired pProp, number of beacons, and number to prioritize per round. There should be no need to edit run_chemstep.py or the SGE wrapper script for the FIRST round of XReal docking. ''If using another virtual library, be sure to update the path in run_chemstep.py to point to your FP library.'' We strongly suggest running ChemSTEP as a job array with 64 CPU slots requested (specified in wrapper). <br /> <br /> '''7. Run ChemSTEP''' with the following command: <br /> qsub launch_chemstep_as_job.sh <br /> <br /> <br /> <br /> Running ChemSTEP successfully will result in the output of a SMILES file within output/complete_info/. You will also get (1) chemstep_algo.log and (2) chemstep_submission.log in the working directory after job submission. chemstep_algo.log contains the running output of the alogorithm in a readable format, with timestamps. chemstep_submission log contains the output of ChemSTEP, as well as any information output by the SGE submission system (python errors, cluster issues, tracebacks if failed). These files will be updated with any information with iterative rounds of ChemSTEP run in the same directory. <br /> <br /> Troubleshooting: if the job is completed and there is no SMI file, check the chemstep_submission.log first. Any traceback error or SGE error should give you some idea of why ChemSTEP failed. Some errors may be due to SGE/wynton issues. If directed, look at a few .out files within /output/jobs/ . If a ChemSTEP run fails on your FIRST run, delete the output and log files, fix what needs to be fixed, and rerun. Errors in subsequent rounds during the chaining step can potentially corrupt chaining files within the /output directory that are needed for prioritization. At the very least, you will definitely have duplicates of beacons and information written to the log files. At this time, if your run fails during iterative rounds, it's best to start from the beginning. <br /> <br /> Prioritized molecules should be built, docked, and their scores fed back into ChemSTEP. More detailed instructions below: <br /> <br /> '''8. Build prioritized molecules (DOCK 3.8).''' /taken from docs.docking.org <br /> source /wynton/group/bks/soft/DOCK-3.8.5/env.sh<br /> <br /> python /wynton/group/bks/soft/DOCK-3.8.5/DOCK3.8/zinc22-3d/submit/submit_building_docker.py --output_folder building_output --bundle_size 1000 --minutes_per_mol 5 --skip_name_check --scheduler sge --container_software apptainer --container_path_or_name /wynton/group/bks/soft/DOCK-3.8.5/building_pipeline.sif smi_round_1.smi<br /> <br /> When building has completed, you must write an SDI file with the complete paths to each built bundle and dock. '''Be sure to change the INDOCK file to save only poses that meet your score pProp score threshold (output by ChemSTEP)!''' Retrieve docking scores as convert to NumPy arrays as outlined above, update the round number when running get_scores.py and convert_scores_to_npy.py! Copy new score and indices files into the directory you ran ChemSTEP in. If you are following along as a tutorial, you should have scores_round_1.npy and indices_round_1.npy from the previous step (from FIRST round of ChemSTEP prioritization). <br /> <br /> <br /> '''9. Set up for iterative rounds of ChemSTEP'''<br /> <br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/run_chemstep_iterative.py .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/launch_chemstep_iterative.sh .<br /> <br /> <br /> '''10. Run ChemSTEP'''<br /> qsub launch_chemstep_as_job.sh [round number]<br /> <br /> For the first iterative round, the round number is [2], and should increase by one for every subsequent round of ChemSTEP. The output will be smi_round_****.smi file. Repeat steps 8-10 for as many rounds as needed. The performance is reported in outputy/complete_info/run_summary.df, which contains the number of beacons selected, the number of molecules docked, the number of hits found, the distance threshold for the selected molecules to dock, and the last added beacon's distance to all previous beacons.</div>

Kholland http://wiki.docking.org/index.php?title=Running_ChemSTEP&diff=16862 2025-09-03T22:45:42Z

<p>Kholland: </p> <hr /> <div>last update: Sept 3 2025 katie. current ver = 0.3.1<br /> <br /> ChemSTEP (Chemical Space Traversal and Exploration Procedure) is an open-source, transparent acceleration algorithm for molecular docking capable of dealing with virtual libraries of several trillion compounds. This wiki page is a guide for BKS lab members to run ChemSTEP on Wynton HPC, using the current version of InifiSee XReal library (1.1T). For more general use directions, please refer to [ChemSTEP Read-the-Docs]. <br /> <br /> At a high-level, ChemSTEP is an iterative process to run in between rounds of docking. The general procedure is as follows: build molecules, dock molecules, convert scores for ChemSTEP, run ChemSTEP. In this case, the first round of building (the &quot;seed set&quot;) has already been done.<br /> <br /> <br /> What you (the user) need: DOCKFILES, directories for (1) docking (2) building and (3) running ChemSTEP<br /> <br /> '''1. Source ChemSTEP virtual environment on Wynton'''<br /> source /wynton/group/bks/work/shared/kholland/chemstep_env/bin/activate<br /> <br /> <br /> '''2. Copy InfiniSee seed-set SDI file into your docking directory'''<br /> This directory should already contain your dockfiles, with INDOCK parameters set to your liking. In this step, we are copying in a split database index file (SDI) containing paths to bundles of db2 files. This seed set contains 100 million molecules sampled randomly from the total virtual library, currently 1.1 trillion molecules. <br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/XR_00_seed_set.wynton.sdi .<br /> <br /> '''3. Dock seed set to your receptor of interest using DOCK 3.8''' /docking directions taken from docs.docking.org. This is meant to be done as you would do a normal LSD. <br /> export MOLECULES_DIR_TO_BIND=[outermost folder containing the molecules to dock]<br /> export DOCKFILES=[path to your dockfiles]<br /> export INPUT_FOLDER=[the folder containing your .sdi file(s)]<br /> export OUTPUT_FOLDER=[where you want the output ]<br /> <br /> /wynton/group/bks/work/bwhall61/needs_github/super_dock3r.sh<br /> <br /> Wait for docking to complete. Next, you must extract all molecule IDs and corresponding DOCK scores from above. To do so, run the following commands in the base docking directory (containing your docking files and output folder) while logged into a dev node:<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/get_scores.py .<br /> python get_scores.py 0<br /> <br /> <br /> This script expects a directory named &quot;output*&quot; within the CWD. If your output from docking follows different naming conventions, vim into get_scores.py and change the path. The output will be a file named &quot;scores_round_0.txt&quot;. For iterative rounds, pass increasing numbers into the command line. i.e. When docking the first round of prioritized molecules, pass [1] for scores_round_1.txt. <br /> <br /> '''4. Convert scores and molecule IDS into NumPY arrays for ChemSTEP recognition. Requires ChemSTEP venv'''<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/convert_scores_to_npy.py .<br /> python convert_scores_to_npy.py 0<br /> <br /> This script expects a txt file with Molecule IDS and DOCK scores. Use the same number you used in step 3. As run above, this will output two files named scores_round_0.npy and indices_round_0.npy that contain line-matched molecule IDs (indices) and their respective docking scores. <br /> <br /> '''5. Enter into or make a directory to run ChemSTEP in. Copy in necessary files for initiating ChemSTEP: params.txt, run_chemstep.py and launch_chemstep_as_job.sh, your score and indices numpy files.'''<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/params.txt .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/run_chemstep.py .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/launch_chemstep_as_job.sh .<br /> <br /> <br /> '''6. Edit params.txt file''' This ONLY needs to be edited for the initial round of ChemSTEP. The parameters outlined here will be carried through the round of ChemSTEP chaining. <br /> seed_indices_file: /absoulte/path/to/your/indices_round_0.npy<br /> seed_scores_file: /absolute/path/to/your/scores_round_0.npy<br /> hit_pprop: 5<br /> n_docked_per_round: 10000000<br /> max_beacons: 150<br /> max_n_rounds: 250<br /> <br /> Be sure that this file reflects your score and indices files for round zero (the seed set). Define your desired pProp, number of beacons, and number to prioritize per round. There should be no need to edit run_chemstep.py or the SGE wrapper script for the FIRST round of XReal docking. ''If using another virtual library, be sure to update the path in run_chemstep.py to point to your FP library.'' We strongly suggest running ChemSTEP as a job array with 64 CPU slots requested (specified in wrapper). <br /> <br /> '''7. Run ChemSTEP''' with the following command: <br /> qsub launch_chemstep_as_job.sh <br /> <br /> <br /> <br /> Running ChemSTEP successfully will result in the output of a SMILES file within output/complete_info/. You will also get (1) chemstep_algo.log and (2) chemstep_submission.log in the working directory after job submission. chemstep_algo.log contains the running output of the alogorithm in a readable format, with timestamps. chemstep_submission log contains the output of ChemSTEP, as well as any information output by the SGE submission system (python errors, cluster issues, tracebacks if failed). These files will be updated with any information with iterative rounds of ChemSTEP run in the same directory. These molecules should be built, docked, and their scores fed back into ChemSTEP. More detailed instructions below: <br /> <br /> '''8. Build prioritized molecules (DOCK 3.8).''' /taken from docs.docking.org <br /> source /wynton/group/bks/soft/DOCK-3.8.5/env.sh<br /> <br /> python /wynton/group/bks/soft/DOCK-3.8.5/DOCK3.8/zinc22-3d/submit/submit_building_docker.py --output_folder building_output --bundle_size 1000 --minutes_per_mol 5 --skip_name_check --scheduler sge --container_software apptainer --container_path_or_name /wynton/group/bks/soft/DOCK-3.8.5/building_pipeline.sif smi_round_1.smi<br /> <br /> When building has completed, you must write an SDI file with the complete paths to each built bundle and dock. '''Be sure to change the INDOCK file to save only poses that meet your score pProp score threshold (output by ChemSTEP)!''' Retrieve docking scores as convert to NumPy arrays as outlined above, update the round number when running get_scores.py and convert_scores_to_npy.py! Copy new score and indices files into the directory you ran ChemSTEP in. If you are following along as a tutorial, you should have scores_round_1.npy and indices_round_1.npy from the previous step (from FIRST round of ChemSTEP prioritization). <br /> <br /> <br /> '''9. Set up for iterative rounds of ChemSTEP'''<br /> <br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/run_chemstep_iterative.py .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/launch_chemstep_iterative.sh .<br /> <br /> <br /> '''10. Run ChemSTEP'''<br /> qsub launch_chemstep_as_job.sh [round number]<br /> <br /> For the first iterative round, the round number is [2], and should increase by one for every subsequent round of ChemSTEP. The output will be smi_round_****.smi file. Repeat steps 8-10 for as many rounds as needed. The performance is reported in outputy/complete_info/run_summary.df, which contains the number of beacons selected, the number of molecules docked, the number of hits found, the distance threshold for the selected molecules to dock, and the last added beacon's distance to all previous beacons.</div>

Kholland http://wiki.docking.org/index.php?title=Running_ChemSTEP&diff=16861 2025-09-03T22:42:36Z

<p>Kholland: </p> <hr /> <div>last update: Sept 3 2025 katie. current ver = 0.3.1<br /> <br /> ChemSTEP (Chemical Space Traversal and Exploration Procedure) is an open-source, transparent acceleration algorithm for molecular docking capable of dealing with virtual libraries of several trillion compounds. This wiki page is a guide for BKS lab members to run ChemSTEP on Wynton HPC, using the current version of InifiSee XReal library (1.1T). For more general use directions, please refer to [ChemSTEP Read-the-Docs]. <br /> <br /> At a high-level, ChemSTEP is an iterative process to run in between rounds of docking. The general procedure is as follows: build molecules, dock molecules, convert scores for ChemSTEP, run ChemSTEP. In this case, the first round of building (the &quot;seed set&quot;) has already been done.<br /> <br /> <br /> What you (the user) need: DOCKFILES, directories for (1) docking (2) building and (3) running ChemSTEP<br /> <br /> '''1. Source ChemSTEP virtual environment on Wynton'''<br /> source /wynton/group/bks/work/shared/kholland/chemstep_env/bin/activate<br /> <br /> <br /> '''2. Copy InfiniSee seed-set SDI file into your docking directory'''<br /> This directory should already contain your dockfiles, with INDOCK parameters set to your liking. In this step, we are copying in a split database index file (SDI) containing paths to bundles of db2 files. This seed set contains 100 million molecules sampled randomly from the total virtual library, currently 1.1 trillion molecules. <br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/XR_00_seed_set.wynton.sdi .<br /> <br /> '''3. Dock seed set to your receptor of interest using DOCK 3.8''' /docking directions taken from docs.docking.org. This is meant to be done as you would do a normal LSD. <br /> export MOLECULES_DIR_TO_BIND=[outermost folder containing the molecules to dock]<br /> export DOCKFILES=[path to your dockfiles]<br /> export INPUT_FOLDER=[the folder containing your .sdi file(s)]<br /> export OUTPUT_FOLDER=[where you want the output ]<br /> <br /> /wynton/group/bks/work/bwhall61/needs_github/super_dock3r.sh<br /> <br /> Wait for docking to complete. Next, you must extract all molecule IDs and corresponding DOCK scores from above. To do so, run the following commands in the base docking directory (containing your docking files and output folder) while logged into a dev node:<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/get_scores.py .<br /> python get_scores.py 0<br /> <br /> <br /> This script expects a directory named &quot;output*&quot; within the CWD. If your output from docking follows different naming conventions, vim into get_scores.py and change the path. The output will be a file named &quot;scores_round_0.txt&quot;. For iterative rounds, pass increasing numbers into the command line. i.e. When docking the first round of prioritized molecules, pass [1] for scores_round_1.txt. <br /> <br /> '''4. Convert scores and molecule IDS into NumPY arrays for ChemSTEP recognition. Requires ChemSTEP venv'''<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/convert_scores_to_npy.py .<br /> python convert_scores_to_npy.py 0<br /> <br /> This script expects a txt file with Molecule IDS and DOCK scores. Use the same number you used in step 3. As run above, this will output two files named scores_round_0.npy and indices_round_0.npy that contain line-matched molecule IDs (indices) and their respective docking scores. <br /> <br /> '''5. Enter into or make a directory to run ChemSTEP in. Copy in necessary files for initiating ChemSTEP: params.txt, run_chemstep.py and launch_chemstep_as_job.sh, your score and indices numpy files.'''<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/params.txt .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/run_chemstep.py .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/launch_chemstep_as_job.sh .<br /> <br /> <br /> '''6. Edit params.txt file''' This ONLY needs to be edited for the initial round of ChemSTEP. The parameters outlined here will be carried through the round of ChemSTEP chaining. <br /> seed_indices_file: /absoulte/path/to/your/indices_round_0.npy<br /> seed_scores_file: /absolute/path/to/your/scores_round_0.npy<br /> hit_pprop: 5<br /> n_docked_per_round: 10000000<br /> max_beacons: 150<br /> max_n_rounds: 250<br /> <br /> Be sure that this file reflects your score and indices files for round zero (the seed set). Define your desired pProp, number of beacons, and number to prioritize per round. There should be no need to edit run_chemstep.py or the SGE wrapper script for the FIRST round of XReal docking. ''If using another virtual library, be sure to update the path in run_chemstep.py to point to your FP library.'' We strongly suggest running ChemSTEP as a job array with 64 CPU slots requested (specified in wrapper). <br /> <br /> '''7. Run ChemSTEP''' with the following command: <br /> qsub launch_chemstep_as_job.sh <br /> <br /> <br /> <br /> Running ChemSTEP successfully will result in the output of a SMILES file within output/complete_info/. You will also get (1) chemstep_algo.log and (2) chemstep_submission.log in the working directory after job submission. chemstep_algo.log contains the running output of the alogorithm in a readable format, with timestamps. chemstep_submission log contains the output of ChemSTEP, as well as any information output by the SGE submission system (python errors, cluster issues, tracebacks if failed). These files will be updated with any information with iterative rounds of ChemSTEP run in the same directory. These molecules should be built, docked, and their scores fed back into ChemSTEP. More detailed instructions below: <br /> <br /> '''8. Build prioritized molecules (DOCK 3.8).''' /taken from docs.docking.org <br /> source /wynton/group/bks/soft/DOCK-3.8.5/env.sh<br /> <br /> python /wynton/group/bks/soft/DOCK-3.8.5/DOCK3.8/zinc22-3d/submit/submit_building_docker.py --output_folder building_output --bundle_size 1000 --minutes_per_mol 1 --skip_name_check --scheduler sge --container_software apptainer --container_path_or_name /wynton/group/bks/soft/DOCK-3.8.5/building_pipeline.sif smi_round_1.smi<br /> <br /> When building has completed, you must write an SDI file with the complete paths to each built bundle and dock. '''Be sure to change the INDOCK file to save only poses that meet your score pProp score threshold (output by ChemSTEP)!''' Retrieve docking scores as convert to NumPy arrays as outlined above, update the round number when running get_scores.py and convert_scores_to_npy.py! Copy new score and indices files into the directory you ran ChemSTEP in. If you are following along as a tutorial, you should have scores_round_1.npy and indices_round_1.npy from the previous step (from FIRST round of ChemSTEP prioritization). <br /> <br /> <br /> '''9. Set up for iterative rounds of ChemSTEP'''<br /> <br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/run_chemstep_iterative.py .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/launch_chemstep_iterative.sh .<br /> <br /> <br /> '''10. Run ChemSTEP'''<br /> qsub launch_chemstep_as_job.sh [round number]<br /> <br /> For the first iterative round, the round number is [2], and should increase by one for every subsequent round of ChemSTEP. The output will be smi_round_****.smi file. Repeat steps 8-10 for as many rounds as needed. The performance is reported in outputy/complete_info/run_summary.df, which contains the number of beacons selected, the number of molecules docked, the number of hits found, the distance threshold for the selected molecules to dock, and the last added beacon's distance to all previous beacons.</div>

Kholland http://wiki.docking.org/index.php?title=Running_ChemSTEP&diff=16860 2025-09-03T22:42:09Z

<p>Kholland: </p> <hr /> <div>last update: Sept 3 2025 katie. current ver = 0.3.1<br /> <br /> ChemSTEP (Chemical Space Traversal and Exploration Procedure) is an open-source, transparent acceleration algorithm for molecular docking capable of dealing with virtual libraries of several trillion compounds. This wiki page is a guide for BKS lab members to run ChemSTEP on Wynton HPC, using the current version of InifiSee XReal library (1.1T). For more general use directions, please refer to [ChemSTEP Read-the-Docs]. <br /> <br /> At a high-level, ChemSTEP is an iterative process to run in between rounds of docking. The general procedure is as follows: build molecules, dock molecules, convert scores for ChemSTEP, run ChemSTEP. In this case, the first round of building (the &quot;seed set&quot;) has already been done.<br /> <br /> <br /> What you (the user) need: DOCKFILES, directories for (1) docking (2) building and (3) running ChemSTEP<br /> <br /> '''1. Source ChemSTEP virtual environment on Wynton'''<br /> source /wynton/group/bks/work/shared/kholland/chemstep_env/bin/activate<br /> <br /> <br /> '''2. Copy InfiniSee seed-set SDI file into your docking directory'''<br /> This directory should already contain your dockfiles, with INDOCK parameters set to your liking. In this step, we are copying in a split database index file (SDI) containing paths to bundles of db2 files. This seed set contains 100 million molecules sampled randomly from the total virtual library, currently 1.1 trillion molecules. <br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/XR_00_seed_set.wynton.sdi .<br /> <br /> '''3. Dock seed set to your receptor of interest using DOCK 3.8''' /docking directions taken from docs.docking.org. This is meant to be done as you would do a normal LSD. <br /> export MOLECULES_DIR_TO_BIND=[outermost folder containing the molecules to dock]<br /> export DOCKFILES=[path to your dockfiles]<br /> export INPUT_FOLDER=[the folder containing your .sdi file(s)]<br /> export OUTPUT_FOLDER=[where you want the output ]<br /> <br /> /wynton/group/bks/work/bwhall61/needs_github/super_dock3r.sh<br /> <br /> Wait for docking to complete. Next, you must extract all molecule IDs and corresponding DOCK scores from above. To do so, run the following commands in the base docking directory (containing your docking files and output folder) while logged into a dev node:<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/get_scores.py .<br /> python3 get_scores.py 0<br /> <br /> <br /> This script expects a directory named &quot;output*&quot; within the CWD. If your output from docking follows different naming conventions, vim into get_scores.py and change the path. The output will be a file named &quot;scores_round_0.txt&quot;. For iterative rounds, pass increasing numbers into the command line. i.e. When docking the first round of prioritized molecules, pass [1] for scores_round_1.txt. <br /> <br /> '''4. Convert scores and molecule IDS into NumPY arrays for ChemSTEP recognition. Requires ChemSTEP venv'''<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/convert_scores_to_npy.py .<br /> python convert_scores_to_npy.py 0<br /> <br /> This script expects a txt file with Molecule IDS and DOCK scores. Use the same number you used in step 3. As run above, this will output two files named scores_round_0.npy and indices_round_0.npy that contain line-matched molecule IDs (indices) and their respective docking scores. <br /> <br /> '''5. Enter into or make a directory to run ChemSTEP in. Copy in necessary files for initiating ChemSTEP: params.txt, run_chemstep.py and launch_chemstep_as_job.sh, your score and indices numpy files.'''<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/params.txt .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/run_chemstep.py .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/launch_chemstep_as_job.sh .<br /> <br /> <br /> '''6. Edit params.txt file''' This ONLY needs to be edited for the initial round of ChemSTEP. The parameters outlined here will be carried through the round of ChemSTEP chaining. <br /> seed_indices_file: /absoulte/path/to/your/indices_round_0.npy<br /> seed_scores_file: /absolute/path/to/your/scores_round_0.npy<br /> hit_pprop: 5<br /> n_docked_per_round: 10000000<br /> max_beacons: 150<br /> max_n_rounds: 250<br /> <br /> Be sure that this file reflects your score and indices files for round zero (the seed set). Define your desired pProp, number of beacons, and number to prioritize per round. There should be no need to edit run_chemstep.py or the SGE wrapper script for the FIRST round of XReal docking. ''If using another virtual library, be sure to update the path in run_chemstep.py to point to your FP library.'' We strongly suggest running ChemSTEP as a job array with 64 CPU slots requested (specified in wrapper). <br /> <br /> '''7. Run ChemSTEP''' with the following command: <br /> qsub launch_chemstep_as_job.sh <br /> <br /> <br /> <br /> Running ChemSTEP successfully will result in the output of a SMILES file within output/complete_info/. You will also get (1) chemstep_algo.log and (2) chemstep_submission.log in the working directory after job submission. chemstep_algo.log contains the running output of the alogorithm in a readable format, with timestamps. chemstep_submission log contains the output of ChemSTEP, as well as any information output by the SGE submission system (python errors, cluster issues, tracebacks if failed). These files will be updated with any information with iterative rounds of ChemSTEP run in the same directory. These molecules should be built, docked, and their scores fed back into ChemSTEP. More detailed instructions below: <br /> <br /> '''8. Build prioritized molecules (DOCK 3.8).''' /taken from docs.docking.org <br /> source /wynton/group/bks/soft/DOCK-3.8.5/env.sh<br /> <br /> python /wynton/group/bks/soft/DOCK-3.8.5/DOCK3.8/zinc22-3d/submit/submit_building_docker.py --output_folder building_output --bundle_size 1000 --minutes_per_mol 1 --skip_name_check --scheduler sge --container_software apptainer --container_path_or_name /wynton/group/bks/soft/DOCK-3.8.5/building_pipeline.sif smi_round_1.smi<br /> <br /> When building has completed, you must write an SDI file with the complete paths to each built bundle and dock. '''Be sure to change the INDOCK file to save only poses that meet your score pProp score threshold (output by ChemSTEP)!''' Retrieve docking scores as convert to NumPy arrays as outlined above, update the round number when running get_scores.py and convert_scores_to_npy.py! Copy new score and indices files into the directory you ran ChemSTEP in. If you are following along as a tutorial, you should have scores_round_1.npy and indices_round_1.npy from the previous step (from FIRST round of ChemSTEP prioritization). <br /> <br /> <br /> '''9. Set up for iterative rounds of ChemSTEP'''<br /> <br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/run_chemstep_iterative.py .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/launch_chemstep_iterative.sh .<br /> <br /> <br /> '''10. Run ChemSTEP'''<br /> qsub launch_chemstep_as_job.sh [round number]<br /> <br /> For the first iterative round, the round number is [2], and should increase by one for every subsequent round of ChemSTEP. The output will be smi_round_****.smi file. Repeat steps 8-10 for as many rounds as needed. The performance is reported in outputy/complete_info/run_summary.df, which contains the number of beacons selected, the number of molecules docked, the number of hits found, the distance threshold for the selected molecules to dock, and the last added beacon's distance to all previous beacons.</div>

Kholland http://wiki.docking.org/index.php?title=Running_ChemSTEP&diff=16859 2025-09-03T22:38:58Z

<p>Kholland: </p> <hr /> <div>last update: Sept 3 2025 katie<br /> <br /> ChemSTEP (Chemical Space Traversal and Exploration Procedure) is an open-source, transparent acceleration algorithm for molecular docking capable of dealing with virtual libraries of several trillion compounds. This wiki page is a guide for BKS lab members to run ChemSTEP on Wynton HPC, using the current version of InifiSee XReal library (1.1T). For more general use directions, please refer to [ChemSTEP Read-the-Docs]. <br /> <br /> At a high-level, ChemSTEP is an iterative process to run in between rounds of docking. The general procedure is as follows: build molecules, dock molecules, convert scores for ChemSTEP, run ChemSTEP. In this case, the first round of building (the &quot;seed set&quot;) has already been done.<br /> <br /> <br /> What you (the user) need: DOCKFILES, directories for (1) docking (2) building and (3) running ChemSTEP<br /> <br /> '''1. Source ChemSTEP virtual environment on Wynton'''<br /> source /wynton/group/bks/work/shared/kholland/chemstep_env/bin/activate<br /> <br /> <br /> '''2. Copy InfiniSee seed-set SDI file into your docking directory'''<br /> This directory should already contain your dockfiles, with INDOCK parameters set to your liking. In this step, we are copying in a split database index file (SDI) containing paths to bundles of db2 files. This seed set contains 100 million molecules sampled randomly from the total virtual library, currently 1.1 trillion molecules. <br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/XR_00_seed_set.wynton.sdi .<br /> <br /> '''3. Dock seed set to your receptor of interest using DOCK 3.8''' /docking directions taken from docs.docking.org. This is meant to be done as you would do a normal LSD. <br /> export MOLECULES_DIR_TO_BIND=[outermost folder containing the molecules to dock]<br /> export DOCKFILES=[path to your dockfiles]<br /> export INPUT_FOLDER=[the folder containing your .sdi file(s)]<br /> export OUTPUT_FOLDER=[where you want the output ]<br /> <br /> /wynton/group/bks/work/bwhall61/needs_github/super_dock3r.sh<br /> <br /> Wait for docking to complete. Next, you must extract all molecule IDs and corresponding DOCK scores from above. To do so, run the following commands in the base docking directory (containing your docking files and output folder) while logged into a dev node:<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/get_scores.py .<br /> python3 get_scores.py 0<br /> <br /> <br /> This script expects a directory named &quot;output*&quot; within the CWD. If your output from docking follows different naming conventions, vim into get_scores.py and change the path. The output will be a file named &quot;scores_round_0.txt&quot;. For iterative rounds, pass increasing numbers into the command line. i.e. When docking the first round of prioritized molecules, pass [1] for scores_round_1.txt. <br /> <br /> '''4. Convert scores and molecule IDS into NumPY arrays for ChemSTEP recognition. Requires ChemSTEP venv'''<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/convert_scores_to_npy.py .<br /> python convert_scores_to_npy.py 0<br /> <br /> This script expects a txt file with Molecule IDS and DOCK scores. Use the same number you used in step 3. As run above, this will output two files named scores_round_0.npy and indices_round_0.npy that contain line-matched molecule IDs (indices) and their respective docking scores. <br /> <br /> '''5. Enter into or make a directory to run ChemSTEP in. Copy in necessary files for initiating ChemSTEP: params.txt, run_chemstep.py and launch_chemstep_as_job.sh, your score and indices numpy files.'''<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/params.txt .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/run_chemstep.py .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/launch_chemstep_as_job.sh .<br /> <br /> <br /> '''6. Edit params.txt file''' This ONLY needs to be edited for the initial round of ChemSTEP. The parameters outlined here will be carried through the round of ChemSTEP chaining. <br /> seed_indices_file: /absoulte/path/to/your/indices_round_0.npy<br /> seed_scores_file: /absolute/path/to/your/scores_round_0.npy<br /> hit_pprop: 5<br /> n_docked_per_round: 10000000<br /> max_beacons: 150<br /> max_n_rounds: 250<br /> <br /> Be sure that this file reflects your score and indices files for round zero (the seed set). Define your desired pProp, number of beacons, and number to prioritize per round. There should be no need to edit run_chemstep.py or the SGE wrapper script for the FIRST round of XReal docking. ''If using another virtual library, be sure to update the path in run_chemstep.py to point to your FP library.'' We strongly suggest running ChemSTEP as a job array with 64 CPU slots requested (specified in wrapper). <br /> <br /> '''7. Run ChemSTEP''' with the following command: <br /> qsub launch_chemstep_as_job.sh <br /> <br /> <br /> When finished, there will be a smi_round_1.smi inside of output/complete_info. These molecules should be built, docked, and their scores fed back into ChemSTEP. More detailed instructions below: <br /> <br /> '''8. Build prioritized molecules (DOCK 3.8).''' /taken from docs.docking.org <br /> source /wynton/group/bks/soft/DOCK-3.8.5/env.sh<br /> <br /> python /wynton/group/bks/soft/DOCK-3.8.5/DOCK3.8/zinc22-3d/submit/submit_building_docker.py --output_folder building_output --bundle_size 1000 --minutes_per_mol 1 --skip_name_check --scheduler sge --container_software apptainer --container_path_or_name /wynton/group/bks/soft/DOCK-3.8.5/building_pipeline.sif smi_round_1.smi<br /> <br /> When building has completed, you must write an SDI file with the complete paths to each built bundle and dock. '''Be sure to change the INDOCK file to save only poses that meet your score pProp score threshold (output by ChemSTEP)!''' Retrieve docking scores as convert to NumPy arrays as outlined above, update the round number when running get_scores.py and convert_scores_to_npy.py! Copy new score and indices files into the directory you ran ChemSTEP in. If you are following along as a tutorial, you should have scores_round_1.npy and indices_round_1.npy from the previous step (from FIRST round of ChemSTEP prioritization). <br /> <br /> <br /> '''9. Set up for iterative rounds of ChemSTEP'''<br /> <br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/run_chemstep_iterative.py .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/launch_chemstep_iterative.sh .<br /> <br /> <br /> '''10. Run ChemSTEP'''<br /> qsub launch_chemstep_as_job.sh [round number]<br /> <br /> For the first iterative round, the round number is [2], and should increase by one for every subsequent round of ChemSTEP. The output will be smi_round_****.smi file. Repeat steps 8-10 for as many rounds as needed. The performance is reported in outputy/complete_info/run_summary.df, which contains the number of beacons selected, the number of molecules docked, the number of hits found, the distance threshold for the selected molecules to dock, and the last added beacon's distance to all previous beacons.</div>

Kholland http://wiki.docking.org/index.php?title=Running_ChemSTEP&diff=16858 2025-09-03T22:37:08Z

<p>Kholland: </p> <hr /> <div>last update: August 26 2025 katie<br /> <br /> ChemSTEP (Chemical Space Traversal and Exploration Procedure) is an open-source, transparent acceleration algorithm for molecular docking capable of dealing with virtual libraries of several trillion compounds. This wiki page is a guide for BKS lab members to run ChemSTEP on Wynton HPC, using the current version of InifiSee XReal library (1.1T). For more general use directions, please refer to [ChemSTEP Read-the-Docs]. <br /> <br /> At a high-level, ChemSTEP is an iterative process to run in between rounds of docking. The general procedure is as follows: build molecules, dock molecules, convert scores for ChemSTEP, run ChemSTEP. In this case, the first round of building (the &quot;seed set&quot;) has already been done.<br /> <br /> <br /> What you (the user) need: DOCKFILES, directories for (1) docking (2) building and (3) running ChemSTEP<br /> <br /> '''1. Source ChemSTEP virtual environment on Wynton'''<br /> source /wynton/group/bks/work/shared/kholland/chemstep_env/bin/activate<br /> <br /> <br /> '''2. Copy InfiniSee seed-set SDI file into your docking directory'''<br /> This directory should already contain your dockfiles, with INDOCK parameters set to your liking. In this step, we are copying in a split database index file (SDI) containing paths to bundles of db2 files. This seed set contains 100 million molecules sampled randomly from the total virtual library, currently 1.1 trillion molecules. <br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/XR_00_seed_set.wynton.sdi .<br /> <br /> '''3. Dock seed set to your receptor of interest using DOCK 3.8''' /docking directions taken from docs.docking.org. This is meant to be done as you would do a normal LSD. <br /> export MOLECULES_DIR_TO_BIND=[outermost folder containing the molecules to dock]<br /> export DOCKFILES=[path to your dockfiles]<br /> export INPUT_FOLDER=[the folder containing your .sdi file(s)]<br /> export OUTPUT_FOLDER=[where you want the output ]<br /> <br /> /wynton/group/bks/work/bwhall61/needs_github/super_dock3r.sh<br /> <br /> Wait for docking to complete. Next, you must extract all molecule IDs and corresponding DOCK scores from above. To do so, run the following commands in the base docking directory (containing your docking files and output folder) while logged into a dev node:<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/get_scores.py .<br /> python3 get_scores.py 0<br /> <br /> <br /> This script expects a directory named &quot;output*&quot; within the CWD. If your output from docking follows different naming conventions, vim into get_scores.py and change the path. The output will be a file named &quot;scores_round_0.txt&quot;. For iterative rounds, pass increasing numbers into the command line. i.e. When docking the first round of prioritized molecules, pass [1] for scores_round_1.txt. <br /> <br /> '''4. Convert scores and molecule IDS into NumPY arrays for ChemSTEP recognition. Requires ChemSTEP venv'''<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/convert_scores_to_npy.py .<br /> python convert_scores_to_npy.py 0<br /> <br /> This script expects a txt file with Molecule IDS and DOCK scores. Use the same number you used in step 3. As run above, this will output two files named scores_round_0.npy and indices_round_0.npy that contain line-matched molecule IDs (indices) and their respective docking scores. <br /> <br /> '''5. Enter into or make a directory to run ChemSTEP in. Copy in necessary files for initiating ChemSTEP: params.txt, run_chemstep.py and launch_chemstep_as_job.sh, your score and indices numpy files.'''<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/params.txt .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/run_chemstep.py .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/launch_chemstep_as_job.sh .<br /> <br /> <br /> '''6. Edit params.txt file''' This ONLY needs to be edited for the initial round of ChemSTEP. The parameters outlined here will be carried through the round of ChemSTEP chaining. <br /> seed_indices_file: /absoulte/path/to/your/indices_round_0.npy<br /> seed_scores_file: /absolute/path/to/your/scores_round_0.npy<br /> hit_pprop: 5<br /> n_docked_per_round: 10000000<br /> max_beacons: 150<br /> max_n_rounds: 250<br /> <br /> Be sure that this file reflects your score and indices files for round zero (the seed set). Define your desired pProp, number of beacons, and number to prioritize per round. There should be no need to edit run_chemstep.py or the SGE wrapper script for the FIRST round of XReal docking. ''If using another virtual library, be sure to update the path in run_chemstep.py to point to your FP library.'' We strongly suggest running ChemSTEP as a job array with 64 CPU slots requested (specified in wrapper). <br /> <br /> '''7. Run ChemSTEP''' with the following command: <br /> qsub launch_chemstep_as_job.sh <br /> <br /> <br /> When finished, there will be a smi_round_1.smi inside of output/complete_info. These molecules should be built, docked, and their scores fed back into ChemSTEP. More detailed instructions below: <br /> <br /> '''8. Build prioritized molecules (DOCK 3.8).''' /taken from docs.docking.org <br /> source /wynton/group/bks/soft/DOCK-3.8.5/env.sh<br /> <br /> python /wynton/group/bks/soft/DOCK-3.8.5/DOCK3.8/zinc22-3d/submit/submit_building_docker.py --output_folder building_output --bundle_size 1000 --minutes_per_mol 1 --skip_name_check --scheduler sge --container_software apptainer --container_path_or_name /wynton/group/bks/soft/DOCK-3.8.5/building_pipeline.sif smi_round_1.smi<br /> <br /> When building has completed, you must write an SDI file with the complete paths to each built bundle and dock. '''Be sure to change the INDOCK file to save only poses that meet your score pProp score threshold (output by ChemSTEP)!''' Retrieve docking scores as convert to NumPy arrays as outlined above, update the round number when running get_scores.py and convert_scores_to_npy.py! Copy new score and indices files into the directory you ran ChemSTEP in. If you are following along as a tutorial, you should have scores_round_1.npy and indices_round_1.npy from the previous step (from FIRST round of ChemSTEP prioritization). Copy these into your ChemSTEP working directory.<br /> <br /> <br /> '''9. Set up for iterative rounds of ChemSTEP'''<br /> <br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/run_chemstep_iterative.py .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/launch_chemstep_iterative.sh .<br /> <br /> <br /> '''10. Run ChemSTEP'''<br /> qsub launch_chemstep_as_job.sh [round number]<br /> <br /> For the first iterative round, the round number is [2], and should increase by one for every subsequent round of ChemSTEP. The output will be smi_round_****.smi file. Repeat steps 8-10 for as many rounds as needed. The performance is reported in outputy/complete_info/run_summary.df, which contains the number of beacons selected, the number of molecules docked, the number of hits found, the distance threshold for the selected molecules to dock, and the last added beacon's distance to all previous beacons.</div>

Kholland http://wiki.docking.org/index.php?title=Running_ChemSTEP&diff=16850 2025-08-27T00:05:54Z

<p>Kholland: </p> <hr /> <div>last update: August 26 2025 katie<br /> <br /> ChemSTEP (Chemical Space Traversal and Exploration Procedure) is an open-source, transparent acceleration algorithm for molecular docking capable of dealing with virtual libraries of several trillion compounds. This wiki page is a guide for BKS lab members to run ChemSTEP on Wynton HPC, using the current version of InifiSee XReal library (1.1T). For more general use directions, please refer to [ChemSTEP Read-the-Docs]. <br /> <br /> At a high-level, ChemSTEP is an iterative process to run in between rounds of docking. The general procedure is as follows: build molecules, dock molecules, convert scores for ChemSTEP, run ChemSTEP. In this case, the first round of building (the &quot;seed set&quot;) has already been done.<br /> <br /> <br /> What you (the user) need: DOCKFILES, directories for (1) docking (2) building and (3) running ChemSTEP<br /> <br /> '''1. Source ChemSTEP virtual environment on Wynton'''<br /> source /wynton/group/bks/work/shared/kholland/chemstep_env/bin/activate<br /> <br /> <br /> '''2. Copy InfiniSee seed-set SDI file into your docking directory'''<br /> This directory should already contain your dockfiles, with INDOCK parameters set to your liking. In this step, we are copying in a split database index file (SDI) containing paths to bundles of db2 files. This seed set contains 100 million molecules sampled randomly from the total virtual library, currently 1.1 trillion molecules. <br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/XR_00_seed_set.wynton.sdi .<br /> <br /> '''3. Dock seed set to your receptor of interest using DOCK 3.8''' /docking directions taken from docs.docking.org. This is meant to be done as you would do a normal LSD. <br /> export MOLECULES_DIR_TO_BIND=[outermost folder containing the molecules to dock]<br /> export DOCKFILES=[path to your dockfiles]<br /> export INPUT_FOLDER=[the folder containing your .sdi file(s)]<br /> export OUTPUT_FOLDER=[where you want the output ]<br /> <br /> /wynton/group/bks/work/bwhall61/needs_github/super_dock3r.sh<br /> <br /> Wait for docking to complete. Next, you must extract all molecule IDs and corresponding DOCK scores from above. To do so, run the following commands in the base docking directory (containing your docking files and output folder) while logged into a dev node:<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/get_scores.py .<br /> python3 get_scores.py 0<br /> <br /> <br /> This script expects a directory named &quot;output*&quot; within the CWD. If your output from docking follows different naming conventions, vim into get_scores.py and change the path. The output will be a file named &quot;scores_round_0.txt&quot;. For iterative rounds, pass increasing numbers into the command line. i.e. When docking the first round of prioritized molecules, pass [1] for scores_round_1.txt. <br /> <br /> '''4. Convert scores and molecule IDS into NumPY arrays for ChemSTEP recognition.'''<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/convert_scores_to_npy.py .<br /> python3 convert_scores_to_npy.py 0<br /> <br /> This script expects a txt file with Molecule IDS and DOCK scores. Use the same number you used in step 3. As run above, this will output two files named scores_round_0.npy and indices_round_0.npy that contain line-matched molecule IDs (indices) and their respective docking scores. <br /> <br /> '''5. Enter into or make a directory to run ChemSTEP in. Copy in necessary files: params.txt, run_chemstep.py and launch_chemstep_as_job.sh, your score and indices numpy files.'''<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/params.txt .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/run_chemstep.py .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/launch_chemstep_as_job.sh .<br /> <br /> <br /> '''6. Edit params.txt file''' This ONLY needs to be edited for the initial round of ChemSTEP. The parameters outlined here will be carried through the round of ChemSTEP chaining. <br /> seed_indices_file: /absoulte/path/to/your/indices_round_0.npy<br /> seed_scores_file: /absolute/path/to/your/scores_round_0.npy<br /> hit_pprop: 5<br /> n_docked_per_round: 10000000<br /> max_beacons: 150<br /> max_n_rounds: 250<br /> <br /> Be sure that this file reflects your score and indices files for round zero (the seed set). Define your desired pProp, number of beacons, and number to prioritize per round. There should be no need to edit run_chemstep.py or the SGE wrapper script for the FIRST round of XReal docking. ''If using another virtual library, be sure to update the path in run_chemstep.py to point to your FP library.'' We strongly suggest running ChemSTEP as a job array with 64 CPU slots requested (specified in wrapper). <br /> <br /> '''7. Run ChemSTEP''' with the following command: <br /> qsub launch_chemstep_as_job.sh <br /> <br /> <br /> When finished, there will be a smi_round_1.smi inside of output/complete_info. These molecules should be built, docked, and their scores fed back into ChemSTEP. More detailed instructions below: <br /> <br /> '''8. Build prioritized molecules (DOCK 3.8).''' /taken from docs.docking.org <br /> source /wynton/group/bks/soft/DOCK-3.8.5/env.sh<br /> <br /> python /wynton/group/bks/soft/DOCK-3.8.5/DOCK3.8/zinc22-3d/submit/submit_building_docker.py --output_folder building_output --bundle_size 1000 --minutes_per_mol 1 --skip_name_check --scheduler sge --container_software apptainer --container_path_or_name /wynton/group/bks/soft/DOCK-3.8.5/building_pipeline.sif smi_round_1.smi<br /> <br /> When building has completed, you must write an SDI file with the complete paths to each built bundle and dock. '''Be sure to change the INDOCK file to save only poses that meet your score pProp score threshold (output by ChemSTEP)!''' Retrieve docking scores as convert to NumPy arrays as outlined above, update the round number when running get_scores.py and convert_scores_to_npy.py! Copy new score and indices files into the directory you ran ChemSTEP in. If you are following along as a tutorial, you should have scores_round_1.npy and indices_round_1.npy from the previous step (from FIRST round of ChemSTEP prioritization). <br /> <br /> <br /> '''9. Set up for iterative rounds of ChemSTEP'''<br /> <br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/run_chemstep_iterative.py .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/launch_chemstep_iterative.sh .<br /> <br /> <br /> '''10. Run ChemSTEP'''<br /> qsub launch_chemstep_as_job.sh [round number]<br /> <br /> For the first iterative round, the round number is [2], and should increase by one for every subsequent round of ChemSTEP. The output will be smi_round_****.smi file. Repeat steps 8-10 for as many rounds as needed. The performance is reported in outputy/complete_info/run_summary.df, which contains the number of beacons selected, the number of molecules docked, the number of hits found, the distance threshold for the selected molecules to dock, and the last added beacon's distance to all previous beacons.</div>

Kholland http://wiki.docking.org/index.php?title=Running_ChemSTEP&diff=16849 2025-08-27T00:02:23Z

<p>Kholland: </p> <hr /> <div>last update: August 20 2025 katie<br /> <br /> ChemSTEP (Chemical Space Traversal and Exploration Procedure) is an open-source, transparent acceleration algorithm for molecular docking capable of dealing with virtual libraries of several trillion compounds. This wiki page is a guide for BKS lab members to run ChemSTEP on Wynton HPC, using the current version of InifiSee XReal library (1.1T). For more general use directions, please refer to [ChemSTEP Read-the-Docs]. <br /> <br /> At a high-level, ChemSTEP is an iterative process to run in between rounds of docking. The general procedure is as follows: build molecules, dock molecules, convert scores for ChemSTEP, run ChemSTEP. In this case, the first round of building (the &quot;seed set&quot;) has already been done.<br /> <br /> <br /> What you (the user) need: DOCKFILES, directories for (1) docking (2) building and (3) running ChemSTEP<br /> <br /> '''1. Source ChemSTEP virtual environment on Wynton'''<br /> source /wynton/group/bks/work/shared/kholland/chemstep_env/bin/activate<br /> <br /> <br /> '''2. Copy InfiniSee seed-set SDI file into your docking directory'''<br /> This directory should already contain your dockfiles, with INDOCK parameters set to your liking. In this step, we are copying in a split database index file (SDI) containing paths to bundles of db2 files. This seed set contains 100 million molecules sampled randomly from the total virtual library, currently 1.1 trillion molecules. <br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/XR_00_seed_set.wynton.sdi .<br /> <br /> '''3. Dock seed set to your receptor of interest using DOCK 3.8''' /docking directions taken from docs.docking.org. This is meant to be done as you would do a normal LSD. <br /> export MOLECULES_DIR_TO_BIND=[outermost folder containing the molecules to dock]<br /> export DOCKFILES=[path to your dockfiles]<br /> export INPUT_FOLDER=[the folder containing your .sdi file(s)]<br /> export OUTPUT_FOLDER=[where you want the output ]<br /> <br /> /wynton/group/bks/work/bwhall61/needs_github/super_dock3r.sh<br /> <br /> Wait for docking to complete. Next, you must extract all molecule IDs and corresponding DOCK scores from above. To do so, run the following commands in the base docking directory (containing your docking files and output folder) while logged into a dev node:<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/get_scores.py .<br /> python3 get_scores.py 0<br /> <br /> <br /> This script expects a directory named &quot;output*&quot; within the CWD. If your output from docking follows different naming conventions, vim into get_scores.py and change the path. The output will be a file named &quot;scores_round_0.txt&quot;. For iterative rounds, pass increasing numbers into the command line. i.e. When docking the first round of prioritized molecules, pass [1] for scores_round_1.txt. <br /> <br /> '''4. Convert scores and molecule IDS into NumPY arrays for ChemSTEP recognition.'''<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/convert_scores_to_npy.py .<br /> python3 convert_scores_to_npy.py 0<br /> <br /> This script expects a txt file with Molecule IDS and DOCK scores. Use the same number you used in step 3. As run above, this will output two files named scores_round_0.npy and indices_round_0.npy that contain line-matched molecule IDs (indices) and their respective docking scores. <br /> <br /> '''5. Enter into or make a directory to run ChemSTEP in. Copy in necessary files: params.txt, run_chemstep.py and launch_chemstep_as_job.sh, your score and indices numpy files.'''<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/params.txt .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/run_chemstep.py .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/launch_chemstep_as_job.sh .<br /> <br /> <br /> '''6. Edit params.txt file''' This ONLY needs to be edited for the initial round of ChemSTEP. The parameters outlined here will be carried through the round of ChemSTEP chaining. <br /> seed_indices_file: /absoulte/path/to/your/indices_round_0.npy<br /> seed_scores_file: /absolute/path/to/your/scores_round_0.npy<br /> hit_pprop: 5<br /> n_docked_per_round: 10000000<br /> max_beacons: 150<br /> max_n_rounds: 250<br /> <br /> Be sure that this file reflects your score and indices files for round zero (the seed set). Define your desired pProp, number of beacons, and number to prioritize per round. There should be no need to edit run_chemstep.py or the SGE wrapper script for the FIRST round of XReal docking. ''If using another virtual library, be sure to update the path in run_chemstep.py to point to your FP library.'' We strongly suggest running ChemSTEP as a job array with 64 CPU slots requested (specified in wrapper). <br /> <br /> '''7. Run ChemSTEP''' with the following command: <br /> qsub launch_chemstep_as_job.sh <br /> <br /> <br /> When finished, there will be a smi_round_1.smi inside of output/complete_info. These molecules should be built, docked, and their scores fed back into ChemSTEP. More detailed instructions below: <br /> <br /> '''8. Build prioritized molecules (DOCK 3.8).''' /taken from docs.docking.org <br /> source /wynton/group/bks/soft/DOCK-3.8.5/env.sh<br /> <br /> python /wynton/group/bks/soft/DOCK-3.8.5/DOCK3.8/zinc22-3d/submit/submit_building_docker.py --output_folder building_output --bundle_size 1000 --minutes_per_mol 1 --skip_name_check --scheduler sge --container_software apptainer --container_path_or_name /wynton/group/bks/soft/DOCK-3.8.5/building_pipeline.sif smi_round_1.smi<br /> <br /> When building has completed, you must write an SDI file with the complete paths to each built bundle and dock. '''Be sure to change the INDOCK file to save only poses that meet your score pProp score threshold (output by ChemSTEP)!''' Retrieve docking scores as convert to NumPy arrays as outlined above, update the round number when running get_scores.py and convert_scores_to_npy.py! Copy new score and indices files into the directory you ran ChemSTEP in. If you are following along as a tutorial, you should have scores_round_1.npy and indices_round_1.npy from the previous step (from FIRST round of ChemSTEP prioritization). <br /> <br /> <br /> '''9. Set up for iterative rounds of ChemSTEP'''<br /> <br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/run_chemstep_iterative.py .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/launch_chemstep_iterative.sh .<br /> <br /> <br /> '''10. Run ChemSTEP'''<br /> qsub launch_chemstep_as_job.sh [round number]<br /> <br /> For the first iterative round, the round number is [2], and should increase by one for every subsequent round of ChemSTEP. The output will be smi_round_****.smi file. Repeat steps 8-10 for as many rounds as needed. The performance is reported in outputy/complete_info/run_summary.df, which contains the number of beacons selected, the number of molecules docked, the number of hits found, the distance threshold for the selected molecules to dock, and the last added beacon's distance to all previous beacons.</div>

Kholland http://wiki.docking.org/index.php?title=Running_ChemSTEP&diff=16848 2025-08-26T23:55:56Z

<p>Kholland: </p> <hr /> <div>last update: August 20 2025 katie<br /> <br /> ChemSTEP (Chemical Space Traversal and Exploration Procedure) is an open-source, transparent acceleration algorithm for molecular docking capable of dealing with virtual libraries of several trillion compounds. This wiki page is a guide for BKS lab members to run ChemSTEP on Wynton HPC, using the current version of InifiSee XReal library (1.1T). For more general use directions, please refer to [ChemSTEP Read-the-Docs]. <br /> <br /> At a high-level, ChemSTEP is an iterative process to run in between rounds of docking. The general procedure is as follows: build molecules, dock molecules, convert scores for ChemSTEP, run ChemSTEP. In this case, the first round of building (the &quot;seed set&quot;) has already been done.<br /> <br /> <br /> What you (the user) need: DOCKFILES, directories for (1) docking (2) building and (3) running ChemSTEP<br /> <br /> '''1. Source ChemSTEP virtual environment on Wynton'''<br /> source /wynton/group/bks/work/shared/kholland/chemstep_env/bin/activate<br /> <br /> <br /> '''2. Copy InfiniSee seed-set SDI file into your docking directory'''<br /> This directory should already contain your dockfiles, with INDOCK parameters set to your liking. In this step, we are copying in a split database index file (SDI) containing paths to bundles of db2 files. This seed set contains 100 million molecules sampled randomly from the total virtual library, currently 1.1 trillion molecules. <br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/XR_00_seed_set.wynton.sdi .<br /> <br /> '''3. Dock seed set to your receptor of interest using DOCK 3.8''' /docking directions taken from docs.docking.org. This is meant to be done as you would do a normal LSD. <br /> export MOLECULES_DIR_TO_BIND=[outermost folder containing the molecules to dock]<br /> export DOCKFILES=[path to your dockfiles]<br /> export INPUT_FOLDER=[the folder containing your .sdi file(s)]<br /> export OUTPUT_FOLDER=[where you want the output ]<br /> <br /> /wynton/group/bks/work/bwhall61/needs_github/super_dock3r.sh<br /> <br /> Wait for docking to complete. Next, you must extract all molecule IDs and corresponding DOCK scores from above. To do so, run the following commands in the base docking directory (containing your docking files and output folder) while logged into a dev node:<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/get_scores.py .<br /> python3 get_scores.py 0<br /> <br /> <br /> This script expects a directory named &quot;output*&quot; within the CWD. If your output from docking follows different naming conventions, vim into get_scores.py and change the path. The output will be a file named &quot;scores_round_0.txt&quot;. For iterative rounds, pass increasing numbers into the command line. i.e. When docking the first round of prioritized molecules, pass [1] for scores_round_1.txt. <br /> <br /> '''4. Convert scores and molecule IDS into NumPY arrays for ChemSTEP recognition.'''<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/convert_scores_to_npy.py .<br /> python3 convert_scores_to_npy.py 0<br /> <br /> This script expects a txt file with Molecule IDS and DOCK scores. Use the same number you used in step 3. As run above, this will output two files named scores_round_0.npy and indices_round_0.npy that contain line-matched molecule IDs (indices) and their respective docking scores. <br /> <br /> '''5. Enter into or make a directory to run ChemSTEP in. Copy in necessary files: params.txt, run_chemstep.py and launch_chemstep_as_job.sh, your score and indices numpy files.'''<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/params.txt .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/run_chemstep.py .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/launch_chemstep_as_job.sh .<br /> <br /> <br /> '''6. Edit params.txt file''' This ONLY needs to be edited for the initial round of ChemSTEP. The parameters outlined here will be carried through the round of ChemSTEP chaining. <br /> seed_indices_file: /absoulte/path/to/your/indices_round_0.npy<br /> seed_scores_file: /absolute/path/to/your/scores_round_0.npy<br /> hit_pprop: 5<br /> n_docked_per_round: 10000000<br /> max_beacons: 150<br /> max_n_rounds: 250<br /> <br /> Be sure that this file reflects your score and indices files for round zero (the seed set). Define your desired pProp, number of beacons, and number to prioritize per round. There should be no need to edit run_chemstep.py or the SGE wrapper script for the FIRST round of XReal docking. ''If using another virtual library, be sure to update the path in run_chemstep.py to point to your FP library.'' We strongly suggest running ChemSTEP as a job array with 64 CPU slots requested (specified in wrapper). <br /> <br /> '''7. Run ChemSTEP''' with the following command: <br /> qsub launch_chemstep_as_job.sh <br /> <br /> <br /> When finished, there will be a smi_round_1.smi inside of output/complete_info. These molecules should be built, docked, and their scores fed back into ChemSTEP. More detailed instructions below: <br /> <br /> '''8. Build prioritized molecules (DOCK 3.8).''' /taken from docs.docking.org <br /> source /wynton/group/bks/soft/DOCK-3.8.5/env.sh<br /> <br /> python /wynton/group/bks/soft/DOCK-3.8.5/DOCK3.8/zinc22-3d/submit/submit_building_docker.py --output_folder building_output --bundle_size 1000 --minutes_per_mol 1 --skip_name_check --scheduler sge --container_software apptainer --container_path_or_name /wynton/group/bks/soft/DOCK-3.8.5/building_pipeline.sif smi_round_1.smi<br /> <br /> When building has completed, you must write an SDI file with the complete paths to each built bundle and dock. '''Be sure to change the INDOCK file to save only poses that meet your score pProp score threshold (output by ChemSTEP)!''' Retrieve docking scores as convert to NumPy arrays as outlined above, update the round number when running get_scores.py and convert_scores_to_npy.py! Copy new score and indices files into the directory you ran ChemSTEP in. If you are following along as a tutorial, you should have scores_round_1.npy and indices_round_1.npy from the previous step (from FIRST round of ChemSTEP prioritization). <br /> <br /> <br /> '''9. Set up for iterative rounds of ChemSTEP'''<br /> <br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/run_chemstep_iterative.py .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/launch_chemstep_iterative.sh .<br /> <br /> <br /> '''10. Run ChemSTEP'''<br /> qsub launch_chemstep_as_job.sh [round number]<br /> <br /> For the first iterative round, the round number is [2], increasing by one for every subsequent round of ChemSTEP. The output will be smi_round_****.smi file. Repeat steps 8-10 for as many rounds as needed. The performance is reported in outputy/complete_info/run_summary.df, which contains the number of beacons selected, the number of molecules docked, the number of hits found, the distance threshold for the selected molecules to dock, and the last added beacon's distance to all previous beacons.</div>

Kholland http://wiki.docking.org/index.php?title=Running_ChemSTEP&diff=16847 2025-08-26T23:54:16Z

<p>Kholland: </p> <hr /> <div>last update: August 20 2025 katie<br /> <br /> ChemSTEP (Chemical Space Traversal and Exploration Procedure) is an open-source, transparent acceleration algorithm for molecular docking capable of dealing with virtual libraries of several trillion compounds. This wiki page is a guide for BKS lab members to run ChemSTEP on Wynton HPC, using the current version of InifiSee XReal library (1.1T). For more general use directions, please refer to [ChemSTEP Read-the-Docs]. <br /> <br /> At a high-level, ChemSTEP is an iterative process to run in between rounds of docking. The general procedure is as follows: build molecules, dock molecules, convert scores for ChemSTEP, run ChemSTEP. In this case, the first round of building (the &quot;seed set&quot;) has already been done.<br /> <br /> <br /> What you (the user) need: DOCKFILES, directories for (1) docking (2) building and (3) running ChemSTEP<br /> <br /> '''1. Source ChemSTEP virtual environment on Wynton'''<br /> source /wynton/group/bks/work/shared/kholland/chemstep_env/bin/activate<br /> <br /> <br /> '''2. Copy InfiniSee seed-set SDI file into your docking directory'''<br /> This directory should already contain your dockfiles, with INDOCK parameters set to your liking. In this step, we are copying in a split database index file (SDI) containing paths to bundles of db2 files. This seed set contains 100 million molecules sampled randomly from the total virtual library, currently 1.1 trillion molecules. <br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/XR_00_seed_set.wynton.sdi .<br /> <br /> '''3. Dock seed set to your receptor of interest using DOCK 3.8''' /docking directions taken from docs.docking.org. This is meant to be done as you would do a normal LSD. <br /> export MOLECULES_DIR_TO_BIND=[outermost folder containing the molecules to dock]<br /> export DOCKFILES=[path to your dockfiles]<br /> export INPUT_FOLDER=[the folder containing your .sdi file(s)]<br /> export OUTPUT_FOLDER=[where you want the output ]<br /> <br /> /wynton/group/bks/work/bwhall61/needs_github/super_dock3r.sh<br /> <br /> Wait for docking to complete. Next, you must extract all molecule IDs and corresponding DOCK scores from above. To do so, run the following commands in the base docking directory (containing your docking files and output folder) while logged into a dev node:<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/get_scores.py .<br /> python3 get_scores.py 0<br /> <br /> <br /> This script expects a directory named &quot;output*&quot; within the CWD. If your output from docking follows different naming conventions, vim into get_scores.py and change the path. The output will be a file named &quot;scores_round_0.txt&quot;. For iterative rounds, pass increasing numbers into the command line. i.e. When docking the first round of prioritized molecules, pass [1] for scores_round_1.txt. <br /> <br /> '''4. Convert scores and molecule IDS into NumPY arrays for ChemSTEP recognition.'''<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/convert_scores_to_npy.py .<br /> python3 convert_scores_to_npy.py 0<br /> <br /> This script expects a txt file with Molecule IDS and DOCK scores. Use the same number you used in step 3. As run above, this will output two files named scores_round_0.npy and indices_round_0.npy that contain line-matched molecule IDs (indices) and their respective docking scores. <br /> <br /> '''5. Enter into or make a directory to run ChemSTEP in. Copy in necessary files: params.txt, run_chemstep.py and launch_chemstep_as_job.sh, your score and indices numpy files.'''<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/params.txt .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/run_chemstep.py .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/launch_chemstep_as_job.sh .<br /> <br /> <br /> '''6. Edit params.txt file''' This ONLY needs to be edited for the initial round of ChemSTEP. The parameters outlined here will be carried through the round of ChemSTEP chaining. <br /> seed_indices_file: /absoulte/path/to/your/indices_round_0.npy<br /> seed_scores_file: /absolute/path/to/your/scores_round_0.npy<br /> hit_pprop: 5<br /> n_docked_per_round: 10000000<br /> max_beacons: 150<br /> max_n_rounds: 250<br /> <br /> Be sure that this file reflects your score and indices files for round zero (the seed set). Define your desired pProp, number of beacons, and number to prioritize per round. There should be no need to edit run_chemstep.py or the SGE wrapper script for the FIRST round of XReal docking. ''If using another virtual library, be sure to update the path in run_chemstep.py to point to your FP library.'' We strongly suggest running ChemSTEP as a job array with 64 CPU slots requested (specified in wrapper). <br /> <br /> '''7. Run ChemSTEP''' with the following command: <br /> qsub launch_chemstep_as_job.sh <br /> <br /> <br /> When finished, there will be a smi_round_1.smi inside of output/complete_info. These molecules should be built, docked, and their scores fed back into ChemSTEP. More detailed instructions below: <br /> <br /> '''8. Build prioritized molecules (DOCK 3.8).''' /taken from docs.docking.org <br /> source /wynton/group/bks/soft/DOCK-3.8.5/env.sh<br /> <br /> python /wynton/group/bks/soft/DOCK-3.8.5/DOCK3.8/zinc22-3d/submit/submit_building_docker.py --output_folder building_output --bundle_size 1000 --minutes_per_mol 1 --skip_name_check --scheduler sge --container_software apptainer --container_path_or_name /wynton/group/bks/soft/DOCK-3.8.5/building_pipeline.sif smi_round_1.smi<br /> <br /> When building has completed, you must write an SDI file with the complete paths to each built bundle and dock. '''Be sure to change the INDOCK file to save only poses that meet your score pProp score threshold (output by ChemSTEP)!''' Retrieve docking scores as convert to NumPy arrays as outlined above, update the round number when running get_scores.py and convert_scores_to_npy.py! Copy new score and indices files into the directory you ran ChemSTEP in. If you are following along as a tutorial, you should have scores_round_1.npy and indices_round_1.npy from the previous step (from FIRST round of ChemSTEP prioritization). <br /> <br /> <br /> '''9. Set up for iterative rounds of ChemSTEP'''<br /> <br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/run_chemstep_iterative.py .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/launch_chemstep_iterative.sh<br /> <br /> <br /> '''10. Run ChemSTEP'''<br /> qsub launch_chemstep_as_job.sh [round number]<br /> <br /> For the first iterative round, the round number is [2], increasing by one for every subsequent round of ChemSTEP. The output will be smi_round_****.smi file. Repeat steps 8-10 for as many rounds as needed. The performance is reported in outputy/complete_info/run_summary.df, which contains the number of beacons selected, the number of molecules docked, the number of hits found, the distance threshold for the selected molecules to dock, and the last added beacon's distance to all previous beacons.</div>

Kholland http://wiki.docking.org/index.php?title=Running_ChemSTEP&diff=16846 2025-08-26T23:42:23Z

<p>Kholland: </p> <hr /> <div>last update: August 20 2025 katie<br /> <br /> ChemSTEP (Chemical Space Traversal and Exploration Procedure) is an open-source, transparent acceleration algorithm for molecular docking capable of dealing with virtual libraries of several trillion compounds. This wiki page is a guide for BKS lab members to run ChemSTEP on Wynton HPC, using the current version of InifiSee XReal library (1.1T). For more general use directions, please refer to [ChemSTEP Read-the-Docs]. <br /> <br /> At a high-level, ChemSTEP is an iterative process to run in between rounds of docking. The general procedure is as follows: build molecules, dock molecules, convert scores for ChemSTEP, run ChemSTEP. In this case, the first round of building (the &quot;seed set&quot;) has already been done.<br /> <br /> <br /> What you (the user) need: DOCKFILES, directories for (1) docking (2) building and (3) running ChemSTEP<br /> <br /> '''1. Source ChemSTEP virtual environment on Wynton'''<br /> source /wynton/group/bks/work/shared/kholland/chemstep_env/bin/activate<br /> <br /> <br /> '''2. Copy InfiniSee seed-set SDI file into your docking directory'''<br /> This directory should already contain your dockfiles, with INDOCK parameters set to your liking. In this step, we are copying in a split database index file (SDI) containing paths to bundles of db2 files. This seed set contains 100 million molecules sampled randomly from the total virtual library, currently 1.1 trillion molecules. <br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/XR_00_seed_set.wynton.sdi .<br /> <br /> '''3. Dock seed set to your receptor of interest using DOCK 3.8''' /docking directions taken from docs.docking.org. This is meant to be done as you would do a normal LSD. <br /> export MOLECULES_DIR_TO_BIND=[outermost folder containing the molecules to dock]<br /> export DOCKFILES=[path to your dockfiles]<br /> export INPUT_FOLDER=[the folder containing your .sdi file(s)]<br /> export OUTPUT_FOLDER=[where you want the output ]<br /> <br /> /wynton/group/bks/work/bwhall61/needs_github/super_dock3r.sh<br /> <br /> Wait for docking to complete. Next, you must extract all molecule IDs and corresponding DOCK scores from above. To do so, run the following commands in the base docking directory (containing your docking files and output folder) while logged into a dev node:<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/get_scores.py .<br /> python3 get_scores.py 0<br /> <br /> <br /> This script expects a directory named &quot;output*&quot; within the CWD. If your output from docking follows different naming conventions, vim into get_scores.py and change the path. The output will be a file named &quot;scores_round_0.txt&quot;. For iterative rounds, pass increasing numbers into the command line. i.e. When docking the first round of prioritized molecules, pass [1] for scores_round_1.txt. <br /> <br /> '''4. Convert scores and molecule IDS into NumPY arrays for ChemSTEP recognition.'''<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/convert_scores_to_npy.py .<br /> python3 convert_scores_to_npy.py 0<br /> <br /> This script expects a txt file with Molecule IDS and DOCK scores. Use the same number you used in step 3. As run above, this will output two files named scores_round_0.npy and indices_round_0.npy that contain line-matched molecule IDs (indices) and their respective docking scores. <br /> <br /> '''5. Enter into or make a directory to run ChemSTEP in. Copy in necessary files: params.txt, run_chemstep.py and launch_chemstep_as_job.sh, your score and indices numpy files.'''<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/params.txt .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/run_chemstep.py .<br /> cp /wynton/group/bks/work/shared/kholland/chemstep_v0p0/launch_chemstep_as_job.sh .<br /> <br /> <br /> '''6. Edit params.txt file''' This ONLY needs to be edited for the initial round of ChemSTEP. The parameters outlined here will be carried through the round of ChemSTEP chaining. <br /> seed_indices_file: /absoulte/path/to/your/indices_round_0.npy<br /> seed_scores_file: /absolute/path/to/your/scores_round_0.npy<br /> hit_pprop: 5<br /> n_docked_per_round: 10000000<br /> max_beacons: 150<br /> max_n_rounds: 250<br /> <br /> Be sure that this file reflects your score and indices files for round zero (the seed set). Define your desired pProp, number of beacons, and number to prioritize per round. There should be no need to edit run_chemstep.py or the SGE wrapper script for the FIRST round of XReal docking. ''If using another virtual library, be sure to update the path in run_chemstep.py to point to your FP library.'' We strongly suggest running ChemSTEP as a job array with 64 CPU slots requested (specified in wrapper). <br /> <br /> '''7. Run ChemSTEP''' with the following command: <br /> qsub launch_chemstep_as_job.sh <br /> <br /> <br /> When finished, there will be a smi_round_1.smi inside of output/complete_info. These molecules should be built, docked, and their scores fed back into ChemSTEP. More detailed instructions below: <br /> <br /> '''8. Build prioritized molecules (DOCK 3.8).''' /taken from docs.docking.org <br /> source /wynton/group/bks/soft/DOCK-3.8.5/env.sh<br /> <br /> python /wynton/group/bks/soft/DOCK-3.8.5/DOCK3.8/zinc22-3d/submit/submit_building_docker.py --output_folder building_output --bundle_size 1000 --minutes_per_mol 1 --skip_name_check --scheduler sge --container_software apptainer --container_path_or_name /wynton/group/bks/soft/DOCK-3.8.5/building_pipeline.sif smi_round_1.smi<br /> <br /> When building has completed, you must write an SDI file with the complete paths to each built bundle and dock. '''Be sure to change the INDOCK file to save only poses that meet your score pProp score threshold (output by ChemSTEP)!''' Retrieve docking scores as convert to NumPy arrays as outlined above, update the round number when running get_scores.py and convert_scores_to_npy.py! Copy new score and indices files into the directory you ran ChemSTEP in. If you are following along as a tutorial, you should have scores_round_1.npy and indices_round_1.npy from the previous step (from FIRST round of ChemSTEP prioritization). <br /> <br /> <br /> '''9. Set up for iterative rounds of ChemSTEP'''<br /> <br /> cp /wynton/group/bks/work/shared/kholland/chemsteo_v0p0/run_chemstep_iterative.py .<br /> <br /> vim/nano launch_chemstep_as_job.sh <br /> <br /> Edit the launch_chemstep_as_job.sh wrapper to call. In this example, we now running our second round of prioritization [2]. For additional rounds of ChemSTEP, it is ESSENTIAL to update this number in the job submission script. This will determine the naming convention of the ChemSTEP algorithm output.<br /> python run_chemstep_iterative.py 2<br /> <br /> <br /> <br /> '''10. Run ChemSTEP'''<br /> qsub launch_chemstep_as_job.sh <br /> <br /> The output will be smi_round_****.smi file. Repeat steps 8-10 for as many rounds as needed. The performance is reported in output_directory/complete_info/run_summary.df, which contains the number of beacons selected, the number of molecules docked, the number of hits found, the distance threshold for the selected molecules to dock, and the last added beacon's distance to all previous beacons.</div>

Kholland