Difference between revisions of "Creating Maps on SmallWorld"

From DISI
Jump to: navigation, search
m (Step 2.5 Incremental Updates)
Line 1: Line 1:
 
Written by Jennifer Young on April 10, 2019
 
Written by Jennifer Young on April 10, 2019
  
==Install==
+
==Install: The developer need only do this once.  For the user, skip to the Run section==
 
+
==The developer need only do this once.  For the user, skip to the Run section==
+
  
 
===Step 1: Create the /sw directory===
 
===Step 1: Create the /sw directory===
Line 23: Line 21:
 
==Run==
 
==Run==
 
===Important! Currently, we only run SmallWorld map generation on n-9-22===
 
===Important! Currently, we only run SmallWorld map generation on n-9-22===
===Step 1: Set SWDIR environment variable===
+
==Step 1: Set SWDIR environment variable==
 
Determine where the /sw directory with the /anon and /maps directory and if using csh, set the SWDIR variable using  
 
Determine where the /sw directory with the /anon and /maps directory and if using csh, set the SWDIR variable using  
 
     setenv SWDIR /srv/db4
 
     setenv SWDIR /srv/db4
Line 30: Line 28:
 
     export SWDIR=/srv/db4
 
     export SWDIR=/srv/db4
  
===Step 2: Generate the map files===
+
==Step 2: Generate the map files==
 
Run sw.jar to convert your smiles file into a map file.  
 
Run sw.jar to convert your smiles file into a map file.  
  
Line 62: Line 60:
 
directory so they can be easily moved to the maps directory when the server is restarted
 
directory so they can be easily moved to the maps directory when the server is restarted
  
===Step 2.5 Incremental Updates===
+
==Step 2.5 Incremental Updates==
  
 
There are several ways to perform incremental updates for SmallWorld maps which are equivalent.
 
There are several ways to perform incremental updates for SmallWorld maps which are equivalent.
Line 82: Line 80:
 
A worked example is located in the SmallWorld version 4 documentation on pages 17-18 in the Incremental Updates Section
 
A worked example is located in the SmallWorld version 4 documentation on pages 17-18 in the Incremental Updates Section
  
==What if you already created a .anon.map file from your new smiles, and simply want to combine the two separate .anon.map files into a single .anon.map file?==
+
===What if you already created a .anon.map file from your new smiles, and simply want to combine the two separate .anon.map files into a single .anon.map file?===
  
 
Make a copy and modify the bash script below with the name of your new map file
 
Make a copy and modify the bash script below with the name of your new map file
Line 112: Line 110:
 
The program will look for the .anon.map files and the .anon.map.cfg for the first and second before proceeding
 
The program will look for the .anon.map files and the .anon.map.cfg for the first and second before proceeding
  
===Step 3: How to Make the New Database Available. Stop the server first!===
+
==Step 3: How to Make the New Database Available. Stop the server first!==
  
 
First, stop the server. You do not need to restart Tomcat every time you want to restart SmallWorld. The TomCat GUI is available at
 
First, stop the server. You do not need to restart Tomcat every time you want to restart SmallWorld. The TomCat GUI is available at
Line 120: Line 118:
 
In the Manager App, in the Applications section, use the Stop and Start buttons for the sw-ws app to restart the SmallWorld server
 
In the Manager App, in the Applications section, use the Stop and Start buttons for the sw-ws app to restart the SmallWorld server
  
===Step 4: Move the Map File into the /maps directory===
+
==Step 4: Move the Map File into the /maps directory==
  
 
If you followed the example above, your .anon.map file should be located in  
 
If you followed the example above, your .anon.map file should be located in  
Line 133: Line 131:
  
  
===Step 5: Restart the Server===
+
==Step 5: Restart the Server==
 
Again, go to the TomCat manager under applications and click Start once you are finished moving around the map files.
 
Again, go to the TomCat manager under applications and click Start once you are finished moving around the map files.

Revision as of 09:28, 14 May 2019

Written by Jennifer Young on April 10, 2019

Install: The developer need only do this once. For the user, skip to the Run section

Step 1: Create the /sw directory

SmallWorld requires a /sw directory which contains /anon and /maps The /sw directory is located on n-9-22 in

   /srv/db4

This directory contains:

The /anon directory contains 12 TB of pre-computed subgraphs

The /maps directory contains text files computed by SmallWorld that map your molecules of choice to the corresponding nodes in the anonymous graph index

Step 2: Find a location for the sw.jar file

Copy the sw.jar file to a reasonable location such as

   /opt/nextmove/sw.jar

Run

Important! Currently, we only run SmallWorld map generation on n-9-22

Step 1: Set SWDIR environment variable

Determine where the /sw directory with the /anon and /maps directory and if using csh, set the SWDIR variable using

   setenv SWDIR /srv/db4

If using bash, set the SWDIR variable using

   export SWDIR=/srv/db4

Step 2: Generate the map files

Run sw.jar to convert your smiles file into a map file.

(Optional: use the time command to see how long this takes)

(Optional: tee your results into a log file to check progress)

Here is an example where wait-ok.smi is your .smi file to map:

   (time java -jar /opt/nextmove/sw.jar map /mnt/nfs/ex9/work/smallworld/wait-ok.smi 
   -T /mnt/nfs/ex9/work/smallworld/scratch_sw 
   -o /mnt/nfs/ex9/work/smallworld/wait-ok.anon.map 
   -u /mnt/nfs/ex9/work/smallworld/wait-ok.anon.unmapped)
    |& tee /mnt/nfs/ex9/work/smallworld/wait-ok_log

This is a long command, so we can break down the different parts.

map The SMILES file you want to map follows the word "map"

-T This is the temp directory you want to write temp files to, rather than writing to /tmp

-o The final name of your output map file

-u Argument to keep track of the unmapped compounds.

Important: The unmapped compounds file may be used to generate extensions and is the same as the .anon.ext.map file mentioned in the SmallWorld documentation


For now, generate the map files in the

   /mnt/nfs/ex9/work/smallworld 

directory so they can be easily moved to the maps directory when the server is restarted

Step 2.5 Incremental Updates

There are several ways to perform incremental updates for SmallWorld maps which are equivalent.

If you just have a new SMILES file input.smi you wish to add to an already existing map named <already_existing>.anon.map, then you can use the sw map command with the --append option provided in the SmallWorld Java Command Line Interface (CLI)

First, if you have not already done so, set an alias "sw" to the SmallWorld Java file as shown below: Make sure the SWDIR environment variable is also set

   setenv SWDIR /srv/db4
   alias sw 'java -jar /opt/nextmove/sw.jar'

Now you can run the sw map command using --append

   sw map input.smi --append --o <already_existing>.anon.map

Another method for performing incremental updates is discussed below. A worked example is located in the SmallWorld version 4 documentation on pages 17-18 in the Incremental Updates Section

What if you already created a .anon.map file from your new smiles, and simply want to combine the two separate .anon.map files into a single .anon.map file?

Make a copy and modify the bash script below with the name of your new map file

   /nfs/ex9/work/smallworld/bash_merge_maps

This script simply combines two map files that have already been created

Replace the OLD_MAPFILE environment variable with the old map

Replace the ADD_MAPFILE environment variable with the map you wish to incrementally add

Replace the NEW_MAPFILE environment variable with the name you want for the new combined map

Updating a new map file in this way does not automatically update the .anon.map.cfg file with the new total number of molecules mapped and other statistics. The create_cfg.py program will automatically generate a new .anon.map.cfg file for the combined map

The create_cfg.py program is located in

   /mnt/nfs/ex9/work/smallworld/create_cfg.py

The create_cfg.py script will automatically generate a cfg file for a map that is composed of two existing SmallWorld Maps. Run as follows:

   python create_cfg.py first second combined

Note: first is first.anon.map second is second.anon.map combined is combined.anon.map

The program will look for the .anon.map files and the .anon.map.cfg for the first and second before proceeding

Step 3: How to Make the New Database Available. Stop the server first!

First, stop the server. You do not need to restart Tomcat every time you want to restart SmallWorld. The TomCat GUI is available at

   http://10.20.9.22:8080/

You need the tunnel to access the private network or this link will not work! Also, the admin username and password is required.

In the Manager App, in the Applications section, use the Stop and Start buttons for the sw-ws app to restart the SmallWorld server

Step 4: Move the Map File into the /maps directory

If you followed the example above, your .anon.map file should be located in

   /mnt/nfs/ex9/work/smallworld

Now, simply move the map file into the "/sw/maps" directory so it can be recognized by SmallWorld.

   mv <your-file-name>.anon.map /mnt/nfs/ex9/work/smallworld/maps

Or, simply

   mv <your-file-name>.anon.map maps


Step 5: Restart the Server

Again, go to the TomCat manager under applications and click Start once you are finished moving around the map files.