Arthor Documentation for Future Developer: Difference between revisions

From DISI
Jump to navigation Jump to search
(131 intermediate revisions by 3 users not shown)
Line 1: Line 1:
Written by Jennifer Young on December 16, 2019. Last edited January 30, 2020
== Introduction ==
[https://www.nextmovesoftware.com/downloads/arthor/documentation/Arthor.pdf Here is the link to Arthor's manual]
* Username: ucsf@nextmovesoftware.com
* Password: <Ask jjiteam@googlegroups.com>


==Install and Set Up on TomCat==
Arthor configurations and the frontend files are consolidated in '''/nfs/soft2/arthor_configs/'''.
Arthor currently runs on n-1-136, which runs CentOS Linux release 7.7.1908 (Core).  You can check the version of CentOS with the following command
    cat /etc/centos-release


Check your current version of Java with the following command:
'''/nfs/soft2/arthor_configs/start_arthor_script.sh''' can start/restart Arthor instances on respective machines.
    java -version


On n-1-136 we are running openjdk version "1.8.0_222", OpenJDK Runtime Environment (build 1.8.0_222-b10), and OpenJDK 64-Bit Server VM (build 25.222-b10, mixed mode)
Launch the script to see the options available.
If Java is not installed, install it using yum


==See this wiki page for more detailed information about installing Tomcat on our cluster==
== How To Download Arthor ==
http://wiki.docking.org/index.php/Tomcat_Installation
# Ssh to nfs-soft2 and become root. Prepare directory
#: <source> mkdir /export/soft2/arthor_configs/arthor-<version> && cd /export/soft2/arthor_configs/arthor-<version> </source>
# [https://www.nextmovesoftware.com/downloads/arthor/ Download Software with this link]
#* Username: ucsf@nextmovesoftware.com
#* Password: <Ask jjiteam@googlegroups.com>
# Go to releases. Look for ''' smallworld-java-<version>.tar.gz ''' and copy the link address.
# Download using wget
#: <source> wget --user ucsf@nextmovesoftware.com --password <Ask jjiteam@googlegroups.com> <link address> </source>
# Decompress the file
#* <source> tar -xvf <file_name> </source>


==Open port for Arthor==
== How To Launch Arthor For The First Time ==
In order for Arthor to be usable in the browser, the port you wish to run it on must be opened.
=== Prepare Files and Directories ===
https://www.thegeekdiary.com/how-to-open-a-ports-in-centos-rhel-7/
# Ssh to nfs-exc and become root
# Open a port in the firewall
#: <source>firewall-cmd --permanent --add-port=<port_number>/tcp
firewall-cmd --reload
</source>
# Go to Arthor Config directory
#: <source>cd /export/soft2/arthor_configs/arthor-<latest_version></source>
# Create an Arthor config file
#: <source>vim <name_of_file>.cfg</source>
#* Add these lines in the file. Check the manual for more options.
#: <source>
DataDir=/local2/public_arthor
MaxConcurrentSearches=6
MaxThreadsPerSearch=8
AutomaticIndex=false
AsyncHitCountMax=20000
Depiction=./depict/bot/svg?w=%w&h=%h&svgunits=px&smi=%s&zoom=0.8&sma=%m&smalim=1
Resolver=https://sw.docking.org/util/smi2mol?smi=%s
</source>
=== Start Arthor Instance ===
# Now ssh into a machine you wish to run an Arthor instance on and become root
# Change your shell to bash if you havn't already
#: <source>bash</source>
# Create a screen
#: <source>screen -S <screen_name></source>
# Prepare Arthor Config Path
#: <source>export ARTHOR_CONFIG="/nfs/soft2/arthor_configs/arthor-<version>/<name_of_config_file>.cfg"</source>
# Launch java
#: <source>java -jar -Dserver.port=<port_number> /nfs/soft2/arthor_configs/arthor-<version>/arthor-server-<version>.war</source>


===Step 1: Check Port Status===
=== Configuration Details ===
Check that the port is not open and that Apache is not showing that port.
    netstat -na | grep <port number you are checking>


    lsof -i -P |grep http
*'''DataDir''': This is the directory where the Arthor data files live.  Location where the index files will be created and loaded from.


===Step 2: Check Port Status in IP Tables===
*'''MaxConcurrentSearches''': Controls the maximum number of searches that can be run concurrently by setting the database pool size. When switching between a large number of databases it can be useful to have a larger pool size, the only trade off is keeping file pointers open.
    iptables-save | grep <port number you are checking>


I skipped Step 3 from the guide, because there was a lot of information in the /etc/services file and I didn't want to edit it and break something.
*'''MaxThreadsPerSearch''': The number of threads to use for both ATDB and ATFP searches


===Step 4: Open Firewall Ports===
*Set '''AutomaticIndex''' to false if you don't want new smiles files added to the data directory to be indexed automatically
I did not include the zone=public section because the stand-alone servers are usually used for private instances of Arthor and SmallWorld.
Run as root.
    firewall-cmd --add-port=<port number you are adding>/tcp --permanent


You need to reload the firewall after a change is made.
*'''AsyncHitCountMax''': The upper-bound for the number of hits to retrieve in background searches.
    firewall-cmd --reload


===Step 5: Check that port is working===
*'''Resolver''': Using Smallworld API, allows input box to take in a SMILE format and automatically draw on the board.
To check that the port is active, run.
    iptables -nL


You should see something along the lines of:
'''Check Arthor manual for more configuration options'''
    ACCEPT    tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:<port number you're adding> ctstate NEW,UNTRACKED


==How to run standalone Arthor instance==
== How to Build Arthor Databases==
We can build Arthor Databases anywhere. Consolidate smiles into one directory so you can index them all one by one.


===Step 1: Use or start a bash shell===
Just use the script located at '''/nfs/home/jjg/scripts/arthor_index_script.sh''' at the directory where you stored the smiles.
You can check your default shell using
    echo $SHELL


If your default shell is csh, use
Here is the content of the script:
    bash
<source>
to start a new bash shell in the current terminal window.  Note that echo $SHELL will show you your default shell regardless of the current shell.
#!/bin/bash


===Step 2: Set your environment variables===
target="*.smi"
    export ARTHOR_DIR=/opt/nextmove/arthor/arthor-3.1-centos7
threads='48'
    export PATH=$ARTHOR_DIR/bin/:$PATH
version="3.6.1"


Make sure the ARTHOR_DIR variable is set to the directory for the latest version of Arthor or whichever version you would like to test.
The PATH environment variable is needed if you wish to use the Arthor tools from the command line


===Step 3: Run the arthor-server.jar===
export ARTHOR_DIR=/nfs/soft2/arthor_configs/arthor-$version/arthor-$version-centos8/
    java -jar /opt/nextmove/arthor/arthor-3.0-rt-beta-linux/java/arthor-server.jar --httpPort <your httpPort>
export PATH=$ARTHOR_DIR/bin/:$PATH


==Setting environment variables for TomCat Server==
for j in $target
Set the environment variables in the setenv.sh file. Note: Be sure to edit the file in the directory corresponding to the latest version of TomCat. As of December 2019, we are running 9.0.27 on n-1-136.
do
        echo 'smi2atdb -v -r -j '$threads' -p '$j' '${j}'.atdb |& tee -a '${j}'_log'
        time smi2atdb -v -r -j $threads -p $j ${j}.atdb |& tee -a ${j}_log
echo 'smi2atfp -j '$threads' -p '$j' '${j}.atfp' |& tee -a '${j}'_log'
        time smi2atfp -j $threads -p $j ${j}.atfp |& tee -a ${j}_log
done


  vim  /opt/tomcat/apache-tomcat-9.0.27/bin/setenv.sh
for i in $target*atdb
do
echo 'atdb2fp -j '$threads' '$i
        time atdb2fp -j $threads $i
chmod 744 $i.fp
done
</source>


Add the line below to the setenv.sh file above, or substitute the path to wherever you currently store the arthor.cfg file
=== Command Details ===
  export ARTHOR_CONFIG=/usr/local/tomcat/arthor.cfg
'''smi2atdb''' creates the atdb files needed for Substructure searching.
*'''-v''' verbose
*'''-r''' Compute and store ring size information in the index allowing ring size (3-10) expressions ([r6] and [Z5])
*'''-j''' is the amount of threads to use to index the smiles file
*'''-p''' stores the position of the original file  
'''atdb2fp''' makes substructure searching faster


Here is an example of the arthor.cfg file:
'''smi2atfp''' creates atfp files for similarity searching
  # Arthor generated config file
  BINDIR=/opt/nextmove/arthor/arthor-2.1.2-centos7/bin
  DATADIR=/usr/local/tomcat/arthor_data
  STAGEDIR=/usr/local/arthor_data/stage
  NTHREADS=64 .
  NODEAFFINITY=true
  SearchAsYouDraw=true
  AutomaticIndex=true
  DEPICTION=./depict/bot/svg?w=%w&h=%h&svgunits=px&smi=%s&zoom=0.8&sma=%m&smalim=1
  RESOLVER=


'''Important parts of the arthor.cfg file'''
==Setting up Round Table==
"Round Table allows you to serve and split chemical searches across multiple host machines.  The implementation provides a lightweight proxy that forwards requests to other Arthor host servers that do the actual search.  Communication is done using the existing Web APIs.
===Setting up Host Server===
# Ssh to nfs-soft2 and become root
# Open a port in the firewall
#: <source>firewall-cmd --permanent --add-port=<port_number>/tcp
firewall-cmd --reload
</source>
# Go to Arthor Config Directory
#: <source>cd /export/soft2/arthor_configs/arthor-<version></source>
# Create Round Table Head configuration file. Here is an example:
# <source>
[RoundTable]
RemoteClient=http://10.20.0.41:8008
RemoteClient=http://10.20.5.19:8008
Resolver=https://sw.docking.org/util/smi2mol?smi=%s
</source>
# Now ssh into a machine you wish to run the round table head on and become root
# Change your shell to bash if you havn't already
#: <source>bash</source>
# Create a screen
#: <source>screen -S <screen_name></source>
# Prepare Arthor Config Path
#: <source>export ARTHOR_CONFIG="/nfs/soft2/arthor_configs/arthor-<version>/<round_table_head>.cfg"</source>
# Launch java
#: <source>java -jar /nfs/soft2/arthor_configs/arthor-<version>/arthor-<version>-centos7/java/arthor.jar --httpPort=<port_number></source>


'''BINDIR''' is the location of the Arthor command line binaries.  These are used to generate the Arthor index files and to perform searches directly on n-1-136. An example of this would be using atdbgrep for substructure search.  
== Active Arthor Instances ==
===Public Arthor===
{| class="wikitable"
|-
! Rocky Linux Machine
! Port
! Round Table Data Directory
! Which Arthor
|-
| arthor
| 10.20.200.100:8080
| /local3/public_arthor/
| Public Arthor
|-
|}


'''DATADIR''' This is the directory where the Arthor data files live. Location where the index files will be created and loaded from.
===Private Arthor===
{| class="wikitable"
|-
! Rocky Linux Machine
! Port
! Round Table Data Directory
! Which Arthor
|-
| arthor
| 10.20.200.100:8081
| /local3/private_arthor/
| Private Arthor
|-
|}


'''STAGEDIR''' Location where the index files will be built before being moved into the DATADIR.
===Super Private Arthor===
{| class="wikitable"
|-
! Rocky Linux Machine
! Port
! Round Table Data Directory
! Which Arthor
|-
| arthor
| 10.20.200.100:8082
| /local3/super_private_arthor/
| Private Arthor
|-
|}


'''NTHREADS''' The number of threads to use for both ATDB and ATFP searches
===Arthor BB, CC, and Covalent===
 
{| class="wikitable"
Set '''AutomaticIndex''' to false if you don't want new smiles files added to the data directory to be indexed automatically
|-
 
! CentOS 7 Machine
==Background==
! Port
Before working with Arthor, it is recommended that you familiarize yourself with the Arthor documentation. Some useful pages to look at include 3-5, 22-25 and 33-39. Of course, reading everything would be the best!
! Data Directory
! Which Arthor
|-
| epyc-A40
| 10.20.200.92:8081
| /local2/arthorbb
| ArthorBB
|-
| epyc-A40
| 10.20.200.92:8082
| /local2/arthorcc
| ArthorCC
|-
| epyc-A40
| 10.20.200.92:8080
| /local2/arthorcovalent
| Arthor Covalent
|-
|}


==Checking Memory Usage==
== Customizing Arthor Frontend To Our Needs (Arthor 4.1) ==
Before building arthor indexes, it's always a good thing to check what percent of the memory is being used. Try to be cautious with how much memory you have left, and make sure to check while building indexes to make sure that you have enough space. To check, run the following command:
'''These instructions only worked and compiled for me in the machine called epyc which is running Rocky 8 Linux operating system.'''


  df -h /<directory with disc>
Summary of changes in index.html:
*Add contact info
*Advertise TLDR
*Remove button for Formula
*Database Information Link


==Building Large Databases==
Summary of changes in index.js:
At the moment, we are building databases of size 500M molecules by merging smile files. There are multiple methods of trying to create large databases, one being merging based off of the same H?? prefix and stopping once the database reaches > 500M molecules (or whatever upperbound you want to use). Here is some python code that simulates this merging process. Essentially the program takes all of the .smi files within an input directory, sorts them lexiographically, and begins merging these .smi files together in order until the size reaches > 500M molecules.
*Hyperlink the results to zinc20 or cartblanche22
 
*Copy SMILES with clipboard button
Feel free to modify it if you think a better method exists.
 
  import subprocess
  import sys
  import os                                                                                                                                                                         
 
  from os import listdir
  from os.path import isfile, join
 
  mypath = "<Path to directory holding .smi files>"
  onlyfiles = [f for f in listdir(mypath) if isfile(join(mypath, f))]
  onlyfiles.sort()
 
  create_fp = True
  cur_mols = 0
  lower_bound = 500000000
  upper_bound = 600000000
  files_to_merge = []
 
  def merge_files(f_t_m):
      arr = f_t_m[0].split(".")
      arr2 = f_t_m[len(f_t_m) - 1].split(".")
      file_name_merge = (arr[0] + "_" + arr2[0] + ".smi")
      print ("File being created: " + file_name_merge)
 
      for file in f_t_m:
        tmp = file.split(".")
        process = subprocess.Popen("cat " + join(mypath, file) + " >> " + file_name_merge, shell=True)
        process.wait()
 
  for file in onlyfiles:
      arr = file.split(".")
 
      if (arr[len(arr) - 1] == "smi"):
        print("Working with " + file)
        mol = sum(1 for line in open(join(mypath, file)))
        print(file, mol, cur_mols)
 
        if (cur_mols + mol > lower_bound):
            if (cur_mols + mol < upper_bound):
              files_to_merge.append(file)
              merge_files(files_to_merge)
              cur_mols = 0
              files_to_merge.clear()
            else:
              merge_files(files_to_merge)
              files_to_merge.clear()
              files_to_merge.append(file)
              merge_files(files_to_merge)
              cur_mols = 0
              files_to_merge.clear()
        else:
            cur_mols += mol
            files_to_merge.append(file)
 
  if (len(files_to_merge) != 0):
      merge_files(files_to_merge)


==Building Arthor Indexes==
Summary of changes in sketcher.js:
Once you've merged the .smi files together, it's time to start building the databases themselves. To do this we use the command
*Input box should be updated as user draws molecule


  smi2atdb -j 0 -p <The .smi file> <The .atdb>
Summary of changes in arthor-swagger.yaml.js:
*Make API point to "/"


The flag "-j 0" enables parallel generation and utilizes all available processors to generate the .atdb file. The "-p" flag stores the offset position in the ATDB file. Since we're building indexes for the Web Application, you must use the "-p" flag when building indexes. Please note that the name of the .smi file should also be the name of the .atdb file. That way, the Web Application knows to use these files together and correctly display the required images. Refer to pages 33-34 in the Arthor documentation for more information.


If there are too many large .smi files and you do not want to manually build each .atdb file, you can use this python script which takes all of the .smi files in the current directory and converts them to .atdb files. Make sure to modify mypath to the directory containing the .smi files. You can change the variable "create_fp" to false if you don't want to create .atdb.fp files (refer to page 9 in the Arthor documentation).
=== Install Prerequisite Packages ===
# Install Apache Maven
#* <source>dnf install maven -y</source>
# Install Node Package Manager (NPM)
#* <source>dnf install npm -y</source>
# In your home directory, create a new directory to hold the files for the upcoming procedures
#* <source>mkdir /mnt/nfs/home/jjg/arthor_build_from_source</source>
# Download these latest Arthors and store them in 'arthor_build_from_source/'. [[Arthor Documentation for Future Developer#How To Download Arthor|Here's how to download Arthor.]]
#* arthor-4.1-source.tar.gz
#* arthor-4.1-centos8.tar.gz
# Extract contents from the tar.gz files
#* <source>tar -xvf arthor-4.1-source.tar.gz
tar -xvf arthor-4.1-centos8.tar.gz</source>
# Install Apache Maven Arthor dependencies through this script
#* <source>
#!/bin/bash


  import subprocess
version=4.1
  import sys
  import os
 
  from os import listdir
  from os.path import isfile, join
 
  mypath = "<Path containing the .smi files"
  onlyfiles = [f for f in listdir(mypath) if isfile(join(mypath, f))]
 
  create_fp = True
 
  for file in onlyfiles:
      arr = file.split(".")
 
      if (arr[len(arr) - 1] == "smi"):
        process = subprocess.Popen("/nfs/ex9/work/xyz/psql/arthor-3.3-centos7/bin/smi2atdb -j 0 -p {0} {1}.atdb".format(join(mypath, file), arr[0]), shell=True)
        process.wait()
 
        print("SUCCESS! {0}.atdb file was created!".format(arr[0]))
 
        if (create_fp):
            process = subprocess.Popen("/nfs/ex9/work/xyz/psql/arthor-3.3-centos7/bin/atdb2fp -j 0 {0}.atdb".format(arr[0]), shell=True)
            process.wait()
     
            print("SUCCESS! {0}.atdb.fp file was created!".format(arr[0]))


==Uploading Indexes to the Web Application==
export ARTHOR_DIR=/mnt/nfs/home/jjg/arthor_build_from_source/arthor-$version/arthor-$version-centos8/java/
One can upload indexes to the Web Application by changing the "DATADIR" variable in the arthor.cfg file to the directory holding the .atdb files. This is already set up on n-1-136 and n-5-34.
export OS=linux
 
export VERSION=$version
==Further Arthor Optimizations==
The following edits can be made to the arthor.cfg to optimize substructure search queries. More information can be found in pages 6-8 in the Arthor Documentation file.


mvn install:install-file -Dfile=$ARTHOR_DIR/arthor-jni-${OS}.jar \
                        -Dpackaging=jar \
                        -DgeneratePom=true \
                        -DartifactId=arthor-jni-${OS} \
                        -DgroupId=com.nextmovesoftware.arthor \
                        -Dversion=$VERSION
mvn install:install-file -Dfile=$ARTHOR_DIR/arthor-jni.jar \
                        -Dpackaging=jar \
                        -DgeneratePom=true \
                        -DartifactId=arthor-jni \
                        -DgroupId=com.nextmovesoftware.arthor \
                        -Dversion=$VERSION
</source>
=== Customizing Index.html ===
Location: /mnt/nfs/home/jjg/arthor_build_from_source/arthor-4.1/arthor-4.1-source/server/src/main/webapp/WEB-INF/static/index.html


'''NodeAffinity NUMA''': optimized flag, pin processing to specific CPU sets to where the data is located in memory. There is a small start-up cost and is most useful for long running services (see Non-Uniform Memory Access (NUMA))
==== Add contact info and tldr ====
# vim /mnt/nfs/home/jjg/arthor_build_from_source/arthor-4.1/arthor-4.1-source/server/src/main/webapp/WEB-INF/static/index.html
# Search for 'arthor_table_list'
#* <source>?arthor_table_list</source>
# Add contact info, a plug to tldr.docking.org, and database link after the div block where arthor_table_list is inside of. It should look like this
#* <source>
      <div class="opt-box-border">
        <label>Databases</label>
        <!-- This will be populated by available databases -->
        <ul id="arthor_table_list">
          <li class="placeholder">Please select a search type</li>
        </ul>
      </div>
      <div class="opt-box-border">
        <label>Ask Questions</label>
        Email us: jjiteam@googlegroups.com
      </div>
      <div class="opt-box-border">
        <label> To Download 100K+ Results</label>
        Sign up for <a href="http://tldr.docking.org/">tldr.docking.org</a> and use arthorbatch
      </div>
      <div class="opt-box-border">
        <label> Database Information</label>
        <a href="https://wiki.docking.org/index.php/Smallworld_and_Arthor_Databases#Smallworld_Databases" target="_blank">Database Information Link</a>
      </div>
</source>


==== Remove Similarity and Formula Buttons ====
# vim /mnt/nfs/home/jjg/arthor_build_from_source/arthor-4.1/arthor-4.1-source/server/src/main/webapp/WEB-INF/static/index.html
# Search for 'arthor_search_list'
#* <source> ?arthor_search_list </source>
# Delete and Replace the whole 'ul' element block with this
#* <source>
          <li value="Similarity" onclick="setSearchType(this)" class="first">
            Similarity
          </li><li value="Substructure" onclick="setSearchType(this)">
            Substructure
          </li><li value="SMARTS" onclick="setSearchType(this)" class="last">
            SMARTS
          </li>
          <!--<li value="Formula" onclick="setSearchType(this)" class="last">
            Formula
          </li>-->
</source>


'''AsyncHitCountAllowed=true|false''' After fetching a page from a substructure or formula search the server will spin off a background process to count the total number of hits. This can be resource intensive for large databases and may not be desirable for servers under heavy load and may not even be needed.  
=== Customize Index.js ===
==== Conditional Hyperlinks ====
Location /mnt/nfs/home/jjg/arthor_build_from_source/arthor-4.1-source/server-ui/src/index.js


# Add the zinc20 hyperlink to the Arthor results. Search for this
#* <source>
$info.append($("<tr>").append($('<td class="identifier">').append(id)));
</source>
# Delete the contents of the whole else block and replace it with this
#* <source>
        if (id.substring(0, 6) == "ZINC00") {
          $info.append(
            $('<tr>').append(
            $('<td class="identifier">').append("<b><a target='_blank' href='https://zinc20.docking.org/substances/"+id+"'>" + id + "</a></b>")));
          $info.append(
            $("<tr>").append(
              $("<td>").append(
                "<span class='table_name'>" + table_name + "</span>",
              ),
            ),
          );
        } else if (id.substring(0,4) == "ZINC" && id.substring(4,6) != "00") {
          $info.append(
            $('<tr>').append(
            $('<td class="identifier">').append("<b><a target='_blank' href='https://cartblanche22.docking.org/substance/"+id+"'>" + id + "</a></b>")));
          $info.append(
            $("<tr>").append(
              $("<td>").append(
                "<span class='table_name'>" + table_name + "</span>",
              ),
            ),
          );
        } else {
          $info.append($("<tr>").append($('<td class="identifier">').append(id)));
          $info.append(
            $("<tr>").append(
              $("<td>").append(
                "<span class='table_name'>" + table_name + "</span>",
              ),
            ),
          );
        }
</source>
==== Copy SMILES Clipboard ====
# Search this
#: <source>?parser</source>
# At the end of the the try block add this
#: <source>
        $info.append(
          $('<tr>').append(
            $('<td>').append('Copy SMILES: <a href="#" onclick="navigator.clipboard.writeText(\'' + parser.smi + '\');"  ><i class="fa fa-clipboard" aria-hidden="true"></i></a>')));
</source>
# The whole try and catch block should look like this
#: <source>
try {
      var parser = new SmiJs.Parser(data);
      if (parser.parse()) {
        $info.append(
          $('<tr>').append(
            $('<td>').append(parser.mf(true))));
        $info.append(
                $('<tr>').append(
                  $('<td>').append(parser.mw())));
        $info.append(
          $('<tr>').append(
            $('<td>').append('Copy SMILES: <a href="#" onclick="navigator.clipboard.writeText(\'' + parser.smi + '\');"  ><i class="fa fa-clipboard" aria-hidden="true"></i></a>')));
      }
    } catch(err) {
      console.log(err);
    }
</source>


'''AsyncHitCountMax=#''' The upper-bound for the number of hits to retrieve in background searches. If very generic queries are issued (e.g. benzene or methane) hundred’s of millions of hits may be counted. Setting this value to anything other than zero (e.g. 10,000) will stop the background search if it exceeds this limit. Note some pathological queries may find very few hits but still end up looking at everything.  
=== Customize Sketcher.js ===
Location: /mnt/nfs/home/jjg/arthor_build_from_source/arthor-4.1-source/server-ui/src/sketcher.js


==== Input Box Updates as User Draws ====
#vim /mnt/nfs/home/jjg/arthor_build_from_source/arthor-4.1-source/server-ui/src/sketcher.js
#Search for this line "var smiles = event.src.smiles();"
#* <source>?var smiles = event.src.smiles();</source>
#Add this new line below it
#* <source>$('#ar_text_input').val(smiles);</source>


'''MaxConcurrentSearches=#''' Controls the maximum number of searches that can be run concurrently by setting the database pool size. The searches may be on the same or different databases. If a search comes in and the pool is full it will have to wait for another search to finish - this increases the request time.

=== Customize arthor-swagger.yaml ===
Location: /mnt/nfs/home/jjg/arthor_build_from_source/arthor-4.1/arthor-4.1-source/server/src/main/webapp/WEB-INF/static/swagger/arthor-swagger.yaml


Typically if each search is using all the processing cores on a machine then additional searches will run at 1/Nth the speed. If the request time is substantially larger that the search time the request had to wait for resources to become available. When switching between a large number of databases it can be useful to have a larger pool size, the only trade off is keeping file pointers open. 
Default: 6
==== Change URL Base Path for API Call ====
# vim /mnt/nfs/home/jjg/arthor_build_from_source/arthor-4.1/arthor-4.1-source/server/src/main/webapp/WEB-INF/static/swagger/arthor-swagger.yaml
# Search for "servers"
#* <source>?basePath</source>
# Remove "/arthor" and keep argument empty


=== Compile/Minify Code through NPM ===
# Install NPM packages and Minify Code
#* <source>
cd /nfs/home/jjg/arthor_build_from_source/arthor-4.1/arthor-4.1-source/server-ui
npm install
npx webpack-dev-server
npx webpack --mode=production
</source>
# If the npx commands don't work, try updating node and npm version to the latest one.
# Build the war file
#* <source>
cd /nfs/home/jjg/arthor_build_from_source/arthor-4.1/arthor-4.1-source/server
mvn install -Pbootable
</source>
# If it works, then the path to new war file is at /mnt/nfs/home/jjg/arthor_build_from_source/arthor-4.1/arthor-4.1-source/server/target/arthor-server-4.1.war


'''Binary Fingerprint Folding''' Arthor uses binary circular fingerprints (ECFP4/radius=2) for similarity. When creating an ATFP index you can specify how large to make your fingerprints. Circular fingerprints are sparser than path based fingerprints (e.g. Daylight) and so can be folded smaller without too much degradation in performance. Folding can significantly reduce the footprint size of a database and improve search speeds. A 256-bit fingerprint takes up 1/4 of the space of 1024-bit and can therefore be traversed 4x faster.
== Adding Arthor to Systemd (example) ==
# Create service files
#: <source>
touch /etc/systemd/system/arthor.service
</source>
# Add this code block in the service file
#: <source>


This is more important for very large databases with billions of compounds, in such instances a minor drop in precision is likely tolerable as ultimately all that happens is some hits may swap places in the hit list.  
[Unit]
Description=Arthor Public Instance
After=network.target


==Virtual Memory==
[Service]
In addition to modifying the arthor.cfg file, virtual memory can also be used to make queries faster. There can still be More information can be found in pages 10-16 in the Arthor Documentation.
Type=exec
Restart=always
RestartSec=3
User=root
ExecStart=/usr/bin/bash /nfs/soft2/arthor_configs/start_arthor_service.sh -arthor
ExecStop=/bin/kill -15 $MAINPID


==Setting up Round Table==
[Install]
This is a new feature in Arthor 3.0 and is currently beta (January 2020). See section 2.4 in the manual
WantedBy=multi-user.target
As explained in the manual, "Round Table allows you to serve and split chemical searches across multiple host machines.  The implementation provides a lightweight proxy that forwards requests to other Arthor host servers that do the actual search.  Communication is done using the existing Web APIs.
</source>
# Change "ExecStart" to other Arthor Instances


Since Arthor requires CentOS 7, as of January 2020 we have 6 servers that are capable of running Arthor with Round Table.
== Restarting Arthor Instance(s) Instructions ==
=== Public, Private Arthor, and Super Private Arthor ===
# ssh to "arthor" server and run
#: <source>
systemctl <start|stop|restart> <arthor|arthorp|arthorc>
</source>


===Setting up Host Server===
=== ArthorBB, ArthorCC, and ArthorCovalent ===
If we want to add machines to the Round Table, for example 'nun' and 'samekh', we need to edit their arthor.cfg file so that when our Local Machine passes commands these secondary servers know to perform the search they are given.
# ssh to "epyc-a40" server and run
 
#: <source>
  $ cat arthor.cfg
systemctl <start|stop|restart> <arthorbb|arthorcc|arthorcovalent>
  MaxThreadsPerSearch=4
</source>
  AutomaticIndex=false
  DATADIR=<Directory where smiles are located>
 
We then run the jar server on each of these host machines containing data on any available port.
 
  java -jar /nfs/ex9/work/xyz/psql/arthor-3.3-centos7/java/arthor.jar --httpPort <port>
 
For our local machine, the arthor.cfg file will look different.
 
  $ cat arthor.cfg
  [RoundTable]
  RemoteClient=http://skynet:<port number where jar server is running>/
  RemoteClient=http://hal:<port number where jar server is running>/
 
Please refer to Section 2 in the RoundTable Documentation file (pages 6-8) for more useful information on configuration.
 
Then run the following command on n-1-136:
  java -jar /nfs/ex9/work/xyz/psql/arthor-3.3-centos7/java/arthor.jar --httpPort <port>
 
===Arthor Round Table Head===
{| class="wikitable"
|-
! CentOS 7 Machine
! Port
! Database
! Arthor Install Location
! Round Table Data Directory
! Active
|-
| n-1-136
| 10.20.10.136:8080/arthor-rt-host/
| (old databases)
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /zinc2/auto_atdb
| not active
|-
| samekh
| 10.20.0.41:8080
| Enamine_REAL_Q2-2020-All-13B
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
| nun
| 10.20.0.40:8080
| Enamine_REAL_Q2-2020-All-41B
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/auto_atdb/
| active
|-
|}


===Arthor Round Table Nodes===
[[Category:Arthor]]
{| class="wikitable"
[[Category:Developer]]
|-
! CentOS 7 Machine
! Port
! Database
! Total Files Size
! Arthor Install Location
! Round Table Data Directory
!Active
|-
| n-1-16
| 10.20.1.16:8008
| Enamine_REAL_Q2-2020-M-13B (aa-al)
| 12 slices: 2.0TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
| n-1-17
| 10.20.1.17
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/auto_atdb/
| active
|-
| n-1-20
| 10.20.1.20
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/auto_atdb/
| active
|-
| n-5-34
| 10.20.5.34
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/auto_atdb/
| active
|-
| n-5-35
| 10.20.5.35
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/auto_atdb/
| active
|-
| n-5-34
| 10.20.5.34
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/auto_atdb/
| active
|-
| qof
| 10.20.9.29
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /export/ex9/work/btingle/auto_atdb
| active
|-
| lamed
| 10.20.9.15
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /export/ex6/work/btingle/auto_atdb
| active
|-
|}

Revision as of 23:48, 26 June 2024

Introduction

Here is the link to Arthor's manual

  • Username: ucsf@nextmovesoftware.com
  • Password: <Ask jjiteam@googlegroups.com>

Arthor configurations and the frontend files are consolidated in /nfs/soft2/arthor_configs/.

/nfs/soft2/arthor_configs/start_arthor_script.sh can start/restart Arthor instances on respective machines.

Launch the script to see the options available.

How To Download Arthor

  1. Ssh to nfs-soft2 and become root. Prepare directory 
     mkdir /export/soft2/arthor_configs/arthor-<version> && cd /export/soft2/arthor_configs/arthor-<version>
  2. Download Software with this link
    • Username: ucsf@nextmovesoftware.com
    • Password: <Ask jjiteam@googlegroups.com>
  3. Go to releases. Look for smallworld-java-<version>.tar.gz and copy the link address.
  4. Download using wget 
     wget --user ucsf@nextmovesoftware.com --password <Ask jjiteam@googlegroups.com> <link address>
  5. Decompress the file 
    •  tar -xvf <file_name>

How To Launch Arthor For The First Time

Prepare Files and Directories

  1. Ssh to nfs-exc and become root
  2. Open a port in the firewall 
    firewall-cmd --permanent --add-port=<port_number>/tcp 
firewall-cmd --reload
  3. Go to Arthor Config directory 
    cd /export/soft2/arthor_configs/arthor-<latest_version>
  4. Create an Arthor config file 
    vim <name_of_file>.cfg
    • Add these lines in the file. Check the manual for more options.
    DataDir=/local2/public_arthor
MaxConcurrentSearches=6
MaxThreadsPerSearch=8
AutomaticIndex=false
AsyncHitCountMax=20000
Depiction=./depict/bot/svg?w=%w&h=%h&svgunits=px&smi=%s&zoom=0.8&sma=%m&smalim=1
Resolver=https://sw.docking.org/util/smi2mol?smi=%s

Start Arthor Instance

  1. Now ssh into a machine you wish to run an Arthor instance on and become root
  2. Change your shell to bash if you havn't already 
    bash
  3. Create a screen 
    screen -S <screen_name>
  4. Prepare Arthor Config Path 
    export ARTHOR_CONFIG="/nfs/soft2/arthor_configs/arthor-<version>/<name_of_config_file>.cfg"
  5. Launch java 
    java -jar -Dserver.port=<port_number> /nfs/soft2/arthor_configs/arthor-<version>/arthor-server-<version>.war

Configuration Details

  • DataDir: This is the directory where the Arthor data files live. Location where the index files will be created and loaded from.
  • MaxConcurrentSearches: Controls the maximum number of searches that can be run concurrently by setting the database pool size. When switching between a large number of databases it can be useful to have a larger pool size, the only trade off is keeping file pointers open.
  • MaxThreadsPerSearch: The number of threads to use for both ATDB and ATFP searches
  • Set AutomaticIndex to false if you don't want new smiles files added to the data directory to be indexed automatically
  • AsyncHitCountMax: The upper-bound for the number of hits to retrieve in background searches.
  • Resolver: Using Smallworld API, allows input box to take in a SMILE format and automatically draw on the board.

Check Arthor manual for more configuration options

How to Build Arthor Databases

We can build Arthor Databases anywhere. Consolidate smiles into one directory so you can index them all one by one.

Just use the script located at /nfs/home/jjg/scripts/arthor_index_script.sh at the directory where you stored the smiles.

Here is the content of the script: 

#!/bin/bash

target="*.smi"
threads='48'
version="3.6.1"


export ARTHOR_DIR=/nfs/soft2/arthor_configs/arthor-$version/arthor-$version-centos8/
export PATH=$ARTHOR_DIR/bin/:$PATH

for j in $target
do
        echo 'smi2atdb -v -r -j '$threads' -p '$j' '${j}'.atdb |& tee -a '${j}'_log'
        time smi2atdb -v -r -j $threads -p $j ${j}.atdb |& tee -a ${j}_log
	echo 'smi2atfp -j '$threads' -p '$j' '${j}.atfp' |& tee -a '${j}'_log'
        time smi2atfp -j $threads -p $j ${j}.atfp |& tee -a ${j}_log
done

for i in $target*atdb
do
	echo 'atdb2fp -j '$threads' '$i
        time atdb2fp -j $threads $i
	chmod 744 $i.fp
done

Command Details

smi2atdb creates the atdb files needed for Substructure searching.

  • -v verbose
  • -r Compute and store ring size information in the index allowing ring size (3-10) expressions ([r6] and [Z5])
  • -j is the amount of threads to use to index the smiles file
  • -p stores the position of the original file

atdb2fp makes substructure searching faster

smi2atfp creates atfp files for similarity searching

Setting up Round Table

"Round Table allows you to serve and split chemical searches across multiple host machines. The implementation provides a lightweight proxy that forwards requests to other Arthor host servers that do the actual search. Communication is done using the existing Web APIs.

Setting up Host Server

  1. Ssh to nfs-soft2 and become root
  2. Open a port in the firewall 
    firewall-cmd --permanent --add-port=<port_number>/tcp 
firewall-cmd --reload
  3. Go to Arthor Config Directory 
    cd /export/soft2/arthor_configs/arthor-<version>
  4. Create Round Table Head configuration file. Here is an example:
  5. [RoundTable]
RemoteClient=http://10.20.0.41:8008
RemoteClient=http://10.20.5.19:8008
Resolver=https://sw.docking.org/util/smi2mol?smi=%s
  6. Now ssh into a machine you wish to run the round table head on and become root
  7. Change your shell to bash if you havn't already 
    bash
  8. Create a screen 
    screen -S <screen_name>
  9. Prepare Arthor Config Path 
    export ARTHOR_CONFIG="/nfs/soft2/arthor_configs/arthor-<version>/<round_table_head>.cfg"
  10. Launch java 
    java -jar /nfs/soft2/arthor_configs/arthor-<version>/arthor-<version>-centos7/java/arthor.jar --httpPort=<port_number>

Active Arthor Instances

Public Arthor

Rocky Linux Machine Port Round Table Data Directory Which Arthor
arthor 10.20.200.100:8080 /local3/public_arthor/ Public Arthor

Private Arthor

Rocky Linux Machine Port Round Table Data Directory Which Arthor
arthor 10.20.200.100:8081 /local3/private_arthor/ Private Arthor

Super Private Arthor

Rocky Linux Machine Port Round Table Data Directory Which Arthor
arthor 10.20.200.100:8082 /local3/super_private_arthor/ Private Arthor

Arthor BB, CC, and Covalent

CentOS 7 Machine Port Data Directory Which Arthor
epyc-A40 10.20.200.92:8081 /local2/arthorbb ArthorBB
epyc-A40 10.20.200.92:8082 /local2/arthorcc ArthorCC
epyc-A40 10.20.200.92:8080 /local2/arthorcovalent Arthor Covalent

Customizing Arthor Frontend To Our Needs (Arthor 4.1)

These instructions only worked and compiled for me in the machine called epyc which is running Rocky 8 Linux operating system.

Summary of changes in index.html:

  • Add contact info
  • Advertise TLDR
  • Remove button for Formula
  • Database Information Link

Summary of changes in index.js:

  • Hyperlink the results to zinc20 or cartblanche22
  • Copy SMILES with clipboard button

Summary of changes in sketcher.js:

  • Input box should be updated as user draws molecule

Summary of changes in arthor-swagger.yaml.js:

  • Make API point to "/"


Install Prerequisite Packages

  1. Install Apache Maven 
    • dnf install maven -y
  2. Install Node Package Manager (NPM) 
    • dnf install npm -y
  3. In your home directory, create a new directory to hold the files for the upcoming procedures 
    • mkdir /mnt/nfs/home/jjg/arthor_build_from_source
  4. Download these latest Arthors and store them in 'arthor_build_from_source/'. Here's how to download Arthor.
    • arthor-4.1-source.tar.gz
    • arthor-4.1-centos8.tar.gz
  5. Extract contents from the tar.gz files 
    • tar -xvf arthor-4.1-source.tar.gz
tar -xvf arthor-4.1-centos8.tar.gz
  6. Install Apache Maven Arthor dependencies through this script 
    • #!/bin/bash

version=4.1

export ARTHOR_DIR=/mnt/nfs/home/jjg/arthor_build_from_source/arthor-$version/arthor-$version-centos8/java/
export OS=linux
export VERSION=$version

mvn install:install-file -Dfile=$ARTHOR_DIR/arthor-jni-${OS}.jar \
                         -Dpackaging=jar \
                         -DgeneratePom=true \
                         -DartifactId=arthor-jni-${OS} \
                         -DgroupId=com.nextmovesoftware.arthor \
                         -Dversion=$VERSION
mvn install:install-file -Dfile=$ARTHOR_DIR/arthor-jni.jar \
                         -Dpackaging=jar \
                         -DgeneratePom=true \
                         -DartifactId=arthor-jni \
                         -DgroupId=com.nextmovesoftware.arthor \
                         -Dversion=$VERSION

Customizing Index.html

Location: /mnt/nfs/home/jjg/arthor_build_from_source/arthor-4.1/arthor-4.1-source/server/src/main/webapp/WEB-INF/static/index.html

Add contact info and tldr

  1. vim /mnt/nfs/home/jjg/arthor_build_from_source/arthor-4.1/arthor-4.1-source/server/src/main/webapp/WEB-INF/static/index.html
  2. Search for 'arthor_table_list' 
    • ?arthor_table_list
  3. Add contact info, a plug to tldr.docking.org, and database link after the div block where arthor_table_list is inside of. It should look like this 
    •       <div class="opt-box-border">
        <label>Databases</label>
        <!-- This will be populated by available databases -->
        <ul id="arthor_table_list">
          <li class="placeholder">Please select a search type</li>
        </ul>
      </div>
      <div class="opt-box-border">
        <label>Ask Questions</label>
        Email us: jjiteam@googlegroups.com
      </div>
      <div class="opt-box-border">
        <label> To Download 100K+ Results</label>
        Sign up for <a href="http://tldr.docking.org/">tldr.docking.org</a> and use arthorbatch
      </div>
      <div class="opt-box-border">
        <label> Database Information</label>
        <a href="https://wiki.docking.org/index.php/Smallworld_and_Arthor_Databases#Smallworld_Databases" target="_blank">Database Information Link</a>
      </div>

Remove Similarity and Formula Buttons

  1. vim /mnt/nfs/home/jjg/arthor_build_from_source/arthor-4.1/arthor-4.1-source/server/src/main/webapp/WEB-INF/static/index.html
  2. Search for 'arthor_search_list' 
    •  ?arthor_search_list
  3. Delete and Replace the whole 'ul' element block with this 
    •           <li value="Similarity" onclick="setSearchType(this)" class="first">
            Similarity
          </li><li value="Substructure" onclick="setSearchType(this)">
            Substructure
          </li><li value="SMARTS" onclick="setSearchType(this)" class="last">
            SMARTS
          </li>
          <!--<li value="Formula" onclick="setSearchType(this)" class="last">
            Formula
          </li>-->

Customize Index.js

Conditional Hyperlinks

Location /mnt/nfs/home/jjg/arthor_build_from_source/arthor-4.1-source/server-ui/src/index.js

  1. Add the zinc20 hyperlink to the Arthor results. Search for this 
    • $info.append($("<tr>").append($('<td class="identifier">').append(id)));
  2. Delete the contents of the whole else block and replace it with this 
    •         if (id.substring(0, 6) == "ZINC00") {
          $info.append(
            $('<tr>').append(
            $('<td class="identifier">').append("<b><a target='_blank' href='https://zinc20.docking.org/substances/"+id+"'>" + id + "</a></b>")));
          $info.append(
            $("<tr>").append(
              $("<td>").append(
                "<span class='table_name'>" + table_name + "</span>",
              ),
            ),
          );
        } else if (id.substring(0,4) == "ZINC" && id.substring(4,6) != "00") {
          $info.append(
            $('<tr>').append(
            $('<td class="identifier">').append("<b><a target='_blank' href='https://cartblanche22.docking.org/substance/"+id+"'>" + id + "</a></b>")));
          $info.append(
            $("<tr>").append(
              $("<td>").append(
                "<span class='table_name'>" + table_name + "</span>",
              ),
            ),
          );
        } else {
          $info.append($("<tr>").append($('<td class="identifier">').append(id)));
          $info.append(
            $("<tr>").append(
              $("<td>").append(
                "<span class='table_name'>" + table_name + "</span>",
              ),
            ),
          );
        }

Copy SMILES Clipboard

  1. Search this 
    ?parser
  2. At the end of the the try block add this 
            $info.append(
          $('<tr>').append(
            $('<td>').append('Copy SMILES: <a href="#" onclick="navigator.clipboard.writeText(\'' + parser.smi + '\');"  ><i class="fa fa-clipboard" aria-hidden="true"></i></a>')));
  3. The whole try and catch block should look like this 
    try {
      var parser = new SmiJs.Parser(data);
      if (parser.parse()) {
        $info.append(
          $('<tr>').append(
            $('<td>').append(parser.mf(true))));
        $info.append(
                $('<tr>').append(
                  $('<td>').append(parser.mw())));
        $info.append(
          $('<tr>').append(
            $('<td>').append('Copy SMILES: <a href="#" onclick="navigator.clipboard.writeText(\'' + parser.smi + '\');"  ><i class="fa fa-clipboard" aria-hidden="true"></i></a>')));
      }
    } catch(err) {
      console.log(err);
    }

Customize Sketcher.js

Location: /mnt/nfs/home/jjg/arthor_build_from_source/arthor-4.1-source/server-ui/src/sketcher.js

Input Box Updates as User Draws

  1. vim /mnt/nfs/home/jjg/arthor_build_from_source/arthor-4.1-source/server-ui/src/sketcher.js
  2. Search for this line "var smiles = event.src.smiles();" 
    • ?var smiles = event.src.smiles();
  3. Add this new line below it 
    • $('#ar_text_input').val(smiles);

Customize arthor-swagger.yaml

Location: /mnt/nfs/home/jjg/arthor_build_from_source/arthor-4.1/arthor-4.1-source/server/src/main/webapp/WEB-INF/static/swagger/arthor-swagger.yaml

Change URL Base Path for API Call

  1. vim /mnt/nfs/home/jjg/arthor_build_from_source/arthor-4.1/arthor-4.1-source/server/src/main/webapp/WEB-INF/static/swagger/arthor-swagger.yaml
  2. Search for "servers" 
    • ?basePath
  3. Remove "/arthor" and keep argument empty

Compile/Minify Code through NPM

  1. Install NPM packages and Minify Code 
    • cd /nfs/home/jjg/arthor_build_from_source/arthor-4.1/arthor-4.1-source/server-ui
npm install
npx webpack-dev-server
npx webpack --mode=production
  2. If the npx commands don't work, try updating node and npm version to the latest one.
  3. Build the war file 
    • cd /nfs/home/jjg/arthor_build_from_source/arthor-4.1/arthor-4.1-source/server
mvn install -Pbootable
  4. If it works, then the path to new war file is at /mnt/nfs/home/jjg/arthor_build_from_source/arthor-4.1/arthor-4.1-source/server/target/arthor-server-4.1.war

Adding Arthor to Systemd (example)

  1. Create service files 
    touch /etc/systemd/system/arthor.service
  2. Add this code block in the service file 
    [Unit]
Description=Arthor Public Instance
After=network.target

[Service]
Type=exec
Restart=always
RestartSec=3
User=root
ExecStart=/usr/bin/bash /nfs/soft2/arthor_configs/start_arthor_service.sh -arthor
ExecStop=/bin/kill -15 $MAINPID

[Install]
WantedBy=multi-user.target
  3. Change "ExecStart" to other Arthor Instances

Restarting Arthor Instance(s) Instructions

Public, Private Arthor, and Super Private Arthor

  1. ssh to "arthor" server and run 
    systemctl <start|stop|restart> <arthor|arthorp|arthorc>

ArthorBB, ArthorCC, and ArthorCovalent

  1. ssh to "epyc-a40" server and run 
    systemctl <start|stop|restart> <arthorbb|arthorcc|arthorcovalent>
Retrieved from "http://wiki.docking.org/index.php?title=Arthor_Documentation_for_Future_Developer&oldid=16216"