DISI - User contributions [en]

Main Page

2023-01-25T00:21:24Z

Jgutie11:

<div id="mainpage"></div>__NOTOC__
Welcome to [[DISI:About | DISI]] (pronounce: /dizzy/ which is short for DISI Is Still Incomplete), a wiki focusing on computational pharmacology and ligand discovery tools hosted by docking.org on behalf of the [[Irwin Lab]] and [[Shoichet Lab]]. This wiki aims to serve several [[:Category:Roles | constituencies]], and is a work in progress. Please [[Contribute |work with us and help us improve]] it.

== WHO are you? ==
This site contains information for lab members and for everyone using our public tools:
* [[Welcome group members | Lab members with ssh]]
* [[Welcome web user | Non members without ssh ]]

== WHY are you here? ==
* [[:Category:Docking | Docking ]]
* [[:Category:Systems pharmacology | Systems pharmacology]]
* [[:Category:Aggregation | Colloidal aggregation of small molecules]]
* [[:Category:Cheminformatics | Chemical informatics]]
* [[DOCKovalent_3.7]] - Covalent docking
* more [[:Category:Topic | topics]]

== Do you have a question? ==
We are trying to organize these pages:

* [[:Category:Manual | WHAT ]] - manuals, such as they are.
* [[:Category:Tutorials | HOW ]] - Tutorials, such as they are.
* [[:Category:Theory | WHY ]] - still working on turning our concept into an idea.
* [[:Category:Article_type | Other ]] - Everything else

== Still haven't found what you're looking for? ==
Try the search bar top right to see if that works...

[[Category:Info]]
[[Category:Organization]]

Main Page

2023-01-25T00:21:12Z

Jgutie11:

File:Dash 1.png

2021-05-14T00:59:52Z

Jgutie11: test 3

== Summary ==
test 3

Arthor Documentation for Future Developer

2021-05-04T17:12:44Z

Jgutie11: /* Arthor Local 8081 (Datasets all local to samekh/nun) */

Written by Jennifer Young on December 16, 2019. Last edited January 05, 2021

==Install and Set Up on TomCat (Method 1)==
Arthor ran on n-1-136, which runs CentOS Linux release 7.7.1908 (Core). You can check the version of CentOS with the following command
cat /etc/centos-release

Check your current version of Java with the following command:
java -version

On n-1-136 we are running openjdk version "1.8.0_222", OpenJDK Runtime Environment (build 1.8.0_222-b10), and OpenJDK 64-Bit Server VM (build 25.222-b10, mixed mode)
If Java is not installed, install it using yum

==See this wiki page for more detailed information about installing Tomcat on our cluster==
http://wiki.docking.org/index.php/Tomcat_Installation

==Open port for Arthor==
In order for Arthor to be usable in the browser, the port you wish to run it on must be opened.
https://www.thegeekdiary.com/how-to-open-a-ports-in-centos-rhel-7/

===Step 1: Check Port Status===
Check that the port is not open and that Apache is not showing that port.
netstat -na | grep <port number you are checking>

lsof -i -P |grep http

===Step 2: Check Port Status in IP Tables===
iptables-save | grep <port number you are checking>

I skipped Step 3 from the guide, because there was a lot of information in the /etc/services file and I didn't want to edit it and break something.

===Step 4: Open Firewall Ports===
I did not include the zone=public section because the stand-alone servers are usually used for private instances of Arthor and SmallWorld.
Run as root.
firewall-cmd --add-port=<port number you are adding>/tcp --permanent

You need to reload the firewall after a change is made.
firewall-cmd --reload

===Step 5: Check that port is working===
To check that the port is active, run.
iptables -nL

You should see something along the lines of:
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:<port number you're adding> ctstate NEW,UNTRACKED

==How to run standalone Arthor instance==

===Step 1: Use or start a bash shell===
You can check your default shell using
echo $SHELL

If your default shell is csh, use
bash
to start a new bash shell in the current terminal window. Note that echo $SHELL will show you your default shell regardless of the current shell.

===Step 2: Set your environment variables===
export ARTHOR_DIR=/opt/nextmove/arthor/arthor-3.3-centos7
export PATH=$ARTHOR_DIR/bin/:$PATH

Make sure the ARTHOR_DIR variable is set to the directory for the latest version of Arthor or whichever version you would like to test.
The PATH environment variable is needed if you wish to use the Arthor tools from the command line

===Step 3: Run the arthor-server.jar===
java -jar /opt/nextmove/arthor/arthor-3.3-centos7/java/arthor.jar --httpPort <your httpPort>

==Setting environment variables for an Arthor Server==
Set the environment variables in the setenv.sh file. Note: Be sure to edit the file in the directory corresponding to the latest version of TomCat. As of December 2019, we are running 9.0.27 on n-1-136.

vim /opt/tomcat/apache-tomcat-9.0.27/bin/setenv.sh

Add the line below to the setenv.sh file above, or substitute the path to wherever you currently store the arthor.cfg file
export ARTHOR_CONFIG=/usr/local/tomcat/arthor.cfg

Here is an example of the arthor.cfg file:
BinDir=/opt/nextmove/arthor/arthor-3.3-centos7/bin
DataDir=/local2/arthor_local_8081/
MaxConcurrentSearches=6
MaxThreadsPerSearch=8
AutomaticIndex=false
AsyncHitCountMax=1000000
Resolver=https://sw.docking.org/util/smi2mol?smi=%s

=== Configuration Details ===

*'''BinDir''': is the location of the Arthor command line binaries. These are used to generate the Arthor index files and to perform searches directly on n-1-136. An example of this would be using atdbgrep for substructure search.

*'''DataDir''': This is the directory where the Arthor data files live. Location where the index files will be created and loaded from.

*'''MaxConcurrentSearches''': Controls the maximum number of searches that can be run concurrently by setting the database pool size. When switching between a large number of databases it can be useful to have a larger pool size, the only trade off is keeping file pointers open.

*'''MaxThreadsPerSearch''': The number of threads to use for both ATDB and ATFP searches

*Set '''AutomaticIndex''' to false if you don't want new smiles files added to the data directory to be indexed automatically

*'''AsyncHitCountMax''': The upper-bound for the number of hits to retrieve in background searches.

*'''Resolver''': Using Smallworld API, allows input box to take in a SMILE format and automatically draw on the board.

Check Arthor manual for more configuration options.

==Background==
Before working with Arthor, it is recommended that you familiarize yourself with the Arthor documentation. Some useful pages to look at include 3-5, 22-25 and 33-39. Of course, reading everything would be the best!

==Checking Disk Space Usage==
Before building arthor indexes, it's always a good thing to check what percent of the memory is being used. Try to be cautious with how much memory you have left, and make sure to check while building indexes to make sure that you have enough space. To check, run the following command:

df -h /<directory with disc>

==Building Large Databases==
At the moment, we are building databases of size 500M molecules by merging smile files. There are multiple methods of trying to create large databases, one being merging based off of the same H?? prefix and stopping once the database reaches > 500M molecules (or whatever upperbound you want to use). Here is some python code that simulates this merging process. Essentially the program takes all of the .smi files within an input directory, sorts them lexiographically, and begins merging these .smi files together in order until the size reaches > 500M molecules.

Feel free to modify it if you think a better method exists.

import subprocess
import sys
import os

from os import listdir
from os.path import isfile, join

mypath = "<Path to directory holding .smi files>"
onlyfiles = [f for f in listdir(mypath) if isfile(join(mypath, f))]
onlyfiles.sort()

create_fp = True
cur_mols = 0
lower_bound = 500000000
upper_bound = 600000000
files_to_merge = []

def merge_files(f_t_m):
arr = f_t_m[0].split(".")
arr2 = f_t_m[len(f_t_m) - 1].split(".")
file_name_merge = (arr[0] + "_" + arr2[0] + ".smi")
print ("File being created: " + file_name_merge)

for file in f_t_m:
tmp = file.split(".")
process = subprocess.Popen("cat " + join(mypath, file) + " >> " + file_name_merge, shell=True)
process.wait()

for file in onlyfiles:
arr = file.split(".")

if (arr[len(arr) - 1] == "smi"):
print("Working with " + file)
mol = sum(1 for line in open(join(mypath, file)))
print(file, mol, cur_mols)

if (cur_mols + mol > lower_bound):
if (cur_mols + mol < upper_bound):
files_to_merge.append(file)
merge_files(files_to_merge)
cur_mols = 0
files_to_merge.clear()
else:
merge_files(files_to_merge)
files_to_merge.clear()
files_to_merge.append(file)
merge_files(files_to_merge)
cur_mols = 0
files_to_merge.clear()
else:
cur_mols += mol
files_to_merge.append(file)

if (len(files_to_merge) != 0):
merge_files(files_to_merge)

==Building Arthor Indexes==
Once you've merged the .smi files together, it's time to start building the databases themselves. To do this we use the command

smi2atdb -j 0 -p <The .smi file> <The .atdb>

The flag "-j 0" enables parallel generation and utilizes all available processors to generate the .atdb file. The "-p" flag stores the offset position in the ATDB file. Since we're building indexes for the Web Application, you must use the "-p" flag when building indexes. Please note that the name of the .smi file should also be the name of the .atdb file. That way, the Web Application knows to use these files together and correctly display the required images. Refer to pages 33-34 in the Arthor documentation for more information.

If there are too many large .smi files and you do not want to manually build each .atdb file, you can use this python script which takes all of the .smi files in the current directory and converts them to .atdb files. Make sure to modify mypath to the directory containing the .smi files. You can change the variable "create_fp" to false if you don't want to create .atdb.fp files (refer to page 9 in the Arthor documentation).

import subprocess
import sys
import os

from os import listdir
from os.path import isfile, join

mypath = "<Path containing the .smi files"
onlyfiles = [f for f in listdir(mypath) if isfile(join(mypath, f))]

create_fp = True

for file in onlyfiles:
arr = file.split(".")

if (arr[len(arr) - 1] == "smi"):
process = subprocess.Popen("/nfs/ex9/work/xyz/psql/arthor-3.3-centos7/bin/smi2atdb -j 0 -p {0} {1}.atdb".format(join(mypath, file), arr[0]), shell=True)
process.wait()

print("SUCCESS! {0}.atdb file was created!".format(arr[0]))

if (create_fp):
process = subprocess.Popen("/nfs/ex9/work/xyz/psql/arthor-3.3-centos7/bin/atdb2fp -j 0 {0}.atdb".format(arr[0]), shell=True)
process.wait()

print("SUCCESS! {0}.atdb.fp file was created!".format(arr[0]))

==Uploading Indexes to the Web Application==
One can upload indexes to the Web Application by changing the "DATADIR" variable in the arthor.cfg file to the directory holding the .atdb files. This is already set up on n-1-136 and n-5-34.

==Virtual Memory==
In addition to modifying the arthor.cfg file, virtual memory can also be used to make queries faster. There can still be More information can be found in pages 10-16 in the Arthor Documentation.

==Setting up Round Table==
This is a new feature in Arthor 3.0 and is currently beta (January 2020). See section 2.4 in the manual
As explained in the manual, "Round Table allows you to serve and split chemical searches across multiple host machines. The implementation provides a lightweight proxy that forwards requests to other Arthor host servers that do the actual search. Communication is done using the existing Web APIs.

Since Arthor requires CentOS 7, as of January 2020 we have 6 servers that are capable of running Arthor with Round Table.

===Setting up Host Server===
If we want to add machines to the Round Table, for example 'nun' and 'samekh', we need to edit their arthor.cfg file so that when our Local Machine passes commands these secondary servers know to perform the search they are given.

$ cat arthor.cfg
MaxThreadsPerSearch=4
AutomaticIndex=false
DATADIR=<Directory where smiles are located>

We then run the jar server on each of these host machines containing data on any available port.

java -jar /nfs/ex9/work/xyz/psql/arthor-3.3-centos7/java/arthor.jar --httpPort <port>

For our local machine, the arthor.cfg file will look different.

$ cat arthor.cfg
[RoundTable]
RemoteClient=http://skynet:<port number where jar server is running>/
RemoteClient=http://hal:<port number where jar server is running>/

Please refer to Section 2 in the RoundTable Documentation file (pages 6-8) for more useful information on configuration.

Then run the following command on n-1-136:

java -jar /nfs/ex9/work/xyz/psql/arthor-3.3-centos7/java/arthor.jar --httpPort <port>

==='''***Arthor configs and frontend code are located in /nfs/exc/arthor_configs/***'''===

===Public Arthor===
{| class="wikitable"
|-
! CentOS 7 Machine
! Port
! Total Files Size
! Round Table Data Directory
! Active
|-
| samekh
| 10.20.0.41:8000
| 2.4TB
| /local2/public_arthor/
| active
|-
| nun
| 10.20.0.40:8000
| 2.4TB
| /local2/public_arthor/
| active
|-
| n-9-22
| 10.20.9.22:8000
| 2.4TB
| /export/db4/public_arthor/
| active
|-
|}

===Arthor Round Table Head===
{| class="wikitable"
|-
! CentOS 7 Machine
! Port
! Database
! Round Table Data Directory
! Active
|-
| samekh
| 10.20.0.41:8080
| REAL-Space-21Q1(private)
| /local2/arthor_database/
| active
|-
| nun
| 10.20.0.40:8080
| REAL_Space_21Q2(super private)
| /local2/arthor_database/
| active
|-
|}

===Arthor Round Table Nodes===
{| class="wikitable"
|-
! CentOS 7 Machine
! Port
! Database
! Total Files Size
! Round Table Data Directory
!Active
|-
| samekh
| 10.20.0.41:8008
| REAL-Space-21Q1-M, REAL-Space-21Q1-S, carboxylates_21Q2
| 4.0TB
| /local2/arthor_database/
| active
|-
| nun
| 10.20.0.40:8008
| 345_acids_20Q4, REAL_Space_21Q2-M, REAL_Space_21Q2-S, acids_21Q2, zinc_21Q1_H01~H25
| 3.1TB
| /local2/arthor_database/
| active
|-
| n-1-17
| 10.20.1.17:8008
|
|
| /local2/arthor_database/
| active
|-
| n-5-32
| 10.20.5.32:8008
| zinc_21Q1_H26~H30
| 6.6TB
| /local2/arthor_database/
| active
|-
| n-5-33
| 10.20.5.33:8008
| acids_21Q2
| 9.1GB
| /local2/arthor_database/
| active
|-
|}

===Arthor Local 8081 (Datasets all local to samekh/nun)===
{| class="wikitable"
|-
! CentOS 7 Machine
! Port
! Database
! Total Files Size
! Round Table Data Directory
!Active
|-
| samekh
| 10.20.0.41:8081
| (TBA)
| (TBA)
| /local2/arthor_local_8081/
| not active
|-
| nun
| 10.20.0.40:8081
| (TBA)
| (TBA)
| /local2/arthor_local_8081/
| not active
|-
|}

== Customizing Arthor Frontend to our needs ==
The frontend Arthor code is located at '''/nfs/exc/arthor_configs/*''' and the '''*''' is based on current running version.
=== Add Arthor Download Options ===
==== For Arthor 3.4: ====
1. vim .extract/webapps/ROOT/WEB-INF/static/index.html

2. search: '''arthor_tsv_link'''

3. in the div with the class=”dropdown-content”, add these link options and change the number accordingly:

<a id="arthor_tsv_link" href="#"> TSV-500</a>
<a id="arthor_tsv_link_5000" href="#"> TSV-5,000</a>
<a id="arthor_tsv_link_50000" href="#"> TSV-50,000</a>
<a id="arthor_tsv_link_100000" href="#"> TSV-100,000</a>
<a id="arthor_tsv_link_max" href="#"> TSV-max</a>
<a id="arthor_csv_link" href="#"> CSV-500</a>
<a id="arthor_csv_link_5000" href="#"> CSV-5,000</a>
<a id="arthor_csv_link_50000" href="#"> CSV-50,000</a>
<a id="arthor_csv_link_100000" href="#"> CSV-100,000</a>
<a id="arthor_csv_link_max" href="#"> CSV-max</a>
<a id="arthor_sdf_link" href="#"> SDF-500</a>
<a id="arthor_sdf_link_5000" href="#"> SDF-5,000</a>
<a id="arthor_sdf_link_50000" href="#"> SDF-50,000</a>
<a id="arthor_sdf_link_100000" href="#"> SDF-100,000</a>
<a id="arthor_sdf_link_max" href="#"> SDF-max</a>

4. then vim .extract/webapps/ROOT/WEB-INF/static/js/index.js

5. search: '''function $(t){'''

6. in the function $(t), add these lines:

if(document.getElementById("arthor_tsv_link")) {
var e=i.a.param({query:s.b.query,type:s.b.type,draw:0,start:0,length:t,flags:s.b.flags}),n=s.b.url+"/dt/"+E(s.b.table)+"/search";i()("#arthor_sdf_link").attr("href",n+".sdf?"+e),i() ("#arthor_tsv_link").attr("href",n+".tsv?"+e),i()("#arthor_csv_link").attr("href",n+".csv?"+e)
}
if (document.getElementById("arthor_tsv_link_5000")) {
var e=i.a.param({query:s.b.query,type:s.b.type,draw:0,start:0,length:5000,flags:s.b.flags}),n=s.b.url+"/dt/"+E(s.b.table)+"/search";i()("#arthor_sdf_link_5000").attr("href",n+".sdf?"+e),i()("#arthor_tsv_link_5000").attr("href",n+".tsv?"+e),i()("#arthor_csv_link_5000").attr("href",n+".csv?"+e)
}
if (document.getElementById("arthor_tsv_link_50000")) {
var e=i.a.param({query:s.b.query,type:s.b.type,draw:0,start:0,length:50000,flags:s.b.flags}),n=s.b.url+"/dt/"+E(s.b.table)+"/search";i()("#arthor_sdf_link_50000").attr("href",n+".sdf?"+e),i()("#arthor_tsv_link_50000").attr("href",n+".tsv?"+e),i()("#arthor_csv_link_50000").attr("href",n+".csv?"+e)
}
if (document.getElementById("arthor_tsv_link_100000")) {
var e=i.a.param({query:s.b.query,type:s.b.type,draw:0,start:0,length:100000,flags:s.b.flags}),n=s.b.url+"/dt/"+E(s.b.table)+"/search";i()("#arthor_sdf_link_100000").attr("href",n+".sdf?"+e),i()("#arthor_tsv_link_100000").attr("href",n+".tsv?"+e),i()("#arthor_csv_link_100000").attr("href",n+".csv?"+e)
}
if (document.getElementById("arthor_tsv_link_max")) {
var e=i.a.param({query:s.b.query,type:s.b.type,draw:0,start:0,length:1000000000,flags:s.b.flags}),n=s.b.url+"/dt/"+E(s.b.table)+"/search";i()("#arthor_sdf_link_max").attr("href",n+".sdf?"+e),i()("#arthor_tsv_link_max").attr("href",n+".tsv?"+e),i()("#arthor_csv_link_max").attr("href",n+".csv?"+e)
}

=== Take out Similarity Button ===
vim .extract/webapps/ROOT/WEB-INF/static/index.html
search: '''Similarity'''
Comment out this line '''< li value="Similarity" onclick="setSearchType(this)" class="first"> Similarity </li >''' //added spaces at the beginning and end so prevent wiki from converting it
Then add "first" in Substructure's class
=== Hyperlink to zinc20 ===
vim .extract/webapps/ROOT/WEB-INF/static/js/index.js
search: '''table_name'''
*find this line "" + d + ""
*replace with '''"<a target='_blank' href='https://zinc20.docking.org/substances/"+d+"'>" + d + "</a>"''' //added spaces at the beginning and end so prevent wiki from converting it

=== Make Input Box Work ===
At the end of the Arthor config file add this:
Resolver=https://sw.docking.org/util/smi2mol?smi=%s
To copy smiles in the input box:
vim .extract/webapps/ROOT/WEB-INF/static/js/index.js
search this: “var e=t.src.smiles()”
add this after the semi-colon
document.getElementById("ar_text_input").value = e;

Arthor Documentation for Future Developer

2021-05-04T17:06:12Z

Jgutie11: /* Arthor Round Table Nodes */

Written by Jennifer Young on December 16, 2019. Last edited January 05, 2021

==Install and Set Up on TomCat (Method 1)==
Arthor ran on n-1-136, which runs CentOS Linux release 7.7.1908 (Core). You can check the version of CentOS with the following command
cat /etc/centos-release

Check your current version of Java with the following command:
java -version

On n-1-136 we are running openjdk version "1.8.0_222", OpenJDK Runtime Environment (build 1.8.0_222-b10), and OpenJDK 64-Bit Server VM (build 25.222-b10, mixed mode)
If Java is not installed, install it using yum

==See this wiki page for more detailed information about installing Tomcat on our cluster==
http://wiki.docking.org/index.php/Tomcat_Installation

==Open port for Arthor==
In order for Arthor to be usable in the browser, the port you wish to run it on must be opened.
https://www.thegeekdiary.com/how-to-open-a-ports-in-centos-rhel-7/

===Step 1: Check Port Status===
Check that the port is not open and that Apache is not showing that port.
netstat -na | grep <port number you are checking>

lsof -i -P |grep http

===Step 2: Check Port Status in IP Tables===
iptables-save | grep <port number you are checking>

I skipped Step 3 from the guide, because there was a lot of information in the /etc/services file and I didn't want to edit it and break something.

===Step 4: Open Firewall Ports===
I did not include the zone=public section because the stand-alone servers are usually used for private instances of Arthor and SmallWorld.
Run as root.
firewall-cmd --add-port=<port number you are adding>/tcp --permanent

You need to reload the firewall after a change is made.
firewall-cmd --reload

===Step 5: Check that port is working===
To check that the port is active, run.
iptables -nL

You should see something along the lines of:
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:<port number you're adding> ctstate NEW,UNTRACKED

==How to run standalone Arthor instance==

===Step 1: Use or start a bash shell===
You can check your default shell using
echo $SHELL

If your default shell is csh, use
bash
to start a new bash shell in the current terminal window. Note that echo $SHELL will show you your default shell regardless of the current shell.

===Step 2: Set your environment variables===
export ARTHOR_DIR=/opt/nextmove/arthor/arthor-3.3-centos7
export PATH=$ARTHOR_DIR/bin/:$PATH

Make sure the ARTHOR_DIR variable is set to the directory for the latest version of Arthor or whichever version you would like to test.
The PATH environment variable is needed if you wish to use the Arthor tools from the command line

===Step 3: Run the arthor-server.jar===
java -jar /opt/nextmove/arthor/arthor-3.3-centos7/java/arthor.jar --httpPort <your httpPort>

==Setting environment variables for an Arthor Server==
Set the environment variables in the setenv.sh file. Note: Be sure to edit the file in the directory corresponding to the latest version of TomCat. As of December 2019, we are running 9.0.27 on n-1-136.

vim /opt/tomcat/apache-tomcat-9.0.27/bin/setenv.sh

Add the line below to the setenv.sh file above, or substitute the path to wherever you currently store the arthor.cfg file
export ARTHOR_CONFIG=/usr/local/tomcat/arthor.cfg

Here is an example of the arthor.cfg file:
BinDir=/opt/nextmove/arthor/arthor-3.3-centos7/bin
DataDir=/local2/arthor_local_8081/
MaxConcurrentSearches=6
MaxThreadsPerSearch=8
AutomaticIndex=false
AsyncHitCountMax=1000000
Resolver=https://sw.docking.org/util/smi2mol?smi=%s

=== Configuration Details ===

*'''BinDir''': is the location of the Arthor command line binaries. These are used to generate the Arthor index files and to perform searches directly on n-1-136. An example of this would be using atdbgrep for substructure search.

*'''DataDir''': This is the directory where the Arthor data files live. Location where the index files will be created and loaded from.

*'''MaxConcurrentSearches''': Controls the maximum number of searches that can be run concurrently by setting the database pool size. When switching between a large number of databases it can be useful to have a larger pool size, the only trade off is keeping file pointers open.

*'''MaxThreadsPerSearch''': The number of threads to use for both ATDB and ATFP searches

*Set '''AutomaticIndex''' to false if you don't want new smiles files added to the data directory to be indexed automatically

*'''AsyncHitCountMax''': The upper-bound for the number of hits to retrieve in background searches.

*'''Resolver''': Using Smallworld API, allows input box to take in a SMILE format and automatically draw on the board.

Check Arthor manual for more configuration options.

==Background==
Before working with Arthor, it is recommended that you familiarize yourself with the Arthor documentation. Some useful pages to look at include 3-5, 22-25 and 33-39. Of course, reading everything would be the best!

==Checking Disk Space Usage==
Before building arthor indexes, it's always a good thing to check what percent of the memory is being used. Try to be cautious with how much memory you have left, and make sure to check while building indexes to make sure that you have enough space. To check, run the following command:

df -h /<directory with disc>

==Building Large Databases==
At the moment, we are building databases of size 500M molecules by merging smile files. There are multiple methods of trying to create large databases, one being merging based off of the same H?? prefix and stopping once the database reaches > 500M molecules (or whatever upperbound you want to use). Here is some python code that simulates this merging process. Essentially the program takes all of the .smi files within an input directory, sorts them lexiographically, and begins merging these .smi files together in order until the size reaches > 500M molecules.

Feel free to modify it if you think a better method exists.

import subprocess
import sys
import os

from os import listdir
from os.path import isfile, join

mypath = "<Path to directory holding .smi files>"
onlyfiles = [f for f in listdir(mypath) if isfile(join(mypath, f))]
onlyfiles.sort()

create_fp = True
cur_mols = 0
lower_bound = 500000000
upper_bound = 600000000
files_to_merge = []

def merge_files(f_t_m):
arr = f_t_m[0].split(".")
arr2 = f_t_m[len(f_t_m) - 1].split(".")
file_name_merge = (arr[0] + "_" + arr2[0] + ".smi")
print ("File being created: " + file_name_merge)

for file in f_t_m:
tmp = file.split(".")
process = subprocess.Popen("cat " + join(mypath, file) + " >> " + file_name_merge, shell=True)
process.wait()

for file in onlyfiles:
arr = file.split(".")

if (arr[len(arr) - 1] == "smi"):
print("Working with " + file)
mol = sum(1 for line in open(join(mypath, file)))
print(file, mol, cur_mols)

if (cur_mols + mol > lower_bound):
if (cur_mols + mol < upper_bound):
files_to_merge.append(file)
merge_files(files_to_merge)
cur_mols = 0
files_to_merge.clear()
else:
merge_files(files_to_merge)
files_to_merge.clear()
files_to_merge.append(file)
merge_files(files_to_merge)
cur_mols = 0
files_to_merge.clear()
else:
cur_mols += mol
files_to_merge.append(file)

if (len(files_to_merge) != 0):
merge_files(files_to_merge)

==Building Arthor Indexes==
Once you've merged the .smi files together, it's time to start building the databases themselves. To do this we use the command

smi2atdb -j 0 -p <The .smi file> <The .atdb>

The flag "-j 0" enables parallel generation and utilizes all available processors to generate the .atdb file. The "-p" flag stores the offset position in the ATDB file. Since we're building indexes for the Web Application, you must use the "-p" flag when building indexes. Please note that the name of the .smi file should also be the name of the .atdb file. That way, the Web Application knows to use these files together and correctly display the required images. Refer to pages 33-34 in the Arthor documentation for more information.

If there are too many large .smi files and you do not want to manually build each .atdb file, you can use this python script which takes all of the .smi files in the current directory and converts them to .atdb files. Make sure to modify mypath to the directory containing the .smi files. You can change the variable "create_fp" to false if you don't want to create .atdb.fp files (refer to page 9 in the Arthor documentation).

import subprocess
import sys
import os

from os import listdir
from os.path import isfile, join

mypath = "<Path containing the .smi files"
onlyfiles = [f for f in listdir(mypath) if isfile(join(mypath, f))]

create_fp = True

for file in onlyfiles:
arr = file.split(".")

if (arr[len(arr) - 1] == "smi"):
process = subprocess.Popen("/nfs/ex9/work/xyz/psql/arthor-3.3-centos7/bin/smi2atdb -j 0 -p {0} {1}.atdb".format(join(mypath, file), arr[0]), shell=True)
process.wait()

print("SUCCESS! {0}.atdb file was created!".format(arr[0]))

if (create_fp):
process = subprocess.Popen("/nfs/ex9/work/xyz/psql/arthor-3.3-centos7/bin/atdb2fp -j 0 {0}.atdb".format(arr[0]), shell=True)
process.wait()

print("SUCCESS! {0}.atdb.fp file was created!".format(arr[0]))

==Uploading Indexes to the Web Application==
One can upload indexes to the Web Application by changing the "DATADIR" variable in the arthor.cfg file to the directory holding the .atdb files. This is already set up on n-1-136 and n-5-34.

==Virtual Memory==
In addition to modifying the arthor.cfg file, virtual memory can also be used to make queries faster. There can still be More information can be found in pages 10-16 in the Arthor Documentation.

==Setting up Round Table==
This is a new feature in Arthor 3.0 and is currently beta (January 2020). See section 2.4 in the manual
As explained in the manual, "Round Table allows you to serve and split chemical searches across multiple host machines. The implementation provides a lightweight proxy that forwards requests to other Arthor host servers that do the actual search. Communication is done using the existing Web APIs.

Since Arthor requires CentOS 7, as of January 2020 we have 6 servers that are capable of running Arthor with Round Table.

===Setting up Host Server===
If we want to add machines to the Round Table, for example 'nun' and 'samekh', we need to edit their arthor.cfg file so that when our Local Machine passes commands these secondary servers know to perform the search they are given.

$ cat arthor.cfg
MaxThreadsPerSearch=4
AutomaticIndex=false
DATADIR=<Directory where smiles are located>

We then run the jar server on each of these host machines containing data on any available port.

java -jar /nfs/ex9/work/xyz/psql/arthor-3.3-centos7/java/arthor.jar --httpPort <port>

For our local machine, the arthor.cfg file will look different.

$ cat arthor.cfg
[RoundTable]
RemoteClient=http://skynet:<port number where jar server is running>/
RemoteClient=http://hal:<port number where jar server is running>/

Please refer to Section 2 in the RoundTable Documentation file (pages 6-8) for more useful information on configuration.

Then run the following command on n-1-136:

java -jar /nfs/ex9/work/xyz/psql/arthor-3.3-centos7/java/arthor.jar --httpPort <port>

==='''***Arthor configs and frontend code are located in /nfs/exc/arthor_configs/***'''===

===Public Arthor===
{| class="wikitable"
|-
! CentOS 7 Machine
! Port
! Total Files Size
! Round Table Data Directory
! Active
|-
| samekh
| 10.20.0.41:8000
| 2.4TB
| /local2/public_arthor/
| active
|-
| nun
| 10.20.0.40:8000
| 2.4TB
| /local2/public_arthor/
| active
|-
| n-9-22
| 10.20.9.22:8000
| 2.4TB
| /export/db4/public_arthor/
| active
|-
|}

===Arthor Round Table Head===
{| class="wikitable"
|-
! CentOS 7 Machine
! Port
! Database
! Round Table Data Directory
! Active
|-
| samekh
| 10.20.0.41:8080
| REAL-Space-21Q1(private)
| /local2/arthor_database/
| active
|-
| nun
| 10.20.0.40:8080
| REAL_Space_21Q2(super private)
| /local2/arthor_database/
| active
|-
|}

===Arthor Round Table Nodes===
{| class="wikitable"
|-
! CentOS 7 Machine
! Port
! Database
! Total Files Size
! Round Table Data Directory
!Active
|-
| samekh
| 10.20.0.41:8008
| REAL-Space-21Q1-M, REAL-Space-21Q1-S, carboxylates_21Q2
| 4.0TB
| /local2/arthor_database/
| active
|-
| nun
| 10.20.0.40:8008
| 345_acids_20Q4, REAL_Space_21Q2-M, REAL_Space_21Q2-S, acids_21Q2, zinc_21Q1_H01~H25
| 3.1TB
| /local2/arthor_database/
| active
|-
| n-1-17
| 10.20.1.17:8008
|
|
| /local2/arthor_database/
| active
|-
| n-5-32
| 10.20.5.32:8008
| zinc_21Q1_H26~H30
| 6.6TB
| /local2/arthor_database/
| active
|-
| n-5-33
| 10.20.5.33:8008
| acids_21Q2
| 9.1GB
| /local2/arthor_database/
| active
|-
|}

===Arthor Local 8081 (Datasets all local to samekh/nun)===
{| class="wikitable"
|-
! CentOS 7 Machine
! Port
! Database
! Total Files Size
! Round Table Data Directory
!Active
|-
| samekh
| 10.20.0.41:8081
| (TBA)
| (TBA)
| /local2/arthor_local_8081/
| active
|-
| nun
| 10.20.0.40:8081
| (TBA)
| (TBA)
| /local2/arthor_local_8081/
| active
|-
|}

== Customizing Arthor Frontend to our needs ==
The frontend Arthor code is located at '''/nfs/exc/arthor_configs/*''' and the '''*''' is based on current running version.
=== Add Arthor Download Options ===
==== For Arthor 3.4: ====
1. vim .extract/webapps/ROOT/WEB-INF/static/index.html

2. search: '''arthor_tsv_link'''

3. in the div with the class=”dropdown-content”, add these link options and change the number accordingly:

<a id="arthor_tsv_link" href="#"> TSV-500</a>
<a id="arthor_tsv_link_5000" href="#"> TSV-5,000</a>
<a id="arthor_tsv_link_50000" href="#"> TSV-50,000</a>
<a id="arthor_tsv_link_100000" href="#"> TSV-100,000</a>
<a id="arthor_tsv_link_max" href="#"> TSV-max</a>
<a id="arthor_csv_link" href="#"> CSV-500</a>
<a id="arthor_csv_link_5000" href="#"> CSV-5,000</a>
<a id="arthor_csv_link_50000" href="#"> CSV-50,000</a>
<a id="arthor_csv_link_100000" href="#"> CSV-100,000</a>
<a id="arthor_csv_link_max" href="#"> CSV-max</a>
<a id="arthor_sdf_link" href="#"> SDF-500</a>
<a id="arthor_sdf_link_5000" href="#"> SDF-5,000</a>
<a id="arthor_sdf_link_50000" href="#"> SDF-50,000</a>
<a id="arthor_sdf_link_100000" href="#"> SDF-100,000</a>
<a id="arthor_sdf_link_max" href="#"> SDF-max</a>

4. then vim .extract/webapps/ROOT/WEB-INF/static/js/index.js

5. search: '''function $(t){'''

6. in the function $(t), add these lines:

if(document.getElementById("arthor_tsv_link")) {
var e=i.a.param({query:s.b.query,type:s.b.type,draw:0,start:0,length:t,flags:s.b.flags}),n=s.b.url+"/dt/"+E(s.b.table)+"/search";i()("#arthor_sdf_link").attr("href",n+".sdf?"+e),i() ("#arthor_tsv_link").attr("href",n+".tsv?"+e),i()("#arthor_csv_link").attr("href",n+".csv?"+e)
}
if (document.getElementById("arthor_tsv_link_5000")) {
var e=i.a.param({query:s.b.query,type:s.b.type,draw:0,start:0,length:5000,flags:s.b.flags}),n=s.b.url+"/dt/"+E(s.b.table)+"/search";i()("#arthor_sdf_link_5000").attr("href",n+".sdf?"+e),i()("#arthor_tsv_link_5000").attr("href",n+".tsv?"+e),i()("#arthor_csv_link_5000").attr("href",n+".csv?"+e)
}
if (document.getElementById("arthor_tsv_link_50000")) {
var e=i.a.param({query:s.b.query,type:s.b.type,draw:0,start:0,length:50000,flags:s.b.flags}),n=s.b.url+"/dt/"+E(s.b.table)+"/search";i()("#arthor_sdf_link_50000").attr("href",n+".sdf?"+e),i()("#arthor_tsv_link_50000").attr("href",n+".tsv?"+e),i()("#arthor_csv_link_50000").attr("href",n+".csv?"+e)
}
if (document.getElementById("arthor_tsv_link_100000")) {
var e=i.a.param({query:s.b.query,type:s.b.type,draw:0,start:0,length:100000,flags:s.b.flags}),n=s.b.url+"/dt/"+E(s.b.table)+"/search";i()("#arthor_sdf_link_100000").attr("href",n+".sdf?"+e),i()("#arthor_tsv_link_100000").attr("href",n+".tsv?"+e),i()("#arthor_csv_link_100000").attr("href",n+".csv?"+e)
}
if (document.getElementById("arthor_tsv_link_max")) {
var e=i.a.param({query:s.b.query,type:s.b.type,draw:0,start:0,length:1000000000,flags:s.b.flags}),n=s.b.url+"/dt/"+E(s.b.table)+"/search";i()("#arthor_sdf_link_max").attr("href",n+".sdf?"+e),i()("#arthor_tsv_link_max").attr("href",n+".tsv?"+e),i()("#arthor_csv_link_max").attr("href",n+".csv?"+e)
}

=== Take out Similarity Button ===
vim .extract/webapps/ROOT/WEB-INF/static/index.html
search: '''Similarity'''
Comment out this line '''< li value="Similarity" onclick="setSearchType(this)" class="first"> Similarity </li >''' //added spaces at the beginning and end so prevent wiki from converting it
Then add "first" in Substructure's class
=== Hyperlink to zinc20 ===
vim .extract/webapps/ROOT/WEB-INF/static/js/index.js
search: '''table_name'''
*find this line "" + d + ""
*replace with '''"<a target='_blank' href='https://zinc20.docking.org/substances/"+d+"'>" + d + "</a>"''' //added spaces at the beginning and end so prevent wiki from converting it

=== Make Input Box Work ===
At the end of the Arthor config file add this:
Resolver=https://sw.docking.org/util/smi2mol?smi=%s
To copy smiles in the input box:
vim .extract/webapps/ROOT/WEB-INF/static/js/index.js
search this: “var e=t.src.smiles()”
add this after the semi-colon
document.getElementById("ar_text_input").value = e;

Arthor Documentation for Future Developer

2021-05-04T00:21:02Z

Jgutie11: /* Arthor Local 8081 (Datasets all local to samekh/nun) */

Written by Jennifer Young on December 16, 2019. Last edited January 05, 2021

==Install and Set Up on TomCat (Method 1)==
Arthor ran on n-1-136, which runs CentOS Linux release 7.7.1908 (Core). You can check the version of CentOS with the following command
cat /etc/centos-release

Check your current version of Java with the following command:
java -version

On n-1-136 we are running openjdk version "1.8.0_222", OpenJDK Runtime Environment (build 1.8.0_222-b10), and OpenJDK 64-Bit Server VM (build 25.222-b10, mixed mode)
If Java is not installed, install it using yum

==See this wiki page for more detailed information about installing Tomcat on our cluster==
http://wiki.docking.org/index.php/Tomcat_Installation

==Open port for Arthor==
In order for Arthor to be usable in the browser, the port you wish to run it on must be opened.
https://www.thegeekdiary.com/how-to-open-a-ports-in-centos-rhel-7/

===Step 1: Check Port Status===
Check that the port is not open and that Apache is not showing that port.
netstat -na | grep <port number you are checking>

lsof -i -P |grep http

===Step 2: Check Port Status in IP Tables===
iptables-save | grep <port number you are checking>

I skipped Step 3 from the guide, because there was a lot of information in the /etc/services file and I didn't want to edit it and break something.

===Step 4: Open Firewall Ports===
I did not include the zone=public section because the stand-alone servers are usually used for private instances of Arthor and SmallWorld.
Run as root.
firewall-cmd --add-port=<port number you are adding>/tcp --permanent

You need to reload the firewall after a change is made.
firewall-cmd --reload

===Step 5: Check that port is working===
To check that the port is active, run.
iptables -nL

You should see something along the lines of:
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:<port number you're adding> ctstate NEW,UNTRACKED

==How to run standalone Arthor instance==

===Step 1: Use or start a bash shell===
You can check your default shell using
echo $SHELL

If your default shell is csh, use
bash
to start a new bash shell in the current terminal window. Note that echo $SHELL will show you your default shell regardless of the current shell.

===Step 2: Set your environment variables===
export ARTHOR_DIR=/opt/nextmove/arthor/arthor-3.3-centos7
export PATH=$ARTHOR_DIR/bin/:$PATH

Make sure the ARTHOR_DIR variable is set to the directory for the latest version of Arthor or whichever version you would like to test.
The PATH environment variable is needed if you wish to use the Arthor tools from the command line

===Step 3: Run the arthor-server.jar===
java -jar /opt/nextmove/arthor/arthor-3.3-centos7/java/arthor.jar --httpPort <your httpPort>

==Setting environment variables for an Arthor Server==
Set the environment variables in the setenv.sh file. Note: Be sure to edit the file in the directory corresponding to the latest version of TomCat. As of December 2019, we are running 9.0.27 on n-1-136.

vim /opt/tomcat/apache-tomcat-9.0.27/bin/setenv.sh

Add the line below to the setenv.sh file above, or substitute the path to wherever you currently store the arthor.cfg file
export ARTHOR_CONFIG=/usr/local/tomcat/arthor.cfg

Here is an example of the arthor.cfg file:
BinDir=/opt/nextmove/arthor/arthor-3.3-centos7/bin
DataDir=/local2/arthor_local_8081/
MaxConcurrentSearches=6
MaxThreadsPerSearch=8
AutomaticIndex=false
AsyncHitCountMax=1000000
Resolver=https://sw.docking.org/util/smi2mol?smi=%s

=== Configuration Details ===

*'''BinDir''': is the location of the Arthor command line binaries. These are used to generate the Arthor index files and to perform searches directly on n-1-136. An example of this would be using atdbgrep for substructure search.

*'''DataDir''': This is the directory where the Arthor data files live. Location where the index files will be created and loaded from.

*'''MaxConcurrentSearches''': Controls the maximum number of searches that can be run concurrently by setting the database pool size. When switching between a large number of databases it can be useful to have a larger pool size, the only trade off is keeping file pointers open.

*'''MaxThreadsPerSearch''': The number of threads to use for both ATDB and ATFP searches

*Set '''AutomaticIndex''' to false if you don't want new smiles files added to the data directory to be indexed automatically

*'''AsyncHitCountMax''': The upper-bound for the number of hits to retrieve in background searches.

*'''Resolver''': Using Smallworld API, allows input box to take in a SMILE format and automatically draw on the board.

Check Arthor manual for more configuration options.

==Background==
Before working with Arthor, it is recommended that you familiarize yourself with the Arthor documentation. Some useful pages to look at include 3-5, 22-25 and 33-39. Of course, reading everything would be the best!

==Checking Disk Space Usage==
Before building arthor indexes, it's always a good thing to check what percent of the memory is being used. Try to be cautious with how much memory you have left, and make sure to check while building indexes to make sure that you have enough space. To check, run the following command:

df -h /<directory with disc>

==Building Large Databases==
At the moment, we are building databases of size 500M molecules by merging smile files. There are multiple methods of trying to create large databases, one being merging based off of the same H?? prefix and stopping once the database reaches > 500M molecules (or whatever upperbound you want to use). Here is some python code that simulates this merging process. Essentially the program takes all of the .smi files within an input directory, sorts them lexiographically, and begins merging these .smi files together in order until the size reaches > 500M molecules.

Feel free to modify it if you think a better method exists.

import subprocess
import sys
import os

from os import listdir
from os.path import isfile, join

mypath = "<Path to directory holding .smi files>"
onlyfiles = [f for f in listdir(mypath) if isfile(join(mypath, f))]
onlyfiles.sort()

create_fp = True
cur_mols = 0
lower_bound = 500000000
upper_bound = 600000000
files_to_merge = []

def merge_files(f_t_m):
arr = f_t_m[0].split(".")
arr2 = f_t_m[len(f_t_m) - 1].split(".")
file_name_merge = (arr[0] + "_" + arr2[0] + ".smi")
print ("File being created: " + file_name_merge)

for file in f_t_m:
tmp = file.split(".")
process = subprocess.Popen("cat " + join(mypath, file) + " >> " + file_name_merge, shell=True)
process.wait()

for file in onlyfiles:
arr = file.split(".")

if (arr[len(arr) - 1] == "smi"):
print("Working with " + file)
mol = sum(1 for line in open(join(mypath, file)))
print(file, mol, cur_mols)

if (cur_mols + mol > lower_bound):
if (cur_mols + mol < upper_bound):
files_to_merge.append(file)
merge_files(files_to_merge)
cur_mols = 0
files_to_merge.clear()
else:
merge_files(files_to_merge)
files_to_merge.clear()
files_to_merge.append(file)
merge_files(files_to_merge)
cur_mols = 0
files_to_merge.clear()
else:
cur_mols += mol
files_to_merge.append(file)

if (len(files_to_merge) != 0):
merge_files(files_to_merge)

==Building Arthor Indexes==
Once you've merged the .smi files together, it's time to start building the databases themselves. To do this we use the command

smi2atdb -j 0 -p <The .smi file> <The .atdb>

The flag "-j 0" enables parallel generation and utilizes all available processors to generate the .atdb file. The "-p" flag stores the offset position in the ATDB file. Since we're building indexes for the Web Application, you must use the "-p" flag when building indexes. Please note that the name of the .smi file should also be the name of the .atdb file. That way, the Web Application knows to use these files together and correctly display the required images. Refer to pages 33-34 in the Arthor documentation for more information.

If there are too many large .smi files and you do not want to manually build each .atdb file, you can use this python script which takes all of the .smi files in the current directory and converts them to .atdb files. Make sure to modify mypath to the directory containing the .smi files. You can change the variable "create_fp" to false if you don't want to create .atdb.fp files (refer to page 9 in the Arthor documentation).

import subprocess
import sys
import os

from os import listdir
from os.path import isfile, join

mypath = "<Path containing the .smi files"
onlyfiles = [f for f in listdir(mypath) if isfile(join(mypath, f))]

create_fp = True

for file in onlyfiles:
arr = file.split(".")

if (arr[len(arr) - 1] == "smi"):
process = subprocess.Popen("/nfs/ex9/work/xyz/psql/arthor-3.3-centos7/bin/smi2atdb -j 0 -p {0} {1}.atdb".format(join(mypath, file), arr[0]), shell=True)
process.wait()

print("SUCCESS! {0}.atdb file was created!".format(arr[0]))

if (create_fp):
process = subprocess.Popen("/nfs/ex9/work/xyz/psql/arthor-3.3-centos7/bin/atdb2fp -j 0 {0}.atdb".format(arr[0]), shell=True)
process.wait()

print("SUCCESS! {0}.atdb.fp file was created!".format(arr[0]))

==Uploading Indexes to the Web Application==
One can upload indexes to the Web Application by changing the "DATADIR" variable in the arthor.cfg file to the directory holding the .atdb files. This is already set up on n-1-136 and n-5-34.

==Virtual Memory==
In addition to modifying the arthor.cfg file, virtual memory can also be used to make queries faster. There can still be More information can be found in pages 10-16 in the Arthor Documentation.

==Setting up Round Table==
This is a new feature in Arthor 3.0 and is currently beta (January 2020). See section 2.4 in the manual
As explained in the manual, "Round Table allows you to serve and split chemical searches across multiple host machines. The implementation provides a lightweight proxy that forwards requests to other Arthor host servers that do the actual search. Communication is done using the existing Web APIs.

Since Arthor requires CentOS 7, as of January 2020 we have 6 servers that are capable of running Arthor with Round Table.

===Setting up Host Server===
If we want to add machines to the Round Table, for example 'nun' and 'samekh', we need to edit their arthor.cfg file so that when our Local Machine passes commands these secondary servers know to perform the search they are given.

$ cat arthor.cfg
MaxThreadsPerSearch=4
AutomaticIndex=false
DATADIR=<Directory where smiles are located>

We then run the jar server on each of these host machines containing data on any available port.

java -jar /nfs/ex9/work/xyz/psql/arthor-3.3-centos7/java/arthor.jar --httpPort <port>

For our local machine, the arthor.cfg file will look different.

$ cat arthor.cfg
[RoundTable]
RemoteClient=http://skynet:<port number where jar server is running>/
RemoteClient=http://hal:<port number where jar server is running>/

Please refer to Section 2 in the RoundTable Documentation file (pages 6-8) for more useful information on configuration.

Then run the following command on n-1-136:

java -jar /nfs/ex9/work/xyz/psql/arthor-3.3-centos7/java/arthor.jar --httpPort <port>

==='''***Arthor configs and frontend code are located in /nfs/exc/arthor_configs/***'''===

===Public Arthor===
{| class="wikitable"
|-
! CentOS 7 Machine
! Port
! Total Files Size
! Round Table Data Directory
! Active
|-
| samekh
| 10.20.0.41:8000
| 2.4TB
| /local2/public_arthor/
| active
|-
| nun
| 10.20.0.40:8000
| 2.4TB
| /local2/public_arthor/
| active
|-
| n-9-22
| 10.20.9.22:8000
| 2.4TB
| /export/db4/public_arthor/
| active
|-
|}

===Arthor Round Table Head===
{| class="wikitable"
|-
! CentOS 7 Machine
! Port
! Database
! Round Table Data Directory
! Active
|-
| samekh
| 10.20.0.41:8080
| REAL-Space-21Q1(private)
| /local2/arthor_database/
| active
|-
| nun
| 10.20.0.40:8080
| REAL_Space_21Q2(super private)
| /local2/arthor_database/
| active
|-
|}

===Arthor Round Table Nodes===
{| class="wikitable"
|-
! CentOS 7 Machine
! Port
! Database
! Total Files Size
! Round Table Data Directory
!Active
|-
| samekh
| 10.20.0.41:8008
| REAL-Space-21Q1-M, REAL-Space-21Q1-S, carboxylates_21Q2
| 4.0TB
| /local2/arthor_database/
| active
|-
| nun
| 10.20.0.40:8008
| 345_acids_20Q4, REAL_Space_21Q2-M, REAL_Space_21Q2-S, acids_21Q2, zinc_21Q1_H01~H25
| 3.1TB
| /local2/arthor_database/
| active
|-
| n-1-17
| 10.20.1.17:8008
|
|
| /local2/arthor_database/
| active
|-
| n-5-32
| 10.20.5.32:8008
| zinc_21Q1_H26~H30
| (TBA)
| /local2/arthor_database/
| active
|-
| n-5-33
| 10.20.5.33:8008
| acids_21Q2
| 9.1GB
| /local2/arthor_database/
| active
|-
|}

===Arthor Local 8081 (Datasets all local to samekh/nun)===
{| class="wikitable"
|-
! CentOS 7 Machine
! Port
! Database
! Total Files Size
! Round Table Data Directory
!Active
|-
| samekh
| 10.20.0.41:8081
| (TBA)
| (TBA)
| /local2/arthor_local_8081/
| active
|-
| nun
| 10.20.0.40:8081
| (TBA)
| (TBA)
| /local2/arthor_local_8081/
| active
|-
|}

== Customizing Arthor Frontend to our needs ==
The frontend Arthor code is located at '''/nfs/exc/arthor_configs/*''' and the '''*''' is based on current running version.
=== Add Arthor Download Options ===
==== For Arthor 3.4: ====
1. vim .extract/webapps/ROOT/WEB-INF/static/index.html

2. search: '''arthor_tsv_link'''

3. in the div with the class=”dropdown-content”, add these link options and change the number accordingly:

<a id="arthor_tsv_link" href="#"> TSV-500</a>
<a id="arthor_tsv_link_5000" href="#"> TSV-5,000</a>
<a id="arthor_tsv_link_50000" href="#"> TSV-50,000</a>
<a id="arthor_tsv_link_100000" href="#"> TSV-100,000</a>
<a id="arthor_tsv_link_max" href="#"> TSV-max</a>
<a id="arthor_csv_link" href="#"> CSV-500</a>
<a id="arthor_csv_link_5000" href="#"> CSV-5,000</a>
<a id="arthor_csv_link_50000" href="#"> CSV-50,000</a>
<a id="arthor_csv_link_100000" href="#"> CSV-100,000</a>
<a id="arthor_csv_link_max" href="#"> CSV-max</a>
<a id="arthor_sdf_link" href="#"> SDF-500</a>
<a id="arthor_sdf_link_5000" href="#"> SDF-5,000</a>
<a id="arthor_sdf_link_50000" href="#"> SDF-50,000</a>
<a id="arthor_sdf_link_100000" href="#"> SDF-100,000</a>
<a id="arthor_sdf_link_max" href="#"> SDF-max</a>

4. then vim .extract/webapps/ROOT/WEB-INF/static/js/index.js

5. search: '''function $(t){'''

6. in the function $(t), add these lines:

if(document.getElementById("arthor_tsv_link")) {
var e=i.a.param({query:s.b.query,type:s.b.type,draw:0,start:0,length:t,flags:s.b.flags}),n=s.b.url+"/dt/"+E(s.b.table)+"/search";i()("#arthor_sdf_link").attr("href",n+".sdf?"+e),i() ("#arthor_tsv_link").attr("href",n+".tsv?"+e),i()("#arthor_csv_link").attr("href",n+".csv?"+e)
}
if (document.getElementById("arthor_tsv_link_5000")) {
var e=i.a.param({query:s.b.query,type:s.b.type,draw:0,start:0,length:5000,flags:s.b.flags}),n=s.b.url+"/dt/"+E(s.b.table)+"/search";i()("#arthor_sdf_link_5000").attr("href",n+".sdf?"+e),i()("#arthor_tsv_link_5000").attr("href",n+".tsv?"+e),i()("#arthor_csv_link_5000").attr("href",n+".csv?"+e)
}
if (document.getElementById("arthor_tsv_link_50000")) {
var e=i.a.param({query:s.b.query,type:s.b.type,draw:0,start:0,length:50000,flags:s.b.flags}),n=s.b.url+"/dt/"+E(s.b.table)+"/search";i()("#arthor_sdf_link_50000").attr("href",n+".sdf?"+e),i()("#arthor_tsv_link_50000").attr("href",n+".tsv?"+e),i()("#arthor_csv_link_50000").attr("href",n+".csv?"+e)
}
if (document.getElementById("arthor_tsv_link_100000")) {
var e=i.a.param({query:s.b.query,type:s.b.type,draw:0,start:0,length:100000,flags:s.b.flags}),n=s.b.url+"/dt/"+E(s.b.table)+"/search";i()("#arthor_sdf_link_100000").attr("href",n+".sdf?"+e),i()("#arthor_tsv_link_100000").attr("href",n+".tsv?"+e),i()("#arthor_csv_link_100000").attr("href",n+".csv?"+e)
}
if (document.getElementById("arthor_tsv_link_max")) {
var e=i.a.param({query:s.b.query,type:s.b.type,draw:0,start:0,length:1000000000,flags:s.b.flags}),n=s.b.url+"/dt/"+E(s.b.table)+"/search";i()("#arthor_sdf_link_max").attr("href",n+".sdf?"+e),i()("#arthor_tsv_link_max").attr("href",n+".tsv?"+e),i()("#arthor_csv_link_max").attr("href",n+".csv?"+e)
}

=== Take out Similarity Button ===
vim .extract/webapps/ROOT/WEB-INF/static/index.html
search: '''Similarity'''
Comment out this line '''< li value="Similarity" onclick="setSearchType(this)" class="first"> Similarity </li >''' //added spaces at the beginning and end so prevent wiki from converting it
Then add "first" in Substructure's class
=== Hyperlink to zinc20 ===
vim .extract/webapps/ROOT/WEB-INF/static/js/index.js
search: '''table_name'''
*find this line "" + d + ""
*replace with '''"<a target='_blank' href='https://zinc20.docking.org/substances/"+d+"'>" + d + "</a>"''' //added spaces at the beginning and end so prevent wiki from converting it

=== Make Input Box Work ===
At the end of the Arthor config file add this:
Resolver=https://sw.docking.org/util/smi2mol?smi=%s
To copy smiles in the input box:
vim .extract/webapps/ROOT/WEB-INF/static/js/index.js
search this: “var e=t.src.smiles()”
add this after the semi-colon
document.getElementById("ar_text_input").value = e;

Arthor Documentation for Future Developer

2021-05-04T00:19:41Z

Jgutie11: /* Setting up Round Table */

Written by Jennifer Young on December 16, 2019. Last edited January 05, 2021

==Install and Set Up on TomCat (Method 1)==
Arthor ran on n-1-136, which runs CentOS Linux release 7.7.1908 (Core). You can check the version of CentOS with the following command
cat /etc/centos-release

Check your current version of Java with the following command:
java -version

On n-1-136 we are running openjdk version "1.8.0_222", OpenJDK Runtime Environment (build 1.8.0_222-b10), and OpenJDK 64-Bit Server VM (build 25.222-b10, mixed mode)
If Java is not installed, install it using yum

==See this wiki page for more detailed information about installing Tomcat on our cluster==
http://wiki.docking.org/index.php/Tomcat_Installation

==Open port for Arthor==
In order for Arthor to be usable in the browser, the port you wish to run it on must be opened.
https://www.thegeekdiary.com/how-to-open-a-ports-in-centos-rhel-7/

===Step 1: Check Port Status===
Check that the port is not open and that Apache is not showing that port.
netstat -na | grep <port number you are checking>

lsof -i -P |grep http

===Step 2: Check Port Status in IP Tables===
iptables-save | grep <port number you are checking>

I skipped Step 3 from the guide, because there was a lot of information in the /etc/services file and I didn't want to edit it and break something.

===Step 4: Open Firewall Ports===
I did not include the zone=public section because the stand-alone servers are usually used for private instances of Arthor and SmallWorld.
Run as root.
firewall-cmd --add-port=<port number you are adding>/tcp --permanent

You need to reload the firewall after a change is made.
firewall-cmd --reload

===Step 5: Check that port is working===
To check that the port is active, run.
iptables -nL

You should see something along the lines of:
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:<port number you're adding> ctstate NEW,UNTRACKED

==How to run standalone Arthor instance==

===Step 1: Use or start a bash shell===
You can check your default shell using
echo $SHELL

If your default shell is csh, use
bash
to start a new bash shell in the current terminal window. Note that echo $SHELL will show you your default shell regardless of the current shell.

===Step 2: Set your environment variables===
export ARTHOR_DIR=/opt/nextmove/arthor/arthor-3.3-centos7
export PATH=$ARTHOR_DIR/bin/:$PATH

Make sure the ARTHOR_DIR variable is set to the directory for the latest version of Arthor or whichever version you would like to test.
The PATH environment variable is needed if you wish to use the Arthor tools from the command line

===Step 3: Run the arthor-server.jar===
java -jar /opt/nextmove/arthor/arthor-3.3-centos7/java/arthor.jar --httpPort <your httpPort>

==Setting environment variables for an Arthor Server==
Set the environment variables in the setenv.sh file. Note: Be sure to edit the file in the directory corresponding to the latest version of TomCat. As of December 2019, we are running 9.0.27 on n-1-136.

vim /opt/tomcat/apache-tomcat-9.0.27/bin/setenv.sh

Add the line below to the setenv.sh file above, or substitute the path to wherever you currently store the arthor.cfg file
export ARTHOR_CONFIG=/usr/local/tomcat/arthor.cfg

Here is an example of the arthor.cfg file:
BinDir=/opt/nextmove/arthor/arthor-3.3-centos7/bin
DataDir=/local2/arthor_local_8081/
MaxConcurrentSearches=6
MaxThreadsPerSearch=8
AutomaticIndex=false
AsyncHitCountMax=1000000
Resolver=https://sw.docking.org/util/smi2mol?smi=%s

=== Configuration Details ===

*'''BinDir''': is the location of the Arthor command line binaries. These are used to generate the Arthor index files and to perform searches directly on n-1-136. An example of this would be using atdbgrep for substructure search.

*'''DataDir''': This is the directory where the Arthor data files live. Location where the index files will be created and loaded from.

*'''MaxConcurrentSearches''': Controls the maximum number of searches that can be run concurrently by setting the database pool size. When switching between a large number of databases it can be useful to have a larger pool size, the only trade off is keeping file pointers open.

*'''MaxThreadsPerSearch''': The number of threads to use for both ATDB and ATFP searches

*Set '''AutomaticIndex''' to false if you don't want new smiles files added to the data directory to be indexed automatically

*'''AsyncHitCountMax''': The upper-bound for the number of hits to retrieve in background searches.

*'''Resolver''': Using Smallworld API, allows input box to take in a SMILE format and automatically draw on the board.

Check Arthor manual for more configuration options.

==Background==
Before working with Arthor, it is recommended that you familiarize yourself with the Arthor documentation. Some useful pages to look at include 3-5, 22-25 and 33-39. Of course, reading everything would be the best!

==Checking Disk Space Usage==
Before building arthor indexes, it's always a good thing to check what percent of the memory is being used. Try to be cautious with how much memory you have left, and make sure to check while building indexes to make sure that you have enough space. To check, run the following command:

df -h /<directory with disc>

==Building Large Databases==
At the moment, we are building databases of size 500M molecules by merging smile files. There are multiple methods of trying to create large databases, one being merging based off of the same H?? prefix and stopping once the database reaches > 500M molecules (or whatever upperbound you want to use). Here is some python code that simulates this merging process. Essentially the program takes all of the .smi files within an input directory, sorts them lexiographically, and begins merging these .smi files together in order until the size reaches > 500M molecules.

Feel free to modify it if you think a better method exists.

import subprocess
import sys
import os

from os import listdir
from os.path import isfile, join

mypath = "<Path to directory holding .smi files>"
onlyfiles = [f for f in listdir(mypath) if isfile(join(mypath, f))]
onlyfiles.sort()

create_fp = True
cur_mols = 0
lower_bound = 500000000
upper_bound = 600000000
files_to_merge = []

def merge_files(f_t_m):
arr = f_t_m[0].split(".")
arr2 = f_t_m[len(f_t_m) - 1].split(".")
file_name_merge = (arr[0] + "_" + arr2[0] + ".smi")
print ("File being created: " + file_name_merge)

for file in f_t_m:
tmp = file.split(".")
process = subprocess.Popen("cat " + join(mypath, file) + " >> " + file_name_merge, shell=True)
process.wait()

for file in onlyfiles:
arr = file.split(".")

if (arr[len(arr) - 1] == "smi"):
print("Working with " + file)
mol = sum(1 for line in open(join(mypath, file)))
print(file, mol, cur_mols)

if (cur_mols + mol > lower_bound):
if (cur_mols + mol < upper_bound):
files_to_merge.append(file)
merge_files(files_to_merge)
cur_mols = 0
files_to_merge.clear()
else:
merge_files(files_to_merge)
files_to_merge.clear()
files_to_merge.append(file)
merge_files(files_to_merge)
cur_mols = 0
files_to_merge.clear()
else:
cur_mols += mol
files_to_merge.append(file)

if (len(files_to_merge) != 0):
merge_files(files_to_merge)

==Building Arthor Indexes==
Once you've merged the .smi files together, it's time to start building the databases themselves. To do this we use the command

smi2atdb -j 0 -p <The .smi file> <The .atdb>

The flag "-j 0" enables parallel generation and utilizes all available processors to generate the .atdb file. The "-p" flag stores the offset position in the ATDB file. Since we're building indexes for the Web Application, you must use the "-p" flag when building indexes. Please note that the name of the .smi file should also be the name of the .atdb file. That way, the Web Application knows to use these files together and correctly display the required images. Refer to pages 33-34 in the Arthor documentation for more information.

If there are too many large .smi files and you do not want to manually build each .atdb file, you can use this python script which takes all of the .smi files in the current directory and converts them to .atdb files. Make sure to modify mypath to the directory containing the .smi files. You can change the variable "create_fp" to false if you don't want to create .atdb.fp files (refer to page 9 in the Arthor documentation).

import subprocess
import sys
import os

from os import listdir
from os.path import isfile, join

mypath = "<Path containing the .smi files"
onlyfiles = [f for f in listdir(mypath) if isfile(join(mypath, f))]

create_fp = True

for file in onlyfiles:
arr = file.split(".")

if (arr[len(arr) - 1] == "smi"):
process = subprocess.Popen("/nfs/ex9/work/xyz/psql/arthor-3.3-centos7/bin/smi2atdb -j 0 -p {0} {1}.atdb".format(join(mypath, file), arr[0]), shell=True)
process.wait()

print("SUCCESS! {0}.atdb file was created!".format(arr[0]))

if (create_fp):
process = subprocess.Popen("/nfs/ex9/work/xyz/psql/arthor-3.3-centos7/bin/atdb2fp -j 0 {0}.atdb".format(arr[0]), shell=True)
process.wait()

print("SUCCESS! {0}.atdb.fp file was created!".format(arr[0]))

==Uploading Indexes to the Web Application==
One can upload indexes to the Web Application by changing the "DATADIR" variable in the arthor.cfg file to the directory holding the .atdb files. This is already set up on n-1-136 and n-5-34.

==Virtual Memory==
In addition to modifying the arthor.cfg file, virtual memory can also be used to make queries faster. There can still be More information can be found in pages 10-16 in the Arthor Documentation.

==Setting up Round Table==
This is a new feature in Arthor 3.0 and is currently beta (January 2020). See section 2.4 in the manual
As explained in the manual, "Round Table allows you to serve and split chemical searches across multiple host machines. The implementation provides a lightweight proxy that forwards requests to other Arthor host servers that do the actual search. Communication is done using the existing Web APIs.

Since Arthor requires CentOS 7, as of January 2020 we have 6 servers that are capable of running Arthor with Round Table.

===Setting up Host Server===
If we want to add machines to the Round Table, for example 'nun' and 'samekh', we need to edit their arthor.cfg file so that when our Local Machine passes commands these secondary servers know to perform the search they are given.

$ cat arthor.cfg
MaxThreadsPerSearch=4
AutomaticIndex=false
DATADIR=<Directory where smiles are located>

We then run the jar server on each of these host machines containing data on any available port.

java -jar /nfs/ex9/work/xyz/psql/arthor-3.3-centos7/java/arthor.jar --httpPort <port>

For our local machine, the arthor.cfg file will look different.

$ cat arthor.cfg
[RoundTable]
RemoteClient=http://skynet:<port number where jar server is running>/
RemoteClient=http://hal:<port number where jar server is running>/

Please refer to Section 2 in the RoundTable Documentation file (pages 6-8) for more useful information on configuration.

Then run the following command on n-1-136:

java -jar /nfs/ex9/work/xyz/psql/arthor-3.3-centos7/java/arthor.jar --httpPort <port>

==='''***Arthor configs and frontend code are located in /nfs/exc/arthor_configs/***'''===

===Public Arthor===
{| class="wikitable"
|-
! CentOS 7 Machine
! Port
! Total Files Size
! Round Table Data Directory
! Active
|-
| samekh
| 10.20.0.41:8000
| 2.4TB
| /local2/public_arthor/
| active
|-
| nun
| 10.20.0.40:8000
| 2.4TB
| /local2/public_arthor/
| active
|-
| n-9-22
| 10.20.9.22:8000
| 2.4TB
| /export/db4/public_arthor/
| active
|-
|}

===Arthor Round Table Head===
{| class="wikitable"
|-
! CentOS 7 Machine
! Port
! Database
! Round Table Data Directory
! Active
|-
| samekh
| 10.20.0.41:8080
| REAL-Space-21Q1(private)
| /local2/arthor_database/
| active
|-
| nun
| 10.20.0.40:8080
| REAL_Space_21Q2(super private)
| /local2/arthor_database/
| active
|-
|}

===Arthor Round Table Nodes===
{| class="wikitable"
|-
! CentOS 7 Machine
! Port
! Database
! Total Files Size
! Round Table Data Directory
!Active
|-
| samekh
| 10.20.0.41:8008
| REAL-Space-21Q1-M, REAL-Space-21Q1-S, carboxylates_21Q2
| 4.0TB
| /local2/arthor_database/
| active
|-
| nun
| 10.20.0.40:8008
| 345_acids_20Q4, REAL_Space_21Q2-M, REAL_Space_21Q2-S, acids_21Q2, zinc_21Q1_H01~H25
| 3.1TB
| /local2/arthor_database/
| active
|-
| n-1-17
| 10.20.1.17:8008
|
|
| /local2/arthor_database/
| active
|-
| n-5-32
| 10.20.5.32:8008
| zinc_21Q1_H26~H30
| (TBA)
| /local2/arthor_database/
| active
|-
| n-5-33
| 10.20.5.33:8008
| acids_21Q2
| 9.1GB
| /local2/arthor_database/
| active
|-
|}

===Arthor Local 8081 (Datasets all local to samekh/nun)===
{| class="wikitable"
|-
! CentOS 7 Machine
! Port
! Database
! Total Files Size
! Round Table Data Directory
!Active
|-
| samekh
| 10.20.0.41:8081
| Enamine_REAL_Q2-2020-All-13B (26 slices)
| 4.5TB
| /local2/arthor_local_8081/
| active
|-
| nun
| 10.20.0.40:8081
| Enamine_REAL_Space_June_2020_S41B (aa-ae, 5 slices), Enamine_REAL_Space_June_2020_M41B (aa-an, 14 slices)
| 4.3TB
| /local2/arthor_local_8081/
| active
|-
|}

== Customizing Arthor Frontend to our needs ==
The frontend Arthor code is located at '''/nfs/exc/arthor_configs/*''' and the '''*''' is based on current running version.
=== Add Arthor Download Options ===
==== For Arthor 3.4: ====
1. vim .extract/webapps/ROOT/WEB-INF/static/index.html

2. search: '''arthor_tsv_link'''

3. in the div with the class=”dropdown-content”, add these link options and change the number accordingly:

<a id="arthor_tsv_link" href="#"> TSV-500</a>
<a id="arthor_tsv_link_5000" href="#"> TSV-5,000</a>
<a id="arthor_tsv_link_50000" href="#"> TSV-50,000</a>
<a id="arthor_tsv_link_100000" href="#"> TSV-100,000</a>
<a id="arthor_tsv_link_max" href="#"> TSV-max</a>
<a id="arthor_csv_link" href="#"> CSV-500</a>
<a id="arthor_csv_link_5000" href="#"> CSV-5,000</a>
<a id="arthor_csv_link_50000" href="#"> CSV-50,000</a>
<a id="arthor_csv_link_100000" href="#"> CSV-100,000</a>
<a id="arthor_csv_link_max" href="#"> CSV-max</a>
<a id="arthor_sdf_link" href="#"> SDF-500</a>
<a id="arthor_sdf_link_5000" href="#"> SDF-5,000</a>
<a id="arthor_sdf_link_50000" href="#"> SDF-50,000</a>
<a id="arthor_sdf_link_100000" href="#"> SDF-100,000</a>
<a id="arthor_sdf_link_max" href="#"> SDF-max</a>

4. then vim .extract/webapps/ROOT/WEB-INF/static/js/index.js

5. search: '''function $(t){'''

6. in the function $(t), add these lines:

if(document.getElementById("arthor_tsv_link")) {
var e=i.a.param({query:s.b.query,type:s.b.type,draw:0,start:0,length:t,flags:s.b.flags}),n=s.b.url+"/dt/"+E(s.b.table)+"/search";i()("#arthor_sdf_link").attr("href",n+".sdf?"+e),i() ("#arthor_tsv_link").attr("href",n+".tsv?"+e),i()("#arthor_csv_link").attr("href",n+".csv?"+e)
}
if (document.getElementById("arthor_tsv_link_5000")) {
var e=i.a.param({query:s.b.query,type:s.b.type,draw:0,start:0,length:5000,flags:s.b.flags}),n=s.b.url+"/dt/"+E(s.b.table)+"/search";i()("#arthor_sdf_link_5000").attr("href",n+".sdf?"+e),i()("#arthor_tsv_link_5000").attr("href",n+".tsv?"+e),i()("#arthor_csv_link_5000").attr("href",n+".csv?"+e)
}
if (document.getElementById("arthor_tsv_link_50000")) {
var e=i.a.param({query:s.b.query,type:s.b.type,draw:0,start:0,length:50000,flags:s.b.flags}),n=s.b.url+"/dt/"+E(s.b.table)+"/search";i()("#arthor_sdf_link_50000").attr("href",n+".sdf?"+e),i()("#arthor_tsv_link_50000").attr("href",n+".tsv?"+e),i()("#arthor_csv_link_50000").attr("href",n+".csv?"+e)
}
if (document.getElementById("arthor_tsv_link_100000")) {
var e=i.a.param({query:s.b.query,type:s.b.type,draw:0,start:0,length:100000,flags:s.b.flags}),n=s.b.url+"/dt/"+E(s.b.table)+"/search";i()("#arthor_sdf_link_100000").attr("href",n+".sdf?"+e),i()("#arthor_tsv_link_100000").attr("href",n+".tsv?"+e),i()("#arthor_csv_link_100000").attr("href",n+".csv?"+e)
}
if (document.getElementById("arthor_tsv_link_max")) {
var e=i.a.param({query:s.b.query,type:s.b.type,draw:0,start:0,length:1000000000,flags:s.b.flags}),n=s.b.url+"/dt/"+E(s.b.table)+"/search";i()("#arthor_sdf_link_max").attr("href",n+".sdf?"+e),i()("#arthor_tsv_link_max").attr("href",n+".tsv?"+e),i()("#arthor_csv_link_max").attr("href",n+".csv?"+e)
}

=== Take out Similarity Button ===
vim .extract/webapps/ROOT/WEB-INF/static/index.html
search: '''Similarity'''
Comment out this line '''< li value="Similarity" onclick="setSearchType(this)" class="first"> Similarity </li >''' //added spaces at the beginning and end so prevent wiki from converting it
Then add "first" in Substructure's class
=== Hyperlink to zinc20 ===
vim .extract/webapps/ROOT/WEB-INF/static/js/index.js
search: '''table_name'''
*find this line "" + d + ""
*replace with '''"<a target='_blank' href='https://zinc20.docking.org/substances/"+d+"'>" + d + "</a>"''' //added spaces at the beginning and end so prevent wiki from converting it

=== Make Input Box Work ===
At the end of the Arthor config file add this:
Resolver=https://sw.docking.org/util/smi2mol?smi=%s
To copy smiles in the input box:
vim .extract/webapps/ROOT/WEB-INF/static/js/index.js
search this: “var e=t.src.smiles()”
add this after the semi-colon
document.getElementById("ar_text_input").value = e;

Arthor Documentation for Future Developer

2021-05-03T23:58:15Z

Jgutie11: /* Public Arthor */

Written by Jennifer Young on December 16, 2019. Last edited January 05, 2021

==Install and Set Up on TomCat (Method 1)==
Arthor ran on n-1-136, which runs CentOS Linux release 7.7.1908 (Core). You can check the version of CentOS with the following command
cat /etc/centos-release

Check your current version of Java with the following command:
java -version

On n-1-136 we are running openjdk version "1.8.0_222", OpenJDK Runtime Environment (build 1.8.0_222-b10), and OpenJDK 64-Bit Server VM (build 25.222-b10, mixed mode)
If Java is not installed, install it using yum

==See this wiki page for more detailed information about installing Tomcat on our cluster==
http://wiki.docking.org/index.php/Tomcat_Installation

==Open port for Arthor==
In order for Arthor to be usable in the browser, the port you wish to run it on must be opened.
https://www.thegeekdiary.com/how-to-open-a-ports-in-centos-rhel-7/

===Step 1: Check Port Status===
Check that the port is not open and that Apache is not showing that port.
netstat -na | grep <port number you are checking>

lsof -i -P |grep http

===Step 2: Check Port Status in IP Tables===
iptables-save | grep <port number you are checking>

I skipped Step 3 from the guide, because there was a lot of information in the /etc/services file and I didn't want to edit it and break something.

===Step 4: Open Firewall Ports===
I did not include the zone=public section because the stand-alone servers are usually used for private instances of Arthor and SmallWorld.
Run as root.
firewall-cmd --add-port=<port number you are adding>/tcp --permanent

You need to reload the firewall after a change is made.
firewall-cmd --reload

===Step 5: Check that port is working===
To check that the port is active, run.
iptables -nL

You should see something along the lines of:
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:<port number you're adding> ctstate NEW,UNTRACKED

==How to run standalone Arthor instance==

===Step 1: Use or start a bash shell===
You can check your default shell using
echo $SHELL

If your default shell is csh, use
bash
to start a new bash shell in the current terminal window. Note that echo $SHELL will show you your default shell regardless of the current shell.

===Step 2: Set your environment variables===
export ARTHOR_DIR=/opt/nextmove/arthor/arthor-3.3-centos7
export PATH=$ARTHOR_DIR/bin/:$PATH

Make sure the ARTHOR_DIR variable is set to the directory for the latest version of Arthor or whichever version you would like to test.
The PATH environment variable is needed if you wish to use the Arthor tools from the command line

===Step 3: Run the arthor-server.jar===
java -jar /opt/nextmove/arthor/arthor-3.3-centos7/java/arthor.jar --httpPort <your httpPort>

==Setting environment variables for an Arthor Server==
Set the environment variables in the setenv.sh file. Note: Be sure to edit the file in the directory corresponding to the latest version of TomCat. As of December 2019, we are running 9.0.27 on n-1-136.

vim /opt/tomcat/apache-tomcat-9.0.27/bin/setenv.sh

Add the line below to the setenv.sh file above, or substitute the path to wherever you currently store the arthor.cfg file
export ARTHOR_CONFIG=/usr/local/tomcat/arthor.cfg

Here is an example of the arthor.cfg file:
BinDir=/opt/nextmove/arthor/arthor-3.3-centos7/bin
DataDir=/local2/arthor_local_8081/
MaxConcurrentSearches=6
MaxThreadsPerSearch=8
AutomaticIndex=false
AsyncHitCountMax=1000000
Resolver=https://sw.docking.org/util/smi2mol?smi=%s

=== Configuration Details ===

*'''BinDir''': is the location of the Arthor command line binaries. These are used to generate the Arthor index files and to perform searches directly on n-1-136. An example of this would be using atdbgrep for substructure search.

*'''DataDir''': This is the directory where the Arthor data files live. Location where the index files will be created and loaded from.

*'''MaxConcurrentSearches''': Controls the maximum number of searches that can be run concurrently by setting the database pool size. When switching between a large number of databases it can be useful to have a larger pool size, the only trade off is keeping file pointers open.

*'''MaxThreadsPerSearch''': The number of threads to use for both ATDB and ATFP searches

*Set '''AutomaticIndex''' to false if you don't want new smiles files added to the data directory to be indexed automatically

*'''AsyncHitCountMax''': The upper-bound for the number of hits to retrieve in background searches.

*'''Resolver''': Using Smallworld API, allows input box to take in a SMILE format and automatically draw on the board.

Check Arthor manual for more configuration options.

==Background==
Before working with Arthor, it is recommended that you familiarize yourself with the Arthor documentation. Some useful pages to look at include 3-5, 22-25 and 33-39. Of course, reading everything would be the best!

==Checking Disk Space Usage==
Before building arthor indexes, it's always a good thing to check what percent of the memory is being used. Try to be cautious with how much memory you have left, and make sure to check while building indexes to make sure that you have enough space. To check, run the following command:

df -h /<directory with disc>

==Building Large Databases==
At the moment, we are building databases of size 500M molecules by merging smile files. There are multiple methods of trying to create large databases, one being merging based off of the same H?? prefix and stopping once the database reaches > 500M molecules (or whatever upperbound you want to use). Here is some python code that simulates this merging process. Essentially the program takes all of the .smi files within an input directory, sorts them lexiographically, and begins merging these .smi files together in order until the size reaches > 500M molecules.

Feel free to modify it if you think a better method exists.

import subprocess
import sys
import os

from os import listdir
from os.path import isfile, join

mypath = "<Path to directory holding .smi files>"
onlyfiles = [f for f in listdir(mypath) if isfile(join(mypath, f))]
onlyfiles.sort()

create_fp = True
cur_mols = 0
lower_bound = 500000000
upper_bound = 600000000
files_to_merge = []

def merge_files(f_t_m):
arr = f_t_m[0].split(".")
arr2 = f_t_m[len(f_t_m) - 1].split(".")
file_name_merge = (arr[0] + "_" + arr2[0] + ".smi")
print ("File being created: " + file_name_merge)

for file in f_t_m:
tmp = file.split(".")
process = subprocess.Popen("cat " + join(mypath, file) + " >> " + file_name_merge, shell=True)
process.wait()

for file in onlyfiles:
arr = file.split(".")

if (arr[len(arr) - 1] == "smi"):
print("Working with " + file)
mol = sum(1 for line in open(join(mypath, file)))
print(file, mol, cur_mols)

if (cur_mols + mol > lower_bound):
if (cur_mols + mol < upper_bound):
files_to_merge.append(file)
merge_files(files_to_merge)
cur_mols = 0
files_to_merge.clear()
else:
merge_files(files_to_merge)
files_to_merge.clear()
files_to_merge.append(file)
merge_files(files_to_merge)
cur_mols = 0
files_to_merge.clear()
else:
cur_mols += mol
files_to_merge.append(file)

if (len(files_to_merge) != 0):
merge_files(files_to_merge)

==Building Arthor Indexes==
Once you've merged the .smi files together, it's time to start building the databases themselves. To do this we use the command

smi2atdb -j 0 -p <The .smi file> <The .atdb>

The flag "-j 0" enables parallel generation and utilizes all available processors to generate the .atdb file. The "-p" flag stores the offset position in the ATDB file. Since we're building indexes for the Web Application, you must use the "-p" flag when building indexes. Please note that the name of the .smi file should also be the name of the .atdb file. That way, the Web Application knows to use these files together and correctly display the required images. Refer to pages 33-34 in the Arthor documentation for more information.

If there are too many large .smi files and you do not want to manually build each .atdb file, you can use this python script which takes all of the .smi files in the current directory and converts them to .atdb files. Make sure to modify mypath to the directory containing the .smi files. You can change the variable "create_fp" to false if you don't want to create .atdb.fp files (refer to page 9 in the Arthor documentation).

import subprocess
import sys
import os

from os import listdir
from os.path import isfile, join

mypath = "<Path containing the .smi files"
onlyfiles = [f for f in listdir(mypath) if isfile(join(mypath, f))]

create_fp = True

for file in onlyfiles:
arr = file.split(".")

if (arr[len(arr) - 1] == "smi"):
process = subprocess.Popen("/nfs/ex9/work/xyz/psql/arthor-3.3-centos7/bin/smi2atdb -j 0 -p {0} {1}.atdb".format(join(mypath, file), arr[0]), shell=True)
process.wait()

print("SUCCESS! {0}.atdb file was created!".format(arr[0]))

if (create_fp):
process = subprocess.Popen("/nfs/ex9/work/xyz/psql/arthor-3.3-centos7/bin/atdb2fp -j 0 {0}.atdb".format(arr[0]), shell=True)
process.wait()

print("SUCCESS! {0}.atdb.fp file was created!".format(arr[0]))

==Uploading Indexes to the Web Application==
One can upload indexes to the Web Application by changing the "DATADIR" variable in the arthor.cfg file to the directory holding the .atdb files. This is already set up on n-1-136 and n-5-34.

==Virtual Memory==
In addition to modifying the arthor.cfg file, virtual memory can also be used to make queries faster. There can still be More information can be found in pages 10-16 in the Arthor Documentation.

==Setting up Round Table==
This is a new feature in Arthor 3.0 and is currently beta (January 2020). See section 2.4 in the manual
As explained in the manual, "Round Table allows you to serve and split chemical searches across multiple host machines. The implementation provides a lightweight proxy that forwards requests to other Arthor host servers that do the actual search. Communication is done using the existing Web APIs.

Since Arthor requires CentOS 7, as of January 2020 we have 6 servers that are capable of running Arthor with Round Table.

===Setting up Host Server===
If we want to add machines to the Round Table, for example 'nun' and 'samekh', we need to edit their arthor.cfg file so that when our Local Machine passes commands these secondary servers know to perform the search they are given.

$ cat arthor.cfg
MaxThreadsPerSearch=4
AutomaticIndex=false
DATADIR=<Directory where smiles are located>

We then run the jar server on each of these host machines containing data on any available port.

java -jar /nfs/ex9/work/xyz/psql/arthor-3.3-centos7/java/arthor.jar --httpPort <port>

For our local machine, the arthor.cfg file will look different.

$ cat arthor.cfg
[RoundTable]
RemoteClient=http://skynet:<port number where jar server is running>/
RemoteClient=http://hal:<port number where jar server is running>/

Please refer to Section 2 in the RoundTable Documentation file (pages 6-8) for more useful information on configuration.

Then run the following command on n-1-136:

java -jar /nfs/ex9/work/xyz/psql/arthor-3.3-centos7/java/arthor.jar --httpPort <port>

===Public Arthor===
{| class="wikitable"
|-
! CentOS 7 Machine
! Port
! Total Files Size
! Round Table Data Directory
! Active
|-
| samekh
| 10.20.0.41:8000
| 2.4TB
| /local2/public_arthor/
| active
|-
| nun
| 10.20.0.40:8000
| 2.4TB
| /local2/public_arthor/
| active
|-
| n-9-22
| 10.20.9.22:8000
| 2.4TB
| /export/db4/public_arthor/
| active
|-
|}

===Arthor Round Table Head===
{| class="wikitable"
|-
! CentOS 7 Machine
! Port
! Database
! Arthor Install Location
! Round Table Data Directory
! Active
|-
| samekh
| 10.20.0.41:8080
| Enamine_REAL_Q2-2020-All-13B
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
| nun
| 10.20.0.40:8080
| Enamine_REAL_Q2-2020-All-41B
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
|}

===Arthor Round Table Nodes===
{| class="wikitable"
|-
! CentOS 7 Machine
! Port
! Database
! Total Files Size
! Arthor Install Location
! Round Table Data Directory
!Active
|-
| samekh
| 10.20.0.41:8008
| Enamine_REAL_Q2-2020-All-13B (26 slices)
| 4.5TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
| nun
| 10.20.0.40:8008
| Enamine_REAL_Space_June_2020_S41B (aa-ae, 5 slices), Enamine_REAL_Space_June_2020_M41B (af-am, 8 slices), zinc22_2d (H04~H25, 22 slices)
| 5.6TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
| n-1-17
| 10.20.1.17:8008
| Enamine_REAL_Space_June_2020_M41B (aa-ae, 5 slices), zinc22_2d (H25~H29, 4 slices)
| 3.7TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
| n-5-32
| 10.20.5.32:8008
| Enamine_REAL_Space_June_2020_M41B (an~az, 13 slices), zinc22_2d (H30, 1 slice)
| 5.6TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
| n-5-33
| 10.20.5.33:8008
| Enamine_REAL_Space_June_2020_M41B (ba-bl, 12 slices)
| 5.3TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
|}

===Arthor Local 8081 (Datasets all local to samekh/nun)===
{| class="wikitable"
|-
! CentOS 7 Machine
! Port
! Database
! Total Files Size
! Arthor Install Location
! Round Table Data Directory
!Active
|-
| samekh
| 10.20.0.41:8081
| Enamine_REAL_Q2-2020-All-13B (26 slices)
| 4.5TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_local_8081/
| active
|-
| nun
| 10.20.0.40:8081
| Enamine_REAL_Space_June_2020_S41B (aa-ae, 5 slices), Enamine_REAL_Space_June_2020_M41B (aa-an, 14 slices)
| 4.3TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_local_8081/
| active
|-
|}

== Customizing Arthor Frontend to our needs ==
The frontend Arthor code is located at '''/nfs/exc/arthor_configs/*''' and the '''*''' is based on current running version.
=== Add Arthor Download Options ===
==== For Arthor 3.4: ====
1. vim .extract/webapps/ROOT/WEB-INF/static/index.html

2. search: '''arthor_tsv_link'''

3. in the div with the class=”dropdown-content”, add these link options and change the number accordingly:

<a id="arthor_tsv_link" href="#"> TSV-500</a>
<a id="arthor_tsv_link_5000" href="#"> TSV-5,000</a>
<a id="arthor_tsv_link_50000" href="#"> TSV-50,000</a>
<a id="arthor_tsv_link_100000" href="#"> TSV-100,000</a>
<a id="arthor_tsv_link_max" href="#"> TSV-max</a>
<a id="arthor_csv_link" href="#"> CSV-500</a>
<a id="arthor_csv_link_5000" href="#"> CSV-5,000</a>
<a id="arthor_csv_link_50000" href="#"> CSV-50,000</a>
<a id="arthor_csv_link_100000" href="#"> CSV-100,000</a>
<a id="arthor_csv_link_max" href="#"> CSV-max</a>
<a id="arthor_sdf_link" href="#"> SDF-500</a>
<a id="arthor_sdf_link_5000" href="#"> SDF-5,000</a>
<a id="arthor_sdf_link_50000" href="#"> SDF-50,000</a>
<a id="arthor_sdf_link_100000" href="#"> SDF-100,000</a>
<a id="arthor_sdf_link_max" href="#"> SDF-max</a>

4. then vim .extract/webapps/ROOT/WEB-INF/static/js/index.js

5. search: '''function $(t){'''

6. in the function $(t), add these lines:

if(document.getElementById("arthor_tsv_link")) {
var e=i.a.param({query:s.b.query,type:s.b.type,draw:0,start:0,length:t,flags:s.b.flags}),n=s.b.url+"/dt/"+E(s.b.table)+"/search";i()("#arthor_sdf_link").attr("href",n+".sdf?"+e),i() ("#arthor_tsv_link").attr("href",n+".tsv?"+e),i()("#arthor_csv_link").attr("href",n+".csv?"+e)
}
if (document.getElementById("arthor_tsv_link_5000")) {
var e=i.a.param({query:s.b.query,type:s.b.type,draw:0,start:0,length:5000,flags:s.b.flags}),n=s.b.url+"/dt/"+E(s.b.table)+"/search";i()("#arthor_sdf_link_5000").attr("href",n+".sdf?"+e),i()("#arthor_tsv_link_5000").attr("href",n+".tsv?"+e),i()("#arthor_csv_link_5000").attr("href",n+".csv?"+e)
}
if (document.getElementById("arthor_tsv_link_50000")) {
var e=i.a.param({query:s.b.query,type:s.b.type,draw:0,start:0,length:50000,flags:s.b.flags}),n=s.b.url+"/dt/"+E(s.b.table)+"/search";i()("#arthor_sdf_link_50000").attr("href",n+".sdf?"+e),i()("#arthor_tsv_link_50000").attr("href",n+".tsv?"+e),i()("#arthor_csv_link_50000").attr("href",n+".csv?"+e)
}
if (document.getElementById("arthor_tsv_link_100000")) {
var e=i.a.param({query:s.b.query,type:s.b.type,draw:0,start:0,length:100000,flags:s.b.flags}),n=s.b.url+"/dt/"+E(s.b.table)+"/search";i()("#arthor_sdf_link_100000").attr("href",n+".sdf?"+e),i()("#arthor_tsv_link_100000").attr("href",n+".tsv?"+e),i()("#arthor_csv_link_100000").attr("href",n+".csv?"+e)
}
if (document.getElementById("arthor_tsv_link_max")) {
var e=i.a.param({query:s.b.query,type:s.b.type,draw:0,start:0,length:1000000000,flags:s.b.flags}),n=s.b.url+"/dt/"+E(s.b.table)+"/search";i()("#arthor_sdf_link_max").attr("href",n+".sdf?"+e),i()("#arthor_tsv_link_max").attr("href",n+".tsv?"+e),i()("#arthor_csv_link_max").attr("href",n+".csv?"+e)
}

=== Take out Similarity Button ===
vim .extract/webapps/ROOT/WEB-INF/static/index.html
search: '''Similarity'''
Comment out this line '''< li value="Similarity" onclick="setSearchType(this)" class="first"> Similarity </li >''' //added spaces at the beginning and end so prevent wiki from converting it
Then add "first" in Substructure's class
=== Hyperlink to zinc20 ===
vim .extract/webapps/ROOT/WEB-INF/static/js/index.js
search: '''table_name'''
*find this line "" + d + ""
*replace with '''"<a target='_blank' href='https://zinc20.docking.org/substances/"+d+"'>" + d + "</a>"''' //added spaces at the beginning and end so prevent wiki from converting it

=== Make Input Box Work ===
At the end of the Arthor config file add this:
Resolver=https://sw.docking.org/util/smi2mol?smi=%s
To copy smiles in the input box:
vim .extract/webapps/ROOT/WEB-INF/static/js/index.js
search this: “var e=t.src.smiles()”
add this after the semi-colon
document.getElementById("ar_text_input").value = e;

How to use Globus

2021-04-06T00:34:16Z

Jgutie11:

== Introduction ==
[https://www.globus.org/ Globus] is a non-profit service for moving, syncing, and sharing large amounts of data asynchronously in the background. Transfers are done from and to, so called, endpoints. In order to perform a file transfer from one location to another using the Globus service, both ends must have an endpoint.

==== Zinc Globus Endpoint is: ====
;ucsfbks#zinc
default path= /mnt/nfs/ex3/published/

== Step 1 ==
====Create an account to log in to Globus. There are a couple ways:====
*[https://app.globus.org/ Use an Organization's login]
**If you are associated to an organization in the drop down list, you can use your organization's login to create an account.
**You can also sign up using a Google account or ORCiD ID
*[https://www.globusid.org/create Create a Globus ID]

== Step 2 ==
====To install and create a Globus Personal Endpoint:====
*Log in.
*On the left side of the page, click the "ENDPOINTS" button on the navigation bar.
*Then, on the top right, click "Create a Personal Endpoint".
*Download based on your Operating System.
*Execute the file and follow the on screen install instructions.
*Double check that your collection is available by going to "ENDPOINTS" then to "Administered By You" which is on the top middle of the page.

'''***Instructions on how to install are provided [https://www.globus.org/globus-connect-personal here] ***'''

== Step 3 ==
Once you have your endpoint, go back to the file manager tab on the navigation bar to begin file transfer.

[[File:Globus1.png|1000px]]

*Click on the collection input box on the left panel and then click the "Your Collections" tab.
*Choose the collection that you made.
*Now you should see something like this:
**[[File:Collection.png|800px]]
*Next, on the right panel, click on the collection input box and type in ucsfbks#zinc
*Click on the ucsfbks#zinc collection.
*Enter authentication credentials. Contact jjiteam@googlegroups.com for login credentials.
*Now both panels are ready to transfer files to and from.
*Choose a file or folder and click the blue start button at the bottom.
**[[File:Ready.png|1000px]]

How to use Globus

2021-04-06T00:31:48Z

Jgutie11:

== Introduction ==
[https://www.globus.org/ Globus] is a non-profit service for moving, syncing, and sharing large amounts of data asynchronously in the background. Transfers are done from and to, so called, endpoints. In order to perform a file transfer from one location to another using the Globus service, both ends must have an endpoint.

==== Zinc Globus Endpoint is: ====
;ucsfbks#zinc
default path= /mnt/nfs/ex3/published/

== Step 1 ==
====Create an account to log in to Globus. There are a couple ways:====
*[https://app.globus.org/ Use an Organization's login]
**If you are associated to an organization in the drop down list, you can use your organization's login to create an account.
**You can also sign up using a Google account or ORCiD ID
*[https://www.globusid.org/create Create a Globus ID]

== Step 2 ==
====To install and create a Globus Personal Endpoint:====
*Log in.
*On the left side of the page, click the "ENDPOINTS" button on the navigation bar.
*Then, on the top right, click "Create a Personal Endpoint".
*Download based on your Operating System.
*Execute the file and follow the on screen install instructions.
*Double check that your collection is available by going to "ENDPOINTS" then to "Administered By You" which is on the top middle of the page.

'''***Instructions on how to install are provided [https://www.globus.org/globus-connect-personal here] ***'''

== Step 3 ==
Once you have your endpoint, go back to the file manager tab on the navigation bar to begin file transfer.

[[File:Globus1.png|1000px]]

*Click on the collection input box on the left panel and then click the "Your Collections" tab.
*Choose the collection that you made.
*Now you should see something like this:
**[[File:Collection.png|800px]]
*Next, on the right panel, click on the collection input box and type in ucsfbks#zinc
*Click on the ucsfbks#zinc collection.
*Enter authentication credentials. Contact jjg9803@gmail.com, khtang015@gmail.com, and jir322@gmail.com for login credentials.
*Now both panels are ready to transfer files to and from.
*Choose a file or folder and click the blue start button at the bottom.
**[[File:Ready.png|1000px]]

Slurm

2021-04-05T22:17:58Z

Jgutie11: /* Migrating to gimel5 */

'''Slurm user-guide'''
=== Submit Jobs with Slurm ===

==== SBATCH-MR (beta) ====
It is a slurm-version of qsub-mr for submitting job on Slurm queueing system. Note: this is have not been extensively tested yet. Please contact me if the script is not working out. We are hoping to fully migrate to Slurm from the out-dated SGE system. Any error report would be helpful - Khanh

New slurm scripts are located in /nfs/soft/tools/utils/sbatch-slice
Just simply replace /nfs/soft/tools/utils/qsub-slice/qsub-mr with /nfs/soft/tools/utils/sbatch-slice/sbatch-mr in your script

To check the status of your job:
By username
$ squeue -u <username>
By jobid
$ squeue -j <job_id>

==== Submit load2d Jobs ====

$ cd <catalog_shortname>
$ source /nfs/exa/work/khtang/ZINC21_load2d/loadenv_zinc21.sh
(development) $ sh /nfs/exa/work/khtang/submit_scripts/sbatch_slice/batch_zinc21.slurm <catalog_shortname>.ism

==== Submit DOCK Jobs ====

* ANACONDA Installation (Python 2.7)

Each user is welcome to download anaconda and install into his/her own folder 
https://www.anaconda.com/distribution/ 
''wget https://repo.anaconda.com/archive/Anaconda2-2019.10-Linux-x86_64.sh'' 
NB: It is also available for Python3, which is our nearest future 

simple installation via ''/bin/sh Anaconda2-2019.10-Linux-x86_64.sh''

After the installation is completed, you need to install a few packages:
conda install -c free bsddb
conda install -c rdkit rdkit
conda install numpy

'''Running DOCK-3.7 with Slurm'''

Here is a “guinea pig project”, which has been done with DOCK-3.7 locally. 
'''GPR40 example:''' /mnt/nfs/home/dudenko/TEST_DOCKING_PROJECT 
'''ChEMBL ligands:''' /mnt/nfs/home/dudenko/CHEMBL4422_active_ligands 
This test calculation should run smoothly, if not, then there is a problem. 
Ultimately, you may need to compare your results with the reference run:
* CHEMBL4422_active_ligands.mol2 - TOP500 scoring poses
* extract_all.sort.uniq.txt - a print-out of scoring details

Slurm queue manager is installed locally at gimel, use it to run this test (and all your future jobs) in parallel. 
Do not forget to set DOCKBASE variable: ''export DOCKBASE=/nfs/soft/dock/versions/dock37/DOCK-3.7.3rc1/''

'''Useful DOCKING commands to remind:'''
$DOCKBASE/docking/setup/setup_db2_zinc15_file_number.py ./ CHEMBL4422_active_ligands_ CHEMBL4422_active_ligands.sdi 100 count
$DOCKBASE/analysis/extract_all.py -s -20
$DOCKBASE/analysis/getposes.py -l 500 -o CHEMBL4422_active_ligands.mol2

'''Useful slurm commands (see https://slurm.schedmd.com/quickstart.html):'''
to see what machine resources are offered by the cluster, do ''sinfo -lNe''
to submit a DOCK-3.7 job, run ''$DOCKBASE/docking/submit/submit_slurm_array.csh''
to see what is happening in the queue, run ''squeue''
to see a detailed info for a specific job: ''scontrol show jobid=_JOBID_''
to delete a job from queue, run ''scancel _JOBID_''

Should your slurm run correctly, type ''squeue'' and you should see something like this:

JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
4187_[637-2091] gimel array_jo dudenko PD 0:00 1 (Resources)
4187_629 gimel array_jo dudenko R 0:00 1 n-1-20
4187_630 gimel array_jo dudenko R 0:00 1 n-5-34
4187_631 gimel array_jo dudenko R 0:00 1 n-1-21
4187_632 gimel array_jo dudenko R 0:00 1 n-5-34
4187_633 gimel array_jo dudenko R 0:00 1 n-1-21
4187_634 gimel array_jo dudenko R 0:00 1 n-5-34
4187_635 gimel array_jo dudenko R 0:00 1 n-5-35
4187_636 gimel array_jo dudenko R 0:00 1 n-5-34
4187_622 gimel array_jo dudenko R 0:01 1 n-5-34
4187_623 gimel array_jo dudenko R 0:01 1 n-5-34
4187_624 gimel array_jo dudenko R 0:01 1 n-5-35
4187_625 gimel array_jo dudenko R 0:01 1 n-1-17

As root at gimel, it is possible to modify a particular job, e.g., ''scontrol update jobid=635 TimeLimit=7-00:00:00''

=== Slurm Installation Guide ===
'''Detailed step-by-step installation instruction (for sysadmins only)'''

==== Setup Node master ====
'''TBA'''

==== Setup Compute Nodes ====

'''Useful links:'''
https://slurm.schedmd.com/quickstart_admin.html
https://wiki.fysik.dtu.dk/niflheim/Slurm_installation

'''node n-1-17''' (Installation of the latest slurm version (20.02.04)? see below in "Migrating to gimel5" section)

* make sure you have there Centos 7: ''cat /etc/redhat-release''
* ''wget https://download.schedmd.com/slurm/slurm-17.02.11.tar.bz2''
* ''yum install readline-devel perl-ExtUtils-MakeMaker.noarch munge-devel pam-devel openssl-devel''
* ''export VER=17.02.11; rpmbuild -ta slurm-$VER.tar.bz2 --without mysql; mv /root/rpmbuild .''

installing built packages from rpmbuild:
* ''yum install rpmbuild/RPMS/x86_64/slurm-plugins-17.02.11-1.el7.x86_64.rpm''
* ''yum install rpmbuild/RPMS/x86_64/slurm-17.02.11-1.el7.x86_64.rpm''
* ''yum install rpmbuild/RPMS/x86_64/slurm-munge-17.02.11-1.el7.x86_64.rpm''

'''setting up munge''':
copy over /etc/munge/munge.key from gimel and put locally to /etc/munge. The key should be identical allover the nodes. 
Munge is a daemon responsible for secure data exchange between nodes. 
Set permissions accordingly: ''chown munge:munge /etc/munge/munge.key; chmod 400 /etc/munge/munge.key'' 

'''starting munge''': ''systemctl enable munge; systemctl start munge''

'''setting up slurm''':
* create a user slurm: adduser slurm.
* all UID/GUIDs of slurm user should be identical allover the nodes. 
Otherwise, one needs to specify a mapping scheme for translating each UID/GUIDs between nodes. 
To edit slurm UID/GUID, do ''vipw'' and replace "slurm line" with slurm:x:XXXXX:YYYYY::/nonexistent:/bin/false 
XXXXX and YYYYY for slurm user can be found at gimel in /etc/passwd 
NB: don't forget to edit /etc/group as well. 
* copy /etc/slurm/slurm.conf from gimel and put locally to /etc/slurm.
* figure out what CPU/Memory resources you have at n-1-17 (see /proc/cpuinfo) and append the following line:
NodeName=n-1-17 NodeAddr=10.20.1.17 CPUs=24 State=UNKNOWN
* append n-1-17 to the partition: PartitionName=gimel Nodes=gimel,n-5-34,n-5-35,n-1-17 Default=YES MaxTime=INFINITE State=UP
* create the following folders:
''mkdir -p /var/spool/slurm-llnl /var/run/slurm-llnl /var/log/slurm-llnl''
''chown -R slurm:slurm /var/spool/slurm-llnl /var/run/slurm-llnl /var/log/slurm-llnl''
* restarting slurm master node at gimel (Centos 6): ''/etc/init.d/slurm restart''
* enabling and starting slurm computing nodes (Centos 7): ''systemctl enable slurmd; systemctl start slurmd''

And last but not least, asking the firewall to allow communication between master node and computing node n-1-17:
* ''firewall-cmd --permanent --zone=public --add-port=6817/tcp'' #slurmctld
* ''firewall-cmd --permanent --zone=public --add-port=6818/tcp'' #slurmd
* ''firewall-cmd --reload''

To see the current situation of the queue, so ''sinfo -lNe'' and you will see:
Wed May 27 09:49:54 2020
NODELIST NODES PARTITION STATE CPUS S:C:T MEMORY TMP_DISK WEIGHT AVAIL_FE REASON
gimel 1 gimel* drained 24 4:6:1 1 0 1 (null) none
n-1-17 1 gimel* idle 24 24:1:1 1 0 1 (null) none
n-5-34 1 gimel* idle 80 80:1:1 1 0 1 (null) none
n-5-35 1 gimel* idle 80 80:1:1 1 0 1 (null) none

To disable a specific node, do ''scontrol update NodeName=n-1-17 State=DRAIN Reason=DRAINED'' 
To return back to service, do ''scontrol update NodeName=n-1-17 State=RESUME''

p.s. Some users/scripts may require csh/tcsh. 
''sudo yum install csh tcsh''

'''Node down after reboot'''
On gimel (master node)
sudo scontrol update NodeName=<node_name> State=RESUME

'''Useful links:'''
https://support.ceci-hpc.be/doc/_contents/QuickStart/SubmittingJobs/SlurmTutorial.html

==== Migrating to gimel5 ====
Reminder: If a GPU Node, add nvidia packages in foreman's puppet classes.
* Make sure you have there Centos 7: ''cat /etc/redhat-release''
* ''wget https://download.schedmd.com/slurm/slurm-20.02.4.tar.bz2''
* ''yum install rpm-build gcc openssl openssl-devel libssh2-devel pam-devel numactl numactl-devel hwloc hwloc-devel lua lua-devel readline-devel rrdtool-devel ncurses-devel gtk2-devel libssh2-devel libibmad libibumad perl-Switch perl-ExtUtils-MakeMaker mysql-devel''
* ''export VER=20.02.4; rpmbuild -ta slurm-$VER.tar.bz2 --with mysql; mv /root/rpmbuild .''

'''Installation on a Compute Node'''
* ''yum install rpmbuild/RPMS/x86_64/slurm-20.02.4-1.el7.x86_64.rpm 
* ''yum install rpmbuild/RPMS/x86_64/slurm-slurmd-20.02.4-1.el7.x86_64.rpm 
* ''systemctl enable slurmd 
* ''systemctl start slurmd

'''Installation for a Backup Controller'''

Currently (04/05/2021): gimel4

* In gimel5's /etc/slurm/slurm.conf, find "BackupController="
** Set the value to ''' gimel4 '''
** Copy the conf file to gimel4
* ''yum install rpmbuild/RPMS/x86_64/slurm-20.02.4-1.el7.x86_64.rpm 
* ''yum install rpmbuild/RPMS/x86_64/slurm-slurmd-20.02.4-1.el7.x86_64.rpm 
* ''yum install rpmbuild/RPMS/x86_64/slurm-slurmctld-20.02.4-1.el7.x86_64.rpm 
* ''systemctl enable slurmctld.service 
* ''systemctl start slurmctld.service

==== GPUs specification ====

- 32-core:
+ n-9-34 (GTX 1080 Ti)
+ n-9-36 (GTX 1080 Ti)
+ n-1-126 (GTX 980)
+ n-1-141 (GTX 980)
- 40-core:
+ n-1-28 (RTX 2080 Super)
+ n-1-38 (RTX 2080 Super)
+ n-1-101 (RTX 2080 Super)
+ n-1-105 (RTX 2080 Super)
+ n-1-124 (RTX 2080 Super)

Back to [[DOCK_3.7]]

[[Category : Slurm]]

Slurm

2021-04-05T21:20:34Z

Jgutie11:

'''Slurm user-guide'''
=== Submit Jobs with Slurm ===

==== SBATCH-MR (beta) ====
It is a slurm-version of qsub-mr for submitting job on Slurm queueing system. Note: this is have not been extensively tested yet. Please contact me if the script is not working out. We are hoping to fully migrate to Slurm from the out-dated SGE system. Any error report would be helpful - Khanh

New slurm scripts are located in /nfs/soft/tools/utils/sbatch-slice
Just simply replace /nfs/soft/tools/utils/qsub-slice/qsub-mr with /nfs/soft/tools/utils/sbatch-slice/sbatch-mr in your script

To check the status of your job:
By username
$ squeue -u <username>
By jobid
$ squeue -j <job_id>

==== Submit load2d Jobs ====

$ cd <catalog_shortname>
$ source /nfs/exa/work/khtang/ZINC21_load2d/loadenv_zinc21.sh
(development) $ sh /nfs/exa/work/khtang/submit_scripts/sbatch_slice/batch_zinc21.slurm <catalog_shortname>.ism

==== Submit DOCK Jobs ====

* ANACONDA Installation (Python 2.7)

Each user is welcome to download anaconda and install into his/her own folder 
https://www.anaconda.com/distribution/ 
''wget https://repo.anaconda.com/archive/Anaconda2-2019.10-Linux-x86_64.sh'' 
NB: It is also available for Python3, which is our nearest future 

simple installation via ''/bin/sh Anaconda2-2019.10-Linux-x86_64.sh''

After the installation is completed, you need to install a few packages:
conda install -c free bsddb
conda install -c rdkit rdkit
conda install numpy

'''Running DOCK-3.7 with Slurm'''

Here is a “guinea pig project”, which has been done with DOCK-3.7 locally. 
'''GPR40 example:''' /mnt/nfs/home/dudenko/TEST_DOCKING_PROJECT 
'''ChEMBL ligands:''' /mnt/nfs/home/dudenko/CHEMBL4422_active_ligands 
This test calculation should run smoothly, if not, then there is a problem. 
Ultimately, you may need to compare your results with the reference run:
* CHEMBL4422_active_ligands.mol2 - TOP500 scoring poses
* extract_all.sort.uniq.txt - a print-out of scoring details

Slurm queue manager is installed locally at gimel, use it to run this test (and all your future jobs) in parallel. 
Do not forget to set DOCKBASE variable: ''export DOCKBASE=/nfs/soft/dock/versions/dock37/DOCK-3.7.3rc1/''

'''Useful DOCKING commands to remind:'''
$DOCKBASE/docking/setup/setup_db2_zinc15_file_number.py ./ CHEMBL4422_active_ligands_ CHEMBL4422_active_ligands.sdi 100 count
$DOCKBASE/analysis/extract_all.py -s -20
$DOCKBASE/analysis/getposes.py -l 500 -o CHEMBL4422_active_ligands.mol2

'''Useful slurm commands (see https://slurm.schedmd.com/quickstart.html):'''
to see what machine resources are offered by the cluster, do ''sinfo -lNe''
to submit a DOCK-3.7 job, run ''$DOCKBASE/docking/submit/submit_slurm_array.csh''
to see what is happening in the queue, run ''squeue''
to see a detailed info for a specific job: ''scontrol show jobid=_JOBID_''
to delete a job from queue, run ''scancel _JOBID_''

Should your slurm run correctly, type ''squeue'' and you should see something like this:

JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
4187_[637-2091] gimel array_jo dudenko PD 0:00 1 (Resources)
4187_629 gimel array_jo dudenko R 0:00 1 n-1-20
4187_630 gimel array_jo dudenko R 0:00 1 n-5-34
4187_631 gimel array_jo dudenko R 0:00 1 n-1-21
4187_632 gimel array_jo dudenko R 0:00 1 n-5-34
4187_633 gimel array_jo dudenko R 0:00 1 n-1-21
4187_634 gimel array_jo dudenko R 0:00 1 n-5-34
4187_635 gimel array_jo dudenko R 0:00 1 n-5-35
4187_636 gimel array_jo dudenko R 0:00 1 n-5-34
4187_622 gimel array_jo dudenko R 0:01 1 n-5-34
4187_623 gimel array_jo dudenko R 0:01 1 n-5-34
4187_624 gimel array_jo dudenko R 0:01 1 n-5-35
4187_625 gimel array_jo dudenko R 0:01 1 n-1-17

As root at gimel, it is possible to modify a particular job, e.g., ''scontrol update jobid=635 TimeLimit=7-00:00:00''

=== Slurm Installation Guide ===
'''Detailed step-by-step installation instruction (for sysadmins only)'''

==== Setup Node master ====
'''TBA'''

==== Setup Compute Nodes ====

'''Useful links:'''
https://slurm.schedmd.com/quickstart_admin.html
https://wiki.fysik.dtu.dk/niflheim/Slurm_installation

'''node n-1-17''' (Installation of the latest slurm version (20.02.04)? see below in "Migrating to gimel5" section)

* make sure you have there Centos 7: ''cat /etc/redhat-release''
* ''wget https://download.schedmd.com/slurm/slurm-17.02.11.tar.bz2''
* ''yum install readline-devel perl-ExtUtils-MakeMaker.noarch munge-devel pam-devel openssl-devel''
* ''export VER=17.02.11; rpmbuild -ta slurm-$VER.tar.bz2 --without mysql; mv /root/rpmbuild .''

installing built packages from rpmbuild:
* ''yum install rpmbuild/RPMS/x86_64/slurm-plugins-17.02.11-1.el7.x86_64.rpm''
* ''yum install rpmbuild/RPMS/x86_64/slurm-17.02.11-1.el7.x86_64.rpm''
* ''yum install rpmbuild/RPMS/x86_64/slurm-munge-17.02.11-1.el7.x86_64.rpm''

'''setting up munge''':
copy over /etc/munge/munge.key from gimel and put locally to /etc/munge. The key should be identical allover the nodes. 
Munge is a daemon responsible for secure data exchange between nodes. 
Set permissions accordingly: ''chown munge:munge /etc/munge/munge.key; chmod 400 /etc/munge/munge.key'' 

'''starting munge''': ''systemctl enable munge; systemctl start munge''

'''setting up slurm''':
* create a user slurm: adduser slurm.
* all UID/GUIDs of slurm user should be identical allover the nodes. 
Otherwise, one needs to specify a mapping scheme for translating each UID/GUIDs between nodes. 
To edit slurm UID/GUID, do ''vipw'' and replace "slurm line" with slurm:x:XXXXX:YYYYY::/nonexistent:/bin/false 
XXXXX and YYYYY for slurm user can be found at gimel in /etc/passwd 
NB: don't forget to edit /etc/group as well. 
* copy /etc/slurm/slurm.conf from gimel and put locally to /etc/slurm.
* figure out what CPU/Memory resources you have at n-1-17 (see /proc/cpuinfo) and append the following line:
NodeName=n-1-17 NodeAddr=10.20.1.17 CPUs=24 State=UNKNOWN
* append n-1-17 to the partition: PartitionName=gimel Nodes=gimel,n-5-34,n-5-35,n-1-17 Default=YES MaxTime=INFINITE State=UP
* create the following folders:
''mkdir -p /var/spool/slurm-llnl /var/run/slurm-llnl /var/log/slurm-llnl''
''chown -R slurm:slurm /var/spool/slurm-llnl /var/run/slurm-llnl /var/log/slurm-llnl''
* restarting slurm master node at gimel (Centos 6): ''/etc/init.d/slurm restart''
* enabling and starting slurm computing nodes (Centos 7): ''systemctl enable slurmd; systemctl start slurmd''

And last but not least, asking the firewall to allow communication between master node and computing node n-1-17:
* ''firewall-cmd --permanent --zone=public --add-port=6817/tcp'' #slurmctld
* ''firewall-cmd --permanent --zone=public --add-port=6818/tcp'' #slurmd
* ''firewall-cmd --reload''

To see the current situation of the queue, so ''sinfo -lNe'' and you will see:
Wed May 27 09:49:54 2020
NODELIST NODES PARTITION STATE CPUS S:C:T MEMORY TMP_DISK WEIGHT AVAIL_FE REASON
gimel 1 gimel* drained 24 4:6:1 1 0 1 (null) none
n-1-17 1 gimel* idle 24 24:1:1 1 0 1 (null) none
n-5-34 1 gimel* idle 80 80:1:1 1 0 1 (null) none
n-5-35 1 gimel* idle 80 80:1:1 1 0 1 (null) none

To disable a specific node, do ''scontrol update NodeName=n-1-17 State=DRAIN Reason=DRAINED'' 
To return back to service, do ''scontrol update NodeName=n-1-17 State=RESUME''

p.s. Some users/scripts may require csh/tcsh. 
''sudo yum install csh tcsh''

'''Node down after reboot'''
On gimel (master node)
sudo scontrol update NodeName=<node_name> State=RESUME

'''Useful links:'''
https://support.ceci-hpc.be/doc/_contents/QuickStart/SubmittingJobs/SlurmTutorial.html

==== Migrating to gimel5 ====
Reminder: If a GPU Node, add nvidia packages in foreman's puppet classes.
* Make sure you have there Centos 7: ''cat /etc/redhat-release''
* ''wget https://download.schedmd.com/slurm/slurm-20.02.4.tar.bz2''
* ''yum install rpm-build gcc openssl openssl-devel libssh2-devel pam-devel numactl numactl-devel hwloc hwloc-devel lua lua-devel readline-devel rrdtool-devel ncurses-devel gtk2-devel libssh2-devel libibmad libibumad perl-Switch perl-ExtUtils-MakeMaker mysql-devel''
* ''export VER=20.02.4; rpmbuild -ta slurm-$VER.tar.bz2 --with mysql; mv /root/rpmbuild .''

'''Installation on a Compute Node'''
* ''yum install rpmbuild/RPMS/x86_64/slurm-20.02.4-1.el7.x86_64.rpm 
* ''yum install rpmbuild/RPMS/x86_64/slurm-slurmd-20.02.4-1.el7.x86_64.rpm 
* ''systemctl enable slurmd 
* ''systemctl start slurmd

'''Installation for a Backup Controller'''

Currently (04/05/2021): gimel4
* ''yum install rpmbuild/RPMS/x86_64/slurm-20.02.4-1.el7.x86_64.rpm 
* ''yum install rpmbuild/RPMS/x86_64/slurm-slurmd-20.02.4-1.el7.x86_64.rpm 
* ''yum install rpmbuild/RPMS/x86_64/slurm-slurmctld-20.02.4-1.el7.x86_64.rpm 
* ''systemctl enable slurmctld.service 
* ''systemctl start slurmctld.service

==== GPUs specification ====

- 32-core:
+ n-9-34 (GTX 1080 Ti)
+ n-9-36 (GTX 1080 Ti)
+ n-1-126 (GTX 980)
+ n-1-141 (GTX 980)
- 40-core:
+ n-1-28 (RTX 2080 Super)
+ n-1-38 (RTX 2080 Super)
+ n-1-101 (RTX 2080 Super)
+ n-1-105 (RTX 2080 Super)
+ n-1-124 (RTX 2080 Super)

Back to [[DOCK_3.7]]

[[Category : Slurm]]

Using LVM To Mount Drives

2021-04-01T20:35:36Z

Jgutie11:

== Format Disk ==
Do these commands as root:
* fdisk -l
** Find disks/RAID disks to format
*parted
*print
*mklabel gpt
*mkpart logical 0GB 9995GB. (based on what we read in result of print above)
*print. (to confirm)
*quit

== Mount storage disks using LVM ==
Reference guide: '''https://www.thegeekdiary.com/redhat-centos-a-beginners-guide-to-lvm-logical-volume-manager/'''

1. Scan devices to be used as physical volumes (PV)
lvmdiskscan

2. Initialize the block devices
pvcreate <drive1> <drive2> <...> ...
example: pvcreate /dev/sdb /dev/sdc /dev/sdd

3. To double check PV
pvdisplay
pvscan
pvs

4. After PV, create a Volume Group (VG)
vgcreate <name of volume> <drive1> <drive2> <...> ...
example: vgcreate soft2 /dev/sdb /dev/sdc /dev/sdd

5. To double check VG
vgs <VG name>
vgdisplay <VG name>

6. After VG, create a Logical Volume (LV)
lvcreate <options> <name of LV> <VG Name>
example: lvcreate -l 100%FREE -n soft_lv soft2
"-l" is for storage space in percent, 100%FREE means use all space in VG
"-n" is for name of LV

7. Double Check LV
lvs /dev/<VG NAME>/<LV NAME>
lvdisplay /dev/<VG NAME>/<LV NAME>
lvscan

8. Create a File System
mkfs.ext4 <options> /dev/<VG NAME>/<LV NAME>
example: mkfs.ext4 /dev/soft2/soft_lv

9. Make the directory:
mkdir <somewhere>
example: mkdir /export/soft2

10. Mount permanently:
vim /etc/fstab
/dev/<VG NAME>/<LV NAME> <somewhere> <type of file system> <mount options> <dump> <fsck>
example:
/dev/soft2/soft_lv /export/soft2 ext4 defaults 1 2

11. Lastly, mount all
mount -a

== RAID1 with LVM on fresh drives ==
1. Check which disk will act as the mirror
fdisk -l

2. Create physical volumes of the drives
pvcreate <disk> <disk> <disk>
example: pvcreate /dev/sdz /dev/sdaa

3. Create a volume group for these drives
vgcreate <name> <disk1> <disk2>
example: vgcreate local2 /dev/sdz /dev/sdaa

4. Create a logical volume with "-m1" flag to say mirror/raid1
lvcreate -l <how much of disk to use in percent> -m1 -n <name> <volume_group>
example: lvcreate -l 100%FREE -m1 -n local2_lv local2

5. Mount permanently:
vim /etc/fstab
/dev/<VG NAME>/<LV NAME> <somewhere> <type of file system> <mount options> <dump> <fsck>
example:
/dev/local2/local2_lv /local2 ext4 defaults 1 2

6. Lastly, mount all
mount -a

== RAID1 with LVM on an existing used logical volume ==
'''Before starting, make sure that the disk that will mirror data is the same size or bigger than the drive being mirrored.'''

1. Check if the volume has free space for lvm logs. Mirror will fail if no free space available.
pvscan
example output:
PV /dev/sda2 VG centos lvm2 [<222.57 GiB / 0 free]
PV /dev/sdb1 VG centos lvm2 [<223.57 GiB / 0 free]
PV /dev/sdd1 VG local2 lvm2 [<7.28 TiB / <5.03 GiB free]
If ''' PV /dev/sdd1 VG local2 lvm2 [<7.28 TiB / 0 free] ''' then do
lvreduce -l -1 --resizefs /dev/<volume_group>/<logical_volume>
example : lvreduce -l -1 --resizefs /dev/local2/zinc_lv

2. Check which disk will act as the mirror
fdisk -l

3. Extend the disk to volume group
vgextend <volume_group> /dev/<disk>
example: vgextend local2 /dev/sdc
To remove extension:
vgreduce <volume_group> /dev/<disk>
example: vgreduce local2 /dev/sdc

4. Convert current working logical volume to RAID1 and assign mirror
lvconvert -m1 /dev/<volume_group>/<logical_volume> /dev/<disk>
example: lvconvert -m1 /dev/local2/zinc_lv /dev/sdc

5. It should say it was successful, to check syncing progress
lvs -a -o +devices

Using LVM To Mount Drives

2021-03-26T23:13:05Z

Jgutie11: /* Partition Disk */

== Format Disk ==
Do these commands as root:
* fdisk -l
** Find disks/RAID disks to format
*parted
*print
*mklabel gpt
*mkpart logical 0GB 9995GB. (based on what we read in result of print above)
*print. (to confirm)
*quit

== Mount storage disks using LVM ==
Reference guide: '''https://www.thegeekdiary.com/redhat-centos-a-beginners-guide-to-lvm-logical-volume-manager/'''

'''Scan devices to be used as physical volumes (PV)'''
* lvmdiskscan

'''Initialize the block devices'''
* pvcreate <drive1> <drive2> <...> ...
** Example: pvcreate /dev/sdb /dev/sdc /dev/sdd

'''To double check PV'''
* pvdisplay
* pvscan
* pvs

'''After PV, create a Volume Group (VG)'''
* vgcreate <name of volume> <drive1> <drive2> <...> ...
** Example: vgcreate soft2 /dev/sdb /dev/sdc /dev/sdd

'''To double check VG'''
* vgs <VG name>
* vgdisplay <VG name>

''' After VG, create a Logical Volume (LV) '''
* lvcreate <options> <name of LV> <VG Name>
** Example: lvcreate -l 100%FREE -n soft_lv soft2
***-l is for storage space in percent, 100%FREE means use all space in VG
***-n is for name of LV

''' Double Check LV '''
* lvs /dev/<VG NAME>/<LV NAME>
* lvdisplay /dev/<VG NAME>/<LV NAME>
* lvscan

'''Last step, create a File System'''
* mkfs.ext4 <options> /dev/<VG NAME>/<LV NAME>
* Example: mkfs.ext4 /dev/soft2/soft_lv

''' It's time to mount '''
Make the directory first:
* mkdir <somewhere>
** Example: mkdir /export/soft2
* mount /dev/<VG NAME>/<LV NAME> <somewhere>
** Example: mount /dev/soft2/soft_lv /export/soft2/
Make it permanent by editing fstab:
* vim /etc/fstab
/dev/<VG NAME>/<LV NAME> <somewhere> <type of file system> <mount options> <dump> <fsck>
example:
/dev/soft2/soft_lv /export/soft2 ext4 defaults 1 2

== RAID1 with LVM on an existing used logical volume ==
'''Before starting, make sure that the disk that will mirror data is the same size or bigger than the drive being mirrored.'''

1. Check if the volume has free space for lvm logs. Mirror will fail if no free space available.
pvscan
example output:
PV /dev/sda2 VG centos lvm2 [<222.57 GiB / 0 free]
PV /dev/sdb1 VG centos lvm2 [<223.57 GiB / 0 free]
PV /dev/sdd1 VG local2 lvm2 [<7.28 TiB / <5.03 GiB free]
If ''' PV /dev/sdd1 VG local2 lvm2 [<7.28 TiB / 0 free] ''' then do
lvreduce -l -1 --resizefs /dev/<volume_group>/<logical_volume>
for example : lvreduce -l -1 --resizefs /dev/local2/zinc_lv

2. Check which disk will act as the mirror
fdisk -l

3. Extend the disk to volume group
vgextend <volume_group> /dev/<disk>
for example: vgextend local2 /dev/sdc
To remove extension:
vgreduce <volume_group> /dev/<disk>
for example: vgreduce local2 /dev/sdc

4. Convert current working logical volume to RAID1 and assign mirror
lvconvert -m1 /dev/<volume_group>/<logical_volume> /dev/<disk>
for example: lvconvert -m1 /dev/local2/zinc_lv /dev/sdc

5. It should say it was successful, to check syncing progress
lvs -a -o +devices

Zfs

2021-03-25T21:26:03Z

Jgutie11: /* Checking Disk Health and Integrity */

ZFS - Zettabyte Filesystem

== ZFS packages installation ==
https://www.symmcom.com/docs/how-tos/storages/how-to-install-zfs-on-centos-7

ZFS rpm pkg must match the CentOS version install in the machine
* Check CentOS version
$ cat /etc/centos-release
CentOS Linux release 7.8.2003 (Core)

* Install ZFS-release package. In this case, you will need to install the package for CentOS 7.8 version
$ yum install http://download.zfsonlinux.org/epel/zfs-release.el7_8.noarch.rpm

* Edit /etc/yum.repos.d/zfs.repo
The ZFS package that we want to install is zfs-kmod
$ vim /etc/yum.repos.d/zfs.repo
There are 2 items to change
[zfs]
name=ZFS on Linux for EL7 - dkms
baseurl=http://download.zfsonlinux.org/epel/7.8/$basearch/
enabled=1 -> change to 0
metadata_expire=7d
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-zfsonlinux
[zfs-kmod]
name=ZFS on Linux for EL7 - kmod
baseurl=http://download.zfsonlinux.org/epel/7.8/kmod/$basearch/
enabled=0 change to 1
metadata_expire=7d
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-zfsonlinux

* Install ZFS
$ yum install zfs

* Useful Commands to see if any disk are avaible
$ lsblk -f
$ fdisk -l
$ fdisk -l /dev/sd* |grep '^Disk' |grep sectors |grep -v experimental

== Beginning ZFS instances ==

There are only two commmands to interact with ZFS.

zpool: used to create a ZFS vdev (virtual device). vdevs are composed of physical devices.
zfs: used to create/interact with a ZFS dataset. ZFS datasets are akin to logical volumes

# zpool creation syntax
zpool create <poolname> <vdev(s)>
# Create a zpool of six raidz2 vdevs, each with six drives. Includes two SSDs to used as a mirrored SLOG and one SSD as an L2ARC read cache. (example commmand was run on qof)
zpool create ex9 raidz2 sda sdb sdc sdd sde sdf raidz2 sdg sdh sdi sdj sdk sdl raidz2 sdm sdn sdo sdp sdq sdr raidz2 sds sdt sdu sdv sdw sdx raidz2 sdy sdz sdaa sdab sdac sdad raidz2 sdae sdaf sdag sdah sdai sdaj log mirror ata-INTEL_SSDSC2KG480G7_BTYM740603E0480BGN ata-INTEL_SSDSC2KG480G7_BTYM7406019K480BGN cache ata-INTEL_SSDSC2KG480G7_BTYM740602GN480BGN
[root@qof ~]# zpool status
pool: ex9
state: ONLINE
scan: none requested
config:
NAME STATE READ WRITE CKSUM
ex9 ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
sda ONLINE 0 0 0
sdb ONLINE 0 0 0
sdc ONLINE 0 0 0
sdd ONLINE 0 0 0
sde ONLINE 0 0 0
sdf ONLINE 0 0 0
raidz2-1 ONLINE 0 0 0
sdg ONLINE 0 0 0
sdh ONLINE 0 0 0
sdi ONLINE 0 0 0
sdj ONLINE 0 0 0
sdk ONLINE 0 0 0
sdl ONLINE 0 0 0
raidz2-2 ONLINE 0 0 0
sdm ONLINE 0 0 0
sdn ONLINE 0 0 0
sdo ONLINE 0 0 0
sdp ONLINE 0 0 0
sdq ONLINE 0 0 0
sdr ONLINE 0 0 0
raidz2-3 ONLINE 0 0 0
sds ONLINE 0 0 0
sdt ONLINE 0 0 0
sdu ONLINE 0 0 0
sdv ONLINE 0 0 0
sdw ONLINE 0 0 0
sdx ONLINE 0 0 0
raidz2-4 ONLINE 0 0 0
sdy ONLINE 0 0 0
sdz ONLINE 0 0 0
sdaa ONLINE 0 0 0
sdab ONLINE 0 0 0
sdac ONLINE 0 0 0
sdad ONLINE 0 0 0
raidz2-5 ONLINE 0 0 0
sdae ONLINE 0 0 0
sdaf ONLINE 0 0 0
sdag ONLINE 0 0 0
sdah ONLINE 0 0 0
sdai ONLINE 0 0 0
sdaj ONLINE 0 0 0
logs
mirror-6 ONLINE 0 0 0
ata-INTEL_SSDSC2KG480G7_BTYM740603E0480BGN ONLINE 0 0 0
ata-INTEL_SSDSC2KG480G7_BTYM7406019K480BGN ONLINE 0 0 0
cache
ata-INTEL_SSDSC2KG480G7_BTYM740602GN480BGN ONLINE 0 0 0

Adding a zfs filesystem:

Using qof as an example, I will create a child filesystem under ex9 named archive that will be mounted under /export/ex9/archive. This archive will be used to backup user data.

-bash-4.2$ zfs list
NAME USED AVAIL REFER MOUNTPOINT
ex9 2.39T 249T 2.39T /export/ex9
-bash-4.2$ sudo zfs create -o mountpoint=/export/ex9/archive ex9/archive
-bash-4.2$ zfs list
NAME USED AVAIL REFER MOUNTPOINT
ex9 2.39T 249T 2.39T /export/ex9
ex9/archive 192K 249T 192K /export/ex9/archive

== Testing ==

NOTE: It is important to reboot machine to see if the zpool stay on.

== Add alias to machine and mount point to puppet ==

Please see [http://wiki.docking.org/index.php/PuppetTricks#Adding_aliases_for_server_on_Alpha this section]

== Adding L2ARC Read Cache to a zpool==
# Look for available SSDs in /dev/disk/by-id/
# Choose an available SSD to use for read cache. Then decide which pool you want to put the cache on.
Syntax: zpool add <zpool name> <cache/log> <path to disk>
$ sudo zpool add ex6 cache /dev/disk/by-id/ata-INTEL_SSDSC2KG480G7_BTYM72830AV6480BGN

== Tuning ZFS options ==
# stores extended attributes as system attributes to improve performance
$ zfs xattr=sa <zfs dataset name>

# Turn on ZFS lz4 compression. Use this for compressible dataset such as many files with text
$ zfs set compression=lz4 <zfs dataset name>

# Turn off access time for improved disk performance (so that the OS doesn't write a new time every time a file is accessed)
$ zfs set atime=off <zfs dataset name>

NOTE: ZFS performance degrades tremendously when the zpool is over 80% used. To avoid this, I have set a quota to 80% of the 248TB in qof/nfs-ex9.
# To set a quota of 200TB on ZFS dataset:
$ zfs set quota=200T <zfs dataset>

# To remove a quota from a ZFS dataset:
$ zfs set quota=none <zfs dataset>

By default, ZFS pools/mounts do not have ACLs active.
# to active access control lists on a zpool
$ sudo zfs set acltype=posixacl <pool name>

== Checking Disk Health and Integrity ==
Print a brief summary of all pools:
zpool list

Print a detailed status of each disk and status of pool:
zpool status

Clear read errors on disk, if not anything serious:
zpool clear <pool_name>

Check data integrity, traverses all the data in the pool once and verifies that all blocks can be read:
zpool scrub <pool_name>

To stop scrub:
zpool scrub -s <pool_name>

== mount after reboot ==
zfs set mountpoint=/export/db2 db2

== when you put in a new disk ==
fdisk -l
to see what is new

sudo zpool create -f /srv/db3 raidz2 /dev/sdaa /dev/sdab /dev/sdac /dev/sdad /dev/sdae /dev/sdaf /dev/sdag /dev/sdah /dev/sdai /dev/sdaj /dev/sdak /dev/sdal
sudo zpool add -f /srv/db3 raidz2 /dev/sdam /dev/sdan /dev/sdao /dev/sdap /dev/sdaq /dev/sdar /dev/sdas /dev/sdat /dev/sdau /dev/sdav /dev/sdaw /dev/sdax

zfs unmount db3

zfs mount db3

= latest =
zpool create -f db3 raidz2 /dev/sdy /dev/sdz /dev/sdaa /dev/sdab /dev/sdac /dev/sdad /dev/sdae /dev/sdaf /dev/sdag /dev/sdah /dev/sdai /dev/sdaj
zpool add -f db3 raidz2 /dev/sdak /dev/sdal /dev/sdam /dev/sdan /dev/sdao /dev/sdap /dev/sdaq /dev/sdar /dev/sdas /dev/sdat /dev/sdau /dev/sdav

zpool create -f db4 raidz2 /dev/sdax /dev/sday /dev/sdaz /dev/sdba /dev/sdbb /dev/sdbc /dev/sdbd /dev/sdbe /dev/sdbf /dev/sdbg /dev/sdbh /dev/sdbi
zpool add -f db4 raidz2 /dev/sdbj /dev/sdbk /dev/sdbl /dev/sdbm /dev/sdbn /dev/sdbo /dev/sdbp /dev/sdbq /dev/sdbr /dev/sdbs /dev/sdbt /dev/sdbu

= Fri Jan 19 2018 =

zpool create -f db5 raidz2 /dev/sdbw /dev/sdbx /dev/sdby /dev/sdbz /dev/sdca /dev/sdcb /dev/sdcc /dev/sdcd /dev/sdce /dev/sdcf /dev/sdcg /dev/sdch
zpool add -f db5 raidz2 /dev/sdci /dev/sdcj /dev/sdck /dev/sdcl /dev/sdcm /dev/sdcn /dev/sdco /dev/sdcp /dev/sdcq /dev/sdcr /dev/sdcs /dev/sdct
zfs mount db5

= Wed Jan 24 2018 =
On tsadi
zpool create -f ex1 mirror /dev/sdaa /dev/sdab /dev/sdac /dev/sdad /dev/sdae
zpool add -f ex1 mirror /dev/sdaf /dev/sdag /dev/sdah /dev/sdai /dev/sdaj
zpool create -f ex2 mirror /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj
zpool add -f ex2 /dev/sdk /dev/sdl /dev/sdm /dev/sdn /dev/sdo
zpool create -f ex3 mirror /dev/sdp /dev/sdq /dev/sdr /dev/sds /dev/sdt
zpool add -f ex3 mirror /dev/sdu /dev/sdv /dev/sdw /dev/sdx /dev/sdy
zpool create -f ex4 mirror /dev/sdz /dev/sdak /dev/sdal
zpool add -f ex4 mirror /dev/sdam /dev/sdan /dev/sdao

On tsadi
zpool create -f ex1 mirror /dev/sdaa /dev/sdab mirror /dev/sdac /dev/sdad mirror /dev/sdae /dev/sdaf mirror /dev/sdag /dev/sdah mirror /dev/sdai /dev/sdaj
zpool create -f ex2 mirror /dev/sdf /dev/sdg mirror /dev/sdh /dev/sdi mirror /dev/sdj /dev/sdk mirror /dev/sdl /dev/sdm mirror /dev/sdn /dev/sdo
zpool create -f ex3 mirror /dev/sdp /dev/sdq mirror /dev/sdr /dev/sds mirro /dev/sdt /dev/sdu mirror /dev/sdv /dev/sdw mirror /dev/sdx /dev/sdy
zpool create -f ex4 mirror /dev/sdz /dev/sdak /dev/sdal mirror /dev/sdam mirror /dev/sdan /dev/sdao

On lamed
zpool create -f ex5 mirror /dev/sdaa /dev/sdab mirror /dev/sdac /dev/sdad mirror /dev/sdae /dev/sdaf mirror /dev/sdag /dev/sdah mirror /dev/sdai /dev/sdaj
zpool create -f ex6 mirror /dev/sda /dev/sdb mirror /dev/sdc /dev/sdd mirror /dev/sde /dev/sdf mirror /dev/sdg /dev/sdh mirror /dev/sdi /dev/sdj
zpool create -f ex7 mirror /dev/sdk /dev/sdl mirror /dev/sdm /dev/sdn mirror /dev/sdo /dev/sdp mirror /dev/sdq /dev/sdr mirror /dev/sds /dev/sdt
zpool create -f ex8 mirror /dev/sdu /dev/sdv mirror /dev/sdw /dev/sdx mirror /dev/sdy /dev/sdz

= Sun Jan 19 2020 =

on mem2, sql system, note sda and sdc are system disks

zpool create -f sql1 raidz2 /dev/sdb /dev/sdc /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/sdk /dev/sdl /dev/sdm
zpool add -f sql1 raidz2 /dev/sdn /dev/sdo /dev/sdp /dev/sdq /dev/sdr /dev/sds /dev/sdt /dev/sdu /dev/sdv /dev/sdw /dev/sdx

transform db4 on n-9-22 from z2 to z0

zpool destroy db4
zpool create -f db4 raidz /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/sdk /dev/sdl /dev/sdm /dev/sdn /dev/sdo /dev/sdp /dev/sdq /dev/sdr /dev/sds /dev/sdt /dev/sdu /dev/sdv /dev/sdw /dev/sdx /dev/sdy

zfs mount
== recovery from accidental pool destruction ==
umount /mnt /mnt2
mdadm -S /dev/md125/dev/md126/dev/md127

sfdisk -d /dev/sda < sda.sfdisk
sfdisk -d /dev/sdb < sdb.sfdisk
sfdisk /dev/sda < sdb.sfdisk

mdadm --detail /dev/md127
mdadm -A -R /dev/md127/dev/sdb2/dev/sda2
mdadm /dev/md127 -a /dev/sda2
mdadm --detail /dev/md127
echo check > /sys/block/md127/md/sync_action
cat /proc/mdstat

mdadm --detail /dev/md126
mdadm -A -R /dev/md126/dev/sdb3/dev/sda3
mdadm /dev/md126 -a /dev/sda3
mdadm --detail /dev/md126
echo check > /sys/block/md126/md/sync_action
cat /proc/mdstat

Also switched the bios to boot from hd2 instead of hd1 (or something)

* Recreate zpool with correct drives
* Point an instance photorec at each of the wiped drives set to recover files of the following types: .gz, .solv (custom definition)

NOTE: If you destroyed your zpool with command 'zpool destroy', you can use the command 'zpool import' to view destroyed pools and recover the pool by doing 'zpool import <zpool name>'.

== Thu Apr 16, 2020 ==
We destroyed old db2 on abacus. We put in 20 new disks 7.68 TB and 2 new 2.5 TB disks

zpool create -f /scratch /dev/sdc /dev/sdd

zpool create -f /srv/db2 raidz2 /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/sdk /dev/sdl /dev/sdm /dev/sdn
zpool add -f /srv/db2 raidz2 /dev/sdo /dev/sdp /dev/sdq /dev/sdr /dev/sds /dev/sdt /dev/sdu /dev/sdv /dev/sdw /dev/sdx

OLD:
sudo zpool create -f /srv/db3 raidz2 /dev/sdaa /dev/sdab /dev/sdac /dev/sdad /dev/sdae /dev/sdaf /dev/sdag /dev/sdah /dev/sdai /dev/sdaj /dev/sdak /dev/sdal
sudo zpool add -f /srv/db3 raidz2 /dev/sdam /dev/sdan /dev/sdao /dev/sdap /dev/sdaq /dev/sdar /dev/sdas /dev/sdat /dev/sdau /dev/sdav /dev/sdaw /dev/sdax

== Mon Apr 20 2020 ==

zpool create -f db2 raidz2 /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/sdk /dev/sdl /dev/sdm /dev/sdn
zpool add -f db2 raidz2 /dev/sdo /dev/sdp /dev/sdq /dev/sdr /dev/sds /dev/sdt /dev/sdu /dev/sdv /dev/sdw /dev/sdx /dev/sdy /dev/sdz

zpool create -f db3 raidz2 /dev/sdaa /dev/sdab /dev/sdac /dev/sdad /dev/sdae /dev/sdaf /dev/sdag /dev/sdah /dev/sdai /dev/sdaj /dev/sdak /dev/sdal
zpool add -f db3 raidz2 /dev/sdam /dev/sdan /dev/sdao /dev/sdap /dev/sdaq /dev/sdar /dev/sdas /dev/sdat /dev/sdau /dev/sdav /dev/sdaw /dev/sdax

zpool create -f db5 raidz2 /dev/sday /dev/sdaz /dev/sdba /dev/sdbb /dev/sdbc /dev/sdbd /dev/sdbe /dev/sdbf /dev/sdbg /dev/sdbh /dev/sdbi /dev/sdbj
zpool add -f db5 raidz2 /dev/sdbk /dev/sdbl /dev/sdbm /dev/sdbn /dev/sdbo /dev/sdbp /dev/sdbq /dev/sdbr /dev/sdbs /dev/sdbt /dev/sdbu /dev/sdbv

== Tue Apr 21 2020 ==

Ben's commands:

fdisk -l 2>/dev/null | grep -o "zfs.*" > disk_ids
split -n 3 disk_ids disk_id_
db2_disks=`cat disk_id_aa`
db3_disks=`cat disk_id_ab`
db5_disks=`cat disk_id_ac`
zpool create -f db2 raidz2 $db2_disks
zpool create -f db3 raidz2 $db3_disks
zpool create -f db5 raidz2 $db5_disks
reboot

Amended commands, Apr 22- based on advice from john that vdevs should be limited to 12 disks each:

fdisk -l 2>/dev/null | grep -o "zfs.*" > disk_ids
split -n 6 disk_ids disk_id_

db2_disks_1=`cat disk_id_aa`
db2_disks_2=`cat disk_id_ab`

db3_disks_1=`cat disk_id_ac`
db3_disks_2=`cat disk_id_ad`

db5_disks_1=`cat disk_id_ae`
db5_disks_2=`cat disk_id_af`

zpool create -f db2 raidz2 $db2_disks_1
zpool add -f db2 raidz2 $db2_disks_2
zpool create -f db3 raidz2 $db3_disks_1
zpool add -f db3 raidz2 $db3_disks_2
zpool create -f db5 raidz2 $db5_disks_1
zpool add -f db5 raidz2 $db5_disks_2

== Mon Jul 20 2020 ==
*sda sdb are system disks
*sdc sdd are 240GB SSD
*sde is 480GB SSD

nfs-exb in n-1-30

zpool create -f exb raidz2 sdf sdg sdh sdi sdj sdk raidz2 sdl sdm sdn sdo sdp sdq raidz2 sdr sds sdt sdu sdv sdw raidz2 sdx sdy sdz sdaa sdab sdac raidz2 sdad sdae sdaf sdag
sdah sdai raidz2 sdaj sdak sdal sdam sdan sdao log mirror sdc sdd cache sde

== Tue Jul 21 2020 ==

nfs-exc in n-1-109:

zpool create -f exc raidz2 sdf sdg sdh sdi sdj sdk\
raidz2 sdl sdm sdn sdo sdp sdq\
raidz2 sdr sds sdt sdu sdv sdw\
raidz2 sdx sdy sdz sdaa sdab sdac\
raidz2 sdad sdae sdaf sdag sdah sdai\
raidz2 sdaj sdak sdal sdam sdan sdao\
log mirror sdc sdd\
cache sde

nfs-exd in n-1-113:

zpool create -f exd raidz2 sdd sde sdf sdg sdh sdi\
raidz2 sdj sdk sdl sdm sdn sdo\
raidz2 sdp sdq sdr sds sdt sdu\
raidz2 sdv sdw sdx sdy sdz sdaa\
raidz2 sdab sdac sdad sdae sdaf sdag\
raidz2 sdah sdai sdaj sdak sdal sdam\
log mirror sdc sdan\
cache sdao

= Troubleshooting =

== zpool destroy : Failed to unmount <device> - device busy ==

The help text will advise you to check lsof or fuser, but really what you need to do is stop the nfs service

systemctl stop nfs
umount /export/ex*
zpool destroy ...
zpool create ...
zpool ...
...
systemctl start nfs

== zpool missing after reboot ==
This is due to zfs-import-cache failed to start at boot time.
# check
$ systemctl status zfs-import-cache.service
# enable at boot time
$ systemctl enable zfs-import-cache.service

== Example: Fixing degraded pool, replacing faulted disk ==
On Feb 22, 2019, one of nfs-ex9's disks became faulty.

-bash-4.2$ '''zpool status'''
pool: ex9
state: DEGRADED
status: One or more devices are faulted in response to persistent errors.
Sufficient replicas exist for the pool to continue functioning in a
degraded state.
action: Replace the faulted device, or use 'zpool clear' to mark the device
repaired.
scan: scrub canceled on Fri Feb 22 11:31:25 2019
config:
raidz2-5 DEGRADED 0 0 0
sdae ONLINE 0 0 0
sdaf ONLINE 0 0 0
sdag ONLINE 0 0 0
sdah FAULTED 18 0 0 too many errors
sdai ONLINE 0 0 0
sdaj ONLINE 0 0 0

I did the following:

-bash-4.2$ '''sudo zpool offline ex9 sdb'''

Then I went to the server room to see that disk 1 still had a red light due to the fault. I pulled the disk out. Inserted a fresh one of the same brand, a Seagate Exos X12. The server detected the new disk and set the disk name as /dev/sdb, just like the one I just pulled out. Finally, I did the following command.

-bash-4.2$ '''sudo zpool replace ex9 /dev/sdah'''
-bash-4.2$ '''zpool status'''
pool: ex9
state: DEGRADED
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Tue Mar 19 14:06:33 2019
1.37G scanned out of 51.8T at 127M/s, 118h33m to go
37.9M resilvered, 0.00% done
.
.
.
raidz2-5 DEGRADED 0 0 0
sdae ONLINE 0 0 0
sdaf ONLINE 0 0 0
sdag ONLINE 0 0 0
replacing-3 DEGRADED 0 0 0
old FAULTED 18 0 0 too many errors
sdah ONLINE 0 0 0 (resilvering)
sdai ONLINE 0 0 0
sdaj ONLINE 0 0 0

Resilvering is the process of a disk being rebuilt from its parity group. Once it is finished, you should be good to go again.

For zayin/nfs-exa, some of the disks are named by id instead of the vdev-id.
raidz2-4 DEGRADED 0 0 0
scsi-35000c500a7da67cb ONLINE 0 0 0
scsi-35000c500a7daa34f ONLINE 0 0 0
scsi-35000c500a7db39db FAULTED 0 0 0 too many errors
scsi-35000c500a7da6b97 ONLINE 0 0 0
scsi-35000c500a7da265b ONLINE 0 0 0
scsi-35000c500a7da740f ONLINE 0 0 0

In this case, we have to determine the vdev name of the new disk disk just got inserted with dmesg. Look for log that mentioning about an new disk
$ dmesg | tail
[14663327.192519] sd 0:0:38:0: ['''sdad'''] Spinning up disk...
[14663327.192756] sd 0:0:38:0: Attached scsi generic sg27 type 0
[14663328.193173] ........................ready
[14663352.681625] sd 0:0:38:0: ['''sdad'''] 27344764928 512-byte logical blocks: (14.0 TB/12.7 TiB)
[14663352.681627] sd 0:0:38:0: [sdad] 4096-byte physical blocks
[14663352.687268] sd 0:0:38:0: [sdad] Write Protect is off
[14663352.687273] sd 0:0:38:0: [sdad] Mode Sense: db 00 10 08
[14663352.690847] sd 0:0:38:0: [sdad] Write cache: enabled, read cache: enabled, supports DPO and FUA
[14663352.732297] sd 0:0:38:0: [sdad] Attached SCSI disk

Once determine the name, we will start the resilvering process
$ zpool replace exa scsi-35000c500a7db39db sdad
# scsi-35000c500a7db39db is the id of the failed disk obtained from zpool status
# sdad is the vdev-id of the new replacement disk determined above

== Disk LED light ==
=== Identify failed disk by LED light ===
By disk_id
# turn light off
$ ledctl locate_off=/dev/disk/by-id/<disk_id>
# turn light on
$ ledctl locate=/dev/disk/by-id/<disk_id>
Example
$ ledctl locate_off=/dev/disk/by-id/scsi-35000c500a7d8137f
$ ledctl locate=/dev/disk/by-id/scsi-35000c500a7d8137f
By vdev
# turn light on
$ ledctl locate_off=/dev/<vdev>
# turn light on
$ ledctl locate=/dev/disk/<vdev>
Example
$ ledctl locate_off=/dev/sdaf
$ ledctl locate=/dev/sdaf

=== Reset light from LED light glitch ===
For qof/nfs-ex9, we had an issue with the disk LED for /dev/sdah still showing up red despite the resilvering occurring. To return the disk LED to a normal status, issue the following command:
$ '''sudo ledctl normal=/dev/<disk vdev id>'''
Example: $ '''sudo ledctl normal=/dev/sdah'''

or for zayin/nfs-exa, disk are identify by id
$ '''sudo ledctl normal=/dev/disk/by-id/<disk id>'''
Example: $ '''sudo ledctl normal=/dev/disk/by-id/scsi-35000c500a7db39db'''

[[Category:Curator]][[Category:Sysadmin]]

Zfs

2021-03-25T21:25:51Z

Jgutie11: /* Checking Disk Health and Integrity */

Zfs

2021-03-25T21:24:14Z

Jgutie11: /* situation */

ZFS - Zettabyte Filesystem

== ZFS packages installation ==
https://www.symmcom.com/docs/how-tos/storages/how-to-install-zfs-on-centos-7

ZFS rpm pkg must match the CentOS version install in the machine
* Check CentOS version
$ cat /etc/centos-release
CentOS Linux release 7.8.2003 (Core)

* Install ZFS-release package. In this case, you will need to install the package for CentOS 7.8 version
$ yum install http://download.zfsonlinux.org/epel/zfs-release.el7_8.noarch.rpm

* Edit /etc/yum.repos.d/zfs.repo
The ZFS package that we want to install is zfs-kmod
$ vim /etc/yum.repos.d/zfs.repo
There are 2 items to change
[zfs]
name=ZFS on Linux for EL7 - dkms
baseurl=http://download.zfsonlinux.org/epel/7.8/$basearch/
enabled=1 -> change to 0
metadata_expire=7d
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-zfsonlinux
[zfs-kmod]
name=ZFS on Linux for EL7 - kmod
baseurl=http://download.zfsonlinux.org/epel/7.8/kmod/$basearch/
enabled=0 change to 1
metadata_expire=7d
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-zfsonlinux

* Install ZFS
$ yum install zfs

* Useful Commands to see if any disk are avaible
$ lsblk -f
$ fdisk -l
$ fdisk -l /dev/sd* |grep '^Disk' |grep sectors |grep -v experimental

== Beginning ZFS instances ==

There are only two commmands to interact with ZFS.

zpool: used to create a ZFS vdev (virtual device). vdevs are composed of physical devices.
zfs: used to create/interact with a ZFS dataset. ZFS datasets are akin to logical volumes

# zpool creation syntax
zpool create <poolname> <vdev(s)>
# Create a zpool of six raidz2 vdevs, each with six drives. Includes two SSDs to used as a mirrored SLOG and one SSD as an L2ARC read cache. (example commmand was run on qof)
zpool create ex9 raidz2 sda sdb sdc sdd sde sdf raidz2 sdg sdh sdi sdj sdk sdl raidz2 sdm sdn sdo sdp sdq sdr raidz2 sds sdt sdu sdv sdw sdx raidz2 sdy sdz sdaa sdab sdac sdad raidz2 sdae sdaf sdag sdah sdai sdaj log mirror ata-INTEL_SSDSC2KG480G7_BTYM740603E0480BGN ata-INTEL_SSDSC2KG480G7_BTYM7406019K480BGN cache ata-INTEL_SSDSC2KG480G7_BTYM740602GN480BGN
[root@qof ~]# zpool status
pool: ex9
state: ONLINE
scan: none requested
config:
NAME STATE READ WRITE CKSUM
ex9 ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
sda ONLINE 0 0 0
sdb ONLINE 0 0 0
sdc ONLINE 0 0 0
sdd ONLINE 0 0 0
sde ONLINE 0 0 0
sdf ONLINE 0 0 0
raidz2-1 ONLINE 0 0 0
sdg ONLINE 0 0 0
sdh ONLINE 0 0 0
sdi ONLINE 0 0 0
sdj ONLINE 0 0 0
sdk ONLINE 0 0 0
sdl ONLINE 0 0 0
raidz2-2 ONLINE 0 0 0
sdm ONLINE 0 0 0
sdn ONLINE 0 0 0
sdo ONLINE 0 0 0
sdp ONLINE 0 0 0
sdq ONLINE 0 0 0
sdr ONLINE 0 0 0
raidz2-3 ONLINE 0 0 0
sds ONLINE 0 0 0
sdt ONLINE 0 0 0
sdu ONLINE 0 0 0
sdv ONLINE 0 0 0
sdw ONLINE 0 0 0
sdx ONLINE 0 0 0
raidz2-4 ONLINE 0 0 0
sdy ONLINE 0 0 0
sdz ONLINE 0 0 0
sdaa ONLINE 0 0 0
sdab ONLINE 0 0 0
sdac ONLINE 0 0 0
sdad ONLINE 0 0 0
raidz2-5 ONLINE 0 0 0
sdae ONLINE 0 0 0
sdaf ONLINE 0 0 0
sdag ONLINE 0 0 0
sdah ONLINE 0 0 0
sdai ONLINE 0 0 0
sdaj ONLINE 0 0 0
logs
mirror-6 ONLINE 0 0 0
ata-INTEL_SSDSC2KG480G7_BTYM740603E0480BGN ONLINE 0 0 0
ata-INTEL_SSDSC2KG480G7_BTYM7406019K480BGN ONLINE 0 0 0
cache
ata-INTEL_SSDSC2KG480G7_BTYM740602GN480BGN ONLINE 0 0 0

Adding a zfs filesystem:

Using qof as an example, I will create a child filesystem under ex9 named archive that will be mounted under /export/ex9/archive. This archive will be used to backup user data.

-bash-4.2$ zfs list
NAME USED AVAIL REFER MOUNTPOINT
ex9 2.39T 249T 2.39T /export/ex9
-bash-4.2$ sudo zfs create -o mountpoint=/export/ex9/archive ex9/archive
-bash-4.2$ zfs list
NAME USED AVAIL REFER MOUNTPOINT
ex9 2.39T 249T 2.39T /export/ex9
ex9/archive 192K 249T 192K /export/ex9/archive

== Testing ==

NOTE: It is important to reboot machine to see if the zpool stay on.

== Add alias to machine and mount point to puppet ==

Please see [http://wiki.docking.org/index.php/PuppetTricks#Adding_aliases_for_server_on_Alpha this section]

== Adding L2ARC Read Cache to a zpool==
# Look for available SSDs in /dev/disk/by-id/
# Choose an available SSD to use for read cache. Then decide which pool you want to put the cache on.
Syntax: zpool add <zpool name> <cache/log> <path to disk>
$ sudo zpool add ex6 cache /dev/disk/by-id/ata-INTEL_SSDSC2KG480G7_BTYM72830AV6480BGN

== Tuning ZFS options ==
# stores extended attributes as system attributes to improve performance
$ zfs xattr=sa <zfs dataset name>

# Turn on ZFS lz4 compression. Use this for compressible dataset such as many files with text
$ zfs set compression=lz4 <zfs dataset name>

# Turn off access time for improved disk performance (so that the OS doesn't write a new time every time a file is accessed)
$ zfs set atime=off <zfs dataset name>

NOTE: ZFS performance degrades tremendously when the zpool is over 80% used. To avoid this, I have set a quota to 80% of the 248TB in qof/nfs-ex9.
# To set a quota of 200TB on ZFS dataset:
$ zfs set quota=200T <zfs dataset>

# To remove a quota from a ZFS dataset:
$ zfs set quota=none <zfs dataset>

By default, ZFS pools/mounts do not have ACLs active.
# to active access control lists on a zpool
$ sudo zfs set acltype=posixacl <pool name>

== Checking Disk Health and Integrity ==
Print a brief summary of all pools:
zpool list

Print a detailed status of each disk and status of pool:
zpool status

Clear read errors on disk, if not anything serious:
zpool clear <pool_name>

Check data integrity, traverses all the data in the pool once and verifies that all blocks can be read:
zpool scrub <pool_name>

== mount after reboot ==
zfs set mountpoint=/export/db2 db2

== when you put in a new disk ==
fdisk -l
to see what is new

sudo zpool create -f /srv/db3 raidz2 /dev/sdaa /dev/sdab /dev/sdac /dev/sdad /dev/sdae /dev/sdaf /dev/sdag /dev/sdah /dev/sdai /dev/sdaj /dev/sdak /dev/sdal
sudo zpool add -f /srv/db3 raidz2 /dev/sdam /dev/sdan /dev/sdao /dev/sdap /dev/sdaq /dev/sdar /dev/sdas /dev/sdat /dev/sdau /dev/sdav /dev/sdaw /dev/sdax

zfs unmount db3

zfs mount db3

= latest =
zpool create -f db3 raidz2 /dev/sdy /dev/sdz /dev/sdaa /dev/sdab /dev/sdac /dev/sdad /dev/sdae /dev/sdaf /dev/sdag /dev/sdah /dev/sdai /dev/sdaj
zpool add -f db3 raidz2 /dev/sdak /dev/sdal /dev/sdam /dev/sdan /dev/sdao /dev/sdap /dev/sdaq /dev/sdar /dev/sdas /dev/sdat /dev/sdau /dev/sdav

zpool create -f db4 raidz2 /dev/sdax /dev/sday /dev/sdaz /dev/sdba /dev/sdbb /dev/sdbc /dev/sdbd /dev/sdbe /dev/sdbf /dev/sdbg /dev/sdbh /dev/sdbi
zpool add -f db4 raidz2 /dev/sdbj /dev/sdbk /dev/sdbl /dev/sdbm /dev/sdbn /dev/sdbo /dev/sdbp /dev/sdbq /dev/sdbr /dev/sdbs /dev/sdbt /dev/sdbu

= Fri Jan 19 2018 =

zpool create -f db5 raidz2 /dev/sdbw /dev/sdbx /dev/sdby /dev/sdbz /dev/sdca /dev/sdcb /dev/sdcc /dev/sdcd /dev/sdce /dev/sdcf /dev/sdcg /dev/sdch
zpool add -f db5 raidz2 /dev/sdci /dev/sdcj /dev/sdck /dev/sdcl /dev/sdcm /dev/sdcn /dev/sdco /dev/sdcp /dev/sdcq /dev/sdcr /dev/sdcs /dev/sdct
zfs mount db5

= Wed Jan 24 2018 =
On tsadi
zpool create -f ex1 mirror /dev/sdaa /dev/sdab /dev/sdac /dev/sdad /dev/sdae
zpool add -f ex1 mirror /dev/sdaf /dev/sdag /dev/sdah /dev/sdai /dev/sdaj
zpool create -f ex2 mirror /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj
zpool add -f ex2 /dev/sdk /dev/sdl /dev/sdm /dev/sdn /dev/sdo
zpool create -f ex3 mirror /dev/sdp /dev/sdq /dev/sdr /dev/sds /dev/sdt
zpool add -f ex3 mirror /dev/sdu /dev/sdv /dev/sdw /dev/sdx /dev/sdy
zpool create -f ex4 mirror /dev/sdz /dev/sdak /dev/sdal
zpool add -f ex4 mirror /dev/sdam /dev/sdan /dev/sdao

On tsadi
zpool create -f ex1 mirror /dev/sdaa /dev/sdab mirror /dev/sdac /dev/sdad mirror /dev/sdae /dev/sdaf mirror /dev/sdag /dev/sdah mirror /dev/sdai /dev/sdaj
zpool create -f ex2 mirror /dev/sdf /dev/sdg mirror /dev/sdh /dev/sdi mirror /dev/sdj /dev/sdk mirror /dev/sdl /dev/sdm mirror /dev/sdn /dev/sdo
zpool create -f ex3 mirror /dev/sdp /dev/sdq mirror /dev/sdr /dev/sds mirro /dev/sdt /dev/sdu mirror /dev/sdv /dev/sdw mirror /dev/sdx /dev/sdy
zpool create -f ex4 mirror /dev/sdz /dev/sdak /dev/sdal mirror /dev/sdam mirror /dev/sdan /dev/sdao

On lamed
zpool create -f ex5 mirror /dev/sdaa /dev/sdab mirror /dev/sdac /dev/sdad mirror /dev/sdae /dev/sdaf mirror /dev/sdag /dev/sdah mirror /dev/sdai /dev/sdaj
zpool create -f ex6 mirror /dev/sda /dev/sdb mirror /dev/sdc /dev/sdd mirror /dev/sde /dev/sdf mirror /dev/sdg /dev/sdh mirror /dev/sdi /dev/sdj
zpool create -f ex7 mirror /dev/sdk /dev/sdl mirror /dev/sdm /dev/sdn mirror /dev/sdo /dev/sdp mirror /dev/sdq /dev/sdr mirror /dev/sds /dev/sdt
zpool create -f ex8 mirror /dev/sdu /dev/sdv mirror /dev/sdw /dev/sdx mirror /dev/sdy /dev/sdz

= Sun Jan 19 2020 =

on mem2, sql system, note sda and sdc are system disks

zpool create -f sql1 raidz2 /dev/sdb /dev/sdc /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/sdk /dev/sdl /dev/sdm
zpool add -f sql1 raidz2 /dev/sdn /dev/sdo /dev/sdp /dev/sdq /dev/sdr /dev/sds /dev/sdt /dev/sdu /dev/sdv /dev/sdw /dev/sdx

transform db4 on n-9-22 from z2 to z0

zpool destroy db4
zpool create -f db4 raidz /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/sdk /dev/sdl /dev/sdm /dev/sdn /dev/sdo /dev/sdp /dev/sdq /dev/sdr /dev/sds /dev/sdt /dev/sdu /dev/sdv /dev/sdw /dev/sdx /dev/sdy

zfs mount
== recovery from accidental pool destruction ==
umount /mnt /mnt2
mdadm -S /dev/md125/dev/md126/dev/md127

sfdisk -d /dev/sda < sda.sfdisk
sfdisk -d /dev/sdb < sdb.sfdisk
sfdisk /dev/sda < sdb.sfdisk

mdadm --detail /dev/md127
mdadm -A -R /dev/md127/dev/sdb2/dev/sda2
mdadm /dev/md127 -a /dev/sda2
mdadm --detail /dev/md127
echo check > /sys/block/md127/md/sync_action
cat /proc/mdstat

mdadm --detail /dev/md126
mdadm -A -R /dev/md126/dev/sdb3/dev/sda3
mdadm /dev/md126 -a /dev/sda3
mdadm --detail /dev/md126
echo check > /sys/block/md126/md/sync_action
cat /proc/mdstat

Also switched the bios to boot from hd2 instead of hd1 (or something)

* Recreate zpool with correct drives
* Point an instance photorec at each of the wiped drives set to recover files of the following types: .gz, .solv (custom definition)

NOTE: If you destroyed your zpool with command 'zpool destroy', you can use the command 'zpool import' to view destroyed pools and recover the pool by doing 'zpool import <zpool name>'.

== Thu Apr 16, 2020 ==
We destroyed old db2 on abacus. We put in 20 new disks 7.68 TB and 2 new 2.5 TB disks

zpool create -f /scratch /dev/sdc /dev/sdd

zpool create -f /srv/db2 raidz2 /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/sdk /dev/sdl /dev/sdm /dev/sdn
zpool add -f /srv/db2 raidz2 /dev/sdo /dev/sdp /dev/sdq /dev/sdr /dev/sds /dev/sdt /dev/sdu /dev/sdv /dev/sdw /dev/sdx

OLD:
sudo zpool create -f /srv/db3 raidz2 /dev/sdaa /dev/sdab /dev/sdac /dev/sdad /dev/sdae /dev/sdaf /dev/sdag /dev/sdah /dev/sdai /dev/sdaj /dev/sdak /dev/sdal
sudo zpool add -f /srv/db3 raidz2 /dev/sdam /dev/sdan /dev/sdao /dev/sdap /dev/sdaq /dev/sdar /dev/sdas /dev/sdat /dev/sdau /dev/sdav /dev/sdaw /dev/sdax

== Mon Apr 20 2020 ==

zpool create -f db2 raidz2 /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/sdk /dev/sdl /dev/sdm /dev/sdn
zpool add -f db2 raidz2 /dev/sdo /dev/sdp /dev/sdq /dev/sdr /dev/sds /dev/sdt /dev/sdu /dev/sdv /dev/sdw /dev/sdx /dev/sdy /dev/sdz

zpool create -f db3 raidz2 /dev/sdaa /dev/sdab /dev/sdac /dev/sdad /dev/sdae /dev/sdaf /dev/sdag /dev/sdah /dev/sdai /dev/sdaj /dev/sdak /dev/sdal
zpool add -f db3 raidz2 /dev/sdam /dev/sdan /dev/sdao /dev/sdap /dev/sdaq /dev/sdar /dev/sdas /dev/sdat /dev/sdau /dev/sdav /dev/sdaw /dev/sdax

zpool create -f db5 raidz2 /dev/sday /dev/sdaz /dev/sdba /dev/sdbb /dev/sdbc /dev/sdbd /dev/sdbe /dev/sdbf /dev/sdbg /dev/sdbh /dev/sdbi /dev/sdbj
zpool add -f db5 raidz2 /dev/sdbk /dev/sdbl /dev/sdbm /dev/sdbn /dev/sdbo /dev/sdbp /dev/sdbq /dev/sdbr /dev/sdbs /dev/sdbt /dev/sdbu /dev/sdbv

== Tue Apr 21 2020 ==

Ben's commands:

fdisk -l 2>/dev/null | grep -o "zfs.*" > disk_ids
split -n 3 disk_ids disk_id_
db2_disks=`cat disk_id_aa`
db3_disks=`cat disk_id_ab`
db5_disks=`cat disk_id_ac`
zpool create -f db2 raidz2 $db2_disks
zpool create -f db3 raidz2 $db3_disks
zpool create -f db5 raidz2 $db5_disks
reboot

Amended commands, Apr 22- based on advice from john that vdevs should be limited to 12 disks each:

fdisk -l 2>/dev/null | grep -o "zfs.*" > disk_ids
split -n 6 disk_ids disk_id_

db2_disks_1=`cat disk_id_aa`
db2_disks_2=`cat disk_id_ab`

db3_disks_1=`cat disk_id_ac`
db3_disks_2=`cat disk_id_ad`

db5_disks_1=`cat disk_id_ae`
db5_disks_2=`cat disk_id_af`

zpool create -f db2 raidz2 $db2_disks_1
zpool add -f db2 raidz2 $db2_disks_2
zpool create -f db3 raidz2 $db3_disks_1
zpool add -f db3 raidz2 $db3_disks_2
zpool create -f db5 raidz2 $db5_disks_1
zpool add -f db5 raidz2 $db5_disks_2

== Mon Jul 20 2020 ==
*sda sdb are system disks
*sdc sdd are 240GB SSD
*sde is 480GB SSD

nfs-exb in n-1-30

zpool create -f exb raidz2 sdf sdg sdh sdi sdj sdk raidz2 sdl sdm sdn sdo sdp sdq raidz2 sdr sds sdt sdu sdv sdw raidz2 sdx sdy sdz sdaa sdab sdac raidz2 sdad sdae sdaf sdag
sdah sdai raidz2 sdaj sdak sdal sdam sdan sdao log mirror sdc sdd cache sde

== Tue Jul 21 2020 ==

nfs-exc in n-1-109:

zpool create -f exc raidz2 sdf sdg sdh sdi sdj sdk\
raidz2 sdl sdm sdn sdo sdp sdq\
raidz2 sdr sds sdt sdu sdv sdw\
raidz2 sdx sdy sdz sdaa sdab sdac\
raidz2 sdad sdae sdaf sdag sdah sdai\
raidz2 sdaj sdak sdal sdam sdan sdao\
log mirror sdc sdd\
cache sde

nfs-exd in n-1-113:

zpool create -f exd raidz2 sdd sde sdf sdg sdh sdi\
raidz2 sdj sdk sdl sdm sdn sdo\
raidz2 sdp sdq sdr sds sdt sdu\
raidz2 sdv sdw sdx sdy sdz sdaa\
raidz2 sdab sdac sdad sdae sdaf sdag\
raidz2 sdah sdai sdaj sdak sdal sdam\
log mirror sdc sdan\
cache sdao

= Troubleshooting =

== zpool destroy : Failed to unmount <device> - device busy ==

The help text will advise you to check lsof or fuser, but really what you need to do is stop the nfs service

systemctl stop nfs
umount /export/ex*
zpool destroy ...
zpool create ...
zpool ...
...
systemctl start nfs

== zpool missing after reboot ==
This is due to zfs-import-cache failed to start at boot time.
# check
$ systemctl status zfs-import-cache.service
# enable at boot time
$ systemctl enable zfs-import-cache.service

== Example: Fixing degraded pool, replacing faulted disk ==
On Feb 22, 2019, one of nfs-ex9's disks became faulty.

-bash-4.2$ '''zpool status'''
pool: ex9
state: DEGRADED
status: One or more devices are faulted in response to persistent errors.
Sufficient replicas exist for the pool to continue functioning in a
degraded state.
action: Replace the faulted device, or use 'zpool clear' to mark the device
repaired.
scan: scrub canceled on Fri Feb 22 11:31:25 2019
config:
raidz2-5 DEGRADED 0 0 0
sdae ONLINE 0 0 0
sdaf ONLINE 0 0 0
sdag ONLINE 0 0 0
sdah FAULTED 18 0 0 too many errors
sdai ONLINE 0 0 0
sdaj ONLINE 0 0 0

I did the following:

-bash-4.2$ '''sudo zpool offline ex9 sdb'''

Then I went to the server room to see that disk 1 still had a red light due to the fault. I pulled the disk out. Inserted a fresh one of the same brand, a Seagate Exos X12. The server detected the new disk and set the disk name as /dev/sdb, just like the one I just pulled out. Finally, I did the following command.

-bash-4.2$ '''sudo zpool replace ex9 /dev/sdah'''
-bash-4.2$ '''zpool status'''
pool: ex9
state: DEGRADED
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Tue Mar 19 14:06:33 2019
1.37G scanned out of 51.8T at 127M/s, 118h33m to go
37.9M resilvered, 0.00% done
.
.
.
raidz2-5 DEGRADED 0 0 0
sdae ONLINE 0 0 0
sdaf ONLINE 0 0 0
sdag ONLINE 0 0 0
replacing-3 DEGRADED 0 0 0
old FAULTED 18 0 0 too many errors
sdah ONLINE 0 0 0 (resilvering)
sdai ONLINE 0 0 0
sdaj ONLINE 0 0 0

Resilvering is the process of a disk being rebuilt from its parity group. Once it is finished, you should be good to go again.

For zayin/nfs-exa, some of the disks are named by id instead of the vdev-id.
raidz2-4 DEGRADED 0 0 0
scsi-35000c500a7da67cb ONLINE 0 0 0
scsi-35000c500a7daa34f ONLINE 0 0 0
scsi-35000c500a7db39db FAULTED 0 0 0 too many errors
scsi-35000c500a7da6b97 ONLINE 0 0 0
scsi-35000c500a7da265b ONLINE 0 0 0
scsi-35000c500a7da740f ONLINE 0 0 0

In this case, we have to determine the vdev name of the new disk disk just got inserted with dmesg. Look for log that mentioning about an new disk
$ dmesg | tail
[14663327.192519] sd 0:0:38:0: ['''sdad'''] Spinning up disk...
[14663327.192756] sd 0:0:38:0: Attached scsi generic sg27 type 0
[14663328.193173] ........................ready
[14663352.681625] sd 0:0:38:0: ['''sdad'''] 27344764928 512-byte logical blocks: (14.0 TB/12.7 TiB)
[14663352.681627] sd 0:0:38:0: [sdad] 4096-byte physical blocks
[14663352.687268] sd 0:0:38:0: [sdad] Write Protect is off
[14663352.687273] sd 0:0:38:0: [sdad] Mode Sense: db 00 10 08
[14663352.690847] sd 0:0:38:0: [sdad] Write cache: enabled, read cache: enabled, supports DPO and FUA
[14663352.732297] sd 0:0:38:0: [sdad] Attached SCSI disk

Once determine the name, we will start the resilvering process
$ zpool replace exa scsi-35000c500a7db39db sdad
# scsi-35000c500a7db39db is the id of the failed disk obtained from zpool status
# sdad is the vdev-id of the new replacement disk determined above

== Disk LED light ==
=== Identify failed disk by LED light ===
By disk_id
# turn light off
$ ledctl locate_off=/dev/disk/by-id/<disk_id>
# turn light on
$ ledctl locate=/dev/disk/by-id/<disk_id>
Example
$ ledctl locate_off=/dev/disk/by-id/scsi-35000c500a7d8137f
$ ledctl locate=/dev/disk/by-id/scsi-35000c500a7d8137f
By vdev
# turn light on
$ ledctl locate_off=/dev/<vdev>
# turn light on
$ ledctl locate=/dev/disk/<vdev>
Example
$ ledctl locate_off=/dev/sdaf
$ ledctl locate=/dev/sdaf

=== Reset light from LED light glitch ===
For qof/nfs-ex9, we had an issue with the disk LED for /dev/sdah still showing up red despite the resilvering occurring. To return the disk LED to a normal status, issue the following command:
$ '''sudo ledctl normal=/dev/<disk vdev id>'''
Example: $ '''sudo ledctl normal=/dev/sdah'''

or for zayin/nfs-exa, disk are identify by id
$ '''sudo ledctl normal=/dev/disk/by-id/<disk id>'''
Example: $ '''sudo ledctl normal=/dev/disk/by-id/scsi-35000c500a7db39db'''

[[Category:Curator]][[Category:Sysadmin]]

Arthor Documentation for Future Developer

2021-03-24T00:22:31Z

Jgutie11: /* Customizing Arthor Code to our needs */

Written by Jennifer Young on December 16, 2019. Last edited January 05, 2021

==Install and Set Up on TomCat (Method 1)==
Arthor ran on n-1-136, which runs CentOS Linux release 7.7.1908 (Core). You can check the version of CentOS with the following command
cat /etc/centos-release

Check your current version of Java with the following command:
java -version

On n-1-136 we are running openjdk version "1.8.0_222", OpenJDK Runtime Environment (build 1.8.0_222-b10), and OpenJDK 64-Bit Server VM (build 25.222-b10, mixed mode)
If Java is not installed, install it using yum

==See this wiki page for more detailed information about installing Tomcat on our cluster==
http://wiki.docking.org/index.php/Tomcat_Installation

==Open port for Arthor==
In order for Arthor to be usable in the browser, the port you wish to run it on must be opened.
https://www.thegeekdiary.com/how-to-open-a-ports-in-centos-rhel-7/

===Step 1: Check Port Status===
Check that the port is not open and that Apache is not showing that port.
netstat -na | grep <port number you are checking>

lsof -i -P |grep http

===Step 2: Check Port Status in IP Tables===
iptables-save | grep <port number you are checking>

I skipped Step 3 from the guide, because there was a lot of information in the /etc/services file and I didn't want to edit it and break something.

===Step 4: Open Firewall Ports===
I did not include the zone=public section because the stand-alone servers are usually used for private instances of Arthor and SmallWorld.
Run as root.
firewall-cmd --add-port=<port number you are adding>/tcp --permanent

You need to reload the firewall after a change is made.
firewall-cmd --reload

===Step 5: Check that port is working===
To check that the port is active, run.
iptables -nL

You should see something along the lines of:
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:<port number you're adding> ctstate NEW,UNTRACKED

==How to run standalone Arthor instance==

===Step 1: Use or start a bash shell===
You can check your default shell using
echo $SHELL

If your default shell is csh, use
bash
to start a new bash shell in the current terminal window. Note that echo $SHELL will show you your default shell regardless of the current shell.

===Step 2: Set your environment variables===
export ARTHOR_DIR=/opt/nextmove/arthor/arthor-3.3-centos7
export PATH=$ARTHOR_DIR/bin/:$PATH

Make sure the ARTHOR_DIR variable is set to the directory for the latest version of Arthor or whichever version you would like to test.
The PATH environment variable is needed if you wish to use the Arthor tools from the command line

===Step 3: Run the arthor-server.jar===
java -jar /opt/nextmove/arthor/arthor-3.3-centos7/java/arthor.jar --httpPort <your httpPort>

==Setting environment variables for an Arthor Server==
Set the environment variables in the setenv.sh file. Note: Be sure to edit the file in the directory corresponding to the latest version of TomCat. As of December 2019, we are running 9.0.27 on n-1-136.

vim /opt/tomcat/apache-tomcat-9.0.27/bin/setenv.sh

Add the line below to the setenv.sh file above, or substitute the path to wherever you currently store the arthor.cfg file
export ARTHOR_CONFIG=/usr/local/tomcat/arthor.cfg

Here is an example of the arthor.cfg file:
BinDir=/opt/nextmove/arthor/arthor-3.3-centos7/bin
DataDir=/local2/arthor_local_8081/
MaxConcurrentSearches=6
MaxThreadsPerSearch=8
AutomaticIndex=false
AsyncHitCountMax=1000000
Resolver=https://sw.docking.org/util/smi2mol?smi=%s

=== Configuration Details ===

*'''BinDir''': is the location of the Arthor command line binaries. These are used to generate the Arthor index files and to perform searches directly on n-1-136. An example of this would be using atdbgrep for substructure search.

*'''DataDir''': This is the directory where the Arthor data files live. Location where the index files will be created and loaded from.

*'''MaxConcurrentSearches''': Controls the maximum number of searches that can be run concurrently by setting the database pool size. When switching between a large number of databases it can be useful to have a larger pool size, the only trade off is keeping file pointers open.

*'''MaxThreadsPerSearch''': The number of threads to use for both ATDB and ATFP searches

*Set '''AutomaticIndex''' to false if you don't want new smiles files added to the data directory to be indexed automatically

*'''AsyncHitCountMax''': The upper-bound for the number of hits to retrieve in background searches.

*'''Resolver''': Using Smallworld API, allows input box to take in a SMILE format and automatically draw on the board.

Check Arthor manual for more configuration options.

==Background==
Before working with Arthor, it is recommended that you familiarize yourself with the Arthor documentation. Some useful pages to look at include 3-5, 22-25 and 33-39. Of course, reading everything would be the best!

==Checking Disk Space Usage==
Before building arthor indexes, it's always a good thing to check what percent of the memory is being used. Try to be cautious with how much memory you have left, and make sure to check while building indexes to make sure that you have enough space. To check, run the following command:

df -h /<directory with disc>

==Building Large Databases==
At the moment, we are building databases of size 500M molecules by merging smile files. There are multiple methods of trying to create large databases, one being merging based off of the same H?? prefix and stopping once the database reaches > 500M molecules (or whatever upperbound you want to use). Here is some python code that simulates this merging process. Essentially the program takes all of the .smi files within an input directory, sorts them lexiographically, and begins merging these .smi files together in order until the size reaches > 500M molecules.

Feel free to modify it if you think a better method exists.

import subprocess
import sys
import os

from os import listdir
from os.path import isfile, join

mypath = "<Path to directory holding .smi files>"
onlyfiles = [f for f in listdir(mypath) if isfile(join(mypath, f))]
onlyfiles.sort()

create_fp = True
cur_mols = 0
lower_bound = 500000000
upper_bound = 600000000
files_to_merge = []

def merge_files(f_t_m):
arr = f_t_m[0].split(".")
arr2 = f_t_m[len(f_t_m) - 1].split(".")
file_name_merge = (arr[0] + "_" + arr2[0] + ".smi")
print ("File being created: " + file_name_merge)

for file in f_t_m:
tmp = file.split(".")
process = subprocess.Popen("cat " + join(mypath, file) + " >> " + file_name_merge, shell=True)
process.wait()

for file in onlyfiles:
arr = file.split(".")

if (arr[len(arr) - 1] == "smi"):
print("Working with " + file)
mol = sum(1 for line in open(join(mypath, file)))
print(file, mol, cur_mols)

if (cur_mols + mol > lower_bound):
if (cur_mols + mol < upper_bound):
files_to_merge.append(file)
merge_files(files_to_merge)
cur_mols = 0
files_to_merge.clear()
else:
merge_files(files_to_merge)
files_to_merge.clear()
files_to_merge.append(file)
merge_files(files_to_merge)
cur_mols = 0
files_to_merge.clear()
else:
cur_mols += mol
files_to_merge.append(file)

if (len(files_to_merge) != 0):
merge_files(files_to_merge)

==Building Arthor Indexes==
Once you've merged the .smi files together, it's time to start building the databases themselves. To do this we use the command

smi2atdb -j 0 -p <The .smi file> <The .atdb>

The flag "-j 0" enables parallel generation and utilizes all available processors to generate the .atdb file. The "-p" flag stores the offset position in the ATDB file. Since we're building indexes for the Web Application, you must use the "-p" flag when building indexes. Please note that the name of the .smi file should also be the name of the .atdb file. That way, the Web Application knows to use these files together and correctly display the required images. Refer to pages 33-34 in the Arthor documentation for more information.

If there are too many large .smi files and you do not want to manually build each .atdb file, you can use this python script which takes all of the .smi files in the current directory and converts them to .atdb files. Make sure to modify mypath to the directory containing the .smi files. You can change the variable "create_fp" to false if you don't want to create .atdb.fp files (refer to page 9 in the Arthor documentation).

import subprocess
import sys
import os

from os import listdir
from os.path import isfile, join

mypath = "<Path containing the .smi files"
onlyfiles = [f for f in listdir(mypath) if isfile(join(mypath, f))]

create_fp = True

for file in onlyfiles:
arr = file.split(".")

if (arr[len(arr) - 1] == "smi"):
process = subprocess.Popen("/nfs/ex9/work/xyz/psql/arthor-3.3-centos7/bin/smi2atdb -j 0 -p {0} {1}.atdb".format(join(mypath, file), arr[0]), shell=True)
process.wait()

print("SUCCESS! {0}.atdb file was created!".format(arr[0]))

if (create_fp):
process = subprocess.Popen("/nfs/ex9/work/xyz/psql/arthor-3.3-centos7/bin/atdb2fp -j 0 {0}.atdb".format(arr[0]), shell=True)
process.wait()

print("SUCCESS! {0}.atdb.fp file was created!".format(arr[0]))

==Uploading Indexes to the Web Application==
One can upload indexes to the Web Application by changing the "DATADIR" variable in the arthor.cfg file to the directory holding the .atdb files. This is already set up on n-1-136 and n-5-34.

==Virtual Memory==
In addition to modifying the arthor.cfg file, virtual memory can also be used to make queries faster. There can still be More information can be found in pages 10-16 in the Arthor Documentation.

==Setting up Round Table==
This is a new feature in Arthor 3.0 and is currently beta (January 2020). See section 2.4 in the manual
As explained in the manual, "Round Table allows you to serve and split chemical searches across multiple host machines. The implementation provides a lightweight proxy that forwards requests to other Arthor host servers that do the actual search. Communication is done using the existing Web APIs.

Since Arthor requires CentOS 7, as of January 2020 we have 6 servers that are capable of running Arthor with Round Table.

===Setting up Host Server===
If we want to add machines to the Round Table, for example 'nun' and 'samekh', we need to edit their arthor.cfg file so that when our Local Machine passes commands these secondary servers know to perform the search they are given.

$ cat arthor.cfg
MaxThreadsPerSearch=4
AutomaticIndex=false
DATADIR=<Directory where smiles are located>

We then run the jar server on each of these host machines containing data on any available port.

java -jar /nfs/ex9/work/xyz/psql/arthor-3.3-centos7/java/arthor.jar --httpPort <port>

For our local machine, the arthor.cfg file will look different.

$ cat arthor.cfg
[RoundTable]
RemoteClient=http://skynet:<port number where jar server is running>/
RemoteClient=http://hal:<port number where jar server is running>/

Please refer to Section 2 in the RoundTable Documentation file (pages 6-8) for more useful information on configuration.

Then run the following command on n-1-136:

java -jar /nfs/ex9/work/xyz/psql/arthor-3.3-centos7/java/arthor.jar --httpPort <port>

===Public Arthor===
{| class="wikitable"
|-
! CentOS 7 Machine
! Port
! Total Files Size
! Arthor Install Location
! Round Table Data Directory
! Active
|-
| samekh
| 10.20.0.41:8000
| 2.4TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/public_arthor/
| active
|-
| nun
| 10.20.0.40:8000
| 2.4TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/public_arthor/
| active
|-
| n-9-22
| 10.20.9.22:8000
| 2.4TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /export/db4/public_arthor/
| active
|-
|}

===Arthor Round Table Head===
{| class="wikitable"
|-
! CentOS 7 Machine
! Port
! Database
! Arthor Install Location
! Round Table Data Directory
! Active
|-
| samekh
| 10.20.0.41:8080
| Enamine_REAL_Q2-2020-All-13B
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
| nun
| 10.20.0.40:8080
| Enamine_REAL_Q2-2020-All-41B
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
|}

===Arthor Round Table Nodes===
{| class="wikitable"
|-
! CentOS 7 Machine
! Port
! Database
! Total Files Size
! Arthor Install Location
! Round Table Data Directory
!Active
|-
| samekh
| 10.20.0.41:8008
| Enamine_REAL_Q2-2020-All-13B (26 slices)
| 4.5TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
| nun
| 10.20.0.40:8008
| Enamine_REAL_Space_June_2020_S41B (aa-ae, 5 slices), Enamine_REAL_Space_June_2020_M41B (af-am, 8 slices), zinc22_2d (H04~H25, 22 slices)
| 5.6TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
| n-1-17
| 10.20.1.17:8008
| Enamine_REAL_Space_June_2020_M41B (aa-ae, 5 slices), zinc22_2d (H25~H29, 4 slices)
| 3.7TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
| n-5-32
| 10.20.5.32:8008
| Enamine_REAL_Space_June_2020_M41B (an~az, 13 slices), zinc22_2d (H30, 1 slice)
| 5.6TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
| n-5-33
| 10.20.5.33:8008
| Enamine_REAL_Space_June_2020_M41B (ba-bl, 12 slices)
| 5.3TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
|}

===Arthor Local 8081 (Datasets all local to samekh/nun)===
{| class="wikitable"
|-
! CentOS 7 Machine
! Port
! Database
! Total Files Size
! Arthor Install Location
! Round Table Data Directory
!Active
|-
| samekh
| 10.20.0.41:8081
| Enamine_REAL_Q2-2020-All-13B (26 slices)
| 4.5TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_local_8081/
| active
|-
| nun
| 10.20.0.40:8081
| Enamine_REAL_Space_June_2020_S41B (aa-ae, 5 slices), Enamine_REAL_Space_June_2020_M41B (aa-an, 14 slices)
| 4.3TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_local_8081/
| active
|-
|}

== Customizing Arthor Frontend to our needs ==
The frontend Arthor code is located at '''/nfs/exc/arthor_configs/*''' and the '''*''' is based on current running version.
=== Add Arthor Download Options ===
==== For Arthor 3.4: ====
1. vim .extract/webapps/ROOT/WEB-INF/static/index.html

2. search: '''arthor_tsv_link'''

3. in the div with the class=”dropdown-content”, add these link options and change the number accordingly:

<a id="arthor_tsv_link" href="#"> TSV-500</a>
<a id="arthor_tsv_link_5000" href="#"> TSV-5,000</a>
<a id="arthor_tsv_link_50000" href="#"> TSV-50,000</a>
<a id="arthor_tsv_link_100000" href="#"> TSV-100,000</a>
<a id="arthor_tsv_link_max" href="#"> TSV-max</a>
<a id="arthor_csv_link" href="#"> CSV-500</a>
<a id="arthor_csv_link_5000" href="#"> CSV-5,000</a>
<a id="arthor_csv_link_50000" href="#"> CSV-50,000</a>
<a id="arthor_csv_link_100000" href="#"> CSV-100,000</a>
<a id="arthor_csv_link_max" href="#"> CSV-max</a>
<a id="arthor_sdf_link" href="#"> SDF-500</a>
<a id="arthor_sdf_link_5000" href="#"> SDF-5,000</a>
<a id="arthor_sdf_link_50000" href="#"> SDF-50,000</a>
<a id="arthor_sdf_link_100000" href="#"> SDF-100,000</a>
<a id="arthor_sdf_link_max" href="#"> SDF-max</a>

4. then vim .extract/webapps/ROOT/WEB-INF/static/js/index.js

5. search: '''function $(t){'''

6. in the function $(t), add these lines:

if(document.getElementById("arthor_tsv_link")) {
var e=i.a.param({query:s.b.query,type:s.b.type,draw:0,start:0,length:t,flags:s.b.flags}),n=s.b.url+"/dt/"+E(s.b.table)+"/search";i()("#arthor_sdf_link").attr("href",n+".sdf?"+e),i() ("#arthor_tsv_link").attr("href",n+".tsv?"+e),i()("#arthor_csv_link").attr("href",n+".csv?"+e)
}
if (document.getElementById("arthor_tsv_link_5000")) {
var e=i.a.param({query:s.b.query,type:s.b.type,draw:0,start:0,length:5000,flags:s.b.flags}),n=s.b.url+"/dt/"+E(s.b.table)+"/search";i()("#arthor_sdf_link_5000").attr("href",n+".sdf?"+e),i()("#arthor_tsv_link_5000").attr("href",n+".tsv?"+e),i()("#arthor_csv_link_5000").attr("href",n+".csv?"+e)
}
if (document.getElementById("arthor_tsv_link_50000")) {
var e=i.a.param({query:s.b.query,type:s.b.type,draw:0,start:0,length:50000,flags:s.b.flags}),n=s.b.url+"/dt/"+E(s.b.table)+"/search";i()("#arthor_sdf_link_50000").attr("href",n+".sdf?"+e),i()("#arthor_tsv_link_50000").attr("href",n+".tsv?"+e),i()("#arthor_csv_link_50000").attr("href",n+".csv?"+e)
}
if (document.getElementById("arthor_tsv_link_100000")) {
var e=i.a.param({query:s.b.query,type:s.b.type,draw:0,start:0,length:100000,flags:s.b.flags}),n=s.b.url+"/dt/"+E(s.b.table)+"/search";i()("#arthor_sdf_link_100000").attr("href",n+".sdf?"+e),i()("#arthor_tsv_link_100000").attr("href",n+".tsv?"+e),i()("#arthor_csv_link_100000").attr("href",n+".csv?"+e)
}
if (document.getElementById("arthor_tsv_link_max")) {
var e=i.a.param({query:s.b.query,type:s.b.type,draw:0,start:0,length:1000000000,flags:s.b.flags}),n=s.b.url+"/dt/"+E(s.b.table)+"/search";i()("#arthor_sdf_link_max").attr("href",n+".sdf?"+e),i()("#arthor_tsv_link_max").attr("href",n+".tsv?"+e),i()("#arthor_csv_link_max").attr("href",n+".csv?"+e)
}

=== Take out Similarity Button ===
vim .extract/webapps/ROOT/WEB-INF/static/index.html
search: '''Similarity'''
Comment out this line '''< li value="Similarity" onclick="setSearchType(this)" class="first"> Similarity </li >''' //added spaces at the beginning and end so prevent wiki from converting it
Then add "first" in Substructure's class
=== Hyperlink to zinc20 ===
vim .extract/webapps/ROOT/WEB-INF/static/js/index.js
search: '''table_name'''
*find this line "" + d + ""
*replace with '''"<a target='_blank' href='https://zinc20.docking.org/substances/"+d+"'>" + d + "</a>"''' //added spaces at the beginning and end so prevent wiki from converting it

=== Make Input Box Work ===
At the end of the Arthor config file add this:
Resolver=https://sw.docking.org/util/smi2mol?smi=%s
To copy smiles in the input box:
vim .extract/webapps/ROOT/WEB-INF/static/js/index.js
search this: “var e=t.src.smiles()”
add this after the semi-colon
document.getElementById("ar_text_input").value = e;

Arthor Documentation for Future Developer

2021-03-24T00:21:54Z

Jgutie11: /* Customizing Arthor Code to our needs */

Written by Jennifer Young on December 16, 2019. Last edited January 05, 2021

==Install and Set Up on TomCat (Method 1)==
Arthor ran on n-1-136, which runs CentOS Linux release 7.7.1908 (Core). You can check the version of CentOS with the following command
cat /etc/centos-release

Check your current version of Java with the following command:
java -version

On n-1-136 we are running openjdk version "1.8.0_222", OpenJDK Runtime Environment (build 1.8.0_222-b10), and OpenJDK 64-Bit Server VM (build 25.222-b10, mixed mode)
If Java is not installed, install it using yum

==See this wiki page for more detailed information about installing Tomcat on our cluster==
http://wiki.docking.org/index.php/Tomcat_Installation

==Open port for Arthor==
In order for Arthor to be usable in the browser, the port you wish to run it on must be opened.
https://www.thegeekdiary.com/how-to-open-a-ports-in-centos-rhel-7/

===Step 1: Check Port Status===
Check that the port is not open and that Apache is not showing that port.
netstat -na | grep <port number you are checking>

lsof -i -P |grep http

===Step 2: Check Port Status in IP Tables===
iptables-save | grep <port number you are checking>

I skipped Step 3 from the guide, because there was a lot of information in the /etc/services file and I didn't want to edit it and break something.

===Step 4: Open Firewall Ports===
I did not include the zone=public section because the stand-alone servers are usually used for private instances of Arthor and SmallWorld.
Run as root.
firewall-cmd --add-port=<port number you are adding>/tcp --permanent

You need to reload the firewall after a change is made.
firewall-cmd --reload

===Step 5: Check that port is working===
To check that the port is active, run.
iptables -nL

You should see something along the lines of:
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:<port number you're adding> ctstate NEW,UNTRACKED

==How to run standalone Arthor instance==

===Step 1: Use or start a bash shell===
You can check your default shell using
echo $SHELL

If your default shell is csh, use
bash
to start a new bash shell in the current terminal window. Note that echo $SHELL will show you your default shell regardless of the current shell.

===Step 2: Set your environment variables===
export ARTHOR_DIR=/opt/nextmove/arthor/arthor-3.3-centos7
export PATH=$ARTHOR_DIR/bin/:$PATH

Make sure the ARTHOR_DIR variable is set to the directory for the latest version of Arthor or whichever version you would like to test.
The PATH environment variable is needed if you wish to use the Arthor tools from the command line

===Step 3: Run the arthor-server.jar===
java -jar /opt/nextmove/arthor/arthor-3.3-centos7/java/arthor.jar --httpPort <your httpPort>

==Setting environment variables for an Arthor Server==
Set the environment variables in the setenv.sh file. Note: Be sure to edit the file in the directory corresponding to the latest version of TomCat. As of December 2019, we are running 9.0.27 on n-1-136.

vim /opt/tomcat/apache-tomcat-9.0.27/bin/setenv.sh

Add the line below to the setenv.sh file above, or substitute the path to wherever you currently store the arthor.cfg file
export ARTHOR_CONFIG=/usr/local/tomcat/arthor.cfg

Here is an example of the arthor.cfg file:
BinDir=/opt/nextmove/arthor/arthor-3.3-centos7/bin
DataDir=/local2/arthor_local_8081/
MaxConcurrentSearches=6
MaxThreadsPerSearch=8
AutomaticIndex=false
AsyncHitCountMax=1000000
Resolver=https://sw.docking.org/util/smi2mol?smi=%s

=== Configuration Details ===

*'''BinDir''': is the location of the Arthor command line binaries. These are used to generate the Arthor index files and to perform searches directly on n-1-136. An example of this would be using atdbgrep for substructure search.

*'''DataDir''': This is the directory where the Arthor data files live. Location where the index files will be created and loaded from.

*'''MaxConcurrentSearches''': Controls the maximum number of searches that can be run concurrently by setting the database pool size. When switching between a large number of databases it can be useful to have a larger pool size, the only trade off is keeping file pointers open.

*'''MaxThreadsPerSearch''': The number of threads to use for both ATDB and ATFP searches

*Set '''AutomaticIndex''' to false if you don't want new smiles files added to the data directory to be indexed automatically

*'''AsyncHitCountMax''': The upper-bound for the number of hits to retrieve in background searches.

*'''Resolver''': Using Smallworld API, allows input box to take in a SMILE format and automatically draw on the board.

Check Arthor manual for more configuration options.

==Background==
Before working with Arthor, it is recommended that you familiarize yourself with the Arthor documentation. Some useful pages to look at include 3-5, 22-25 and 33-39. Of course, reading everything would be the best!

==Checking Disk Space Usage==
Before building arthor indexes, it's always a good thing to check what percent of the memory is being used. Try to be cautious with how much memory you have left, and make sure to check while building indexes to make sure that you have enough space. To check, run the following command:

df -h /<directory with disc>

==Building Large Databases==
At the moment, we are building databases of size 500M molecules by merging smile files. There are multiple methods of trying to create large databases, one being merging based off of the same H?? prefix and stopping once the database reaches > 500M molecules (or whatever upperbound you want to use). Here is some python code that simulates this merging process. Essentially the program takes all of the .smi files within an input directory, sorts them lexiographically, and begins merging these .smi files together in order until the size reaches > 500M molecules.

Feel free to modify it if you think a better method exists.

import subprocess
import sys
import os

from os import listdir
from os.path import isfile, join

mypath = "<Path to directory holding .smi files>"
onlyfiles = [f for f in listdir(mypath) if isfile(join(mypath, f))]
onlyfiles.sort()

create_fp = True
cur_mols = 0
lower_bound = 500000000
upper_bound = 600000000
files_to_merge = []

def merge_files(f_t_m):
arr = f_t_m[0].split(".")
arr2 = f_t_m[len(f_t_m) - 1].split(".")
file_name_merge = (arr[0] + "_" + arr2[0] + ".smi")
print ("File being created: " + file_name_merge)

for file in f_t_m:
tmp = file.split(".")
process = subprocess.Popen("cat " + join(mypath, file) + " >> " + file_name_merge, shell=True)
process.wait()

for file in onlyfiles:
arr = file.split(".")

if (arr[len(arr) - 1] == "smi"):
print("Working with " + file)
mol = sum(1 for line in open(join(mypath, file)))
print(file, mol, cur_mols)

if (cur_mols + mol > lower_bound):
if (cur_mols + mol < upper_bound):
files_to_merge.append(file)
merge_files(files_to_merge)
cur_mols = 0
files_to_merge.clear()
else:
merge_files(files_to_merge)
files_to_merge.clear()
files_to_merge.append(file)
merge_files(files_to_merge)
cur_mols = 0
files_to_merge.clear()
else:
cur_mols += mol
files_to_merge.append(file)

if (len(files_to_merge) != 0):
merge_files(files_to_merge)

==Building Arthor Indexes==
Once you've merged the .smi files together, it's time to start building the databases themselves. To do this we use the command

smi2atdb -j 0 -p <The .smi file> <The .atdb>

The flag "-j 0" enables parallel generation and utilizes all available processors to generate the .atdb file. The "-p" flag stores the offset position in the ATDB file. Since we're building indexes for the Web Application, you must use the "-p" flag when building indexes. Please note that the name of the .smi file should also be the name of the .atdb file. That way, the Web Application knows to use these files together and correctly display the required images. Refer to pages 33-34 in the Arthor documentation for more information.

If there are too many large .smi files and you do not want to manually build each .atdb file, you can use this python script which takes all of the .smi files in the current directory and converts them to .atdb files. Make sure to modify mypath to the directory containing the .smi files. You can change the variable "create_fp" to false if you don't want to create .atdb.fp files (refer to page 9 in the Arthor documentation).

import subprocess
import sys
import os

from os import listdir
from os.path import isfile, join

mypath = "<Path containing the .smi files"
onlyfiles = [f for f in listdir(mypath) if isfile(join(mypath, f))]

create_fp = True

for file in onlyfiles:
arr = file.split(".")

if (arr[len(arr) - 1] == "smi"):
process = subprocess.Popen("/nfs/ex9/work/xyz/psql/arthor-3.3-centos7/bin/smi2atdb -j 0 -p {0} {1}.atdb".format(join(mypath, file), arr[0]), shell=True)
process.wait()

print("SUCCESS! {0}.atdb file was created!".format(arr[0]))

if (create_fp):
process = subprocess.Popen("/nfs/ex9/work/xyz/psql/arthor-3.3-centos7/bin/atdb2fp -j 0 {0}.atdb".format(arr[0]), shell=True)
process.wait()

print("SUCCESS! {0}.atdb.fp file was created!".format(arr[0]))

==Uploading Indexes to the Web Application==
One can upload indexes to the Web Application by changing the "DATADIR" variable in the arthor.cfg file to the directory holding the .atdb files. This is already set up on n-1-136 and n-5-34.

==Virtual Memory==
In addition to modifying the arthor.cfg file, virtual memory can also be used to make queries faster. There can still be More information can be found in pages 10-16 in the Arthor Documentation.

==Setting up Round Table==
This is a new feature in Arthor 3.0 and is currently beta (January 2020). See section 2.4 in the manual
As explained in the manual, "Round Table allows you to serve and split chemical searches across multiple host machines. The implementation provides a lightweight proxy that forwards requests to other Arthor host servers that do the actual search. Communication is done using the existing Web APIs.

Since Arthor requires CentOS 7, as of January 2020 we have 6 servers that are capable of running Arthor with Round Table.

===Setting up Host Server===
If we want to add machines to the Round Table, for example 'nun' and 'samekh', we need to edit their arthor.cfg file so that when our Local Machine passes commands these secondary servers know to perform the search they are given.

$ cat arthor.cfg
MaxThreadsPerSearch=4
AutomaticIndex=false
DATADIR=<Directory where smiles are located>

We then run the jar server on each of these host machines containing data on any available port.

java -jar /nfs/ex9/work/xyz/psql/arthor-3.3-centos7/java/arthor.jar --httpPort <port>

For our local machine, the arthor.cfg file will look different.

$ cat arthor.cfg
[RoundTable]
RemoteClient=http://skynet:<port number where jar server is running>/
RemoteClient=http://hal:<port number where jar server is running>/

Please refer to Section 2 in the RoundTable Documentation file (pages 6-8) for more useful information on configuration.

Then run the following command on n-1-136:

java -jar /nfs/ex9/work/xyz/psql/arthor-3.3-centos7/java/arthor.jar --httpPort <port>

===Public Arthor===
{| class="wikitable"
|-
! CentOS 7 Machine
! Port
! Total Files Size
! Arthor Install Location
! Round Table Data Directory
! Active
|-
| samekh
| 10.20.0.41:8000
| 2.4TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/public_arthor/
| active
|-
| nun
| 10.20.0.40:8000
| 2.4TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/public_arthor/
| active
|-
| n-9-22
| 10.20.9.22:8000
| 2.4TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /export/db4/public_arthor/
| active
|-
|}

===Arthor Round Table Head===
{| class="wikitable"
|-
! CentOS 7 Machine
! Port
! Database
! Arthor Install Location
! Round Table Data Directory
! Active
|-
| samekh
| 10.20.0.41:8080
| Enamine_REAL_Q2-2020-All-13B
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
| nun
| 10.20.0.40:8080
| Enamine_REAL_Q2-2020-All-41B
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
|}

===Arthor Round Table Nodes===
{| class="wikitable"
|-
! CentOS 7 Machine
! Port
! Database
! Total Files Size
! Arthor Install Location
! Round Table Data Directory
!Active
|-
| samekh
| 10.20.0.41:8008
| Enamine_REAL_Q2-2020-All-13B (26 slices)
| 4.5TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
| nun
| 10.20.0.40:8008
| Enamine_REAL_Space_June_2020_S41B (aa-ae, 5 slices), Enamine_REAL_Space_June_2020_M41B (af-am, 8 slices), zinc22_2d (H04~H25, 22 slices)
| 5.6TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
| n-1-17
| 10.20.1.17:8008
| Enamine_REAL_Space_June_2020_M41B (aa-ae, 5 slices), zinc22_2d (H25~H29, 4 slices)
| 3.7TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
| n-5-32
| 10.20.5.32:8008
| Enamine_REAL_Space_June_2020_M41B (an~az, 13 slices), zinc22_2d (H30, 1 slice)
| 5.6TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
| n-5-33
| 10.20.5.33:8008
| Enamine_REAL_Space_June_2020_M41B (ba-bl, 12 slices)
| 5.3TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
|}

===Arthor Local 8081 (Datasets all local to samekh/nun)===
{| class="wikitable"
|-
! CentOS 7 Machine
! Port
! Database
! Total Files Size
! Arthor Install Location
! Round Table Data Directory
!Active
|-
| samekh
| 10.20.0.41:8081
| Enamine_REAL_Q2-2020-All-13B (26 slices)
| 4.5TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_local_8081/
| active
|-
| nun
| 10.20.0.40:8081
| Enamine_REAL_Space_June_2020_S41B (aa-ae, 5 slices), Enamine_REAL_Space_June_2020_M41B (aa-an, 14 slices)
| 4.3TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_local_8081/
| active
|-
|}

== Customizing Arthor Code to our needs ==
The frontend Arthor code is located at '''/nfs/exc/arthor_configs/*''' and the '''*''' is based on current running version.
=== Add Arthor Download Options ===
==== For Arthor 3.4: ====
1. vim .extract/webapps/ROOT/WEB-INF/static/index.html

2. search: '''arthor_tsv_link'''

3. in the div with the class=”dropdown-content”, add these link options and change the number accordingly:

<a id="arthor_tsv_link" href="#"> TSV-500</a>
<a id="arthor_tsv_link_5000" href="#"> TSV-5,000</a>
<a id="arthor_tsv_link_50000" href="#"> TSV-50,000</a>
<a id="arthor_tsv_link_100000" href="#"> TSV-100,000</a>
<a id="arthor_tsv_link_max" href="#"> TSV-max</a>
<a id="arthor_csv_link" href="#"> CSV-500</a>
<a id="arthor_csv_link_5000" href="#"> CSV-5,000</a>
<a id="arthor_csv_link_50000" href="#"> CSV-50,000</a>
<a id="arthor_csv_link_100000" href="#"> CSV-100,000</a>
<a id="arthor_csv_link_max" href="#"> CSV-max</a>
<a id="arthor_sdf_link" href="#"> SDF-500</a>
<a id="arthor_sdf_link_5000" href="#"> SDF-5,000</a>
<a id="arthor_sdf_link_50000" href="#"> SDF-50,000</a>
<a id="arthor_sdf_link_100000" href="#"> SDF-100,000</a>
<a id="arthor_sdf_link_max" href="#"> SDF-max</a>

4. then vim .extract/webapps/ROOT/WEB-INF/static/js/index.js

5. search: '''function $(t){'''

6. in the function $(t), add these lines:

if(document.getElementById("arthor_tsv_link")) {
var e=i.a.param({query:s.b.query,type:s.b.type,draw:0,start:0,length:t,flags:s.b.flags}),n=s.b.url+"/dt/"+E(s.b.table)+"/search";i()("#arthor_sdf_link").attr("href",n+".sdf?"+e),i() ("#arthor_tsv_link").attr("href",n+".tsv?"+e),i()("#arthor_csv_link").attr("href",n+".csv?"+e)
}
if (document.getElementById("arthor_tsv_link_5000")) {
var e=i.a.param({query:s.b.query,type:s.b.type,draw:0,start:0,length:5000,flags:s.b.flags}),n=s.b.url+"/dt/"+E(s.b.table)+"/search";i()("#arthor_sdf_link_5000").attr("href",n+".sdf?"+e),i()("#arthor_tsv_link_5000").attr("href",n+".tsv?"+e),i()("#arthor_csv_link_5000").attr("href",n+".csv?"+e)
}
if (document.getElementById("arthor_tsv_link_50000")) {
var e=i.a.param({query:s.b.query,type:s.b.type,draw:0,start:0,length:50000,flags:s.b.flags}),n=s.b.url+"/dt/"+E(s.b.table)+"/search";i()("#arthor_sdf_link_50000").attr("href",n+".sdf?"+e),i()("#arthor_tsv_link_50000").attr("href",n+".tsv?"+e),i()("#arthor_csv_link_50000").attr("href",n+".csv?"+e)
}
if (document.getElementById("arthor_tsv_link_100000")) {
var e=i.a.param({query:s.b.query,type:s.b.type,draw:0,start:0,length:100000,flags:s.b.flags}),n=s.b.url+"/dt/"+E(s.b.table)+"/search";i()("#arthor_sdf_link_100000").attr("href",n+".sdf?"+e),i()("#arthor_tsv_link_100000").attr("href",n+".tsv?"+e),i()("#arthor_csv_link_100000").attr("href",n+".csv?"+e)
}
if (document.getElementById("arthor_tsv_link_max")) {
var e=i.a.param({query:s.b.query,type:s.b.type,draw:0,start:0,length:1000000000,flags:s.b.flags}),n=s.b.url+"/dt/"+E(s.b.table)+"/search";i()("#arthor_sdf_link_max").attr("href",n+".sdf?"+e),i()("#arthor_tsv_link_max").attr("href",n+".tsv?"+e),i()("#arthor_csv_link_max").attr("href",n+".csv?"+e)
}

=== Take out Similarity Button ===
vim .extract/webapps/ROOT/WEB-INF/static/index.html
search: '''Similarity'''
Comment out this line '''< li value="Similarity" onclick="setSearchType(this)" class="first"> Similarity </li >''' //added spaces at the beginning and end so prevent wiki from converting it
Then add "first" in Substructure's class
=== Hyperlink to zinc20 ===
vim .extract/webapps/ROOT/WEB-INF/static/js/index.js
search: '''table_name'''
*find this line "" + d + ""
*replace with '''"<a target='_blank' href='https://zinc20.docking.org/substances/"+d+"'>" + d + "</a>"''' //added spaces at the beginning and end so prevent wiki from converting it

=== Make Input Box Work ===
At the end of the Arthor config file add this:
Resolver=https://sw.docking.org/util/smi2mol?smi=%s
To copy smiles in the input box:
vim .extract/webapps/ROOT/WEB-INF/static/js/index.js
search this: “var e=t.src.smiles()”
add this after the semi-colon
document.getElementById("ar_text_input").value = e;

Arthor Documentation for Future Developer

2021-03-23T00:08:25Z

Jgutie11: /* Hyperlink to zinc20 */

Written by Jennifer Young on December 16, 2019. Last edited January 05, 2021

==Install and Set Up on TomCat (Method 1)==
Arthor ran on n-1-136, which runs CentOS Linux release 7.7.1908 (Core). You can check the version of CentOS with the following command
cat /etc/centos-release

Check your current version of Java with the following command:
java -version

On n-1-136 we are running openjdk version "1.8.0_222", OpenJDK Runtime Environment (build 1.8.0_222-b10), and OpenJDK 64-Bit Server VM (build 25.222-b10, mixed mode)
If Java is not installed, install it using yum

==See this wiki page for more detailed information about installing Tomcat on our cluster==
http://wiki.docking.org/index.php/Tomcat_Installation

==Open port for Arthor==
In order for Arthor to be usable in the browser, the port you wish to run it on must be opened.
https://www.thegeekdiary.com/how-to-open-a-ports-in-centos-rhel-7/

===Step 1: Check Port Status===
Check that the port is not open and that Apache is not showing that port.
netstat -na | grep <port number you are checking>

lsof -i -P |grep http

===Step 2: Check Port Status in IP Tables===
iptables-save | grep <port number you are checking>

I skipped Step 3 from the guide, because there was a lot of information in the /etc/services file and I didn't want to edit it and break something.

===Step 4: Open Firewall Ports===
I did not include the zone=public section because the stand-alone servers are usually used for private instances of Arthor and SmallWorld.
Run as root.
firewall-cmd --add-port=<port number you are adding>/tcp --permanent

You need to reload the firewall after a change is made.
firewall-cmd --reload

===Step 5: Check that port is working===
To check that the port is active, run.
iptables -nL

You should see something along the lines of:
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:<port number you're adding> ctstate NEW,UNTRACKED

==How to run standalone Arthor instance==

===Step 1: Use or start a bash shell===
You can check your default shell using
echo $SHELL

If your default shell is csh, use
bash
to start a new bash shell in the current terminal window. Note that echo $SHELL will show you your default shell regardless of the current shell.

===Step 2: Set your environment variables===
export ARTHOR_DIR=/opt/nextmove/arthor/arthor-3.3-centos7
export PATH=$ARTHOR_DIR/bin/:$PATH

Make sure the ARTHOR_DIR variable is set to the directory for the latest version of Arthor or whichever version you would like to test.
The PATH environment variable is needed if you wish to use the Arthor tools from the command line

===Step 3: Run the arthor-server.jar===
java -jar /opt/nextmove/arthor/arthor-3.3-centos7/java/arthor.jar --httpPort <your httpPort>

==Setting environment variables for an Arthor Server==
Set the environment variables in the setenv.sh file. Note: Be sure to edit the file in the directory corresponding to the latest version of TomCat. As of December 2019, we are running 9.0.27 on n-1-136.

vim /opt/tomcat/apache-tomcat-9.0.27/bin/setenv.sh

Add the line below to the setenv.sh file above, or substitute the path to wherever you currently store the arthor.cfg file
export ARTHOR_CONFIG=/usr/local/tomcat/arthor.cfg

Here is an example of the arthor.cfg file:
BinDir=/opt/nextmove/arthor/arthor-3.3-centos7/bin
DataDir=/local2/arthor_local_8081/
MaxConcurrentSearches=6
MaxThreadsPerSearch=8
AutomaticIndex=false
AsyncHitCountMax=1000000
Resolver=https://sw.docking.org/util/smi2mol?smi=%s

=== Configuration Details ===

*'''BinDir''': is the location of the Arthor command line binaries. These are used to generate the Arthor index files and to perform searches directly on n-1-136. An example of this would be using atdbgrep for substructure search.

*'''DataDir''': This is the directory where the Arthor data files live. Location where the index files will be created and loaded from.

*'''MaxConcurrentSearches''': Controls the maximum number of searches that can be run concurrently by setting the database pool size. When switching between a large number of databases it can be useful to have a larger pool size, the only trade off is keeping file pointers open.

*'''MaxThreadsPerSearch''': The number of threads to use for both ATDB and ATFP searches

*Set '''AutomaticIndex''' to false if you don't want new smiles files added to the data directory to be indexed automatically

*'''AsyncHitCountMax''': The upper-bound for the number of hits to retrieve in background searches.

*'''Resolver''': Using Smallworld API, allows input box to take in a SMILE format and automatically draw on the board.

Check Arthor manual for more configuration options.

==Background==
Before working with Arthor, it is recommended that you familiarize yourself with the Arthor documentation. Some useful pages to look at include 3-5, 22-25 and 33-39. Of course, reading everything would be the best!

==Checking Disk Space Usage==
Before building arthor indexes, it's always a good thing to check what percent of the memory is being used. Try to be cautious with how much memory you have left, and make sure to check while building indexes to make sure that you have enough space. To check, run the following command:

df -h /<directory with disc>

==Building Large Databases==
At the moment, we are building databases of size 500M molecules by merging smile files. There are multiple methods of trying to create large databases, one being merging based off of the same H?? prefix and stopping once the database reaches > 500M molecules (or whatever upperbound you want to use). Here is some python code that simulates this merging process. Essentially the program takes all of the .smi files within an input directory, sorts them lexiographically, and begins merging these .smi files together in order until the size reaches > 500M molecules.

Feel free to modify it if you think a better method exists.

import subprocess
import sys
import os

from os import listdir
from os.path import isfile, join

mypath = "<Path to directory holding .smi files>"
onlyfiles = [f for f in listdir(mypath) if isfile(join(mypath, f))]
onlyfiles.sort()

create_fp = True
cur_mols = 0
lower_bound = 500000000
upper_bound = 600000000
files_to_merge = []

def merge_files(f_t_m):
arr = f_t_m[0].split(".")
arr2 = f_t_m[len(f_t_m) - 1].split(".")
file_name_merge = (arr[0] + "_" + arr2[0] + ".smi")
print ("File being created: " + file_name_merge)

for file in f_t_m:
tmp = file.split(".")
process = subprocess.Popen("cat " + join(mypath, file) + " >> " + file_name_merge, shell=True)
process.wait()

for file in onlyfiles:
arr = file.split(".")

if (arr[len(arr) - 1] == "smi"):
print("Working with " + file)
mol = sum(1 for line in open(join(mypath, file)))
print(file, mol, cur_mols)

if (cur_mols + mol > lower_bound):
if (cur_mols + mol < upper_bound):
files_to_merge.append(file)
merge_files(files_to_merge)
cur_mols = 0
files_to_merge.clear()
else:
merge_files(files_to_merge)
files_to_merge.clear()
files_to_merge.append(file)
merge_files(files_to_merge)
cur_mols = 0
files_to_merge.clear()
else:
cur_mols += mol
files_to_merge.append(file)

if (len(files_to_merge) != 0):
merge_files(files_to_merge)

==Building Arthor Indexes==
Once you've merged the .smi files together, it's time to start building the databases themselves. To do this we use the command

smi2atdb -j 0 -p <The .smi file> <The .atdb>

The flag "-j 0" enables parallel generation and utilizes all available processors to generate the .atdb file. The "-p" flag stores the offset position in the ATDB file. Since we're building indexes for the Web Application, you must use the "-p" flag when building indexes. Please note that the name of the .smi file should also be the name of the .atdb file. That way, the Web Application knows to use these files together and correctly display the required images. Refer to pages 33-34 in the Arthor documentation for more information.

If there are too many large .smi files and you do not want to manually build each .atdb file, you can use this python script which takes all of the .smi files in the current directory and converts them to .atdb files. Make sure to modify mypath to the directory containing the .smi files. You can change the variable "create_fp" to false if you don't want to create .atdb.fp files (refer to page 9 in the Arthor documentation).

import subprocess
import sys
import os

from os import listdir
from os.path import isfile, join

mypath = "<Path containing the .smi files"
onlyfiles = [f for f in listdir(mypath) if isfile(join(mypath, f))]

create_fp = True

for file in onlyfiles:
arr = file.split(".")

if (arr[len(arr) - 1] == "smi"):
process = subprocess.Popen("/nfs/ex9/work/xyz/psql/arthor-3.3-centos7/bin/smi2atdb -j 0 -p {0} {1}.atdb".format(join(mypath, file), arr[0]), shell=True)
process.wait()

print("SUCCESS! {0}.atdb file was created!".format(arr[0]))

if (create_fp):
process = subprocess.Popen("/nfs/ex9/work/xyz/psql/arthor-3.3-centos7/bin/atdb2fp -j 0 {0}.atdb".format(arr[0]), shell=True)
process.wait()

print("SUCCESS! {0}.atdb.fp file was created!".format(arr[0]))

==Uploading Indexes to the Web Application==
One can upload indexes to the Web Application by changing the "DATADIR" variable in the arthor.cfg file to the directory holding the .atdb files. This is already set up on n-1-136 and n-5-34.

==Virtual Memory==
In addition to modifying the arthor.cfg file, virtual memory can also be used to make queries faster. There can still be More information can be found in pages 10-16 in the Arthor Documentation.

==Setting up Round Table==
This is a new feature in Arthor 3.0 and is currently beta (January 2020). See section 2.4 in the manual
As explained in the manual, "Round Table allows you to serve and split chemical searches across multiple host machines. The implementation provides a lightweight proxy that forwards requests to other Arthor host servers that do the actual search. Communication is done using the existing Web APIs.

Since Arthor requires CentOS 7, as of January 2020 we have 6 servers that are capable of running Arthor with Round Table.

===Setting up Host Server===
If we want to add machines to the Round Table, for example 'nun' and 'samekh', we need to edit their arthor.cfg file so that when our Local Machine passes commands these secondary servers know to perform the search they are given.

$ cat arthor.cfg
MaxThreadsPerSearch=4
AutomaticIndex=false
DATADIR=<Directory where smiles are located>

We then run the jar server on each of these host machines containing data on any available port.

java -jar /nfs/ex9/work/xyz/psql/arthor-3.3-centos7/java/arthor.jar --httpPort <port>

For our local machine, the arthor.cfg file will look different.

$ cat arthor.cfg
[RoundTable]
RemoteClient=http://skynet:<port number where jar server is running>/
RemoteClient=http://hal:<port number where jar server is running>/

Please refer to Section 2 in the RoundTable Documentation file (pages 6-8) for more useful information on configuration.

Then run the following command on n-1-136:

java -jar /nfs/ex9/work/xyz/psql/arthor-3.3-centos7/java/arthor.jar --httpPort <port>

===Public Arthor===
{| class="wikitable"
|-
! CentOS 7 Machine
! Port
! Total Files Size
! Arthor Install Location
! Round Table Data Directory
! Active
|-
| samekh
| 10.20.0.41:8000
| 2.4TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/public_arthor/
| active
|-
| nun
| 10.20.0.40:8000
| 2.4TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/public_arthor/
| active
|-
| n-9-22
| 10.20.9.22:8000
| 2.4TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /export/db4/public_arthor/
| active
|-
|}

===Arthor Round Table Head===
{| class="wikitable"
|-
! CentOS 7 Machine
! Port
! Database
! Arthor Install Location
! Round Table Data Directory
! Active
|-
| samekh
| 10.20.0.41:8080
| Enamine_REAL_Q2-2020-All-13B
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
| nun
| 10.20.0.40:8080
| Enamine_REAL_Q2-2020-All-41B
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
|}

===Arthor Round Table Nodes===
{| class="wikitable"
|-
! CentOS 7 Machine
! Port
! Database
! Total Files Size
! Arthor Install Location
! Round Table Data Directory
!Active
|-
| samekh
| 10.20.0.41:8008
| Enamine_REAL_Q2-2020-All-13B (26 slices)
| 4.5TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
| nun
| 10.20.0.40:8008
| Enamine_REAL_Space_June_2020_S41B (aa-ae, 5 slices), Enamine_REAL_Space_June_2020_M41B (af-am, 8 slices), zinc22_2d (H04~H25, 22 slices)
| 5.6TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
| n-1-17
| 10.20.1.17:8008
| Enamine_REAL_Space_June_2020_M41B (aa-ae, 5 slices), zinc22_2d (H25~H29, 4 slices)
| 3.7TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
| n-5-32
| 10.20.5.32:8008
| Enamine_REAL_Space_June_2020_M41B (an~az, 13 slices), zinc22_2d (H30, 1 slice)
| 5.6TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
| n-5-33
| 10.20.5.33:8008
| Enamine_REAL_Space_June_2020_M41B (ba-bl, 12 slices)
| 5.3TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
|}

===Arthor Local 8081 (Datasets all local to samekh/nun)===
{| class="wikitable"
|-
! CentOS 7 Machine
! Port
! Database
! Total Files Size
! Arthor Install Location
! Round Table Data Directory
!Active
|-
| samekh
| 10.20.0.41:8081
| Enamine_REAL_Q2-2020-All-13B (26 slices)
| 4.5TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_local_8081/
| active
|-
| nun
| 10.20.0.40:8081
| Enamine_REAL_Space_June_2020_S41B (aa-ae, 5 slices), Enamine_REAL_Space_June_2020_M41B (aa-an, 14 slices)
| 4.3TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_local_8081/
| active
|-
|}

== Customizing Arthor Code to our needs ==
If Arthor Sever is launched through "java -jar /opt/nextmove/arthor/arthor-3.3.2-centos7/java/arthor.jar --httpPort=<port>", find the directory where this line of code was executed. Once found do '''ls -a''', there should be a hidden directory called .extract.
=== Change Arthor Download Size (Hardcoded) ===
vim .extract/webapps/ROOT/WEB-INF/static/js/index.js
?#arthor_sdf_link //search this
*Look for 0!==arguments[0]?arguments[0]:<number>
*Change number to desirec amount
=== Change Arthor Download Size (Options) ===
vim .extract/webapps/ROOT/WEB-INF/static/index.html
search this: “res-download”
in the div with the class=”dropdown-content”
add these link options and change the number accordingly:
<a id="arthor_tsv_link" href="#"> TSV-500</a>
<a id="arthor_tsv_link_1000" href="#"> TSV-1,000</a>
<a id="arthor_tsv_link_5000" href="#"> TSV-5,000</a>
<a id="arthor_csv_link" href="#"> CSV-500</a>
<a id="arthor_csv_link_1000" href="#"> CSV-1,000</a>
<a id="arthor_csv_link_5000" href="#"> CSV-5,000</a>
<a id="arthor_sdf_link" href="#"> SDF-500</a>
<a id="arthor_sdf_link_1000" href="#"> SDF-1,000</a>
<a id="arthor_sdf_link_5000" href="#"> SDF-5,000</a>
then vim .extract/webapps/ROOT/WEB-INF/static/js/index.js
For arthor 3.3.2 search this: “function $(){”
For arthor 3.3 search this: “function Fs()” or “#arthor_tsv_link”
Separate this function from the rest of the code from beginning and end of the function
For arthor 3.3.2, edit the function for example:
function $(){
if(document.getElementById("arthor_tsv_link")) {
var t=arguments.length>0&&void 0!==arguments[0]?
arguments[0]:500,e=i.a.param({query:arthor.query,type:arthor.type,draw:0,start:0,length:t,flags:arthor.flags}),n=arthor.url+"/dt/"+L(arthor.table)+"/search";i()
("#arthor_sdf_link").attr("href",n+".sdf?"+e),i()("#arthor_tsv_link").attr("href",n+".tsv?"+e),i()("#arthor_csv_link").attr("href",n+".csv?"+e)
}
if (document.getElementById("arthor_tsv_link_1000")) {
var t=arguments.length>0&&void 0!==arguments[0]?
arguments[0]:1000,e=i.a.param({query:arthor.query,type:arthor.type,draw:0,start:0,length:t,flags:arthor.flags}),n=arthor.url+"/dt/"+L(arthor.table)+"/search";i()
("#arthor_sdf_link_1000").attr("href",n+".sdf?"+e),i()("#arthor_tsv_link_1000").attr("href",n+".tsv?"+e),i()("#arthor_csv_link_1000").attr("href",n+".csv?"+e)
}
if (document.getElementById("arthor_tsv_link_5000")) {
var t=arguments.length>0&&void 0!==arguments[0]?
arguments[0]:5000,e=i.a.param({query:arthor.query,type:arthor.type,draw:0,start:0,length:t,flags:arthor.flags}),n=arthor.url+"/dt/"+L(arthor.table)+"/search";i()
("#arthor_sdf_link_5000").attr("href",n+".sdf?"+e),i()("#arthor_tsv_link_5000").attr("href",n+".tsv?"+e),i()("#arthor_csv_link_5000").attr("href",n+".csv?"+e)
}
}
For arthor 3.3, edit the function for example:
function Fs(){
if(document.getElementById("arthor_tsv_link")) {
var t = arguments.length>0&&void 0!==arguments[0]?
arguments[0]:500,e=i.a.param({query:arthor.query,type:arthor.type,draw:0,start:0,length:t,flags:arthor.flags}),n=arthor.url+"/dt/"+xs(arthor.table)+"/search";i()
("#arthor_sdf_link").attr("href",n+".sdf?"+e),i()("#arthor_tsv_link").attr("href",n+".tsv?"+e),i()("#arthor_csv_link").attr("href",n+".csv?"+e)
}
if (document.getElementById("arthor_tsv_link_1000")) {
var t=arguments.length>0&&void 0!==arguments[0]?
arguments[0]:1000,e=i.a.param({query:arthor.query,type:arthor.type,draw:0,start:0,length:t,flags:arthor.flags}),n=arthor.url+"/dt/"+xs(arthor.table)+"/search";i()
("#arthor_sdf_link_1000").attr("href",n+".sdf?"+e),i()("#arthor_tsv_link_1000").attr("href",n+".tsv?"+e),i()("#arthor_csv_link_1000").attr("href",n+".csv?"+e)
}
if (document.getElementById("arthor_tsv_link_5000")) {
var t=arguments.length>0&&void 0!==arguments[0]?
arguments[0]:5000,e=i.a.param({query:arthor.query,type:arthor.type,draw:0,start:0,length:t,flags:arthor.flags}),n=arthor.url+"/dt/"+xs(arthor.table)+"/search";i()
("#arthor_sdf_link_5000").attr("href",n+".sdf?"+e),i()("#arthor_tsv_link_5000").attr("href",n+".tsv?"+e),i()("#arthor_csv_link_5000").attr("href",n+".csv?"+e)
}
}
=== Take out Similarity Button ===
vim .extract/webapps/ROOT/WEB-INF/static/index.html
?Similarity //search this
*Comment out this line '''< li value="Similarity" onclick="setSearchType(this)" class="first"> Similarity </li >''' //added spaces at the beginning and end so prevent wiki from converting it
=== Hyperlink to zinc20 ===
vim .extract/webapps/ROOT/WEB-INF/static/js/index.js
?table_name //search this
*find this line "" + d + ""
*replace with '''"<a target='_blank' href='https://zinc20.docking.org/substances/"+d+"'>" + d + "</a>"''' //added spaces at the beginning and end so prevent wiki from converting it

=== Make Input Box Work ===
At the end of the Arthor config file add this:
Resolver=https://sw.docking.org/util/smi2mol?smi=%s
To copy smiles in the input box:
vim .extract/webapps/ROOT/WEB-INF/static/js/index.js
search this: “var e=t.src.smiles()”
add this after the semi-colon
document.getElementById("ar_text_input").value = e;

Using LVM To Mount Drives

2021-03-05T21:38:43Z

Jgutie11: /* Mount Local2 using LVM */

Using LVM To Mount Drives

2021-03-05T00:24:34Z

Jgutie11: /* Mount Local2 using LVM */

Using LVM To Mount Drives

2021-03-05T00:24:01Z

Jgutie11:

== Format Disk ==
Do these commands as root:
* fdisk -l
** Find disks/RAID disks to format
*parted
*print
*mklabel gpt
*mkpart logical 0GB 9995GB. (based on what we read in result of print above)
*print. (to confirm)
*quit

== Mount Local2 using LVM ==
* pvcreate /dev/sdb1
* vgcreate zinc /dev/sdb1
* lvcreate -l 100%FREE -n zinc_lv zinc
* mkfs.ext4 -L zinc /dev/mapper/zinc-zinc_lv
* mkdir /local2
* vim /etc/fstab
** /dev/zinc/zinc_lv /local2 ext4 defaults 1 2
* mount /local2

== Partition Disk ==
* https://www.techotopia.com/index.php/Adding_a_New_Disk_Drive_to_a_CentOS_System

Using LVM To Mount Drives

2021-03-04T22:37:33Z

Jgutie11:

== Format Disk ==
Do these commands as root:
* fdisk -l
** Find disks/RAID disks to format
*parted
*print
*mklabel gpt
*mkpart logical 0GB 9995GB. (based on what we read in result of print above)
*print. (to confirm)
*quit

== Mount Local2 ==
* pvcreate /dev/sdb1
* vgcreate zinc /dev/sdb1
* lvcreate -l 100%FREE -n zinc_lv zinc
* mkfs.ext4 -L zinc /dev/mapper/zinc-zinc_lv
* mkdir /local2
* vim /etc/fstab
** /dev/zinc/zinc_lv /local2 ext4 defaults 1 2
* mount /local2

== Partition Disk ==
* https://www.techotopia.com/index.php/Adding_a_New_Disk_Drive_to_a_CentOS_System

Arthor Documentation for Future Developer

2021-03-01T20:46:12Z

Jgutie11: /* Customizing Arthor Code to our needs */

Written by Jennifer Young on December 16, 2019. Last edited January 05, 2021

==Install and Set Up on TomCat (Method 1)==
Arthor ran on n-1-136, which runs CentOS Linux release 7.7.1908 (Core). You can check the version of CentOS with the following command
cat /etc/centos-release

Check your current version of Java with the following command:
java -version

On n-1-136 we are running openjdk version "1.8.0_222", OpenJDK Runtime Environment (build 1.8.0_222-b10), and OpenJDK 64-Bit Server VM (build 25.222-b10, mixed mode)
If Java is not installed, install it using yum

==See this wiki page for more detailed information about installing Tomcat on our cluster==
http://wiki.docking.org/index.php/Tomcat_Installation

==Open port for Arthor==
In order for Arthor to be usable in the browser, the port you wish to run it on must be opened.
https://www.thegeekdiary.com/how-to-open-a-ports-in-centos-rhel-7/

===Step 1: Check Port Status===
Check that the port is not open and that Apache is not showing that port.
netstat -na | grep <port number you are checking>

lsof -i -P |grep http

===Step 2: Check Port Status in IP Tables===
iptables-save | grep <port number you are checking>

I skipped Step 3 from the guide, because there was a lot of information in the /etc/services file and I didn't want to edit it and break something.

===Step 4: Open Firewall Ports===
I did not include the zone=public section because the stand-alone servers are usually used for private instances of Arthor and SmallWorld.
Run as root.
firewall-cmd --add-port=<port number you are adding>/tcp --permanent

You need to reload the firewall after a change is made.
firewall-cmd --reload

===Step 5: Check that port is working===
To check that the port is active, run.
iptables -nL

You should see something along the lines of:
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:<port number you're adding> ctstate NEW,UNTRACKED

==How to run standalone Arthor instance==

===Step 1: Use or start a bash shell===
You can check your default shell using
echo $SHELL

If your default shell is csh, use
bash
to start a new bash shell in the current terminal window. Note that echo $SHELL will show you your default shell regardless of the current shell.

===Step 2: Set your environment variables===
export ARTHOR_DIR=/opt/nextmove/arthor/arthor-3.3-centos7
export PATH=$ARTHOR_DIR/bin/:$PATH

Make sure the ARTHOR_DIR variable is set to the directory for the latest version of Arthor or whichever version you would like to test.
The PATH environment variable is needed if you wish to use the Arthor tools from the command line

===Step 3: Run the arthor-server.jar===
java -jar /opt/nextmove/arthor/arthor-3.3-centos7/java/arthor.jar --httpPort <your httpPort>

==Setting environment variables for an Arthor Server==
Set the environment variables in the setenv.sh file. Note: Be sure to edit the file in the directory corresponding to the latest version of TomCat. As of December 2019, we are running 9.0.27 on n-1-136.

vim /opt/tomcat/apache-tomcat-9.0.27/bin/setenv.sh

Add the line below to the setenv.sh file above, or substitute the path to wherever you currently store the arthor.cfg file
export ARTHOR_CONFIG=/usr/local/tomcat/arthor.cfg

Here is an example of the arthor.cfg file:
BinDir=/opt/nextmove/arthor/arthor-3.3-centos7/bin
DataDir=/local2/arthor_local_8081/
MaxConcurrentSearches=6
MaxThreadsPerSearch=8
AutomaticIndex=false
AsyncHitCountMax=1000000
Resolver=https://sw.docking.org/util/smi2mol?smi=%s

=== Configuration Details ===

*'''BinDir''': is the location of the Arthor command line binaries. These are used to generate the Arthor index files and to perform searches directly on n-1-136. An example of this would be using atdbgrep for substructure search.

*'''DataDir''': This is the directory where the Arthor data files live. Location where the index files will be created and loaded from.

*'''MaxConcurrentSearches''': Controls the maximum number of searches that can be run concurrently by setting the database pool size. When switching between a large number of databases it can be useful to have a larger pool size, the only trade off is keeping file pointers open.

*'''MaxThreadsPerSearch''': The number of threads to use for both ATDB and ATFP searches

*Set '''AutomaticIndex''' to false if you don't want new smiles files added to the data directory to be indexed automatically

*'''AsyncHitCountMax''': The upper-bound for the number of hits to retrieve in background searches.

*'''Resolver''': Using Smallworld API, allows input box to take in a SMILE format and automatically draw on the board.

Check Arthor manual for more configuration options.

==Background==
Before working with Arthor, it is recommended that you familiarize yourself with the Arthor documentation. Some useful pages to look at include 3-5, 22-25 and 33-39. Of course, reading everything would be the best!

==Checking Disk Space Usage==
Before building arthor indexes, it's always a good thing to check what percent of the memory is being used. Try to be cautious with how much memory you have left, and make sure to check while building indexes to make sure that you have enough space. To check, run the following command:

df -h /<directory with disc>

==Building Large Databases==
At the moment, we are building databases of size 500M molecules by merging smile files. There are multiple methods of trying to create large databases, one being merging based off of the same H?? prefix and stopping once the database reaches > 500M molecules (or whatever upperbound you want to use). Here is some python code that simulates this merging process. Essentially the program takes all of the .smi files within an input directory, sorts them lexiographically, and begins merging these .smi files together in order until the size reaches > 500M molecules.

Feel free to modify it if you think a better method exists.

import subprocess
import sys
import os

from os import listdir
from os.path import isfile, join

mypath = "<Path to directory holding .smi files>"
onlyfiles = [f for f in listdir(mypath) if isfile(join(mypath, f))]
onlyfiles.sort()

create_fp = True
cur_mols = 0
lower_bound = 500000000
upper_bound = 600000000
files_to_merge = []

def merge_files(f_t_m):
arr = f_t_m[0].split(".")
arr2 = f_t_m[len(f_t_m) - 1].split(".")
file_name_merge = (arr[0] + "_" + arr2[0] + ".smi")
print ("File being created: " + file_name_merge)

for file in f_t_m:
tmp = file.split(".")
process = subprocess.Popen("cat " + join(mypath, file) + " >> " + file_name_merge, shell=True)
process.wait()

for file in onlyfiles:
arr = file.split(".")

if (arr[len(arr) - 1] == "smi"):
print("Working with " + file)
mol = sum(1 for line in open(join(mypath, file)))
print(file, mol, cur_mols)

if (cur_mols + mol > lower_bound):
if (cur_mols + mol < upper_bound):
files_to_merge.append(file)
merge_files(files_to_merge)
cur_mols = 0
files_to_merge.clear()
else:
merge_files(files_to_merge)
files_to_merge.clear()
files_to_merge.append(file)
merge_files(files_to_merge)
cur_mols = 0
files_to_merge.clear()
else:
cur_mols += mol
files_to_merge.append(file)

if (len(files_to_merge) != 0):
merge_files(files_to_merge)

==Building Arthor Indexes==
Once you've merged the .smi files together, it's time to start building the databases themselves. To do this we use the command

smi2atdb -j 0 -p <The .smi file> <The .atdb>

The flag "-j 0" enables parallel generation and utilizes all available processors to generate the .atdb file. The "-p" flag stores the offset position in the ATDB file. Since we're building indexes for the Web Application, you must use the "-p" flag when building indexes. Please note that the name of the .smi file should also be the name of the .atdb file. That way, the Web Application knows to use these files together and correctly display the required images. Refer to pages 33-34 in the Arthor documentation for more information.

If there are too many large .smi files and you do not want to manually build each .atdb file, you can use this python script which takes all of the .smi files in the current directory and converts them to .atdb files. Make sure to modify mypath to the directory containing the .smi files. You can change the variable "create_fp" to false if you don't want to create .atdb.fp files (refer to page 9 in the Arthor documentation).

import subprocess
import sys
import os

from os import listdir
from os.path import isfile, join

mypath = "<Path containing the .smi files"
onlyfiles = [f for f in listdir(mypath) if isfile(join(mypath, f))]

create_fp = True

for file in onlyfiles:
arr = file.split(".")

if (arr[len(arr) - 1] == "smi"):
process = subprocess.Popen("/nfs/ex9/work/xyz/psql/arthor-3.3-centos7/bin/smi2atdb -j 0 -p {0} {1}.atdb".format(join(mypath, file), arr[0]), shell=True)
process.wait()

print("SUCCESS! {0}.atdb file was created!".format(arr[0]))

if (create_fp):
process = subprocess.Popen("/nfs/ex9/work/xyz/psql/arthor-3.3-centos7/bin/atdb2fp -j 0 {0}.atdb".format(arr[0]), shell=True)
process.wait()

print("SUCCESS! {0}.atdb.fp file was created!".format(arr[0]))

==Uploading Indexes to the Web Application==
One can upload indexes to the Web Application by changing the "DATADIR" variable in the arthor.cfg file to the directory holding the .atdb files. This is already set up on n-1-136 and n-5-34.

==Virtual Memory==
In addition to modifying the arthor.cfg file, virtual memory can also be used to make queries faster. There can still be More information can be found in pages 10-16 in the Arthor Documentation.

==Setting up Round Table==
This is a new feature in Arthor 3.0 and is currently beta (January 2020). See section 2.4 in the manual
As explained in the manual, "Round Table allows you to serve and split chemical searches across multiple host machines. The implementation provides a lightweight proxy that forwards requests to other Arthor host servers that do the actual search. Communication is done using the existing Web APIs.

Since Arthor requires CentOS 7, as of January 2020 we have 6 servers that are capable of running Arthor with Round Table.

===Setting up Host Server===
If we want to add machines to the Round Table, for example 'nun' and 'samekh', we need to edit their arthor.cfg file so that when our Local Machine passes commands these secondary servers know to perform the search they are given.

$ cat arthor.cfg
MaxThreadsPerSearch=4
AutomaticIndex=false
DATADIR=<Directory where smiles are located>

We then run the jar server on each of these host machines containing data on any available port.

java -jar /nfs/ex9/work/xyz/psql/arthor-3.3-centos7/java/arthor.jar --httpPort <port>

For our local machine, the arthor.cfg file will look different.

$ cat arthor.cfg
[RoundTable]
RemoteClient=http://skynet:<port number where jar server is running>/
RemoteClient=http://hal:<port number where jar server is running>/

Please refer to Section 2 in the RoundTable Documentation file (pages 6-8) for more useful information on configuration.

Then run the following command on n-1-136:

java -jar /nfs/ex9/work/xyz/psql/arthor-3.3-centos7/java/arthor.jar --httpPort <port>

===Public Arthor===
{| class="wikitable"
|-
! CentOS 7 Machine
! Port
! Total Files Size
! Arthor Install Location
! Round Table Data Directory
! Active
|-
| samekh
| 10.20.0.41:8000
| 2.4TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/public_arthor/
| active
|-
| nun
| 10.20.0.40:8000
| 2.4TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/public_arthor/
| active
|-
| n-9-22
| 10.20.9.22:8000
| 2.4TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /export/db4/public_arthor/
| active
|-
|}

===Arthor Round Table Head===
{| class="wikitable"
|-
! CentOS 7 Machine
! Port
! Database
! Arthor Install Location
! Round Table Data Directory
! Active
|-
| samekh
| 10.20.0.41:8080
| Enamine_REAL_Q2-2020-All-13B
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
| nun
| 10.20.0.40:8080
| Enamine_REAL_Q2-2020-All-41B
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
|}

===Arthor Round Table Nodes===
{| class="wikitable"
|-
! CentOS 7 Machine
! Port
! Database
! Total Files Size
! Arthor Install Location
! Round Table Data Directory
!Active
|-
| samekh
| 10.20.0.41:8008
| Enamine_REAL_Q2-2020-All-13B (26 slices)
| 4.5TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
| nun
| 10.20.0.40:8008
| Enamine_REAL_Space_June_2020_S41B (aa-ae, 5 slices), Enamine_REAL_Space_June_2020_M41B (af-am, 8 slices), zinc22_2d (H04~H25, 22 slices)
| 5.6TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
| n-1-17
| 10.20.1.17:8008
| Enamine_REAL_Space_June_2020_M41B (aa-ae, 5 slices), zinc22_2d (H25~H29, 4 slices)
| 3.7TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
| n-5-32
| 10.20.5.32:8008
| Enamine_REAL_Space_June_2020_M41B (an~az, 13 slices), zinc22_2d (H30, 1 slice)
| 5.6TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
| n-5-33
| 10.20.5.33:8008
| Enamine_REAL_Space_June_2020_M41B (ba-bl, 12 slices)
| 5.3TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
|}

===Arthor Local 8081 (Datasets all local to samekh/nun)===
{| class="wikitable"
|-
! CentOS 7 Machine
! Port
! Database
! Total Files Size
! Arthor Install Location
! Round Table Data Directory
!Active
|-
| samekh
| 10.20.0.41:8081
| Enamine_REAL_Q2-2020-All-13B (26 slices)
| 4.5TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_local_8081/
| active
|-
| nun
| 10.20.0.40:8081
| Enamine_REAL_Space_June_2020_S41B (aa-ae, 5 slices), Enamine_REAL_Space_June_2020_M41B (aa-an, 14 slices)
| 4.3TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_local_8081/
| active
|-
|}

== Customizing Arthor Code to our needs ==
If Arthor Sever is launched through "java -jar /opt/nextmove/arthor/arthor-3.3.2-centos7/java/arthor.jar --httpPort=<port>", find the directory where this line of code was executed. Once found do '''ls -a''', there should be a hidden directory called .extract.
=== Change Arthor Download Size (Hardcoded) ===
vim .extract/webapps/ROOT/WEB-INF/static/js/index.js
?#arthor_sdf_link //search this
*Look for 0!==arguments[0]?arguments[0]:<number>
*Change number to desirec amount
=== Change Arthor Download Size (Options) ===
vim .extract/webapps/ROOT/WEB-INF/static/index.html
search this: “res-download”
in the div with the class=”dropdown-content”
add these link options and change the number accordingly:
<a id="arthor_tsv_link" href="#"> TSV-500</a>
<a id="arthor_tsv_link_1000" href="#"> TSV-1,000</a>
<a id="arthor_tsv_link_5000" href="#"> TSV-5,000</a>
<a id="arthor_csv_link" href="#"> CSV-500</a>
<a id="arthor_csv_link_1000" href="#"> CSV-1,000</a>
<a id="arthor_csv_link_5000" href="#"> CSV-5,000</a>
<a id="arthor_sdf_link" href="#"> SDF-500</a>
<a id="arthor_sdf_link_1000" href="#"> SDF-1,000</a>
<a id="arthor_sdf_link_5000" href="#"> SDF-5,000</a>
then vim .extract/webapps/ROOT/WEB-INF/static/js/index.js
For arthor 3.3.2 search this: “function $(){”
For arthor 3.3 search this: “function Fs()” or “#arthor_tsv_link”
Separate this function from the rest of the code from beginning and end of the function
For arthor 3.3.2, edit the function for example:
function $(){
if(document.getElementById("arthor_tsv_link")) {
var t=arguments.length>0&&void 0!==arguments[0]?
arguments[0]:500,e=i.a.param({query:arthor.query,type:arthor.type,draw:0,start:0,length:t,flags:arthor.flags}),n=arthor.url+"/dt/"+L(arthor.table)+"/search";i()
("#arthor_sdf_link").attr("href",n+".sdf?"+e),i()("#arthor_tsv_link").attr("href",n+".tsv?"+e),i()("#arthor_csv_link").attr("href",n+".csv?"+e)
}
if (document.getElementById("arthor_tsv_link_1000")) {
var t=arguments.length>0&&void 0!==arguments[0]?
arguments[0]:1000,e=i.a.param({query:arthor.query,type:arthor.type,draw:0,start:0,length:t,flags:arthor.flags}),n=arthor.url+"/dt/"+L(arthor.table)+"/search";i()
("#arthor_sdf_link_1000").attr("href",n+".sdf?"+e),i()("#arthor_tsv_link_1000").attr("href",n+".tsv?"+e),i()("#arthor_csv_link_1000").attr("href",n+".csv?"+e)
}
if (document.getElementById("arthor_tsv_link_5000")) {
var t=arguments.length>0&&void 0!==arguments[0]?
arguments[0]:5000,e=i.a.param({query:arthor.query,type:arthor.type,draw:0,start:0,length:t,flags:arthor.flags}),n=arthor.url+"/dt/"+L(arthor.table)+"/search";i()
("#arthor_sdf_link_5000").attr("href",n+".sdf?"+e),i()("#arthor_tsv_link_5000").attr("href",n+".tsv?"+e),i()("#arthor_csv_link_5000").attr("href",n+".csv?"+e)
}
}
For arthor 3.3, edit the function for example:
function Fs(){
if(document.getElementById("arthor_tsv_link")) {
var t = arguments.length>0&&void 0!==arguments[0]?
arguments[0]:500,e=i.a.param({query:arthor.query,type:arthor.type,draw:0,start:0,length:t,flags:arthor.flags}),n=arthor.url+"/dt/"+xs(arthor.table)+"/search";i()
("#arthor_sdf_link").attr("href",n+".sdf?"+e),i()("#arthor_tsv_link").attr("href",n+".tsv?"+e),i()("#arthor_csv_link").attr("href",n+".csv?"+e)
}
if (document.getElementById("arthor_tsv_link_1000")) {
var t=arguments.length>0&&void 0!==arguments[0]?
arguments[0]:1000,e=i.a.param({query:arthor.query,type:arthor.type,draw:0,start:0,length:t,flags:arthor.flags}),n=arthor.url+"/dt/"+xs(arthor.table)+"/search";i()
("#arthor_sdf_link_1000").attr("href",n+".sdf?"+e),i()("#arthor_tsv_link_1000").attr("href",n+".tsv?"+e),i()("#arthor_csv_link_1000").attr("href",n+".csv?"+e)
}
if (document.getElementById("arthor_tsv_link_5000")) {
var t=arguments.length>0&&void 0!==arguments[0]?
arguments[0]:5000,e=i.a.param({query:arthor.query,type:arthor.type,draw:0,start:0,length:t,flags:arthor.flags}),n=arthor.url+"/dt/"+xs(arthor.table)+"/search";i()
("#arthor_sdf_link_5000").attr("href",n+".sdf?"+e),i()("#arthor_tsv_link_5000").attr("href",n+".tsv?"+e),i()("#arthor_csv_link_5000").attr("href",n+".csv?"+e)
}
}
=== Take out Similarity Button ===
vim .extract/webapps/ROOT/WEB-INF/static/index.html
?Similarity //search this
*Comment out this line '''< li value="Similarity" onclick="setSearchType(this)" class="first"> Similarity </li >''' //added spaces at the beginning and end so prevent wiki from converting it
=== Hyperlink to zinc20 ===
vim .extract/webapps/ROOT/WEB-INF/static/js/index.js
?table_name //search this
*find this line "" + h + ""
*replace with '''"<a target='_blank' href='https://zinc20.docking.org/substances/"+h+"'>" + h + "</a>"''' //added spaces at the beginning and end so prevent wiki from converting it
=== Make Input Box Work ===
At the end of the Arthor config file add this:
Resolver=https://sw.docking.org/util/smi2mol?smi=%s
To copy smiles in the input box:
vim .extract/webapps/ROOT/WEB-INF/static/js/index.js
search this: “var e=t.src.smiles()”
add this after the semi-colon
document.getElementById("ar_text_input").value = e;

Arthor Documentation for Future Developer

2021-03-01T20:42:12Z

Jgutie11: /* Customizing Arthor Code to our needs */

Written by Jennifer Young on December 16, 2019. Last edited January 05, 2021

==Install and Set Up on TomCat (Method 1)==
Arthor ran on n-1-136, which runs CentOS Linux release 7.7.1908 (Core). You can check the version of CentOS with the following command
cat /etc/centos-release

Check your current version of Java with the following command:
java -version

On n-1-136 we are running openjdk version "1.8.0_222", OpenJDK Runtime Environment (build 1.8.0_222-b10), and OpenJDK 64-Bit Server VM (build 25.222-b10, mixed mode)
If Java is not installed, install it using yum

==See this wiki page for more detailed information about installing Tomcat on our cluster==
http://wiki.docking.org/index.php/Tomcat_Installation

==Open port for Arthor==
In order for Arthor to be usable in the browser, the port you wish to run it on must be opened.
https://www.thegeekdiary.com/how-to-open-a-ports-in-centos-rhel-7/

===Step 1: Check Port Status===
Check that the port is not open and that Apache is not showing that port.
netstat -na | grep <port number you are checking>

lsof -i -P |grep http

===Step 2: Check Port Status in IP Tables===
iptables-save | grep <port number you are checking>

I skipped Step 3 from the guide, because there was a lot of information in the /etc/services file and I didn't want to edit it and break something.

===Step 4: Open Firewall Ports===
I did not include the zone=public section because the stand-alone servers are usually used for private instances of Arthor and SmallWorld.
Run as root.
firewall-cmd --add-port=<port number you are adding>/tcp --permanent

You need to reload the firewall after a change is made.
firewall-cmd --reload

===Step 5: Check that port is working===
To check that the port is active, run.
iptables -nL

You should see something along the lines of:
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:<port number you're adding> ctstate NEW,UNTRACKED

==How to run standalone Arthor instance==

===Step 1: Use or start a bash shell===
You can check your default shell using
echo $SHELL

If your default shell is csh, use
bash
to start a new bash shell in the current terminal window. Note that echo $SHELL will show you your default shell regardless of the current shell.

===Step 2: Set your environment variables===
export ARTHOR_DIR=/opt/nextmove/arthor/arthor-3.3-centos7
export PATH=$ARTHOR_DIR/bin/:$PATH

Make sure the ARTHOR_DIR variable is set to the directory for the latest version of Arthor or whichever version you would like to test.
The PATH environment variable is needed if you wish to use the Arthor tools from the command line

===Step 3: Run the arthor-server.jar===
java -jar /opt/nextmove/arthor/arthor-3.3-centos7/java/arthor.jar --httpPort <your httpPort>

==Setting environment variables for an Arthor Server==
Set the environment variables in the setenv.sh file. Note: Be sure to edit the file in the directory corresponding to the latest version of TomCat. As of December 2019, we are running 9.0.27 on n-1-136.

vim /opt/tomcat/apache-tomcat-9.0.27/bin/setenv.sh

Add the line below to the setenv.sh file above, or substitute the path to wherever you currently store the arthor.cfg file
export ARTHOR_CONFIG=/usr/local/tomcat/arthor.cfg

Here is an example of the arthor.cfg file:
BinDir=/opt/nextmove/arthor/arthor-3.3-centos7/bin
DataDir=/local2/arthor_local_8081/
MaxConcurrentSearches=6
MaxThreadsPerSearch=8
AutomaticIndex=false
AsyncHitCountMax=1000000
Resolver=https://sw.docking.org/util/smi2mol?smi=%s

=== Configuration Details ===

*'''BinDir''': is the location of the Arthor command line binaries. These are used to generate the Arthor index files and to perform searches directly on n-1-136. An example of this would be using atdbgrep for substructure search.

*'''DataDir''': This is the directory where the Arthor data files live. Location where the index files will be created and loaded from.

*'''MaxConcurrentSearches''': Controls the maximum number of searches that can be run concurrently by setting the database pool size. When switching between a large number of databases it can be useful to have a larger pool size, the only trade off is keeping file pointers open.

*'''MaxThreadsPerSearch''': The number of threads to use for both ATDB and ATFP searches

*Set '''AutomaticIndex''' to false if you don't want new smiles files added to the data directory to be indexed automatically

*'''AsyncHitCountMax''': The upper-bound for the number of hits to retrieve in background searches.

*'''Resolver''': Using Smallworld API, allows input box to take in a SMILE format and automatically draw on the board.

Check Arthor manual for more configuration options.

==Background==
Before working with Arthor, it is recommended that you familiarize yourself with the Arthor documentation. Some useful pages to look at include 3-5, 22-25 and 33-39. Of course, reading everything would be the best!

==Checking Disk Space Usage==
Before building arthor indexes, it's always a good thing to check what percent of the memory is being used. Try to be cautious with how much memory you have left, and make sure to check while building indexes to make sure that you have enough space. To check, run the following command:

df -h /<directory with disc>

==Building Large Databases==
At the moment, we are building databases of size 500M molecules by merging smile files. There are multiple methods of trying to create large databases, one being merging based off of the same H?? prefix and stopping once the database reaches > 500M molecules (or whatever upperbound you want to use). Here is some python code that simulates this merging process. Essentially the program takes all of the .smi files within an input directory, sorts them lexiographically, and begins merging these .smi files together in order until the size reaches > 500M molecules.

Feel free to modify it if you think a better method exists.

import subprocess
import sys
import os

from os import listdir
from os.path import isfile, join

mypath = "<Path to directory holding .smi files>"
onlyfiles = [f for f in listdir(mypath) if isfile(join(mypath, f))]
onlyfiles.sort()

create_fp = True
cur_mols = 0
lower_bound = 500000000
upper_bound = 600000000
files_to_merge = []

def merge_files(f_t_m):
arr = f_t_m[0].split(".")
arr2 = f_t_m[len(f_t_m) - 1].split(".")
file_name_merge = (arr[0] + "_" + arr2[0] + ".smi")
print ("File being created: " + file_name_merge)

for file in f_t_m:
tmp = file.split(".")
process = subprocess.Popen("cat " + join(mypath, file) + " >> " + file_name_merge, shell=True)
process.wait()

for file in onlyfiles:
arr = file.split(".")

if (arr[len(arr) - 1] == "smi"):
print("Working with " + file)
mol = sum(1 for line in open(join(mypath, file)))
print(file, mol, cur_mols)

if (cur_mols + mol > lower_bound):
if (cur_mols + mol < upper_bound):
files_to_merge.append(file)
merge_files(files_to_merge)
cur_mols = 0
files_to_merge.clear()
else:
merge_files(files_to_merge)
files_to_merge.clear()
files_to_merge.append(file)
merge_files(files_to_merge)
cur_mols = 0
files_to_merge.clear()
else:
cur_mols += mol
files_to_merge.append(file)

if (len(files_to_merge) != 0):
merge_files(files_to_merge)

==Building Arthor Indexes==
Once you've merged the .smi files together, it's time to start building the databases themselves. To do this we use the command

smi2atdb -j 0 -p <The .smi file> <The .atdb>

The flag "-j 0" enables parallel generation and utilizes all available processors to generate the .atdb file. The "-p" flag stores the offset position in the ATDB file. Since we're building indexes for the Web Application, you must use the "-p" flag when building indexes. Please note that the name of the .smi file should also be the name of the .atdb file. That way, the Web Application knows to use these files together and correctly display the required images. Refer to pages 33-34 in the Arthor documentation for more information.

If there are too many large .smi files and you do not want to manually build each .atdb file, you can use this python script which takes all of the .smi files in the current directory and converts them to .atdb files. Make sure to modify mypath to the directory containing the .smi files. You can change the variable "create_fp" to false if you don't want to create .atdb.fp files (refer to page 9 in the Arthor documentation).

import subprocess
import sys
import os

from os import listdir
from os.path import isfile, join

mypath = "<Path containing the .smi files"
onlyfiles = [f for f in listdir(mypath) if isfile(join(mypath, f))]

create_fp = True

for file in onlyfiles:
arr = file.split(".")

if (arr[len(arr) - 1] == "smi"):
process = subprocess.Popen("/nfs/ex9/work/xyz/psql/arthor-3.3-centos7/bin/smi2atdb -j 0 -p {0} {1}.atdb".format(join(mypath, file), arr[0]), shell=True)
process.wait()

print("SUCCESS! {0}.atdb file was created!".format(arr[0]))

if (create_fp):
process = subprocess.Popen("/nfs/ex9/work/xyz/psql/arthor-3.3-centos7/bin/atdb2fp -j 0 {0}.atdb".format(arr[0]), shell=True)
process.wait()

print("SUCCESS! {0}.atdb.fp file was created!".format(arr[0]))

==Uploading Indexes to the Web Application==
One can upload indexes to the Web Application by changing the "DATADIR" variable in the arthor.cfg file to the directory holding the .atdb files. This is already set up on n-1-136 and n-5-34.

==Virtual Memory==
In addition to modifying the arthor.cfg file, virtual memory can also be used to make queries faster. There can still be More information can be found in pages 10-16 in the Arthor Documentation.

==Setting up Round Table==
This is a new feature in Arthor 3.0 and is currently beta (January 2020). See section 2.4 in the manual
As explained in the manual, "Round Table allows you to serve and split chemical searches across multiple host machines. The implementation provides a lightweight proxy that forwards requests to other Arthor host servers that do the actual search. Communication is done using the existing Web APIs.

Since Arthor requires CentOS 7, as of January 2020 we have 6 servers that are capable of running Arthor with Round Table.

===Setting up Host Server===
If we want to add machines to the Round Table, for example 'nun' and 'samekh', we need to edit their arthor.cfg file so that when our Local Machine passes commands these secondary servers know to perform the search they are given.

$ cat arthor.cfg
MaxThreadsPerSearch=4
AutomaticIndex=false
DATADIR=<Directory where smiles are located>

We then run the jar server on each of these host machines containing data on any available port.

java -jar /nfs/ex9/work/xyz/psql/arthor-3.3-centos7/java/arthor.jar --httpPort <port>

For our local machine, the arthor.cfg file will look different.

$ cat arthor.cfg
[RoundTable]
RemoteClient=http://skynet:<port number where jar server is running>/
RemoteClient=http://hal:<port number where jar server is running>/

Please refer to Section 2 in the RoundTable Documentation file (pages 6-8) for more useful information on configuration.

Then run the following command on n-1-136:

java -jar /nfs/ex9/work/xyz/psql/arthor-3.3-centos7/java/arthor.jar --httpPort <port>

===Public Arthor===
{| class="wikitable"
|-
! CentOS 7 Machine
! Port
! Total Files Size
! Arthor Install Location
! Round Table Data Directory
! Active
|-
| samekh
| 10.20.0.41:8000
| 2.4TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/public_arthor/
| active
|-
| nun
| 10.20.0.40:8000
| 2.4TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/public_arthor/
| active
|-
| n-9-22
| 10.20.9.22:8000
| 2.4TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /export/db4/public_arthor/
| active
|-
|}

===Arthor Round Table Head===
{| class="wikitable"
|-
! CentOS 7 Machine
! Port
! Database
! Arthor Install Location
! Round Table Data Directory
! Active
|-
| samekh
| 10.20.0.41:8080
| Enamine_REAL_Q2-2020-All-13B
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
| nun
| 10.20.0.40:8080
| Enamine_REAL_Q2-2020-All-41B
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
|}

===Arthor Round Table Nodes===
{| class="wikitable"
|-
! CentOS 7 Machine
! Port
! Database
! Total Files Size
! Arthor Install Location
! Round Table Data Directory
!Active
|-
| samekh
| 10.20.0.41:8008
| Enamine_REAL_Q2-2020-All-13B (26 slices)
| 4.5TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
| nun
| 10.20.0.40:8008
| Enamine_REAL_Space_June_2020_S41B (aa-ae, 5 slices), Enamine_REAL_Space_June_2020_M41B (af-am, 8 slices), zinc22_2d (H04~H25, 22 slices)
| 5.6TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
| n-1-17
| 10.20.1.17:8008
| Enamine_REAL_Space_June_2020_M41B (aa-ae, 5 slices), zinc22_2d (H25~H29, 4 slices)
| 3.7TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
| n-5-32
| 10.20.5.32:8008
| Enamine_REAL_Space_June_2020_M41B (an~az, 13 slices), zinc22_2d (H30, 1 slice)
| 5.6TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
| n-5-33
| 10.20.5.33:8008
| Enamine_REAL_Space_June_2020_M41B (ba-bl, 12 slices)
| 5.3TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
|}

===Arthor Local 8081 (Datasets all local to samekh/nun)===
{| class="wikitable"
|-
! CentOS 7 Machine
! Port
! Database
! Total Files Size
! Arthor Install Location
! Round Table Data Directory
!Active
|-
| samekh
| 10.20.0.41:8081
| Enamine_REAL_Q2-2020-All-13B (26 slices)
| 4.5TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_local_8081/
| active
|-
| nun
| 10.20.0.40:8081
| Enamine_REAL_Space_June_2020_S41B (aa-ae, 5 slices), Enamine_REAL_Space_June_2020_M41B (aa-an, 14 slices)
| 4.3TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_local_8081/
| active
|-
|}

== Customizing Arthor Code to our needs ==
If Arthor Sever is launched through "java -jar /opt/nextmove/arthor/arthor-3.3.2-centos7/java/arthor.jar --httpPort=<port>", find the directory where this line of code was executed. Once found do '''ls -a''', there should be a hidden directory called .extract.
=== Change Arthor Download Size (Hardcoded) ===
vim .extract/webapps/ROOT/WEB-INF/static/js/index.js
?#arthor_sdf_link //search this
*Look for 0!==arguments[0]?arguments[0]:<number>
*Change number to desirec amount
=== Change Arthor Download Size (Options) ===
vim .extract/webapps/ROOT/WEB-INF/static/index.html
search this: “res-download”
in the div with the class=”dropdown-content”
add these link options and change the number accordingly:
<a id="arthor_tsv_link" href="#"> TSV-500</a>
<a id="arthor_tsv_link_1000" href="#"> TSV-1,000</a>
<a id="arthor_tsv_link_5000" href="#"> TSV-5,000</a>
<a id="arthor_csv_link" href="#"> CSV-500</a>
<a id="arthor_csv_link_1000" href="#"> CSV-1,000</a>
<a id="arthor_csv_link_5000" href="#"> CSV-5,000</a>
<a id="arthor_sdf_link" href="#"> SDF-500</a>
<a id="arthor_sdf_link_1000" href="#"> SDF-1,000</a>
<a id="arthor_sdf_link_5000" href="#"> SDF-5,000</a>
then vim .extract/webapps/ROOT/WEB-INF/static/js/index.js
For arthor 3.3.2 search this: “function $(){”
For arthor 3.3 search this: “function Fs()” or “#arthor_tsv_link”
Separate this function from the rest of the code from beginning and end of the function
For arthor 3.3.2, edit the function for example:
function $(){
if(document.getElementById("arthor_tsv_link")) {
var t=arguments.length>0&&void 0!==arguments[0]?
arguments[0]:500,e=i.a.param({query:arthor.query,type:arthor.type,draw:0,start:0,length:t,flags:arthor.flags}),n=arthor.url+"/dt/"+L(arthor.table)+"/search";i()
("#arthor_sdf_link").attr("href",n+".sdf?"+e),i()("#arthor_tsv_link").attr("href",n+".tsv?"+e),i()("#arthor_csv_link").attr("href",n+".csv?"+e)
}
if (document.getElementById("arthor_tsv_link_1000")) {
var t=arguments.length>0&&void 0!==arguments[0]?
arguments[0]:1000,e=i.a.param({query:arthor.query,type:arthor.type,draw:0,start:0,length:t,flags:arthor.flags}),n=arthor.url+"/dt/"+L(arthor.table)+"/search";i()
("#arthor_sdf_link_1000").attr("href",n+".sdf?"+e),i()("#arthor_tsv_link_1000").attr("href",n+".tsv?"+e),i()("#arthor_csv_link_1000").attr("href",n+".csv?"+e)
}
if (document.getElementById("arthor_tsv_link_5000")) {
var t=arguments.length>0&&void 0!==arguments[0]?
arguments[0]:5000,e=i.a.param({query:arthor.query,type:arthor.type,draw:0,start:0,length:t,flags:arthor.flags}),n=arthor.url+"/dt/"+L(arthor.table)+"/search";i()
("#arthor_sdf_link_5000").attr("href",n+".sdf?"+e),i()("#arthor_tsv_link_5000").attr("href",n+".tsv?"+e),i()("#arthor_csv_link_5000").attr("href",n+".csv?"+e)
}
}
For arthor 3.3, edit the function for example:
function Fs(){
if(document.getElementById("arthor_tsv_link")) {
var t = arguments.length>0&&void 0!==arguments[0]?
arguments[0]:500,e=i.a.param({query:arthor.query,type:arthor.type,draw:0,start:0,length:t,flags:arthor.flags}),n=arthor.url+"/dt/"+xs(arthor.table)+"/search";i()
("#arthor_sdf_link").attr("href",n+".sdf?"+e),i()("#arthor_tsv_link").attr("href",n+".tsv?"+e),i()("#arthor_csv_link").attr("href",n+".csv?"+e)
}
if (document.getElementById("arthor_tsv_link_1000")) {
var t=arguments.length>0&&void 0!==arguments[0]?
arguments[0]:1000,e=i.a.param({query:arthor.query,type:arthor.type,draw:0,start:0,length:t,flags:arthor.flags}),n=arthor.url+"/dt/"+xs(arthor.table)+"/search";i()
("#arthor_sdf_link_1000").attr("href",n+".sdf?"+e),i()("#arthor_tsv_link_1000").attr("href",n+".tsv?"+e),i()("#arthor_csv_link_1000").attr("href",n+".csv?"+e)
}
if (document.getElementById("arthor_tsv_link_5000")) {
var t=arguments.length>0&&void 0!==arguments[0]?
arguments[0]:5000,e=i.a.param({query:arthor.query,type:arthor.type,draw:0,start:0,length:t,flags:arthor.flags}),n=arthor.url+"/dt/"+xs(arthor.table)+"/search";i()
("#arthor_sdf_link_5000").attr("href",n+".sdf?"+e),i()("#arthor_tsv_link_5000").attr("href",n+".tsv?"+e),i()("#arthor_csv_link_5000").attr("href",n+".csv?"+e)
}
}
=== Take out Similarity Button ===
vim .extract/webapps/ROOT/WEB-INF/static/index.html
?Similarity //search this
*Comment out this line '''< li value="Similarity" onclick="setSearchType(this)" class="first"> Similarity </li >''' //added spaces at the beginning and end so prevent wiki from converting it
=== Hyperlink to zinc20 ===
vim .extract/webapps/ROOT/WEB-INF/static/js/index.js
?table_name //search this
*find this line "" + h + ""
*replace with '''"<a target='_blank' href='https://zinc20.docking.org/substances/"+h+"'>" + h + "</a>"''' //added spaces at the beginning and end so prevent wiki from converting it
=== Make Input Box Work ===
At the end of the Arthor config file add this:
Resolver=https://sw.docking.org/util/smi2mol?smi=%s

List of Docking.org websites

2021-02-03T00:43:36Z

Jgutie11:

==List==
{| class="wikitable"
|-
! scope="col"| Website
! scope="col"| Server
! scope="col"| Port
|-
! scope="row"| Advisor
| Unknown
| Unknown
|-
! scope="row"| BKSLab
| gimel
| 5002
|-
! scope="row"| DOCKBlaster
| Unknown
| Unknow
|-
! scope="row"| DUDE
| Unknown
| Unknown
|-
! scope="row"| Excipient
| gimel2
| 8093
|-
! scope="row"| IrwinLab
| gimel
| 5004
|-
! scope="row"| Tools18(tldr)
| gimel2
| 5000
|-
! scope="row"| Transportal
| Omega
| 8123
|}
{ Sysadmin }
[[Category:Delete]]

Replacing failed disk on Server

2021-01-26T01:18:21Z

Jgutie11: /* Auto-check Disk Machines Python Script */

== How to check if Disk failed==
===Check for the light on disk===

Solid Yellow => Fail

Blinking Yellow => Predictive Failure (going to fail soon)

Green => Normal
=== Replace disk instruction===
* Determine what machine the disk below to
* Press the red button on the disk to turn it off.
* Gently pull a little bit out (NOT all the way) and wait for 10 sec until it stops spinning before pulling all the way out.
* Find replacement with a similar disk with the same specs
* Carefully unscrew the disk from disk holder (if the disk holder part on the replacement is the same then you don't have to).

=== Auto-check Disk Machines Python Script ===
In gimel5, there is a python script that runs every day at 12am through crontab under s_jjg.

The file is located at: '''/nfs/home/jjg/python_scripts/check_for_failed_disks.py'''

This script ssh-es into the machines below and runs a command to list the status of disks. (Does not include cluster 0 machines)
machines: abacus, n-9-22, tsadi, lamed, qof, zayin, n-1-30, n-1-109, n-1-113, shin
data pools: db2, db3, db5, db4, ex1, ex2, ex3, ex4, ex5, ex6, ex7, ex8, ex9, exa, exb, exc, exd, db

If a disk in any of the listed machines report that a disk has failed, the script will email the sysadmins.

Example output:
pool: db2
state: ONLINE
pool: db3
state: ONLINE
pool: db5
state: ONLINE
pool: db4
state: ONLINE
pool: ex1
state: ONLINE
pool: ex2
state: ONLINE
pool: ex3
state: ONLINE
pool: ex4
state: ONLINE
pool: ex5
state: ONLINE
pool: ex6
state: ONLINE
pool: ex7
state: ONLINE
pool: ex8
state: ONLINE
pool: ex9
state: ONLINE
pool: exa
state: ONLINE
pool: exb
state: ONLINE
pool: exc
state: ONLINE
pool: exd
state: ONLINE
----------------------------------------------------------------------------
pool: db2
EID:Slt DID State DG Size Intf Med SED PI SeSz Model Sp Type
8:0 35 Onln 0 3.637 TB SAS HDD N N 512B ST4000NM0023 U -
8:1 10 Onln 0 3.637 TB SAS HDD N N 512B ST4000NM0023 U -
8:2 18 Onln 0 3.637 TB SAS HDD N N 512B ST4000NM0023 U -
8:3 12 Onln 0 3.637 TB SAS HDD N N 512B ST4000NM0023 U -
8:4 16 Onln 0 3.637 TB SAS HDD N N 512B ST4000NM0023 U -
8:5 11 Onln 0 3.637 TB SAS HDD N N 512B ST4000NM0023 U -
8:6 32 Onln 0 3.637 TB SAS HDD N N 512B ST4000NM0023 U -
8:7 13 Onln 0 3.637 TB SAS HDD N N 512B ST4000NM0023 U -
8:8 41 Onln 0 3.637 TB SAS HDD N N 512B ST4000NM0023 U -
8:9 33 Onln 0 3.637 TB SAS HDD N N 512B WD4001FYYG-01SL3 U -
8:10 20 Onln 0 3.637 TB SAS HDD N N 512B ST4000NM0023 U -
8:11 27 Onln 0 3.637 TB SAS HDD N N 512B ST4000NM0023 U -
8:12 23 Onln 0 3.637 TB SAS HDD N N 512B ST4000NM0023 U -
8:13 25 Onln 0 3.637 TB SAS HDD N N 512B ST4000NM0023 U -
8:14 14 Onln 0 3.637 TB SAS HDD N N 512B ST4000NM0023 U -
8:15 42 Onln 0 3.637 TB SAS HDD N N 512B ST4000NM0023 U -
8:16 19 Onln 0 3.637 TB SAS HDD N N 512B ST4000NM0023 U -
8:17 39 Onln 0 3.637 TB SAS HDD N N 512B ST4000NM0023 U -
8:18 40 Onln 0 3.637 TB SAS HDD N N 512B MB4000JEFNC U -
8:19 29 Onln 0 3.637 TB SAS HDD N N 512B ST4000NM0023 U -
8:20 26 Onln 0 3.637 TB SAS HDD N N 512B ST4000NM0023 U -
8:21 36 Onln 0 3.637 TB SAS HDD N N 512B ST4000NM0023 U -
8:22 34 Onln 0 3.637 TB SAS HDD N N 512B ST4000NM0023 U -

== How to check if disk is failed or install correctly==
=== On Cluster 0 's machines ===
1. Log into gimel as root
$ ssh root@sgehead1.bkslab.org
2. Log in as root to the machine that you determined from earlier
$ ssh root@<machine_name>
Example: RAID 3,6,7 belongs to nfshead2
3. Run this command
$ /opt/compaq/hpacucli/bld/hpacucli ctrl all show config

Output Example:
Smart Array P800 in Slot 1 (sn: PAFGF0N9SXQ0MX)
array A (SATA, Unused Space: 0 MB)
logicaldrive 1 (5.5 TB, RAID 1+0, OK)
physicaldrive 1E:1:1 (port 1E:box 1:bay 1, SATA, 1 TB, OK)
physicaldrive 1E:1:2 (port 1E:box 1:bay 2, SATA, 1 TB, OK)
physicaldrive 1E:1:3 (port 1E:box 1:bay 3, SATA, 1 TB, OK)
physicaldrive 1E:1:4 (port 1E:box 1:bay 4, SATA, 1 TB, OK)
physicaldrive 1E:1:5 (port 1E:box 1:bay 5, SATA, 1 TB, OK)
physicaldrive 1E:1:6 (port 1E:box 1:bay 6, SATA, 1 TB, OK)
physicaldrive 1E:1:7 (port 1E:box 1:bay 7, SATA, 1 TB, OK)
physicaldrive 1E:1:8 (port 1E:box 1:bay 8, SATA, 1 TB, OK)
physicaldrive 1E:1:9 (port 1E:box 1:bay 9, SATA, 1 TB, OK)
physicaldrive 1E:1:10 (port 1E:box 1:bay 10, SATA, 1 TB, OK)
physicaldrive 1E:1:11 (port 1E:box 1:bay 11, SATA, 1 TB, OK)
physicaldrive 1E:1:12 (port 1E:box 1:bay 12, SATA, 1 TB, OK)
array B (SATA, Unused Space: 0 MB)
logicaldrive 2 (5.5 TB, RAID 1+0, OK)
physicaldrive 2E:1:1 (port 2E:box 1:bay 1, SATA, 1 TB, OK)
physicaldrive 2E:1:2 (port 2E:box 1:bay 2, SATA, 1 TB, Predictive Failure)
physicaldrive 2E:1:3 (port 2E:box 1:bay 3, SATA, 1 TB, OK)
physicaldrive 2E:1:4 (port 2E:box 1:bay 4, SATA, 1 TB, OK)
physicaldrive 2E:1:5 (port 2E:box 1:bay 5, SATA, 1 TB, OK)
physicaldrive 2E:1:6 (port 2E:box 1:bay 6, SATA, 1 TB, OK)
physicaldrive 2E:1:7 (port 2E:box 1:bay 7, SATA, 1 TB, OK)
physicaldrive 2E:1:8 (port 2E:box 1:bay 8, SATA, 1 TB, OK)
physicaldrive 2E:1:9 (port 2E:box 1:bay 9, SATA, 1 TB, OK)
physicaldrive 2E:1:10 (port 2E:box 1:bay 10, SATA, 1 TB, OK)
physicaldrive 2E:1:11 (port 2E:box 1:bay 11, SATA, 1 TB, OK)
physicaldrive 2E:1:12 (port 2E:box 1:bay 12, SATA, 1 TB, OK)
array C (SATA, Unused Space: 0 MB)
logicaldrive 3 (5.5 TB, RAID 1+0, Ready for Rebuild)
physicaldrive 2E:2:1 (port 2E:box 2:bay 1, SATA, 1 TB, OK)
physicaldrive 2E:2:2 (port 2E:box 2:bay 2, SATA, 1 TB, OK)
physicaldrive 2E:2:3 (port 2E:box 2:bay 3, SATA, 1 TB, OK)
physicaldrive 2E:2:4 (port 2E:box 2:bay 4, SATA, 1 TB, OK)
physicaldrive 2E:2:5 (port 2E:box 2:bay 5, SATA, 1 TB, OK)
physicaldrive 2E:2:6 (port 2E:box 2:bay 6, SATA, 1 TB, OK)
physicaldrive 2E:2:7 (port 2E:box 2:bay 7, SATA, 1 TB, OK)
physicaldrive 2E:2:8 (port 2E:box 2:bay 8, SATA, 1 TB, OK)
physicaldrive 2E:2:9 (port 2E:box 2:bay 9, SATA, 1 TB, OK)
physicaldrive 2E:2:10 (port 2E:box 2:bay 10, SATA, 1 TB, OK)
physicaldrive 2E:2:11 (port 2E:box 2:bay 11, SATA, 1 TB, OK)
physicaldrive 2E:2:12 (port 2E:box 2:bay 12, SATA, 1 TB, OK)
Expander 243 (WWID: 50014380031A4B00, Port: 1E, Box: 1)
Expander 245 (WWID: 5001438005396E00, Port: 2E, Box: 2)
Expander 246 (WWID: 500143800460A600, Port: 2E, Box: 1)
Expander 248 (WWID: 50014380055E913F)
Enclosure SEP (Vendor ID HP, Model MSA60) 241 (WWID: 50014380031A4B25, Port: 1E, Box: 1)
Enclosure SEP (Vendor ID HP, Model MSA60) 242 (WWID: 5001438005396E25, Port: 2E, Box: 2)
Enclosure SEP (Vendor ID HP, Model MSA60) 244 (WWID: 500143800460A625, Port: 2E, Box: 1)
SEP (Vendor ID HP, Model P800) 247 (WWID: 50014380055E913E)

=== On '''shin'''===
As root
/opt/MegaRAID/storcli/storcli64 /c0 /eall /sall show all
<pre>Drive /c0/e8/s18 :
================

-----------------------------------------------------------------------------
EID:Slt DID State DG Size Intf Med SED PI SeSz Model Sp Type
-----------------------------------------------------------------------------
8:18 24 Failed 0 3.637 TB SAS HDD N N 512B ST4000NM0023 U -
-----------------------------------------------------------------------------

EID-Enclosure Device ID|Slt-Slot No.|DID-Device ID|DG-DriveGroup
DHS-Dedicated Hot Spare|UGood-Unconfigured Good|GHS-Global Hotspare
UBad-Unconfigured Bad|Onln-Online|Offln-Offline|Intf-Interface
Med-Media Type|SED-Self Encryptive Drive|PI-Protection Info
SeSz-Sector Size|Sp-Spun|U-Up|D-Down|T-Transition|F-Foreign
UGUnsp-Unsupported|UGShld-UnConfigured shielded|HSPShld-Hotspare shielded
CFShld-Configured shielded|Cpybck-CopyBack|CBShld-Copyback Shielded

Drive /c0/e8/s18 - Detailed Information :
=======================================

Drive /c0/e8/s18 State :
======================
Shield Counter = 0
Media Error Count = 0
Other Error Count = 16
BBM Error Count = 0
Drive Temperature = 32C (89.60 F)
Predictive Failure Count = 0
S.M.A.R.T alert flagged by drive = No

Drive /c0/e8/s18 Device attributes :
==================================
SN = Z1Z2S2TL0000C4216E9V
Manufacturer Id = SEAGATE
Model Number = ST4000NM0023
NAND Vendor = NA
WWN = 5000C50057DB2A28
Firmware Revision = 0003
Firmware Release Number = 03290003
Raw size = 3.638 TB [0x1d1c0beb0 Sectors]
Coerced size = 3.637 TB [0x1d1b00000 Sectors]
Non Coerced size = 3.637 TB [0x1d1b0beb0 Sectors]
Device Speed = 6.0Gb/s
Link Speed = 6.0Gb/s
Write cache = N/A
Logical Sector Size = 512B
Physical Sector Size = 512B
Connector Name = Port 0 - 3 & Port 4 - 7 </pre>

=== On ZFS machines ===
$ zpool status
For instruction on how to identify and replace failed disk on ZFS system. [http://wiki.docking.org/index.php/Zfs#Example:_Fixing_degraded_pool.2C_replacing_faulted_disk '''Read here''']

=== On Any Raid1 Configurations ===
Steps to fix a hard drive failure that is in a raid 1 configuration:

The following demonstrates what a failed disk looks like:

[root@myServer ~]# cat /proc/mdstat 
Personalities : [raid1] 
md0 : active raid1 sdb1[0] sda1[2](F) 
128384 blocks [2/1] [U_] 
md1 : active raid1 sdb2[0] sda2[2](F) 
16779776 blocks [2/1] [U_] 
md2 : active raid1 sdb3[0] sda3[2](F) 
139379840 blocks [2/1] [U_] 
unused devices: <none> 

[root@myServer ~]# smartctl -a /dev/sda 
smartctl 5.42 2011-10-20 r3458 [x86_64-linux-2.6.18-371.1.2.el5] (local build) 
Copyright (C) 2002-11 by Bruce Allen, http://smartmontools.sourceforge.net 
Short INQUIRY response, skip product id 
A mandatory SMART command failed: exiting. To continue, add one or more '-T permissive' options. 

[root@myServer ~]# smartctl -a /dev/sdb 
smartctl 5.42 2011-10-20 r3458 [x86_64-linux-2.6.18-371.1.2.el5] (local build) 
Copyright (C) 2002-11 by Bruce Allen, http://smartmontools.sourceforge.net 
=== START OF INFORMATION SECTION === 
Model Family: Seagate Barracuda 7200.10 
Device Model: ST3160815AS 
Serial Number: 9RA6DZP8 
Firmware Version: 4.AAB 
User Capacity: 160,041,885,696 bytes [160 GB] 
Sector Size: 512 bytes logical/physical 
Device is: In smartctl database [for details use: -P show] 
ATA Version is: 7 
ATA Standard is: Exact ATA specification draft version not indicated 
Local Time is: Mon Sep 8 15:50:48 2014 PDT 
SMART support is: Available - device has SMART capability. 
SMART support is: Enabled 
=== START OF READ SMART DATA SECTION === 
SMART overall-health self-assessment test result: PASSED 

There is a lot more that gets printed, but I cut it out. 

So /dev/sda has clearly failed. 

Take note of the GOOD disk serial number so I leave that one in when I replace it: 
Serial Number: 9RA6DZP8 

Mark and remove failed disk from raid: 

[root@myServer ~]# mdadm --manage /dev/md0 --fail /dev/sda1 
mdadm: set /dev/sda1 faulty in /dev/md0 
[root@myServer ~]# mdadm --manage /dev/md1 --fail /dev/sda2 
mdadm: set /dev/sda2 faulty in /dev/md1 
[root@myServer ~]# mdadm --manage /dev/md2 --fail /dev/sda3 
mdadm: set /dev/sda3 faulty in /dev/md2 
[root@myServer ~]# mdadm --manage /dev/md0 --remove /dev/sda1 
mdadm: hot removed /dev/sda1 
[root@myServer ~]# mdadm --manage /dev/md1 --remove /dev/sda2 
mdadm: hot removed /dev/sda2 
[root@myServer ~]# mdadm --manage /dev/md2 --remove /dev/sda3 
mdadm: hot removed /dev/sda3 

Make sure grub is installed on the good disk and that grub.conf is updated:

[root@myServer ~]# grub-install /dev/sdb 
Installation finished. No error reported. 
This is the contents of the device map /boot/grub/device.map. 
Check if this is correct or not. 
If any of the lines is incorrect, fix it and re-run the script `grub-install'. 
This device map was generated by anaconda 
(hd0) /dev/sda 
(hd1) /dev/sdb 

Take note of the which hd partition corresponds with the good disk, ie hd1 in this case. 

[root@myServer ~]# vim /boot/grub/menu.lst 
Add fallback=1 right after default=0 
Go to the bottom section where you should find some kernel stanzas. 
Copy the first of them and paste the stanza before the first existing stanza; replace root (hd0,0) with root (hd1,0) 
Should look like this: 
[...] 
title CentOS (2.6.18-128.el5) 
root (hd1,0) 
kernel /vmlinuz-2.6.18-128.el5 ro root=/dev/VolGroup00/LogVol00 
initrd /initrd-2.6.18-128.el5.img 
title CentOS (2.6.18-128.el5) 
root (hd0,0) 
kernel /vmlinuz-2.6.18-128.el5 ro root=/dev/ 
initrd /initrd-2.6.18-128.el5.img 

Save and quit 

[root@myServer ~]# mv /boot/initramfs-$(uname -r).img /boot/initramfs-$(uname -r).img.bak 
[root@myServer ~]# mkinitrd /boot/initramfs-$(uname -r).img $(uname -r) 
[root@myServer ~]# init 0 

Swap the bad drive with the new drive and boot the machine. 

Once it's booted: 

Check the device names with cat /proc/mdstat and/or fisk -l. 
The newly installed drive on myServer was named /dev/sda. 

[root@myServer ~]# modeprobe raid1 
[root@myServer ~]# modeprobe linear 

Copy the partitions from one disk to the other:

[root@myServer ~]# sfdisk -d /dev/sdb | sfdisk --force /dev/sda 
[root@myServer ~]# sfdisk -l => sanity check 

Add the new disk to the raid array:

[root@myServer ~]# mdadm --manage /dev/md0 --add /dev/sda1 
mdadm: added /dev/sda1 
[root@myServer ~]# mdadm --manage /dev/md1 --add /dev/sda2 
mdadm: added /dev/sda2 
[root@myServer ~]# mdadm --manage /dev/md2 --add /dev/sda3 
mdadm: added /dev/sda3 

Sanity check:
[root@myServer ~]# cat /proc/mdstat 
Personalities : [raid1] [linear] 
md0 : active raid1 sda1[1] sdb1[0] 
128384 blocks [2/2] [UU] 
md1 : active raid1 sda2[2] sdb2[0] 
16779776 blocks [2/1] [U_] 
[>....................] recovery = 3.2% (548864/16779776) finish=8.8min speed=30492K/sec 
md2 : active raid1 sda3[2] sdb3[0] 
139379840 blocks [2/1] [U_] 
resync=DELAYED 
unused devices: <none> 

That's it! :) 
[[ Category: Ben ]]
[[ Category : Sysadmin ]]
[[Category:Tutorials]]

Replacing failed disk on Server

2021-01-26T00:46:08Z

Jgutie11:

== How to check if Disk failed==
===Check for the light on disk===

Solid Yellow => Fail

Blinking Yellow => Predictive Failure (going to fail soon)

Green => Normal
=== Replace disk instruction===
* Determine what machine the disk below to
* Press the red button on the disk to turn it off.
* Gently pull a little bit out (NOT all the way) and wait for 10 sec until it stops spinning before pulling all the way out.
* Find replacement with a similar disk with the same specs
* Carefully unscrew the disk from disk holder (if the disk holder part on the replacement is the same then you don't have to).

=== Auto-check Disk Machines Python Script ===
In gimel5, there is a python script that runs every day at 12am through crontab under s_jjg.

The file is located at: '''/nfs/home/jjg/python_scripts/check_for_failed_disks.py'''

This script ssh-es into the machines below and runs a command to list the status of database. (Does not include cluster 0 machines)
abacus
db2
db3
db5
n-9-22
db4
tsadi
ex1
ex2
ex3
ex4
lamed
ex5
ex6
ex7
ex8
qof
ex9
zayin
exa
n-1-30
exb
n-1-109
exc
n-1-113
exd
shin
db

== How to check if disk is failed or install correctly==
=== On Cluster 0 's machines ===
1. Log into gimel as root
$ ssh root@sgehead1.bkslab.org
2. Log in as root to the machine that you determined from earlier
$ ssh root@<machine_name>
Example: RAID 3,6,7 belongs to nfshead2
3. Run this command
$ /opt/compaq/hpacucli/bld/hpacucli ctrl all show config

Output Example:
Smart Array P800 in Slot 1 (sn: PAFGF0N9SXQ0MX)
array A (SATA, Unused Space: 0 MB)
logicaldrive 1 (5.5 TB, RAID 1+0, OK)
physicaldrive 1E:1:1 (port 1E:box 1:bay 1, SATA, 1 TB, OK)
physicaldrive 1E:1:2 (port 1E:box 1:bay 2, SATA, 1 TB, OK)
physicaldrive 1E:1:3 (port 1E:box 1:bay 3, SATA, 1 TB, OK)
physicaldrive 1E:1:4 (port 1E:box 1:bay 4, SATA, 1 TB, OK)
physicaldrive 1E:1:5 (port 1E:box 1:bay 5, SATA, 1 TB, OK)
physicaldrive 1E:1:6 (port 1E:box 1:bay 6, SATA, 1 TB, OK)
physicaldrive 1E:1:7 (port 1E:box 1:bay 7, SATA, 1 TB, OK)
physicaldrive 1E:1:8 (port 1E:box 1:bay 8, SATA, 1 TB, OK)
physicaldrive 1E:1:9 (port 1E:box 1:bay 9, SATA, 1 TB, OK)
physicaldrive 1E:1:10 (port 1E:box 1:bay 10, SATA, 1 TB, OK)
physicaldrive 1E:1:11 (port 1E:box 1:bay 11, SATA, 1 TB, OK)
physicaldrive 1E:1:12 (port 1E:box 1:bay 12, SATA, 1 TB, OK)
array B (SATA, Unused Space: 0 MB)
logicaldrive 2 (5.5 TB, RAID 1+0, OK)
physicaldrive 2E:1:1 (port 2E:box 1:bay 1, SATA, 1 TB, OK)
physicaldrive 2E:1:2 (port 2E:box 1:bay 2, SATA, 1 TB, Predictive Failure)
physicaldrive 2E:1:3 (port 2E:box 1:bay 3, SATA, 1 TB, OK)
physicaldrive 2E:1:4 (port 2E:box 1:bay 4, SATA, 1 TB, OK)
physicaldrive 2E:1:5 (port 2E:box 1:bay 5, SATA, 1 TB, OK)
physicaldrive 2E:1:6 (port 2E:box 1:bay 6, SATA, 1 TB, OK)
physicaldrive 2E:1:7 (port 2E:box 1:bay 7, SATA, 1 TB, OK)
physicaldrive 2E:1:8 (port 2E:box 1:bay 8, SATA, 1 TB, OK)
physicaldrive 2E:1:9 (port 2E:box 1:bay 9, SATA, 1 TB, OK)
physicaldrive 2E:1:10 (port 2E:box 1:bay 10, SATA, 1 TB, OK)
physicaldrive 2E:1:11 (port 2E:box 1:bay 11, SATA, 1 TB, OK)
physicaldrive 2E:1:12 (port 2E:box 1:bay 12, SATA, 1 TB, OK)
array C (SATA, Unused Space: 0 MB)
logicaldrive 3 (5.5 TB, RAID 1+0, Ready for Rebuild)
physicaldrive 2E:2:1 (port 2E:box 2:bay 1, SATA, 1 TB, OK)
physicaldrive 2E:2:2 (port 2E:box 2:bay 2, SATA, 1 TB, OK)
physicaldrive 2E:2:3 (port 2E:box 2:bay 3, SATA, 1 TB, OK)
physicaldrive 2E:2:4 (port 2E:box 2:bay 4, SATA, 1 TB, OK)
physicaldrive 2E:2:5 (port 2E:box 2:bay 5, SATA, 1 TB, OK)
physicaldrive 2E:2:6 (port 2E:box 2:bay 6, SATA, 1 TB, OK)
physicaldrive 2E:2:7 (port 2E:box 2:bay 7, SATA, 1 TB, OK)
physicaldrive 2E:2:8 (port 2E:box 2:bay 8, SATA, 1 TB, OK)
physicaldrive 2E:2:9 (port 2E:box 2:bay 9, SATA, 1 TB, OK)
physicaldrive 2E:2:10 (port 2E:box 2:bay 10, SATA, 1 TB, OK)
physicaldrive 2E:2:11 (port 2E:box 2:bay 11, SATA, 1 TB, OK)
physicaldrive 2E:2:12 (port 2E:box 2:bay 12, SATA, 1 TB, OK)
Expander 243 (WWID: 50014380031A4B00, Port: 1E, Box: 1)
Expander 245 (WWID: 5001438005396E00, Port: 2E, Box: 2)
Expander 246 (WWID: 500143800460A600, Port: 2E, Box: 1)
Expander 248 (WWID: 50014380055E913F)
Enclosure SEP (Vendor ID HP, Model MSA60) 241 (WWID: 50014380031A4B25, Port: 1E, Box: 1)
Enclosure SEP (Vendor ID HP, Model MSA60) 242 (WWID: 5001438005396E25, Port: 2E, Box: 2)
Enclosure SEP (Vendor ID HP, Model MSA60) 244 (WWID: 500143800460A625, Port: 2E, Box: 1)
SEP (Vendor ID HP, Model P800) 247 (WWID: 50014380055E913E)

=== On '''shin'''===
As root
/opt/MegaRAID/storcli/storcli64 /c0 /eall /sall show all
<pre>Drive /c0/e8/s18 :
================

-----------------------------------------------------------------------------
EID:Slt DID State DG Size Intf Med SED PI SeSz Model Sp Type
-----------------------------------------------------------------------------
8:18 24 Failed 0 3.637 TB SAS HDD N N 512B ST4000NM0023 U -
-----------------------------------------------------------------------------

EID-Enclosure Device ID|Slt-Slot No.|DID-Device ID|DG-DriveGroup
DHS-Dedicated Hot Spare|UGood-Unconfigured Good|GHS-Global Hotspare
UBad-Unconfigured Bad|Onln-Online|Offln-Offline|Intf-Interface
Med-Media Type|SED-Self Encryptive Drive|PI-Protection Info
SeSz-Sector Size|Sp-Spun|U-Up|D-Down|T-Transition|F-Foreign
UGUnsp-Unsupported|UGShld-UnConfigured shielded|HSPShld-Hotspare shielded
CFShld-Configured shielded|Cpybck-CopyBack|CBShld-Copyback Shielded

Drive /c0/e8/s18 - Detailed Information :
=======================================

Drive /c0/e8/s18 State :
======================
Shield Counter = 0
Media Error Count = 0
Other Error Count = 16
BBM Error Count = 0
Drive Temperature = 32C (89.60 F)
Predictive Failure Count = 0
S.M.A.R.T alert flagged by drive = No

Drive /c0/e8/s18 Device attributes :
==================================
SN = Z1Z2S2TL0000C4216E9V
Manufacturer Id = SEAGATE
Model Number = ST4000NM0023
NAND Vendor = NA
WWN = 5000C50057DB2A28
Firmware Revision = 0003
Firmware Release Number = 03290003
Raw size = 3.638 TB [0x1d1c0beb0 Sectors]
Coerced size = 3.637 TB [0x1d1b00000 Sectors]
Non Coerced size = 3.637 TB [0x1d1b0beb0 Sectors]
Device Speed = 6.0Gb/s
Link Speed = 6.0Gb/s
Write cache = N/A
Logical Sector Size = 512B
Physical Sector Size = 512B
Connector Name = Port 0 - 3 & Port 4 - 7 </pre>

=== On ZFS machines ===
$ zpool status
For instruction on how to identify and replace failed disk on ZFS system. [http://wiki.docking.org/index.php/Zfs#Example:_Fixing_degraded_pool.2C_replacing_faulted_disk '''Read here''']

=== On Any Raid1 Configurations ===
Steps to fix a hard drive failure that is in a raid 1 configuration:

The following demonstrates what a failed disk looks like:

[root@myServer ~]# cat /proc/mdstat 
Personalities : [raid1] 
md0 : active raid1 sdb1[0] sda1[2](F) 
128384 blocks [2/1] [U_] 
md1 : active raid1 sdb2[0] sda2[2](F) 
16779776 blocks [2/1] [U_] 
md2 : active raid1 sdb3[0] sda3[2](F) 
139379840 blocks [2/1] [U_] 
unused devices: <none> 

[root@myServer ~]# smartctl -a /dev/sda 
smartctl 5.42 2011-10-20 r3458 [x86_64-linux-2.6.18-371.1.2.el5] (local build) 
Copyright (C) 2002-11 by Bruce Allen, http://smartmontools.sourceforge.net 
Short INQUIRY response, skip product id 
A mandatory SMART command failed: exiting. To continue, add one or more '-T permissive' options. 

[root@myServer ~]# smartctl -a /dev/sdb 
smartctl 5.42 2011-10-20 r3458 [x86_64-linux-2.6.18-371.1.2.el5] (local build) 
Copyright (C) 2002-11 by Bruce Allen, http://smartmontools.sourceforge.net 
=== START OF INFORMATION SECTION === 
Model Family: Seagate Barracuda 7200.10 
Device Model: ST3160815AS 
Serial Number: 9RA6DZP8 
Firmware Version: 4.AAB 
User Capacity: 160,041,885,696 bytes [160 GB] 
Sector Size: 512 bytes logical/physical 
Device is: In smartctl database [for details use: -P show] 
ATA Version is: 7 
ATA Standard is: Exact ATA specification draft version not indicated 
Local Time is: Mon Sep 8 15:50:48 2014 PDT 
SMART support is: Available - device has SMART capability. 
SMART support is: Enabled 
=== START OF READ SMART DATA SECTION === 
SMART overall-health self-assessment test result: PASSED 

There is a lot more that gets printed, but I cut it out. 

So /dev/sda has clearly failed. 

Take note of the GOOD disk serial number so I leave that one in when I replace it: 
Serial Number: 9RA6DZP8 

Mark and remove failed disk from raid: 

[root@myServer ~]# mdadm --manage /dev/md0 --fail /dev/sda1 
mdadm: set /dev/sda1 faulty in /dev/md0 
[root@myServer ~]# mdadm --manage /dev/md1 --fail /dev/sda2 
mdadm: set /dev/sda2 faulty in /dev/md1 
[root@myServer ~]# mdadm --manage /dev/md2 --fail /dev/sda3 
mdadm: set /dev/sda3 faulty in /dev/md2 
[root@myServer ~]# mdadm --manage /dev/md0 --remove /dev/sda1 
mdadm: hot removed /dev/sda1 
[root@myServer ~]# mdadm --manage /dev/md1 --remove /dev/sda2 
mdadm: hot removed /dev/sda2 
[root@myServer ~]# mdadm --manage /dev/md2 --remove /dev/sda3 
mdadm: hot removed /dev/sda3 

Make sure grub is installed on the good disk and that grub.conf is updated:

[root@myServer ~]# grub-install /dev/sdb 
Installation finished. No error reported. 
This is the contents of the device map /boot/grub/device.map. 
Check if this is correct or not. 
If any of the lines is incorrect, fix it and re-run the script `grub-install'. 
This device map was generated by anaconda 
(hd0) /dev/sda 
(hd1) /dev/sdb 

Take note of the which hd partition corresponds with the good disk, ie hd1 in this case. 

[root@myServer ~]# vim /boot/grub/menu.lst 
Add fallback=1 right after default=0 
Go to the bottom section where you should find some kernel stanzas. 
Copy the first of them and paste the stanza before the first existing stanza; replace root (hd0,0) with root (hd1,0) 
Should look like this: 
[...] 
title CentOS (2.6.18-128.el5) 
root (hd1,0) 
kernel /vmlinuz-2.6.18-128.el5 ro root=/dev/VolGroup00/LogVol00 
initrd /initrd-2.6.18-128.el5.img 
title CentOS (2.6.18-128.el5) 
root (hd0,0) 
kernel /vmlinuz-2.6.18-128.el5 ro root=/dev/ 
initrd /initrd-2.6.18-128.el5.img 

Save and quit 

[root@myServer ~]# mv /boot/initramfs-$(uname -r).img /boot/initramfs-$(uname -r).img.bak 
[root@myServer ~]# mkinitrd /boot/initramfs-$(uname -r).img $(uname -r) 
[root@myServer ~]# init 0 

Swap the bad drive with the new drive and boot the machine. 

Once it's booted: 

Check the device names with cat /proc/mdstat and/or fisk -l. 
The newly installed drive on myServer was named /dev/sda. 

[root@myServer ~]# modeprobe raid1 
[root@myServer ~]# modeprobe linear 

Copy the partitions from one disk to the other:

[root@myServer ~]# sfdisk -d /dev/sdb | sfdisk --force /dev/sda 
[root@myServer ~]# sfdisk -l => sanity check 

Add the new disk to the raid array:

[root@myServer ~]# mdadm --manage /dev/md0 --add /dev/sda1 
mdadm: added /dev/sda1 
[root@myServer ~]# mdadm --manage /dev/md1 --add /dev/sda2 
mdadm: added /dev/sda2 
[root@myServer ~]# mdadm --manage /dev/md2 --add /dev/sda3 
mdadm: added /dev/sda3 

Sanity check:
[root@myServer ~]# cat /proc/mdstat 
Personalities : [raid1] [linear] 
md0 : active raid1 sda1[1] sdb1[0] 
128384 blocks [2/2] [UU] 
md1 : active raid1 sda2[2] sdb2[0] 
16779776 blocks [2/1] [U_] 
[>....................] recovery = 3.2% (548864/16779776) finish=8.8min speed=30492K/sec 
md2 : active raid1 sda3[2] sdb3[0] 
139379840 blocks [2/1] [U_] 
resync=DELAYED 
unused devices: <none> 

That's it! :) 
[[ Category: Ben ]]
[[ Category : Sysadmin ]]
[[Category:Tutorials]]

Replacing failed disk on Server

2021-01-26T00:38:01Z

Jgutie11: /* How to check if Disk failed */

== How to check if Disk failed==
===Check for the light on disk===

Solid Yellow => Fail

Blinking Yellow => Predictive Failure (going to fail soon)

Green => Normal
=== Replace disk instruction===
* Determine what machine the disk below to
* Press the red button on the disk to turn it off.
* Gently pull a little bit out (NOT all the way) and wait for 10 sec until it stops spinning before pulling all the way out.
* Find replacement with a similar disk with the same specs
* Carefully unscrew the disk from disk holder (if the disk holder part on the replacement is the same then you don't have to).

=== Auto-check Disk Machines Python Script ===
In gimel5, there is a python script that runs every day at 12am through crontab under s_jjg.

The file is located at: '''/nfs/home/jjg/python_scripts/check_for_failed_disks.py'''

This script ssh-es into the machines below and runs a command to list the status of database.
abacus
db2
db3
db5
n-9-22
db4
tsadi
ex1
ex2
ex3
ex4
lamed
ex5
ex6
ex7
ex8
qof
ex9
zayin
exa
n-1-30
exb
n-1-109
exc
n-1-113
exd
shin
db

== How to check if disk is failed or install correctly==
=== On Cluster 0 's machines ===
1. Log into gimel as root
$ ssh root@sgehead1.bkslab.org
2. Log in as root to the machine that you determined from earlier
$ ssh root@<machine_name>
Example: RAID 3,6,7 belongs to nfshead2
3. Run this command
$ /opt/compaq/hpacucli/bld/hpacucli ctrl all show config

Output Example:
Smart Array P800 in Slot 1 (sn: PAFGF0N9SXQ0MX)
array A (SATA, Unused Space: 0 MB)
logicaldrive 1 (5.5 TB, RAID 1+0, OK)
physicaldrive 1E:1:1 (port 1E:box 1:bay 1, SATA, 1 TB, OK)
physicaldrive 1E:1:2 (port 1E:box 1:bay 2, SATA, 1 TB, OK)
physicaldrive 1E:1:3 (port 1E:box 1:bay 3, SATA, 1 TB, OK)
physicaldrive 1E:1:4 (port 1E:box 1:bay 4, SATA, 1 TB, OK)
physicaldrive 1E:1:5 (port 1E:box 1:bay 5, SATA, 1 TB, OK)
physicaldrive 1E:1:6 (port 1E:box 1:bay 6, SATA, 1 TB, OK)
physicaldrive 1E:1:7 (port 1E:box 1:bay 7, SATA, 1 TB, OK)
physicaldrive 1E:1:8 (port 1E:box 1:bay 8, SATA, 1 TB, OK)
physicaldrive 1E:1:9 (port 1E:box 1:bay 9, SATA, 1 TB, OK)
physicaldrive 1E:1:10 (port 1E:box 1:bay 10, SATA, 1 TB, OK)
physicaldrive 1E:1:11 (port 1E:box 1:bay 11, SATA, 1 TB, OK)
physicaldrive 1E:1:12 (port 1E:box 1:bay 12, SATA, 1 TB, OK)
array B (SATA, Unused Space: 0 MB)
logicaldrive 2 (5.5 TB, RAID 1+0, OK)
physicaldrive 2E:1:1 (port 2E:box 1:bay 1, SATA, 1 TB, OK)
physicaldrive 2E:1:2 (port 2E:box 1:bay 2, SATA, 1 TB, Predictive Failure)
physicaldrive 2E:1:3 (port 2E:box 1:bay 3, SATA, 1 TB, OK)
physicaldrive 2E:1:4 (port 2E:box 1:bay 4, SATA, 1 TB, OK)
physicaldrive 2E:1:5 (port 2E:box 1:bay 5, SATA, 1 TB, OK)
physicaldrive 2E:1:6 (port 2E:box 1:bay 6, SATA, 1 TB, OK)
physicaldrive 2E:1:7 (port 2E:box 1:bay 7, SATA, 1 TB, OK)
physicaldrive 2E:1:8 (port 2E:box 1:bay 8, SATA, 1 TB, OK)
physicaldrive 2E:1:9 (port 2E:box 1:bay 9, SATA, 1 TB, OK)
physicaldrive 2E:1:10 (port 2E:box 1:bay 10, SATA, 1 TB, OK)
physicaldrive 2E:1:11 (port 2E:box 1:bay 11, SATA, 1 TB, OK)
physicaldrive 2E:1:12 (port 2E:box 1:bay 12, SATA, 1 TB, OK)
array C (SATA, Unused Space: 0 MB)
logicaldrive 3 (5.5 TB, RAID 1+0, Ready for Rebuild)
physicaldrive 2E:2:1 (port 2E:box 2:bay 1, SATA, 1 TB, OK)
physicaldrive 2E:2:2 (port 2E:box 2:bay 2, SATA, 1 TB, OK)
physicaldrive 2E:2:3 (port 2E:box 2:bay 3, SATA, 1 TB, OK)
physicaldrive 2E:2:4 (port 2E:box 2:bay 4, SATA, 1 TB, OK)
physicaldrive 2E:2:5 (port 2E:box 2:bay 5, SATA, 1 TB, OK)
physicaldrive 2E:2:6 (port 2E:box 2:bay 6, SATA, 1 TB, OK)
physicaldrive 2E:2:7 (port 2E:box 2:bay 7, SATA, 1 TB, OK)
physicaldrive 2E:2:8 (port 2E:box 2:bay 8, SATA, 1 TB, OK)
physicaldrive 2E:2:9 (port 2E:box 2:bay 9, SATA, 1 TB, OK)
physicaldrive 2E:2:10 (port 2E:box 2:bay 10, SATA, 1 TB, OK)
physicaldrive 2E:2:11 (port 2E:box 2:bay 11, SATA, 1 TB, OK)
physicaldrive 2E:2:12 (port 2E:box 2:bay 12, SATA, 1 TB, OK)
Expander 243 (WWID: 50014380031A4B00, Port: 1E, Box: 1)
Expander 245 (WWID: 5001438005396E00, Port: 2E, Box: 2)
Expander 246 (WWID: 500143800460A600, Port: 2E, Box: 1)
Expander 248 (WWID: 50014380055E913F)
Enclosure SEP (Vendor ID HP, Model MSA60) 241 (WWID: 50014380031A4B25, Port: 1E, Box: 1)
Enclosure SEP (Vendor ID HP, Model MSA60) 242 (WWID: 5001438005396E25, Port: 2E, Box: 2)
Enclosure SEP (Vendor ID HP, Model MSA60) 244 (WWID: 500143800460A625, Port: 2E, Box: 1)
SEP (Vendor ID HP, Model P800) 247 (WWID: 50014380055E913E)

=== On '''shin'''===
As root
/opt/MegaRAID/storcli/storcli64 /c0 /eall /sall show all
<pre>Drive /c0/e8/s18 :
================

-----------------------------------------------------------------------------
EID:Slt DID State DG Size Intf Med SED PI SeSz Model Sp Type
-----------------------------------------------------------------------------
8:18 24 Failed 0 3.637 TB SAS HDD N N 512B ST4000NM0023 U -
-----------------------------------------------------------------------------

EID-Enclosure Device ID|Slt-Slot No.|DID-Device ID|DG-DriveGroup
DHS-Dedicated Hot Spare|UGood-Unconfigured Good|GHS-Global Hotspare
UBad-Unconfigured Bad|Onln-Online|Offln-Offline|Intf-Interface
Med-Media Type|SED-Self Encryptive Drive|PI-Protection Info
SeSz-Sector Size|Sp-Spun|U-Up|D-Down|T-Transition|F-Foreign
UGUnsp-Unsupported|UGShld-UnConfigured shielded|HSPShld-Hotspare shielded
CFShld-Configured shielded|Cpybck-CopyBack|CBShld-Copyback Shielded

Drive /c0/e8/s18 - Detailed Information :
=======================================

Drive /c0/e8/s18 State :
======================
Shield Counter = 0
Media Error Count = 0
Other Error Count = 16
BBM Error Count = 0
Drive Temperature = 32C (89.60 F)
Predictive Failure Count = 0
S.M.A.R.T alert flagged by drive = No

Drive /c0/e8/s18 Device attributes :
==================================
SN = Z1Z2S2TL0000C4216E9V
Manufacturer Id = SEAGATE
Model Number = ST4000NM0023
NAND Vendor = NA
WWN = 5000C50057DB2A28
Firmware Revision = 0003
Firmware Release Number = 03290003
Raw size = 3.638 TB [0x1d1c0beb0 Sectors]
Coerced size = 3.637 TB [0x1d1b00000 Sectors]
Non Coerced size = 3.637 TB [0x1d1b0beb0 Sectors]
Device Speed = 6.0Gb/s
Link Speed = 6.0Gb/s
Write cache = N/A
Logical Sector Size = 512B
Physical Sector Size = 512B
Connector Name = Port 0 - 3 & Port 4 - 7 </pre>

=== On ZFS machines ===
$ zpool status
For instruction on how to identify and replace failed disk on ZFS system. [http://wiki.docking.org/index.php/Zfs#Example:_Fixing_degraded_pool.2C_replacing_faulted_disk '''Read here''']

=== On Any Raid1 Configurations ===
Steps to fix a hard drive failure that is in a raid 1 configuration:

The following demonstrates what a failed disk looks like:

[root@myServer ~]# cat /proc/mdstat 
Personalities : [raid1] 
md0 : active raid1 sdb1[0] sda1[2](F) 
128384 blocks [2/1] [U_] 
md1 : active raid1 sdb2[0] sda2[2](F) 
16779776 blocks [2/1] [U_] 
md2 : active raid1 sdb3[0] sda3[2](F) 
139379840 blocks [2/1] [U_] 
unused devices: <none> 

[root@myServer ~]# smartctl -a /dev/sda 
smartctl 5.42 2011-10-20 r3458 [x86_64-linux-2.6.18-371.1.2.el5] (local build) 
Copyright (C) 2002-11 by Bruce Allen, http://smartmontools.sourceforge.net 
Short INQUIRY response, skip product id 
A mandatory SMART command failed: exiting. To continue, add one or more '-T permissive' options. 

[root@myServer ~]# smartctl -a /dev/sdb 
smartctl 5.42 2011-10-20 r3458 [x86_64-linux-2.6.18-371.1.2.el5] (local build) 
Copyright (C) 2002-11 by Bruce Allen, http://smartmontools.sourceforge.net 
=== START OF INFORMATION SECTION === 
Model Family: Seagate Barracuda 7200.10 
Device Model: ST3160815AS 
Serial Number: 9RA6DZP8 
Firmware Version: 4.AAB 
User Capacity: 160,041,885,696 bytes [160 GB] 
Sector Size: 512 bytes logical/physical 
Device is: In smartctl database [for details use: -P show] 
ATA Version is: 7 
ATA Standard is: Exact ATA specification draft version not indicated 
Local Time is: Mon Sep 8 15:50:48 2014 PDT 
SMART support is: Available - device has SMART capability. 
SMART support is: Enabled 
=== START OF READ SMART DATA SECTION === 
SMART overall-health self-assessment test result: PASSED 

There is a lot more that gets printed, but I cut it out. 

So /dev/sda has clearly failed. 

Take note of the GOOD disk serial number so I leave that one in when I replace it: 
Serial Number: 9RA6DZP8 

Mark and remove failed disk from raid: 

[root@myServer ~]# mdadm --manage /dev/md0 --fail /dev/sda1 
mdadm: set /dev/sda1 faulty in /dev/md0 
[root@myServer ~]# mdadm --manage /dev/md1 --fail /dev/sda2 
mdadm: set /dev/sda2 faulty in /dev/md1 
[root@myServer ~]# mdadm --manage /dev/md2 --fail /dev/sda3 
mdadm: set /dev/sda3 faulty in /dev/md2 
[root@myServer ~]# mdadm --manage /dev/md0 --remove /dev/sda1 
mdadm: hot removed /dev/sda1 
[root@myServer ~]# mdadm --manage /dev/md1 --remove /dev/sda2 
mdadm: hot removed /dev/sda2 
[root@myServer ~]# mdadm --manage /dev/md2 --remove /dev/sda3 
mdadm: hot removed /dev/sda3 

Make sure grub is installed on the good disk and that grub.conf is updated:

[root@myServer ~]# grub-install /dev/sdb 
Installation finished. No error reported. 
This is the contents of the device map /boot/grub/device.map. 
Check if this is correct or not. 
If any of the lines is incorrect, fix it and re-run the script `grub-install'. 
This device map was generated by anaconda 
(hd0) /dev/sda 
(hd1) /dev/sdb 

Take note of the which hd partition corresponds with the good disk, ie hd1 in this case. 

[root@myServer ~]# vim /boot/grub/menu.lst 
Add fallback=1 right after default=0 
Go to the bottom section where you should find some kernel stanzas. 
Copy the first of them and paste the stanza before the first existing stanza; replace root (hd0,0) with root (hd1,0) 
Should look like this: 
[...] 
title CentOS (2.6.18-128.el5) 
root (hd1,0) 
kernel /vmlinuz-2.6.18-128.el5 ro root=/dev/VolGroup00/LogVol00 
initrd /initrd-2.6.18-128.el5.img 
title CentOS (2.6.18-128.el5) 
root (hd0,0) 
kernel /vmlinuz-2.6.18-128.el5 ro root=/dev/ 
initrd /initrd-2.6.18-128.el5.img 

Save and quit 

[root@myServer ~]# mv /boot/initramfs-$(uname -r).img /boot/initramfs-$(uname -r).img.bak 
[root@myServer ~]# mkinitrd /boot/initramfs-$(uname -r).img $(uname -r) 
[root@myServer ~]# init 0 

Swap the bad drive with the new drive and boot the machine. 

Once it's booted: 

Check the device names with cat /proc/mdstat and/or fisk -l. 
The newly installed drive on myServer was named /dev/sda. 

[root@myServer ~]# modeprobe raid1 
[root@myServer ~]# modeprobe linear 

Copy the partitions from one disk to the other:

[root@myServer ~]# sfdisk -d /dev/sdb | sfdisk --force /dev/sda 
[root@myServer ~]# sfdisk -l => sanity check 

Add the new disk to the raid array:

[root@myServer ~]# mdadm --manage /dev/md0 --add /dev/sda1 
mdadm: added /dev/sda1 
[root@myServer ~]# mdadm --manage /dev/md1 --add /dev/sda2 
mdadm: added /dev/sda2 
[root@myServer ~]# mdadm --manage /dev/md2 --add /dev/sda3 
mdadm: added /dev/sda3 

Sanity check:
[root@myServer ~]# cat /proc/mdstat 
Personalities : [raid1] [linear] 
md0 : active raid1 sda1[1] sdb1[0] 
128384 blocks [2/2] [UU] 
md1 : active raid1 sda2[2] sdb2[0] 
16779776 blocks [2/1] [U_] 
[>....................] recovery = 3.2% (548864/16779776) finish=8.8min speed=30492K/sec 
md2 : active raid1 sda3[2] sdb3[0] 
139379840 blocks [2/1] [U_] 
resync=DELAYED 
unused devices: <none> 

That's it! :) 
[[ Category: Ben ]]
[[ Category : Sysadmin ]]
[[Category:Tutorials]]

Replacing failed disk on Server

2021-01-26T00:37:36Z

Jgutie11: /* How to check if Disk failed */

== How to check if Disk failed==
===Check for the light on disk===

Solid Yellow => Fail

Blinking Yellow => Predictive Failure (going to fail soon)

Green => Normal
=== Replace disk instruction===
* Determine what machine the disk below to
* Press the red button on the disk to turn it off.
* Gently pull a little bit out (NOT all the way) and wait for 10 sec until it stops spinning before pulling all the way out.
* Find replacement with a similar disk with the same specs
* Carefully unscrew the disk from disk holder (if the disk holder part on the replacement is the same then you don't have to).

=== Auto-check Disk Machines Python Script ===
In gimel5, there is a python script that runs every day at 12am through crontab under s_jjg.

The file is located at: '''/nfs/home/jjg/python_scripts/check_for_failed_disks.py'''

This script ssh-es into the machines below and runs a command to list the status of database.
abacus
db2
db3
db5
n-9-22
db4
tsadi
ex1
ex2
ex3
ex4
lamed
ex5
ex6
ex7
ex8
qof
ex9
zayin
exa
n-1-30
exb
n-1-109
exc
n-1-113
exd

== How to check if disk is failed or install correctly==
=== On Cluster 0 's machines ===
1. Log into gimel as root
$ ssh root@sgehead1.bkslab.org
2. Log in as root to the machine that you determined from earlier
$ ssh root@<machine_name>
Example: RAID 3,6,7 belongs to nfshead2
3. Run this command
$ /opt/compaq/hpacucli/bld/hpacucli ctrl all show config

Output Example:
Smart Array P800 in Slot 1 (sn: PAFGF0N9SXQ0MX)
array A (SATA, Unused Space: 0 MB)
logicaldrive 1 (5.5 TB, RAID 1+0, OK)
physicaldrive 1E:1:1 (port 1E:box 1:bay 1, SATA, 1 TB, OK)
physicaldrive 1E:1:2 (port 1E:box 1:bay 2, SATA, 1 TB, OK)
physicaldrive 1E:1:3 (port 1E:box 1:bay 3, SATA, 1 TB, OK)
physicaldrive 1E:1:4 (port 1E:box 1:bay 4, SATA, 1 TB, OK)
physicaldrive 1E:1:5 (port 1E:box 1:bay 5, SATA, 1 TB, OK)
physicaldrive 1E:1:6 (port 1E:box 1:bay 6, SATA, 1 TB, OK)
physicaldrive 1E:1:7 (port 1E:box 1:bay 7, SATA, 1 TB, OK)
physicaldrive 1E:1:8 (port 1E:box 1:bay 8, SATA, 1 TB, OK)
physicaldrive 1E:1:9 (port 1E:box 1:bay 9, SATA, 1 TB, OK)
physicaldrive 1E:1:10 (port 1E:box 1:bay 10, SATA, 1 TB, OK)
physicaldrive 1E:1:11 (port 1E:box 1:bay 11, SATA, 1 TB, OK)
physicaldrive 1E:1:12 (port 1E:box 1:bay 12, SATA, 1 TB, OK)
array B (SATA, Unused Space: 0 MB)
logicaldrive 2 (5.5 TB, RAID 1+0, OK)
physicaldrive 2E:1:1 (port 2E:box 1:bay 1, SATA, 1 TB, OK)
physicaldrive 2E:1:2 (port 2E:box 1:bay 2, SATA, 1 TB, Predictive Failure)
physicaldrive 2E:1:3 (port 2E:box 1:bay 3, SATA, 1 TB, OK)
physicaldrive 2E:1:4 (port 2E:box 1:bay 4, SATA, 1 TB, OK)
physicaldrive 2E:1:5 (port 2E:box 1:bay 5, SATA, 1 TB, OK)
physicaldrive 2E:1:6 (port 2E:box 1:bay 6, SATA, 1 TB, OK)
physicaldrive 2E:1:7 (port 2E:box 1:bay 7, SATA, 1 TB, OK)
physicaldrive 2E:1:8 (port 2E:box 1:bay 8, SATA, 1 TB, OK)
physicaldrive 2E:1:9 (port 2E:box 1:bay 9, SATA, 1 TB, OK)
physicaldrive 2E:1:10 (port 2E:box 1:bay 10, SATA, 1 TB, OK)
physicaldrive 2E:1:11 (port 2E:box 1:bay 11, SATA, 1 TB, OK)
physicaldrive 2E:1:12 (port 2E:box 1:bay 12, SATA, 1 TB, OK)
array C (SATA, Unused Space: 0 MB)
logicaldrive 3 (5.5 TB, RAID 1+0, Ready for Rebuild)
physicaldrive 2E:2:1 (port 2E:box 2:bay 1, SATA, 1 TB, OK)
physicaldrive 2E:2:2 (port 2E:box 2:bay 2, SATA, 1 TB, OK)
physicaldrive 2E:2:3 (port 2E:box 2:bay 3, SATA, 1 TB, OK)
physicaldrive 2E:2:4 (port 2E:box 2:bay 4, SATA, 1 TB, OK)
physicaldrive 2E:2:5 (port 2E:box 2:bay 5, SATA, 1 TB, OK)
physicaldrive 2E:2:6 (port 2E:box 2:bay 6, SATA, 1 TB, OK)
physicaldrive 2E:2:7 (port 2E:box 2:bay 7, SATA, 1 TB, OK)
physicaldrive 2E:2:8 (port 2E:box 2:bay 8, SATA, 1 TB, OK)
physicaldrive 2E:2:9 (port 2E:box 2:bay 9, SATA, 1 TB, OK)
physicaldrive 2E:2:10 (port 2E:box 2:bay 10, SATA, 1 TB, OK)
physicaldrive 2E:2:11 (port 2E:box 2:bay 11, SATA, 1 TB, OK)
physicaldrive 2E:2:12 (port 2E:box 2:bay 12, SATA, 1 TB, OK)
Expander 243 (WWID: 50014380031A4B00, Port: 1E, Box: 1)
Expander 245 (WWID: 5001438005396E00, Port: 2E, Box: 2)
Expander 246 (WWID: 500143800460A600, Port: 2E, Box: 1)
Expander 248 (WWID: 50014380055E913F)
Enclosure SEP (Vendor ID HP, Model MSA60) 241 (WWID: 50014380031A4B25, Port: 1E, Box: 1)
Enclosure SEP (Vendor ID HP, Model MSA60) 242 (WWID: 5001438005396E25, Port: 2E, Box: 2)
Enclosure SEP (Vendor ID HP, Model MSA60) 244 (WWID: 500143800460A625, Port: 2E, Box: 1)
SEP (Vendor ID HP, Model P800) 247 (WWID: 50014380055E913E)

=== On '''shin'''===
As root
/opt/MegaRAID/storcli/storcli64 /c0 /eall /sall show all
<pre>Drive /c0/e8/s18 :
================

-----------------------------------------------------------------------------
EID:Slt DID State DG Size Intf Med SED PI SeSz Model Sp Type
-----------------------------------------------------------------------------
8:18 24 Failed 0 3.637 TB SAS HDD N N 512B ST4000NM0023 U -
-----------------------------------------------------------------------------

EID-Enclosure Device ID|Slt-Slot No.|DID-Device ID|DG-DriveGroup
DHS-Dedicated Hot Spare|UGood-Unconfigured Good|GHS-Global Hotspare
UBad-Unconfigured Bad|Onln-Online|Offln-Offline|Intf-Interface
Med-Media Type|SED-Self Encryptive Drive|PI-Protection Info
SeSz-Sector Size|Sp-Spun|U-Up|D-Down|T-Transition|F-Foreign
UGUnsp-Unsupported|UGShld-UnConfigured shielded|HSPShld-Hotspare shielded
CFShld-Configured shielded|Cpybck-CopyBack|CBShld-Copyback Shielded

Drive /c0/e8/s18 - Detailed Information :
=======================================

Drive /c0/e8/s18 State :
======================
Shield Counter = 0
Media Error Count = 0
Other Error Count = 16
BBM Error Count = 0
Drive Temperature = 32C (89.60 F)
Predictive Failure Count = 0
S.M.A.R.T alert flagged by drive = No

Drive /c0/e8/s18 Device attributes :
==================================
SN = Z1Z2S2TL0000C4216E9V
Manufacturer Id = SEAGATE
Model Number = ST4000NM0023
NAND Vendor = NA
WWN = 5000C50057DB2A28
Firmware Revision = 0003
Firmware Release Number = 03290003
Raw size = 3.638 TB [0x1d1c0beb0 Sectors]
Coerced size = 3.637 TB [0x1d1b00000 Sectors]
Non Coerced size = 3.637 TB [0x1d1b0beb0 Sectors]
Device Speed = 6.0Gb/s
Link Speed = 6.0Gb/s
Write cache = N/A
Logical Sector Size = 512B
Physical Sector Size = 512B
Connector Name = Port 0 - 3 & Port 4 - 7 </pre>

=== On ZFS machines ===
$ zpool status
For instruction on how to identify and replace failed disk on ZFS system. [http://wiki.docking.org/index.php/Zfs#Example:_Fixing_degraded_pool.2C_replacing_faulted_disk '''Read here''']

=== On Any Raid1 Configurations ===
Steps to fix a hard drive failure that is in a raid 1 configuration:

The following demonstrates what a failed disk looks like:

[root@myServer ~]# cat /proc/mdstat 
Personalities : [raid1] 
md0 : active raid1 sdb1[0] sda1[2](F) 
128384 blocks [2/1] [U_] 
md1 : active raid1 sdb2[0] sda2[2](F) 
16779776 blocks [2/1] [U_] 
md2 : active raid1 sdb3[0] sda3[2](F) 
139379840 blocks [2/1] [U_] 
unused devices: <none> 

[root@myServer ~]# smartctl -a /dev/sda 
smartctl 5.42 2011-10-20 r3458 [x86_64-linux-2.6.18-371.1.2.el5] (local build) 
Copyright (C) 2002-11 by Bruce Allen, http://smartmontools.sourceforge.net 
Short INQUIRY response, skip product id 
A mandatory SMART command failed: exiting. To continue, add one or more '-T permissive' options. 

[root@myServer ~]# smartctl -a /dev/sdb 
smartctl 5.42 2011-10-20 r3458 [x86_64-linux-2.6.18-371.1.2.el5] (local build) 
Copyright (C) 2002-11 by Bruce Allen, http://smartmontools.sourceforge.net 
=== START OF INFORMATION SECTION === 
Model Family: Seagate Barracuda 7200.10 
Device Model: ST3160815AS 
Serial Number: 9RA6DZP8 
Firmware Version: 4.AAB 
User Capacity: 160,041,885,696 bytes [160 GB] 
Sector Size: 512 bytes logical/physical 
Device is: In smartctl database [for details use: -P show] 
ATA Version is: 7 
ATA Standard is: Exact ATA specification draft version not indicated 
Local Time is: Mon Sep 8 15:50:48 2014 PDT 
SMART support is: Available - device has SMART capability. 
SMART support is: Enabled 
=== START OF READ SMART DATA SECTION === 
SMART overall-health self-assessment test result: PASSED 

There is a lot more that gets printed, but I cut it out. 

So /dev/sda has clearly failed. 

Take note of the GOOD disk serial number so I leave that one in when I replace it: 
Serial Number: 9RA6DZP8 

Mark and remove failed disk from raid: 

[root@myServer ~]# mdadm --manage /dev/md0 --fail /dev/sda1 
mdadm: set /dev/sda1 faulty in /dev/md0 
[root@myServer ~]# mdadm --manage /dev/md1 --fail /dev/sda2 
mdadm: set /dev/sda2 faulty in /dev/md1 
[root@myServer ~]# mdadm --manage /dev/md2 --fail /dev/sda3 
mdadm: set /dev/sda3 faulty in /dev/md2 
[root@myServer ~]# mdadm --manage /dev/md0 --remove /dev/sda1 
mdadm: hot removed /dev/sda1 
[root@myServer ~]# mdadm --manage /dev/md1 --remove /dev/sda2 
mdadm: hot removed /dev/sda2 
[root@myServer ~]# mdadm --manage /dev/md2 --remove /dev/sda3 
mdadm: hot removed /dev/sda3 

Make sure grub is installed on the good disk and that grub.conf is updated:

[root@myServer ~]# grub-install /dev/sdb 
Installation finished. No error reported. 
This is the contents of the device map /boot/grub/device.map. 
Check if this is correct or not. 
If any of the lines is incorrect, fix it and re-run the script `grub-install'. 
This device map was generated by anaconda 
(hd0) /dev/sda 
(hd1) /dev/sdb 

Take note of the which hd partition corresponds with the good disk, ie hd1 in this case. 

[root@myServer ~]# vim /boot/grub/menu.lst 
Add fallback=1 right after default=0 
Go to the bottom section where you should find some kernel stanzas. 
Copy the first of them and paste the stanza before the first existing stanza; replace root (hd0,0) with root (hd1,0) 
Should look like this: 
[...] 
title CentOS (2.6.18-128.el5) 
root (hd1,0) 
kernel /vmlinuz-2.6.18-128.el5 ro root=/dev/VolGroup00/LogVol00 
initrd /initrd-2.6.18-128.el5.img 
title CentOS (2.6.18-128.el5) 
root (hd0,0) 
kernel /vmlinuz-2.6.18-128.el5 ro root=/dev/ 
initrd /initrd-2.6.18-128.el5.img 

Save and quit 

[root@myServer ~]# mv /boot/initramfs-$(uname -r).img /boot/initramfs-$(uname -r).img.bak 
[root@myServer ~]# mkinitrd /boot/initramfs-$(uname -r).img $(uname -r) 
[root@myServer ~]# init 0 

Swap the bad drive with the new drive and boot the machine. 

Once it's booted: 

Check the device names with cat /proc/mdstat and/or fisk -l. 
The newly installed drive on myServer was named /dev/sda. 

[root@myServer ~]# modeprobe raid1 
[root@myServer ~]# modeprobe linear 

Copy the partitions from one disk to the other:

[root@myServer ~]# sfdisk -d /dev/sdb | sfdisk --force /dev/sda 
[root@myServer ~]# sfdisk -l => sanity check 

Add the new disk to the raid array:

[root@myServer ~]# mdadm --manage /dev/md0 --add /dev/sda1 
mdadm: added /dev/sda1 
[root@myServer ~]# mdadm --manage /dev/md1 --add /dev/sda2 
mdadm: added /dev/sda2 
[root@myServer ~]# mdadm --manage /dev/md2 --add /dev/sda3 
mdadm: added /dev/sda3 

Sanity check:
[root@myServer ~]# cat /proc/mdstat 
Personalities : [raid1] [linear] 
md0 : active raid1 sda1[1] sdb1[0] 
128384 blocks [2/2] [UU] 
md1 : active raid1 sda2[2] sdb2[0] 
16779776 blocks [2/1] [U_] 
[>....................] recovery = 3.2% (548864/16779776) finish=8.8min speed=30492K/sec 
md2 : active raid1 sda3[2] sdb3[0] 
139379840 blocks [2/1] [U_] 
resync=DELAYED 
unused devices: <none> 

That's it! :) 
[[ Category: Ben ]]
[[ Category : Sysadmin ]]
[[Category:Tutorials]]

How to Replace a Failed Disk

2021-01-25T22:49:47Z

Jgutie11:

Steps to fix a hard drive failure that is in a raid 1 configuration:

The following demonstrates what a failed disk looks like:

[root@myServer ~]# cat /proc/mdstat 
Personalities : [raid1] 
md0 : active raid1 sdb1[0] sda1[2](F) 
128384 blocks [2/1] [U_] 
md1 : active raid1 sdb2[0] sda2[2](F) 
16779776 blocks [2/1] [U_] 
md2 : active raid1 sdb3[0] sda3[2](F) 
139379840 blocks [2/1] [U_] 
unused devices: <none> 

[root@myServer ~]# smartctl -a /dev/sda 
smartctl 5.42 2011-10-20 r3458 [x86_64-linux-2.6.18-371.1.2.el5] (local build) 
Copyright (C) 2002-11 by Bruce Allen, http://smartmontools.sourceforge.net 
Short INQUIRY response, skip product id 
A mandatory SMART command failed: exiting. To continue, add one or more '-T permissive' options. 

[root@myServer ~]# smartctl -a /dev/sdb 
smartctl 5.42 2011-10-20 r3458 [x86_64-linux-2.6.18-371.1.2.el5] (local build) 
Copyright (C) 2002-11 by Bruce Allen, http://smartmontools.sourceforge.net 
=== START OF INFORMATION SECTION === 
Model Family: Seagate Barracuda 7200.10 
Device Model: ST3160815AS 
Serial Number: 9RA6DZP8 
Firmware Version: 4.AAB 
User Capacity: 160,041,885,696 bytes [160 GB] 
Sector Size: 512 bytes logical/physical 
Device is: In smartctl database [for details use: -P show] 
ATA Version is: 7 
ATA Standard is: Exact ATA specification draft version not indicated 
Local Time is: Mon Sep 8 15:50:48 2014 PDT 
SMART support is: Available - device has SMART capability. 
SMART support is: Enabled 
=== START OF READ SMART DATA SECTION === 
SMART overall-health self-assessment test result: PASSED 

There is a lot more that gets printed, but I cut it out. 

So /dev/sda has clearly failed. 

Take note of the GOOD disk serial number so I leave that one in when I replace it: 
Serial Number: 9RA6DZP8 

Mark and remove failed disk from raid: 

[root@myServer ~]# mdadm --manage /dev/md0 --fail /dev/sda1 
mdadm: set /dev/sda1 faulty in /dev/md0 
[root@myServer ~]# mdadm --manage /dev/md1 --fail /dev/sda2 
mdadm: set /dev/sda2 faulty in /dev/md1 
[root@myServer ~]# mdadm --manage /dev/md2 --fail /dev/sda3 
mdadm: set /dev/sda3 faulty in /dev/md2 
[root@myServer ~]# mdadm --manage /dev/md0 --remove /dev/sda1 
mdadm: hot removed /dev/sda1 
[root@myServer ~]# mdadm --manage /dev/md1 --remove /dev/sda2 
mdadm: hot removed /dev/sda2 
[root@myServer ~]# mdadm --manage /dev/md2 --remove /dev/sda3 
mdadm: hot removed /dev/sda3 

Make sure grub is installed on the good disk and that grub.conf is updated:

[root@myServer ~]# grub-install /dev/sdb 
Installation finished. No error reported. 
This is the contents of the device map /boot/grub/device.map. 
Check if this is correct or not. 
If any of the lines is incorrect, fix it and re-run the script `grub-install'. 
This device map was generated by anaconda 
(hd0) /dev/sda 
(hd1) /dev/sdb 

Take note of the which hd partition corresponds with the good disk, ie hd1 in this case. 

[root@myServer ~]# vim /boot/grub/menu.lst 
Add fallback=1 right after default=0 
Go to the bottom section where you should find some kernel stanzas. 
Copy the first of them and paste the stanza before the first existing stanza; replace root (hd0,0) with root (hd1,0) 
Should look like this: 
[...] 
title CentOS (2.6.18-128.el5) 
root (hd1,0) 
kernel /vmlinuz-2.6.18-128.el5 ro root=/dev/VolGroup00/LogVol00 
initrd /initrd-2.6.18-128.el5.img 
title CentOS (2.6.18-128.el5) 
root (hd0,0) 
kernel /vmlinuz-2.6.18-128.el5 ro root=/dev/ 
initrd /initrd-2.6.18-128.el5.img 

Save and quit 

[root@myServer ~]# mv /boot/initramfs-$(uname -r).img /boot/initramfs-$(uname -r).img.bak 
[root@myServer ~]# mkinitrd /boot/initramfs-$(uname -r).img $(uname -r) 
[root@myServer ~]# init 0 

Swap the bad drive with the new drive and boot the machine. 

Once it's booted: 

Check the device names with cat /proc/mdstat and/or fisk -l. 
The newly installed drive on myServer was named /dev/sda. 

[root@myServer ~]# modeprobe raid1 
[root@myServer ~]# modeprobe linear 

Copy the partitions from one disk to the other:

[root@myServer ~]# sfdisk -d /dev/sdb | sfdisk --force /dev/sda 
[root@myServer ~]# sfdisk -l => sanity check 

Add the new disk to the raid array:

[root@myServer ~]# mdadm --manage /dev/md0 --add /dev/sda1 
mdadm: added /dev/sda1 
[root@myServer ~]# mdadm --manage /dev/md1 --add /dev/sda2 
mdadm: added /dev/sda2 
[root@myServer ~]# mdadm --manage /dev/md2 --add /dev/sda3 
mdadm: added /dev/sda3 

Sanity check:
[root@myServer ~]# cat /proc/mdstat 
Personalities : [raid1] [linear] 
md0 : active raid1 sda1[1] sdb1[0] 
128384 blocks [2/2] [UU] 
md1 : active raid1 sda2[2] sdb2[0] 
16779776 blocks [2/1] [U_] 
[>....................] recovery = 3.2% (548864/16779776) finish=8.8min speed=30492K/sec 
md2 : active raid1 sda3[2] sdb3[0] 
139379840 blocks [2/1] [U_] 
resync=DELAYED 
unused devices: <none> 

That's it! :) 

[[Category:Sysadmin]]
[[Category:Tutorials]]
[[Category:Delete]]

Replacing failed disk on Server

2021-01-25T22:49:21Z

Jgutie11:

== How to check if Disk failed==
===Check for the light on disk===

Solid Yellow => Fail

Blinking Yellow => Predictive Failure (going to fail soon)

Green => Normal
=== Replace disk instruction===
* Determine what machine the disk below to
* Press the red button on the disk to turn it off.
* Gently pull a little bit out (NOT all the way) and wait for 10 sec until it stops spinning before pulling all the way out.
* Find replacement with a similar disk with the same specs
* Carefully unscrew the disk from disk holder (if the disk holder part on the replacement is the same then you don't have to).

== How to check if disk is failed or install correctly==
=== On Cluster 0 's machines ===
1. Log into gimel as root
$ ssh root@sgehead1.bkslab.org
2. Log in as root to the machine that you determined from earlier
$ ssh root@<machine_name>
Example: RAID 3,6,7 belongs to nfshead2
3. Run this command
$ /opt/compaq/hpacucli/bld/hpacucli ctrl all show config

Output Example:
Smart Array P800 in Slot 1 (sn: PAFGF0N9SXQ0MX)
array A (SATA, Unused Space: 0 MB)
logicaldrive 1 (5.5 TB, RAID 1+0, OK)
physicaldrive 1E:1:1 (port 1E:box 1:bay 1, SATA, 1 TB, OK)
physicaldrive 1E:1:2 (port 1E:box 1:bay 2, SATA, 1 TB, OK)
physicaldrive 1E:1:3 (port 1E:box 1:bay 3, SATA, 1 TB, OK)
physicaldrive 1E:1:4 (port 1E:box 1:bay 4, SATA, 1 TB, OK)
physicaldrive 1E:1:5 (port 1E:box 1:bay 5, SATA, 1 TB, OK)
physicaldrive 1E:1:6 (port 1E:box 1:bay 6, SATA, 1 TB, OK)
physicaldrive 1E:1:7 (port 1E:box 1:bay 7, SATA, 1 TB, OK)
physicaldrive 1E:1:8 (port 1E:box 1:bay 8, SATA, 1 TB, OK)
physicaldrive 1E:1:9 (port 1E:box 1:bay 9, SATA, 1 TB, OK)
physicaldrive 1E:1:10 (port 1E:box 1:bay 10, SATA, 1 TB, OK)
physicaldrive 1E:1:11 (port 1E:box 1:bay 11, SATA, 1 TB, OK)
physicaldrive 1E:1:12 (port 1E:box 1:bay 12, SATA, 1 TB, OK)
array B (SATA, Unused Space: 0 MB)
logicaldrive 2 (5.5 TB, RAID 1+0, OK)
physicaldrive 2E:1:1 (port 2E:box 1:bay 1, SATA, 1 TB, OK)
physicaldrive 2E:1:2 (port 2E:box 1:bay 2, SATA, 1 TB, Predictive Failure)
physicaldrive 2E:1:3 (port 2E:box 1:bay 3, SATA, 1 TB, OK)
physicaldrive 2E:1:4 (port 2E:box 1:bay 4, SATA, 1 TB, OK)
physicaldrive 2E:1:5 (port 2E:box 1:bay 5, SATA, 1 TB, OK)
physicaldrive 2E:1:6 (port 2E:box 1:bay 6, SATA, 1 TB, OK)
physicaldrive 2E:1:7 (port 2E:box 1:bay 7, SATA, 1 TB, OK)
physicaldrive 2E:1:8 (port 2E:box 1:bay 8, SATA, 1 TB, OK)
physicaldrive 2E:1:9 (port 2E:box 1:bay 9, SATA, 1 TB, OK)
physicaldrive 2E:1:10 (port 2E:box 1:bay 10, SATA, 1 TB, OK)
physicaldrive 2E:1:11 (port 2E:box 1:bay 11, SATA, 1 TB, OK)
physicaldrive 2E:1:12 (port 2E:box 1:bay 12, SATA, 1 TB, OK)
array C (SATA, Unused Space: 0 MB)
logicaldrive 3 (5.5 TB, RAID 1+0, Ready for Rebuild)
physicaldrive 2E:2:1 (port 2E:box 2:bay 1, SATA, 1 TB, OK)
physicaldrive 2E:2:2 (port 2E:box 2:bay 2, SATA, 1 TB, OK)
physicaldrive 2E:2:3 (port 2E:box 2:bay 3, SATA, 1 TB, OK)
physicaldrive 2E:2:4 (port 2E:box 2:bay 4, SATA, 1 TB, OK)
physicaldrive 2E:2:5 (port 2E:box 2:bay 5, SATA, 1 TB, OK)
physicaldrive 2E:2:6 (port 2E:box 2:bay 6, SATA, 1 TB, OK)
physicaldrive 2E:2:7 (port 2E:box 2:bay 7, SATA, 1 TB, OK)
physicaldrive 2E:2:8 (port 2E:box 2:bay 8, SATA, 1 TB, OK)
physicaldrive 2E:2:9 (port 2E:box 2:bay 9, SATA, 1 TB, OK)
physicaldrive 2E:2:10 (port 2E:box 2:bay 10, SATA, 1 TB, OK)
physicaldrive 2E:2:11 (port 2E:box 2:bay 11, SATA, 1 TB, OK)
physicaldrive 2E:2:12 (port 2E:box 2:bay 12, SATA, 1 TB, OK)
Expander 243 (WWID: 50014380031A4B00, Port: 1E, Box: 1)
Expander 245 (WWID: 5001438005396E00, Port: 2E, Box: 2)
Expander 246 (WWID: 500143800460A600, Port: 2E, Box: 1)
Expander 248 (WWID: 50014380055E913F)
Enclosure SEP (Vendor ID HP, Model MSA60) 241 (WWID: 50014380031A4B25, Port: 1E, Box: 1)
Enclosure SEP (Vendor ID HP, Model MSA60) 242 (WWID: 5001438005396E25, Port: 2E, Box: 2)
Enclosure SEP (Vendor ID HP, Model MSA60) 244 (WWID: 500143800460A625, Port: 2E, Box: 1)
SEP (Vendor ID HP, Model P800) 247 (WWID: 50014380055E913E)

=== On '''shin'''===
As root
/opt/MegaRAID/storcli/storcli64 /c0 /eall /sall show all
<pre>Drive /c0/e8/s18 :
================

-----------------------------------------------------------------------------
EID:Slt DID State DG Size Intf Med SED PI SeSz Model Sp Type
-----------------------------------------------------------------------------
8:18 24 Failed 0 3.637 TB SAS HDD N N 512B ST4000NM0023 U -
-----------------------------------------------------------------------------

EID-Enclosure Device ID|Slt-Slot No.|DID-Device ID|DG-DriveGroup
DHS-Dedicated Hot Spare|UGood-Unconfigured Good|GHS-Global Hotspare
UBad-Unconfigured Bad|Onln-Online|Offln-Offline|Intf-Interface
Med-Media Type|SED-Self Encryptive Drive|PI-Protection Info
SeSz-Sector Size|Sp-Spun|U-Up|D-Down|T-Transition|F-Foreign
UGUnsp-Unsupported|UGShld-UnConfigured shielded|HSPShld-Hotspare shielded
CFShld-Configured shielded|Cpybck-CopyBack|CBShld-Copyback Shielded

Drive /c0/e8/s18 - Detailed Information :
=======================================

Drive /c0/e8/s18 State :
======================
Shield Counter = 0
Media Error Count = 0
Other Error Count = 16
BBM Error Count = 0
Drive Temperature = 32C (89.60 F)
Predictive Failure Count = 0
S.M.A.R.T alert flagged by drive = No

Drive /c0/e8/s18 Device attributes :
==================================
SN = Z1Z2S2TL0000C4216E9V
Manufacturer Id = SEAGATE
Model Number = ST4000NM0023
NAND Vendor = NA
WWN = 5000C50057DB2A28
Firmware Revision = 0003
Firmware Release Number = 03290003
Raw size = 3.638 TB [0x1d1c0beb0 Sectors]
Coerced size = 3.637 TB [0x1d1b00000 Sectors]
Non Coerced size = 3.637 TB [0x1d1b0beb0 Sectors]
Device Speed = 6.0Gb/s
Link Speed = 6.0Gb/s
Write cache = N/A
Logical Sector Size = 512B
Physical Sector Size = 512B
Connector Name = Port 0 - 3 & Port 4 - 7 </pre>

=== On ZFS machines ===
$ zpool status
For instruction on how to identify and replace failed disk on ZFS system. [http://wiki.docking.org/index.php/Zfs#Example:_Fixing_degraded_pool.2C_replacing_faulted_disk '''Read here''']

=== On Any Raid1 Configurations ===
Steps to fix a hard drive failure that is in a raid 1 configuration:

The following demonstrates what a failed disk looks like:

[root@myServer ~]# cat /proc/mdstat 
Personalities : [raid1] 
md0 : active raid1 sdb1[0] sda1[2](F) 
128384 blocks [2/1] [U_] 
md1 : active raid1 sdb2[0] sda2[2](F) 
16779776 blocks [2/1] [U_] 
md2 : active raid1 sdb3[0] sda3[2](F) 
139379840 blocks [2/1] [U_] 
unused devices: <none> 

[root@myServer ~]# smartctl -a /dev/sda 
smartctl 5.42 2011-10-20 r3458 [x86_64-linux-2.6.18-371.1.2.el5] (local build) 
Copyright (C) 2002-11 by Bruce Allen, http://smartmontools.sourceforge.net 
Short INQUIRY response, skip product id 
A mandatory SMART command failed: exiting. To continue, add one or more '-T permissive' options. 

[root@myServer ~]# smartctl -a /dev/sdb 
smartctl 5.42 2011-10-20 r3458 [x86_64-linux-2.6.18-371.1.2.el5] (local build) 
Copyright (C) 2002-11 by Bruce Allen, http://smartmontools.sourceforge.net 
=== START OF INFORMATION SECTION === 
Model Family: Seagate Barracuda 7200.10 
Device Model: ST3160815AS 
Serial Number: 9RA6DZP8 
Firmware Version: 4.AAB 
User Capacity: 160,041,885,696 bytes [160 GB] 
Sector Size: 512 bytes logical/physical 
Device is: In smartctl database [for details use: -P show] 
ATA Version is: 7 
ATA Standard is: Exact ATA specification draft version not indicated 
Local Time is: Mon Sep 8 15:50:48 2014 PDT 
SMART support is: Available - device has SMART capability. 
SMART support is: Enabled 
=== START OF READ SMART DATA SECTION === 
SMART overall-health self-assessment test result: PASSED 

There is a lot more that gets printed, but I cut it out. 

So /dev/sda has clearly failed. 

Take note of the GOOD disk serial number so I leave that one in when I replace it: 
Serial Number: 9RA6DZP8 

Mark and remove failed disk from raid: 

[root@myServer ~]# mdadm --manage /dev/md0 --fail /dev/sda1 
mdadm: set /dev/sda1 faulty in /dev/md0 
[root@myServer ~]# mdadm --manage /dev/md1 --fail /dev/sda2 
mdadm: set /dev/sda2 faulty in /dev/md1 
[root@myServer ~]# mdadm --manage /dev/md2 --fail /dev/sda3 
mdadm: set /dev/sda3 faulty in /dev/md2 
[root@myServer ~]# mdadm --manage /dev/md0 --remove /dev/sda1 
mdadm: hot removed /dev/sda1 
[root@myServer ~]# mdadm --manage /dev/md1 --remove /dev/sda2 
mdadm: hot removed /dev/sda2 
[root@myServer ~]# mdadm --manage /dev/md2 --remove /dev/sda3 
mdadm: hot removed /dev/sda3 

Make sure grub is installed on the good disk and that grub.conf is updated:

[root@myServer ~]# grub-install /dev/sdb 
Installation finished. No error reported. 
This is the contents of the device map /boot/grub/device.map. 
Check if this is correct or not. 
If any of the lines is incorrect, fix it and re-run the script `grub-install'. 
This device map was generated by anaconda 
(hd0) /dev/sda 
(hd1) /dev/sdb 

Take note of the which hd partition corresponds with the good disk, ie hd1 in this case. 

[root@myServer ~]# vim /boot/grub/menu.lst 
Add fallback=1 right after default=0 
Go to the bottom section where you should find some kernel stanzas. 
Copy the first of them and paste the stanza before the first existing stanza; replace root (hd0,0) with root (hd1,0) 
Should look like this: 
[...] 
title CentOS (2.6.18-128.el5) 
root (hd1,0) 
kernel /vmlinuz-2.6.18-128.el5 ro root=/dev/VolGroup00/LogVol00 
initrd /initrd-2.6.18-128.el5.img 
title CentOS (2.6.18-128.el5) 
root (hd0,0) 
kernel /vmlinuz-2.6.18-128.el5 ro root=/dev/ 
initrd /initrd-2.6.18-128.el5.img 

Save and quit 

[root@myServer ~]# mv /boot/initramfs-$(uname -r).img /boot/initramfs-$(uname -r).img.bak 
[root@myServer ~]# mkinitrd /boot/initramfs-$(uname -r).img $(uname -r) 
[root@myServer ~]# init 0 

Swap the bad drive with the new drive and boot the machine. 

Once it's booted: 

Check the device names with cat /proc/mdstat and/or fisk -l. 
The newly installed drive on myServer was named /dev/sda. 

[root@myServer ~]# modeprobe raid1 
[root@myServer ~]# modeprobe linear 

Copy the partitions from one disk to the other:

[root@myServer ~]# sfdisk -d /dev/sdb | sfdisk --force /dev/sda 
[root@myServer ~]# sfdisk -l => sanity check 

Add the new disk to the raid array:

[root@myServer ~]# mdadm --manage /dev/md0 --add /dev/sda1 
mdadm: added /dev/sda1 
[root@myServer ~]# mdadm --manage /dev/md1 --add /dev/sda2 
mdadm: added /dev/sda2 
[root@myServer ~]# mdadm --manage /dev/md2 --add /dev/sda3 
mdadm: added /dev/sda3 

Sanity check:
[root@myServer ~]# cat /proc/mdstat 
Personalities : [raid1] [linear] 
md0 : active raid1 sda1[1] sdb1[0] 
128384 blocks [2/2] [UU] 
md1 : active raid1 sda2[2] sdb2[0] 
16779776 blocks [2/1] [U_] 
[>....................] recovery = 3.2% (548864/16779776) finish=8.8min speed=30492K/sec 
md2 : active raid1 sda3[2] sdb3[0] 
139379840 blocks [2/1] [U_] 
resync=DELAYED 
unused devices: <none> 

That's it! :) 
[[ Category: Ben ]]
[[ Category : Sysadmin ]]
[[Category:Tutorials]]

Mount smallworld disks

2021-01-11T23:18:08Z

Jgutie11:

yum install epel-release -y
yum install ntfs-3g -y
then
ls -ldt /dev/s* |head
to get recent new disks after plugging them in

then
mount /dev/sdap1 /mnt/dsk1
mount /dev/sdaq1 /mnt/dsk2

then, bob is your uncle.

=== If on private network and not set up for NAT to internet, do this... ===
On CentOS 7 systems do:
ifconfig
*find which port is private
nmtui
*enter port settings
*remove existing gateway
*replace with 10.20.0.6
*save and exit
systemctl restart network
systemctl restart NetworkConfig

[[Category:Sysadmin]]

Mount smallworld disks

2021-01-08T18:31:05Z

Jgutie11:

Arthor Documentation for Future Developer

2021-01-07T01:45:50Z

Jgutie11:

Written by Jennifer Young on December 16, 2019. Last edited January 05, 2021

==Install and Set Up on TomCat (Method 1)==
Arthor ran on n-1-136, which runs CentOS Linux release 7.7.1908 (Core). You can check the version of CentOS with the following command
cat /etc/centos-release

Check your current version of Java with the following command:
java -version

On n-1-136 we are running openjdk version "1.8.0_222", OpenJDK Runtime Environment (build 1.8.0_222-b10), and OpenJDK 64-Bit Server VM (build 25.222-b10, mixed mode)
If Java is not installed, install it using yum

==See this wiki page for more detailed information about installing Tomcat on our cluster==
http://wiki.docking.org/index.php/Tomcat_Installation

==Open port for Arthor==
In order for Arthor to be usable in the browser, the port you wish to run it on must be opened.
https://www.thegeekdiary.com/how-to-open-a-ports-in-centos-rhel-7/

===Step 1: Check Port Status===
Check that the port is not open and that Apache is not showing that port.
netstat -na | grep <port number you are checking>

lsof -i -P |grep http

===Step 2: Check Port Status in IP Tables===
iptables-save | grep <port number you are checking>

I skipped Step 3 from the guide, because there was a lot of information in the /etc/services file and I didn't want to edit it and break something.

===Step 4: Open Firewall Ports===
I did not include the zone=public section because the stand-alone servers are usually used for private instances of Arthor and SmallWorld.
Run as root.
firewall-cmd --add-port=<port number you are adding>/tcp --permanent

You need to reload the firewall after a change is made.
firewall-cmd --reload

===Step 5: Check that port is working===
To check that the port is active, run.
iptables -nL

You should see something along the lines of:
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:<port number you're adding> ctstate NEW,UNTRACKED

==How to run standalone Arthor instance==

===Step 1: Use or start a bash shell===
You can check your default shell using
echo $SHELL

If your default shell is csh, use
bash
to start a new bash shell in the current terminal window. Note that echo $SHELL will show you your default shell regardless of the current shell.

===Step 2: Set your environment variables===
export ARTHOR_DIR=/opt/nextmove/arthor/arthor-3.3-centos7
export PATH=$ARTHOR_DIR/bin/:$PATH

Make sure the ARTHOR_DIR variable is set to the directory for the latest version of Arthor or whichever version you would like to test.
The PATH environment variable is needed if you wish to use the Arthor tools from the command line

===Step 3: Run the arthor-server.jar===
java -jar /opt/nextmove/arthor/arthor-3.3-centos7/java/arthor.jar --httpPort <your httpPort>

==Setting environment variables for an Arthor Server==
Set the environment variables in the setenv.sh file. Note: Be sure to edit the file in the directory corresponding to the latest version of TomCat. As of December 2019, we are running 9.0.27 on n-1-136.

vim /opt/tomcat/apache-tomcat-9.0.27/bin/setenv.sh

Add the line below to the setenv.sh file above, or substitute the path to wherever you currently store the arthor.cfg file
export ARTHOR_CONFIG=/usr/local/tomcat/arthor.cfg

Here is an example of the arthor.cfg file:
BinDir=/opt/nextmove/arthor/arthor-3.3-centos7/bin
DataDir=/local2/arthor_local_8081/
MaxConcurrentSearches=6
MaxThreadsPerSearch=8
AutomaticIndex=false
AsyncHitCountMax=1000000
Resolver=https://sw.docking.org/util/smi2mol?smi=%s

=== Configuration Details ===

*'''BinDir''': is the location of the Arthor command line binaries. These are used to generate the Arthor index files and to perform searches directly on n-1-136. An example of this would be using atdbgrep for substructure search.

*'''DataDir''': This is the directory where the Arthor data files live. Location where the index files will be created and loaded from.

*'''MaxConcurrentSearches''': Controls the maximum number of searches that can be run concurrently by setting the database pool size. When switching between a large number of databases it can be useful to have a larger pool size, the only trade off is keeping file pointers open.

*'''MaxThreadsPerSearch''': The number of threads to use for both ATDB and ATFP searches

*Set '''AutomaticIndex''' to false if you don't want new smiles files added to the data directory to be indexed automatically

*'''AsyncHitCountMax''': The upper-bound for the number of hits to retrieve in background searches.

*'''Resolver''': Using Smallworld API, allows input box to take in a SMILE format and automatically draw on the board.

Check Arthor manual for more configuration options.

==Background==
Before working with Arthor, it is recommended that you familiarize yourself with the Arthor documentation. Some useful pages to look at include 3-5, 22-25 and 33-39. Of course, reading everything would be the best!

==Checking Disk Space Usage==
Before building arthor indexes, it's always a good thing to check what percent of the memory is being used. Try to be cautious with how much memory you have left, and make sure to check while building indexes to make sure that you have enough space. To check, run the following command:

df -h /<directory with disc>

==Building Large Databases==
At the moment, we are building databases of size 500M molecules by merging smile files. There are multiple methods of trying to create large databases, one being merging based off of the same H?? prefix and stopping once the database reaches > 500M molecules (or whatever upperbound you want to use). Here is some python code that simulates this merging process. Essentially the program takes all of the .smi files within an input directory, sorts them lexiographically, and begins merging these .smi files together in order until the size reaches > 500M molecules.

Feel free to modify it if you think a better method exists.

import subprocess
import sys
import os

from os import listdir
from os.path import isfile, join

mypath = "<Path to directory holding .smi files>"
onlyfiles = [f for f in listdir(mypath) if isfile(join(mypath, f))]
onlyfiles.sort()

create_fp = True
cur_mols = 0
lower_bound = 500000000
upper_bound = 600000000
files_to_merge = []

def merge_files(f_t_m):
arr = f_t_m[0].split(".")
arr2 = f_t_m[len(f_t_m) - 1].split(".")
file_name_merge = (arr[0] + "_" + arr2[0] + ".smi")
print ("File being created: " + file_name_merge)

for file in f_t_m:
tmp = file.split(".")
process = subprocess.Popen("cat " + join(mypath, file) + " >> " + file_name_merge, shell=True)
process.wait()

for file in onlyfiles:
arr = file.split(".")

if (arr[len(arr) - 1] == "smi"):
print("Working with " + file)
mol = sum(1 for line in open(join(mypath, file)))
print(file, mol, cur_mols)

if (cur_mols + mol > lower_bound):
if (cur_mols + mol < upper_bound):
files_to_merge.append(file)
merge_files(files_to_merge)
cur_mols = 0
files_to_merge.clear()
else:
merge_files(files_to_merge)
files_to_merge.clear()
files_to_merge.append(file)
merge_files(files_to_merge)
cur_mols = 0
files_to_merge.clear()
else:
cur_mols += mol
files_to_merge.append(file)

if (len(files_to_merge) != 0):
merge_files(files_to_merge)

==Building Arthor Indexes==
Once you've merged the .smi files together, it's time to start building the databases themselves. To do this we use the command

smi2atdb -j 0 -p <The .smi file> <The .atdb>

The flag "-j 0" enables parallel generation and utilizes all available processors to generate the .atdb file. The "-p" flag stores the offset position in the ATDB file. Since we're building indexes for the Web Application, you must use the "-p" flag when building indexes. Please note that the name of the .smi file should also be the name of the .atdb file. That way, the Web Application knows to use these files together and correctly display the required images. Refer to pages 33-34 in the Arthor documentation for more information.

If there are too many large .smi files and you do not want to manually build each .atdb file, you can use this python script which takes all of the .smi files in the current directory and converts them to .atdb files. Make sure to modify mypath to the directory containing the .smi files. You can change the variable "create_fp" to false if you don't want to create .atdb.fp files (refer to page 9 in the Arthor documentation).

import subprocess
import sys
import os

from os import listdir
from os.path import isfile, join

mypath = "<Path containing the .smi files"
onlyfiles = [f for f in listdir(mypath) if isfile(join(mypath, f))]

create_fp = True

for file in onlyfiles:
arr = file.split(".")

if (arr[len(arr) - 1] == "smi"):
process = subprocess.Popen("/nfs/ex9/work/xyz/psql/arthor-3.3-centos7/bin/smi2atdb -j 0 -p {0} {1}.atdb".format(join(mypath, file), arr[0]), shell=True)
process.wait()

print("SUCCESS! {0}.atdb file was created!".format(arr[0]))

if (create_fp):
process = subprocess.Popen("/nfs/ex9/work/xyz/psql/arthor-3.3-centos7/bin/atdb2fp -j 0 {0}.atdb".format(arr[0]), shell=True)
process.wait()

print("SUCCESS! {0}.atdb.fp file was created!".format(arr[0]))

==Uploading Indexes to the Web Application==
One can upload indexes to the Web Application by changing the "DATADIR" variable in the arthor.cfg file to the directory holding the .atdb files. This is already set up on n-1-136 and n-5-34.

==Virtual Memory==
In addition to modifying the arthor.cfg file, virtual memory can also be used to make queries faster. There can still be More information can be found in pages 10-16 in the Arthor Documentation.

==Setting up Round Table==
This is a new feature in Arthor 3.0 and is currently beta (January 2020). See section 2.4 in the manual
As explained in the manual, "Round Table allows you to serve and split chemical searches across multiple host machines. The implementation provides a lightweight proxy that forwards requests to other Arthor host servers that do the actual search. Communication is done using the existing Web APIs.

Since Arthor requires CentOS 7, as of January 2020 we have 6 servers that are capable of running Arthor with Round Table.

===Setting up Host Server===
If we want to add machines to the Round Table, for example 'nun' and 'samekh', we need to edit their arthor.cfg file so that when our Local Machine passes commands these secondary servers know to perform the search they are given.

$ cat arthor.cfg
MaxThreadsPerSearch=4
AutomaticIndex=false
DATADIR=<Directory where smiles are located>

We then run the jar server on each of these host machines containing data on any available port.

java -jar /nfs/ex9/work/xyz/psql/arthor-3.3-centos7/java/arthor.jar --httpPort <port>

For our local machine, the arthor.cfg file will look different.

$ cat arthor.cfg
[RoundTable]
RemoteClient=http://skynet:<port number where jar server is running>/
RemoteClient=http://hal:<port number where jar server is running>/

Please refer to Section 2 in the RoundTable Documentation file (pages 6-8) for more useful information on configuration.

Then run the following command on n-1-136:

java -jar /nfs/ex9/work/xyz/psql/arthor-3.3-centos7/java/arthor.jar --httpPort <port>

===Public Arthor===
{| class="wikitable"
|-
! CentOS 7 Machine
! Port
! Total Files Size
! Arthor Install Location
! Round Table Data Directory
! Active
|-
| samekh
| 10.20.0.41:8000
| 2.4TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/public_arthor/
| active
|-
| nun
| 10.20.0.40:8000
| 2.4TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/public_arthor/
| active
|-
| n-9-22
| 10.20.9.22:8000
| 2.4TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /export/db4/public_arthor/
| active
|-
|}

===Arthor Round Table Head===
{| class="wikitable"
|-
! CentOS 7 Machine
! Port
! Database
! Arthor Install Location
! Round Table Data Directory
! Active
|-
| samekh
| 10.20.0.41:8080
| Enamine_REAL_Q2-2020-All-13B
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
| nun
| 10.20.0.40:8080
| Enamine_REAL_Q2-2020-All-41B
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
|}

===Arthor Round Table Nodes===
{| class="wikitable"
|-
! CentOS 7 Machine
! Port
! Database
! Total Files Size
! Arthor Install Location
! Round Table Data Directory
!Active
|-
| samekh
| 10.20.0.41:8008
| Enamine_REAL_Q2-2020-All-13B (26 slices)
| 4.5TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
| nun
| 10.20.0.40:8008
| Enamine_REAL_Space_June_2020_S41B (aa-ae, 5 slices), Enamine_REAL_Space_June_2020_M41B (af-am, 8 slices), zinc22_2d (H04~H25, 22 slices)
| 5.6TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
| n-1-17
| 10.20.1.17:8008
| Enamine_REAL_Space_June_2020_M41B (aa-ae, 5 slices), zinc22_2d (H25~H29, 4 slices)
| 3.7TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
| n-5-32
| 10.20.5.32:8008
| Enamine_REAL_Space_June_2020_M41B (an~az, 13 slices), zinc22_2d (H30, 1 slice)
| 5.6TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
| n-5-33
| 10.20.5.33:8008
| Enamine_REAL_Space_June_2020_M41B (ba-bl, 12 slices)
| 5.3TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
|}

===Arthor Local 8081 (Datasets all local to samekh/nun)===
{| class="wikitable"
|-
! CentOS 7 Machine
! Port
! Database
! Total Files Size
! Arthor Install Location
! Round Table Data Directory
!Active
|-
| samekh
| 10.20.0.41:8081
| Enamine_REAL_Q2-2020-All-13B (26 slices)
| 4.5TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_local_8081/
| active
|-
| nun
| 10.20.0.40:8081
| Enamine_REAL_Space_June_2020_S41B (aa-ae, 5 slices), Enamine_REAL_Space_June_2020_M41B (aa-an, 14 slices)
| 4.3TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_local_8081/
| active
|-
|}

== Customizing Arthor Code to our needs ==
If Arthor Sever is launched through "java -jar /opt/nextmove/arthor/arthor-3.3-centos7/java/arthor.jar --httpPort=<port>", find the directory where this line of code was executed. Once found do '''ls -a''', there should be a hidden directory called .extract.
=== Change Arthor Download Size (Hardcoded) ===
vim .extract/webapps/ROOT/WEB-INF/static/js/index.js
?#arthor_sdf_link //search this
*Look for 0!==arguments[0]?arguments[0]:<number>
*Change number to desirec amount
=== Take out Similarity Button ===
vim .extract/webapps/ROOT/WEB-INF/static/index.html
?Similarity //search this
*Comment out this line '''< li value="Similarity" onclick="setSearchType(this)" class="first"> Similarity </li >''' //added spaces at the beginning and end so prevent wiki from converting it
=== Hyperlink to zinc20 ===
vim .extract/webapps/ROOT/WEB-INF/static/js/index.js
?table_name //search this
*find this line "" + h + ""
*replace with '''"<a target='_blank' href='https://zinc20.docking.org/substances/"+h+"'>" + h + "</a>"''' //added spaces at the beginning and end so prevent wiki from converting it

Arthor Documentation for Future Developer

2021-01-07T01:04:02Z

Jgutie11: /* Configuration Details */

Written by Jennifer Young on December 16, 2019. Last edited January 05, 2021

==Install and Set Up on TomCat (Method 1)==
Arthor ran on n-1-136, which runs CentOS Linux release 7.7.1908 (Core). You can check the version of CentOS with the following command
cat /etc/centos-release

Check your current version of Java with the following command:
java -version

On n-1-136 we are running openjdk version "1.8.0_222", OpenJDK Runtime Environment (build 1.8.0_222-b10), and OpenJDK 64-Bit Server VM (build 25.222-b10, mixed mode)
If Java is not installed, install it using yum

==See this wiki page for more detailed information about installing Tomcat on our cluster==
http://wiki.docking.org/index.php/Tomcat_Installation

==Open port for Arthor==
In order for Arthor to be usable in the browser, the port you wish to run it on must be opened.
https://www.thegeekdiary.com/how-to-open-a-ports-in-centos-rhel-7/

===Step 1: Check Port Status===
Check that the port is not open and that Apache is not showing that port.
netstat -na | grep <port number you are checking>

lsof -i -P |grep http

===Step 2: Check Port Status in IP Tables===
iptables-save | grep <port number you are checking>

I skipped Step 3 from the guide, because there was a lot of information in the /etc/services file and I didn't want to edit it and break something.

===Step 4: Open Firewall Ports===
I did not include the zone=public section because the stand-alone servers are usually used for private instances of Arthor and SmallWorld.
Run as root.
firewall-cmd --add-port=<port number you are adding>/tcp --permanent

You need to reload the firewall after a change is made.
firewall-cmd --reload

===Step 5: Check that port is working===
To check that the port is active, run.
iptables -nL

You should see something along the lines of:
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:<port number you're adding> ctstate NEW,UNTRACKED

==How to run standalone Arthor instance==

===Step 1: Use or start a bash shell===
You can check your default shell using
echo $SHELL

If your default shell is csh, use
bash
to start a new bash shell in the current terminal window. Note that echo $SHELL will show you your default shell regardless of the current shell.

===Step 2: Set your environment variables===
export ARTHOR_DIR=/opt/nextmove/arthor/arthor-3.3-centos7
export PATH=$ARTHOR_DIR/bin/:$PATH

Make sure the ARTHOR_DIR variable is set to the directory for the latest version of Arthor or whichever version you would like to test.
The PATH environment variable is needed if you wish to use the Arthor tools from the command line

===Step 3: Run the arthor-server.jar===
java -jar /opt/nextmove/arthor/arthor-3.3-centos7/java/arthor.jar --httpPort <your httpPort>

==Setting environment variables for an Arthor Server==
Set the environment variables in the setenv.sh file. Note: Be sure to edit the file in the directory corresponding to the latest version of TomCat. As of December 2019, we are running 9.0.27 on n-1-136.

vim /opt/tomcat/apache-tomcat-9.0.27/bin/setenv.sh

Add the line below to the setenv.sh file above, or substitute the path to wherever you currently store the arthor.cfg file
export ARTHOR_CONFIG=/usr/local/tomcat/arthor.cfg

Here is an example of the arthor.cfg file:
BinDir=/opt/nextmove/arthor/arthor-3.3-centos7/bin
DataDir=/local2/arthor_local_8081/
MaxConcurrentSearches=6
MaxThreadsPerSearch=8
AutomaticIndex=false
AsyncHitCountMax=1000000
Resolver=https://sw.docking.org/util/smi2mol?smi=%s

=== Configuration Details ===

*'''BinDir''': is the location of the Arthor command line binaries. These are used to generate the Arthor index files and to perform searches directly on n-1-136. An example of this would be using atdbgrep for substructure search.

*'''DataDir''': This is the directory where the Arthor data files live. Location where the index files will be created and loaded from.

*'''MaxConcurrentSearches''': Controls the maximum number of searches that can be run concurrently by setting the database pool size. When switching between a large number of databases it can be useful to have a larger pool size, the only trade off is keeping file pointers open.

*'''MaxThreadsPerSearch''': The number of threads to use for both ATDB and ATFP searches

*Set '''AutomaticIndex''' to false if you don't want new smiles files added to the data directory to be indexed automatically

*'''AsyncHitCountMax''': The upper-bound for the number of hits to retrieve in background searches.

*'''Resolver''': Using Smallworld API, allows input box to take in a SMILE format and automatically draw on the board.

Check Arthor manual for more configuration options.

==Background==
Before working with Arthor, it is recommended that you familiarize yourself with the Arthor documentation. Some useful pages to look at include 3-5, 22-25 and 33-39. Of course, reading everything would be the best!

==Checking Disk Space Usage==
Before building arthor indexes, it's always a good thing to check what percent of the memory is being used. Try to be cautious with how much memory you have left, and make sure to check while building indexes to make sure that you have enough space. To check, run the following command:

df -h /<directory with disc>

==Building Large Databases==
At the moment, we are building databases of size 500M molecules by merging smile files. There are multiple methods of trying to create large databases, one being merging based off of the same H?? prefix and stopping once the database reaches > 500M molecules (or whatever upperbound you want to use). Here is some python code that simulates this merging process. Essentially the program takes all of the .smi files within an input directory, sorts them lexiographically, and begins merging these .smi files together in order until the size reaches > 500M molecules.

Feel free to modify it if you think a better method exists.

import subprocess
import sys
import os

from os import listdir
from os.path import isfile, join

mypath = "<Path to directory holding .smi files>"
onlyfiles = [f for f in listdir(mypath) if isfile(join(mypath, f))]
onlyfiles.sort()

create_fp = True
cur_mols = 0
lower_bound = 500000000
upper_bound = 600000000
files_to_merge = []

def merge_files(f_t_m):
arr = f_t_m[0].split(".")
arr2 = f_t_m[len(f_t_m) - 1].split(".")
file_name_merge = (arr[0] + "_" + arr2[0] + ".smi")
print ("File being created: " + file_name_merge)

for file in f_t_m:
tmp = file.split(".")
process = subprocess.Popen("cat " + join(mypath, file) + " >> " + file_name_merge, shell=True)
process.wait()

for file in onlyfiles:
arr = file.split(".")

if (arr[len(arr) - 1] == "smi"):
print("Working with " + file)
mol = sum(1 for line in open(join(mypath, file)))
print(file, mol, cur_mols)

if (cur_mols + mol > lower_bound):
if (cur_mols + mol < upper_bound):
files_to_merge.append(file)
merge_files(files_to_merge)
cur_mols = 0
files_to_merge.clear()
else:
merge_files(files_to_merge)
files_to_merge.clear()
files_to_merge.append(file)
merge_files(files_to_merge)
cur_mols = 0
files_to_merge.clear()
else:
cur_mols += mol
files_to_merge.append(file)

if (len(files_to_merge) != 0):
merge_files(files_to_merge)

==Building Arthor Indexes==
Once you've merged the .smi files together, it's time to start building the databases themselves. To do this we use the command

smi2atdb -j 0 -p <The .smi file> <The .atdb>

The flag "-j 0" enables parallel generation and utilizes all available processors to generate the .atdb file. The "-p" flag stores the offset position in the ATDB file. Since we're building indexes for the Web Application, you must use the "-p" flag when building indexes. Please note that the name of the .smi file should also be the name of the .atdb file. That way, the Web Application knows to use these files together and correctly display the required images. Refer to pages 33-34 in the Arthor documentation for more information.

If there are too many large .smi files and you do not want to manually build each .atdb file, you can use this python script which takes all of the .smi files in the current directory and converts them to .atdb files. Make sure to modify mypath to the directory containing the .smi files. You can change the variable "create_fp" to false if you don't want to create .atdb.fp files (refer to page 9 in the Arthor documentation).

import subprocess
import sys
import os

from os import listdir
from os.path import isfile, join

mypath = "<Path containing the .smi files"
onlyfiles = [f for f in listdir(mypath) if isfile(join(mypath, f))]

create_fp = True

for file in onlyfiles:
arr = file.split(".")

if (arr[len(arr) - 1] == "smi"):
process = subprocess.Popen("/nfs/ex9/work/xyz/psql/arthor-3.3-centos7/bin/smi2atdb -j 0 -p {0} {1}.atdb".format(join(mypath, file), arr[0]), shell=True)
process.wait()

print("SUCCESS! {0}.atdb file was created!".format(arr[0]))

if (create_fp):
process = subprocess.Popen("/nfs/ex9/work/xyz/psql/arthor-3.3-centos7/bin/atdb2fp -j 0 {0}.atdb".format(arr[0]), shell=True)
process.wait()

print("SUCCESS! {0}.atdb.fp file was created!".format(arr[0]))

==Uploading Indexes to the Web Application==
One can upload indexes to the Web Application by changing the "DATADIR" variable in the arthor.cfg file to the directory holding the .atdb files. This is already set up on n-1-136 and n-5-34.

==Virtual Memory==
In addition to modifying the arthor.cfg file, virtual memory can also be used to make queries faster. There can still be More information can be found in pages 10-16 in the Arthor Documentation.

==Setting up Round Table==
This is a new feature in Arthor 3.0 and is currently beta (January 2020). See section 2.4 in the manual
As explained in the manual, "Round Table allows you to serve and split chemical searches across multiple host machines. The implementation provides a lightweight proxy that forwards requests to other Arthor host servers that do the actual search. Communication is done using the existing Web APIs.

Since Arthor requires CentOS 7, as of January 2020 we have 6 servers that are capable of running Arthor with Round Table.

===Setting up Host Server===
If we want to add machines to the Round Table, for example 'nun' and 'samekh', we need to edit their arthor.cfg file so that when our Local Machine passes commands these secondary servers know to perform the search they are given.

$ cat arthor.cfg
MaxThreadsPerSearch=4
AutomaticIndex=false
DATADIR=<Directory where smiles are located>

We then run the jar server on each of these host machines containing data on any available port.

java -jar /nfs/ex9/work/xyz/psql/arthor-3.3-centos7/java/arthor.jar --httpPort <port>

For our local machine, the arthor.cfg file will look different.

$ cat arthor.cfg
[RoundTable]
RemoteClient=http://skynet:<port number where jar server is running>/
RemoteClient=http://hal:<port number where jar server is running>/

Please refer to Section 2 in the RoundTable Documentation file (pages 6-8) for more useful information on configuration.

Then run the following command on n-1-136:

java -jar /nfs/ex9/work/xyz/psql/arthor-3.3-centos7/java/arthor.jar --httpPort <port>

===Public Arthor===
{| class="wikitable"
|-
! CentOS 7 Machine
! Port
! Total Files Size
! Arthor Install Location
! Round Table Data Directory
! Active
|-
| samekh
| 10.20.0.41:8000
| 2.4TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/public_arthor/
| active
|-
| nun
| 10.20.0.40:8000
| 2.4TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/public_arthor/
| active
|-
| n-9-22
| 10.20.9.22:8000
| 2.4TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /export/db4/public_arthor/
| active
|-
|}

===Arthor Round Table Head===
{| class="wikitable"
|-
! CentOS 7 Machine
! Port
! Database
! Arthor Install Location
! Round Table Data Directory
! Active
|-
| samekh
| 10.20.0.41:8080
| Enamine_REAL_Q2-2020-All-13B
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
| nun
| 10.20.0.40:8080
| Enamine_REAL_Q2-2020-All-41B
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
|}

===Arthor Round Table Nodes===
{| class="wikitable"
|-
! CentOS 7 Machine
! Port
! Database
! Total Files Size
! Arthor Install Location
! Round Table Data Directory
!Active
|-
| samekh
| 10.20.0.41:8008
| Enamine_REAL_Q2-2020-All-13B (26 slices)
| 4.5TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
| nun
| 10.20.0.40:8008
| Enamine_REAL_Space_June_2020_S41B (aa-ae, 5 slices), Enamine_REAL_Space_June_2020_M41B (af-am, 8 slices), zinc22_2d (H04~H25, 22 slices)
| 5.6TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
| n-1-17
| 10.20.1.17:8008
| Enamine_REAL_Space_June_2020_M41B (aa-ae, 5 slices), zinc22_2d (H25~H29, 4 slices)
| 3.7TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
| n-5-32
| 10.20.5.32:8008
| Enamine_REAL_Space_June_2020_M41B (an~az, 13 slices), zinc22_2d (H30, 1 slice)
| 5.6TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
| n-5-33
| 10.20.5.33:8008
| Enamine_REAL_Space_June_2020_M41B (ba-bl, 12 slices)
| 5.3TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
|}

===Arthor Local 8081 (Datasets all local to samekh/nun)===
{| class="wikitable"
|-
! CentOS 7 Machine
! Port
! Database
! Total Files Size
! Arthor Install Location
! Round Table Data Directory
!Active
|-
| samekh
| 10.20.0.41:8081
| Enamine_REAL_Q2-2020-All-13B (26 slices)
| 4.5TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_local_8081/
| active
|-
| nun
| 10.20.0.40:8081
| Enamine_REAL_Space_June_2020_S41B (aa-ae, 5 slices), Enamine_REAL_Space_June_2020_M41B (aa-an, 14 slices)
| 4.3TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_local_8081/
| active
|-
|}

Arthor Documentation for Future Developer

2021-01-07T01:03:23Z

Jgutie11: /* Further Arthor Optimizations */

Written by Jennifer Young on December 16, 2019. Last edited January 05, 2021

==Install and Set Up on TomCat (Method 1)==
Arthor ran on n-1-136, which runs CentOS Linux release 7.7.1908 (Core). You can check the version of CentOS with the following command
cat /etc/centos-release

Check your current version of Java with the following command:
java -version

On n-1-136 we are running openjdk version "1.8.0_222", OpenJDK Runtime Environment (build 1.8.0_222-b10), and OpenJDK 64-Bit Server VM (build 25.222-b10, mixed mode)
If Java is not installed, install it using yum

==See this wiki page for more detailed information about installing Tomcat on our cluster==
http://wiki.docking.org/index.php/Tomcat_Installation

==Open port for Arthor==
In order for Arthor to be usable in the browser, the port you wish to run it on must be opened.
https://www.thegeekdiary.com/how-to-open-a-ports-in-centos-rhel-7/

===Step 1: Check Port Status===
Check that the port is not open and that Apache is not showing that port.
netstat -na | grep <port number you are checking>

lsof -i -P |grep http

===Step 2: Check Port Status in IP Tables===
iptables-save | grep <port number you are checking>

I skipped Step 3 from the guide, because there was a lot of information in the /etc/services file and I didn't want to edit it and break something.

===Step 4: Open Firewall Ports===
I did not include the zone=public section because the stand-alone servers are usually used for private instances of Arthor and SmallWorld.
Run as root.
firewall-cmd --add-port=<port number you are adding>/tcp --permanent

You need to reload the firewall after a change is made.
firewall-cmd --reload

===Step 5: Check that port is working===
To check that the port is active, run.
iptables -nL

You should see something along the lines of:
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:<port number you're adding> ctstate NEW,UNTRACKED

==How to run standalone Arthor instance==

===Step 1: Use or start a bash shell===
You can check your default shell using
echo $SHELL

If your default shell is csh, use
bash
to start a new bash shell in the current terminal window. Note that echo $SHELL will show you your default shell regardless of the current shell.

===Step 2: Set your environment variables===
export ARTHOR_DIR=/opt/nextmove/arthor/arthor-3.3-centos7
export PATH=$ARTHOR_DIR/bin/:$PATH

Make sure the ARTHOR_DIR variable is set to the directory for the latest version of Arthor or whichever version you would like to test.
The PATH environment variable is needed if you wish to use the Arthor tools from the command line

===Step 3: Run the arthor-server.jar===
java -jar /opt/nextmove/arthor/arthor-3.3-centos7/java/arthor.jar --httpPort <your httpPort>

==Setting environment variables for an Arthor Server==
Set the environment variables in the setenv.sh file. Note: Be sure to edit the file in the directory corresponding to the latest version of TomCat. As of December 2019, we are running 9.0.27 on n-1-136.

vim /opt/tomcat/apache-tomcat-9.0.27/bin/setenv.sh

Add the line below to the setenv.sh file above, or substitute the path to wherever you currently store the arthor.cfg file
export ARTHOR_CONFIG=/usr/local/tomcat/arthor.cfg

Here is an example of the arthor.cfg file:
BinDir=/opt/nextmove/arthor/arthor-3.3-centos7/bin
DataDir=/local2/arthor_local_8081/
MaxConcurrentSearches=6
MaxThreadsPerSearch=8
AutomaticIndex=false
AsyncHitCountMax=1000000
Resolver=https://sw.docking.org/util/smi2mol?smi=%s

=== Configuration Details ===

*'''BinDir''': is the location of the Arthor command line binaries. These are used to generate the Arthor index files and to perform searches directly on n-1-136. An example of this would be using atdbgrep for substructure search.

*'''DataDir''': This is the directory where the Arthor data files live. Location where the index files will be created and loaded from.

*'''MaxConcurrentSearches''': Controls the maximum number of searches that can be run concurrently by setting the database pool size. When switching between a large number of databases it can be useful to have a larger pool size, the only trade off is keeping file pointers open.

*'''MaxThreadsPerSearch''': The number of threads to use for both ATDB and ATFP searches

*Set '''AutomaticIndex''' to false if you don't want new smiles files added to the data directory to be indexed automatically

*'''AsyncHitCountMax''': The upper-bound for the number of hits to retrieve in background searches.

*'''Resolver''': Using Smallworld API, allows input box to take in a SMILE format and automatically draw on the board.

==Background==
Before working with Arthor, it is recommended that you familiarize yourself with the Arthor documentation. Some useful pages to look at include 3-5, 22-25 and 33-39. Of course, reading everything would be the best!

==Checking Disk Space Usage==
Before building arthor indexes, it's always a good thing to check what percent of the memory is being used. Try to be cautious with how much memory you have left, and make sure to check while building indexes to make sure that you have enough space. To check, run the following command:

df -h /<directory with disc>

==Building Large Databases==
At the moment, we are building databases of size 500M molecules by merging smile files. There are multiple methods of trying to create large databases, one being merging based off of the same H?? prefix and stopping once the database reaches > 500M molecules (or whatever upperbound you want to use). Here is some python code that simulates this merging process. Essentially the program takes all of the .smi files within an input directory, sorts them lexiographically, and begins merging these .smi files together in order until the size reaches > 500M molecules.

Feel free to modify it if you think a better method exists.

import subprocess
import sys
import os

from os import listdir
from os.path import isfile, join

mypath = "<Path to directory holding .smi files>"
onlyfiles = [f for f in listdir(mypath) if isfile(join(mypath, f))]
onlyfiles.sort()

create_fp = True
cur_mols = 0
lower_bound = 500000000
upper_bound = 600000000
files_to_merge = []

def merge_files(f_t_m):
arr = f_t_m[0].split(".")
arr2 = f_t_m[len(f_t_m) - 1].split(".")
file_name_merge = (arr[0] + "_" + arr2[0] + ".smi")
print ("File being created: " + file_name_merge)

for file in f_t_m:
tmp = file.split(".")
process = subprocess.Popen("cat " + join(mypath, file) + " >> " + file_name_merge, shell=True)
process.wait()

for file in onlyfiles:
arr = file.split(".")

if (arr[len(arr) - 1] == "smi"):
print("Working with " + file)
mol = sum(1 for line in open(join(mypath, file)))
print(file, mol, cur_mols)

if (cur_mols + mol > lower_bound):
if (cur_mols + mol < upper_bound):
files_to_merge.append(file)
merge_files(files_to_merge)
cur_mols = 0
files_to_merge.clear()
else:
merge_files(files_to_merge)
files_to_merge.clear()
files_to_merge.append(file)
merge_files(files_to_merge)
cur_mols = 0
files_to_merge.clear()
else:
cur_mols += mol
files_to_merge.append(file)

if (len(files_to_merge) != 0):
merge_files(files_to_merge)

==Building Arthor Indexes==
Once you've merged the .smi files together, it's time to start building the databases themselves. To do this we use the command

smi2atdb -j 0 -p <The .smi file> <The .atdb>

The flag "-j 0" enables parallel generation and utilizes all available processors to generate the .atdb file. The "-p" flag stores the offset position in the ATDB file. Since we're building indexes for the Web Application, you must use the "-p" flag when building indexes. Please note that the name of the .smi file should also be the name of the .atdb file. That way, the Web Application knows to use these files together and correctly display the required images. Refer to pages 33-34 in the Arthor documentation for more information.

If there are too many large .smi files and you do not want to manually build each .atdb file, you can use this python script which takes all of the .smi files in the current directory and converts them to .atdb files. Make sure to modify mypath to the directory containing the .smi files. You can change the variable "create_fp" to false if you don't want to create .atdb.fp files (refer to page 9 in the Arthor documentation).

import subprocess
import sys
import os

from os import listdir
from os.path import isfile, join

mypath = "<Path containing the .smi files"
onlyfiles = [f for f in listdir(mypath) if isfile(join(mypath, f))]

create_fp = True

for file in onlyfiles:
arr = file.split(".")

if (arr[len(arr) - 1] == "smi"):
process = subprocess.Popen("/nfs/ex9/work/xyz/psql/arthor-3.3-centos7/bin/smi2atdb -j 0 -p {0} {1}.atdb".format(join(mypath, file), arr[0]), shell=True)
process.wait()

print("SUCCESS! {0}.atdb file was created!".format(arr[0]))

if (create_fp):
process = subprocess.Popen("/nfs/ex9/work/xyz/psql/arthor-3.3-centos7/bin/atdb2fp -j 0 {0}.atdb".format(arr[0]), shell=True)
process.wait()

print("SUCCESS! {0}.atdb.fp file was created!".format(arr[0]))

==Uploading Indexes to the Web Application==
One can upload indexes to the Web Application by changing the "DATADIR" variable in the arthor.cfg file to the directory holding the .atdb files. This is already set up on n-1-136 and n-5-34.

==Virtual Memory==
In addition to modifying the arthor.cfg file, virtual memory can also be used to make queries faster. There can still be More information can be found in pages 10-16 in the Arthor Documentation.

==Setting up Round Table==
This is a new feature in Arthor 3.0 and is currently beta (January 2020). See section 2.4 in the manual
As explained in the manual, "Round Table allows you to serve and split chemical searches across multiple host machines. The implementation provides a lightweight proxy that forwards requests to other Arthor host servers that do the actual search. Communication is done using the existing Web APIs.

Since Arthor requires CentOS 7, as of January 2020 we have 6 servers that are capable of running Arthor with Round Table.

===Setting up Host Server===
If we want to add machines to the Round Table, for example 'nun' and 'samekh', we need to edit their arthor.cfg file so that when our Local Machine passes commands these secondary servers know to perform the search they are given.

$ cat arthor.cfg
MaxThreadsPerSearch=4
AutomaticIndex=false
DATADIR=<Directory where smiles are located>

We then run the jar server on each of these host machines containing data on any available port.

java -jar /nfs/ex9/work/xyz/psql/arthor-3.3-centos7/java/arthor.jar --httpPort <port>

For our local machine, the arthor.cfg file will look different.

$ cat arthor.cfg
[RoundTable]
RemoteClient=http://skynet:<port number where jar server is running>/
RemoteClient=http://hal:<port number where jar server is running>/

Please refer to Section 2 in the RoundTable Documentation file (pages 6-8) for more useful information on configuration.

Then run the following command on n-1-136:

java -jar /nfs/ex9/work/xyz/psql/arthor-3.3-centos7/java/arthor.jar --httpPort <port>

===Public Arthor===
{| class="wikitable"
|-
! CentOS 7 Machine
! Port
! Total Files Size
! Arthor Install Location
! Round Table Data Directory
! Active
|-
| samekh
| 10.20.0.41:8000
| 2.4TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/public_arthor/
| active
|-
| nun
| 10.20.0.40:8000
| 2.4TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/public_arthor/
| active
|-
| n-9-22
| 10.20.9.22:8000
| 2.4TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /export/db4/public_arthor/
| active
|-
|}

===Arthor Round Table Head===
{| class="wikitable"
|-
! CentOS 7 Machine
! Port
! Database
! Arthor Install Location
! Round Table Data Directory
! Active
|-
| samekh
| 10.20.0.41:8080
| Enamine_REAL_Q2-2020-All-13B
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
| nun
| 10.20.0.40:8080
| Enamine_REAL_Q2-2020-All-41B
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
|}

===Arthor Round Table Nodes===
{| class="wikitable"
|-
! CentOS 7 Machine
! Port
! Database
! Total Files Size
! Arthor Install Location
! Round Table Data Directory
!Active
|-
| samekh
| 10.20.0.41:8008
| Enamine_REAL_Q2-2020-All-13B (26 slices)
| 4.5TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
| nun
| 10.20.0.40:8008
| Enamine_REAL_Space_June_2020_S41B (aa-ae, 5 slices), Enamine_REAL_Space_June_2020_M41B (af-am, 8 slices), zinc22_2d (H04~H25, 22 slices)
| 5.6TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
| n-1-17
| 10.20.1.17:8008
| Enamine_REAL_Space_June_2020_M41B (aa-ae, 5 slices), zinc22_2d (H25~H29, 4 slices)
| 3.7TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
| n-5-32
| 10.20.5.32:8008
| Enamine_REAL_Space_June_2020_M41B (an~az, 13 slices), zinc22_2d (H30, 1 slice)
| 5.6TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
| n-5-33
| 10.20.5.33:8008
| Enamine_REAL_Space_June_2020_M41B (ba-bl, 12 slices)
| 5.3TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
|}

===Arthor Local 8081 (Datasets all local to samekh/nun)===
{| class="wikitable"
|-
! CentOS 7 Machine
! Port
! Database
! Total Files Size
! Arthor Install Location
! Round Table Data Directory
!Active
|-
| samekh
| 10.20.0.41:8081
| Enamine_REAL_Q2-2020-All-13B (26 slices)
| 4.5TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_local_8081/
| active
|-
| nun
| 10.20.0.40:8081
| Enamine_REAL_Space_June_2020_S41B (aa-ae, 5 slices), Enamine_REAL_Space_June_2020_M41B (aa-an, 14 slices)
| 4.3TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_local_8081/
| active
|-
|}

Arthor Documentation for Future Developer

2021-01-07T01:02:45Z

Jgutie11: /* Setting environment variables for an Arthor Server */

Written by Jennifer Young on December 16, 2019. Last edited January 05, 2021

==Install and Set Up on TomCat (Method 1)==
Arthor ran on n-1-136, which runs CentOS Linux release 7.7.1908 (Core). You can check the version of CentOS with the following command
cat /etc/centos-release

Check your current version of Java with the following command:
java -version

On n-1-136 we are running openjdk version "1.8.0_222", OpenJDK Runtime Environment (build 1.8.0_222-b10), and OpenJDK 64-Bit Server VM (build 25.222-b10, mixed mode)
If Java is not installed, install it using yum

==See this wiki page for more detailed information about installing Tomcat on our cluster==
http://wiki.docking.org/index.php/Tomcat_Installation

==Open port for Arthor==
In order for Arthor to be usable in the browser, the port you wish to run it on must be opened.
https://www.thegeekdiary.com/how-to-open-a-ports-in-centos-rhel-7/

===Step 1: Check Port Status===
Check that the port is not open and that Apache is not showing that port.
netstat -na | grep <port number you are checking>

lsof -i -P |grep http

===Step 2: Check Port Status in IP Tables===
iptables-save | grep <port number you are checking>

I skipped Step 3 from the guide, because there was a lot of information in the /etc/services file and I didn't want to edit it and break something.

===Step 4: Open Firewall Ports===
I did not include the zone=public section because the stand-alone servers are usually used for private instances of Arthor and SmallWorld.
Run as root.
firewall-cmd --add-port=<port number you are adding>/tcp --permanent

You need to reload the firewall after a change is made.
firewall-cmd --reload

===Step 5: Check that port is working===
To check that the port is active, run.
iptables -nL

You should see something along the lines of:
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:<port number you're adding> ctstate NEW,UNTRACKED

==How to run standalone Arthor instance==

===Step 1: Use or start a bash shell===
You can check your default shell using
echo $SHELL

If your default shell is csh, use
bash
to start a new bash shell in the current terminal window. Note that echo $SHELL will show you your default shell regardless of the current shell.

===Step 2: Set your environment variables===
export ARTHOR_DIR=/opt/nextmove/arthor/arthor-3.3-centos7
export PATH=$ARTHOR_DIR/bin/:$PATH

Make sure the ARTHOR_DIR variable is set to the directory for the latest version of Arthor or whichever version you would like to test.
The PATH environment variable is needed if you wish to use the Arthor tools from the command line

===Step 3: Run the arthor-server.jar===
java -jar /opt/nextmove/arthor/arthor-3.3-centos7/java/arthor.jar --httpPort <your httpPort>

==Setting environment variables for an Arthor Server==
Set the environment variables in the setenv.sh file. Note: Be sure to edit the file in the directory corresponding to the latest version of TomCat. As of December 2019, we are running 9.0.27 on n-1-136.

vim /opt/tomcat/apache-tomcat-9.0.27/bin/setenv.sh

Add the line below to the setenv.sh file above, or substitute the path to wherever you currently store the arthor.cfg file
export ARTHOR_CONFIG=/usr/local/tomcat/arthor.cfg

Here is an example of the arthor.cfg file:
BinDir=/opt/nextmove/arthor/arthor-3.3-centos7/bin
DataDir=/local2/arthor_local_8081/
MaxConcurrentSearches=6
MaxThreadsPerSearch=8
AutomaticIndex=false
AsyncHitCountMax=1000000
Resolver=https://sw.docking.org/util/smi2mol?smi=%s

=== Configuration Details ===

*'''BinDir''': is the location of the Arthor command line binaries. These are used to generate the Arthor index files and to perform searches directly on n-1-136. An example of this would be using atdbgrep for substructure search.

*'''DataDir''': This is the directory where the Arthor data files live. Location where the index files will be created and loaded from.

*'''MaxConcurrentSearches''': Controls the maximum number of searches that can be run concurrently by setting the database pool size. When switching between a large number of databases it can be useful to have a larger pool size, the only trade off is keeping file pointers open.

*'''MaxThreadsPerSearch''': The number of threads to use for both ATDB and ATFP searches

*Set '''AutomaticIndex''' to false if you don't want new smiles files added to the data directory to be indexed automatically

*'''AsyncHitCountMax''': The upper-bound for the number of hits to retrieve in background searches.

*'''Resolver''': Using Smallworld API, allows input box to take in a SMILE format and automatically draw on the board.

==Background==
Before working with Arthor, it is recommended that you familiarize yourself with the Arthor documentation. Some useful pages to look at include 3-5, 22-25 and 33-39. Of course, reading everything would be the best!

==Checking Disk Space Usage==
Before building arthor indexes, it's always a good thing to check what percent of the memory is being used. Try to be cautious with how much memory you have left, and make sure to check while building indexes to make sure that you have enough space. To check, run the following command:

df -h /<directory with disc>

==Building Large Databases==
At the moment, we are building databases of size 500M molecules by merging smile files. There are multiple methods of trying to create large databases, one being merging based off of the same H?? prefix and stopping once the database reaches > 500M molecules (or whatever upperbound you want to use). Here is some python code that simulates this merging process. Essentially the program takes all of the .smi files within an input directory, sorts them lexiographically, and begins merging these .smi files together in order until the size reaches > 500M molecules.

Feel free to modify it if you think a better method exists.

import subprocess
import sys
import os

from os import listdir
from os.path import isfile, join

mypath = "<Path to directory holding .smi files>"
onlyfiles = [f for f in listdir(mypath) if isfile(join(mypath, f))]
onlyfiles.sort()

create_fp = True
cur_mols = 0
lower_bound = 500000000
upper_bound = 600000000
files_to_merge = []

def merge_files(f_t_m):
arr = f_t_m[0].split(".")
arr2 = f_t_m[len(f_t_m) - 1].split(".")
file_name_merge = (arr[0] + "_" + arr2[0] + ".smi")
print ("File being created: " + file_name_merge)

for file in f_t_m:
tmp = file.split(".")
process = subprocess.Popen("cat " + join(mypath, file) + " >> " + file_name_merge, shell=True)
process.wait()

for file in onlyfiles:
arr = file.split(".")

if (arr[len(arr) - 1] == "smi"):
print("Working with " + file)
mol = sum(1 for line in open(join(mypath, file)))
print(file, mol, cur_mols)

if (cur_mols + mol > lower_bound):
if (cur_mols + mol < upper_bound):
files_to_merge.append(file)
merge_files(files_to_merge)
cur_mols = 0
files_to_merge.clear()
else:
merge_files(files_to_merge)
files_to_merge.clear()
files_to_merge.append(file)
merge_files(files_to_merge)
cur_mols = 0
files_to_merge.clear()
else:
cur_mols += mol
files_to_merge.append(file)

if (len(files_to_merge) != 0):
merge_files(files_to_merge)

==Building Arthor Indexes==
Once you've merged the .smi files together, it's time to start building the databases themselves. To do this we use the command

smi2atdb -j 0 -p <The .smi file> <The .atdb>

The flag "-j 0" enables parallel generation and utilizes all available processors to generate the .atdb file. The "-p" flag stores the offset position in the ATDB file. Since we're building indexes for the Web Application, you must use the "-p" flag when building indexes. Please note that the name of the .smi file should also be the name of the .atdb file. That way, the Web Application knows to use these files together and correctly display the required images. Refer to pages 33-34 in the Arthor documentation for more information.

If there are too many large .smi files and you do not want to manually build each .atdb file, you can use this python script which takes all of the .smi files in the current directory and converts them to .atdb files. Make sure to modify mypath to the directory containing the .smi files. You can change the variable "create_fp" to false if you don't want to create .atdb.fp files (refer to page 9 in the Arthor documentation).

import subprocess
import sys
import os

from os import listdir
from os.path import isfile, join

mypath = "<Path containing the .smi files"
onlyfiles = [f for f in listdir(mypath) if isfile(join(mypath, f))]

create_fp = True

for file in onlyfiles:
arr = file.split(".")

if (arr[len(arr) - 1] == "smi"):
process = subprocess.Popen("/nfs/ex9/work/xyz/psql/arthor-3.3-centos7/bin/smi2atdb -j 0 -p {0} {1}.atdb".format(join(mypath, file), arr[0]), shell=True)
process.wait()

print("SUCCESS! {0}.atdb file was created!".format(arr[0]))

if (create_fp):
process = subprocess.Popen("/nfs/ex9/work/xyz/psql/arthor-3.3-centos7/bin/atdb2fp -j 0 {0}.atdb".format(arr[0]), shell=True)
process.wait()

print("SUCCESS! {0}.atdb.fp file was created!".format(arr[0]))

==Uploading Indexes to the Web Application==
One can upload indexes to the Web Application by changing the "DATADIR" variable in the arthor.cfg file to the directory holding the .atdb files. This is already set up on n-1-136 and n-5-34.

==Further Arthor Optimizations==
The following edits can be made to the arthor.cfg to optimize substructure search queries. More information can be found in pages 6-8 in the Arthor Documentation file.

'''NodeAffinity NUMA''': optimized flag, pin processing to specific CPU sets to where the data is located in memory. There is a small start-up cost and is most useful for long running services (see Non-Uniform Memory Access (NUMA))

'''AsyncHitCountAllowed=true|false''' After fetching a page from a substructure or formula search the server will spin off a background process to count the total number of hits. This can be resource intensive for large databases and may not be desirable for servers under heavy load and may not even be needed.

'''AsyncHitCountMax=#''' The upper-bound for the number of hits to retrieve in background searches. If very generic queries are issued (e.g. benzene or methane) hundred’s of millions of hits may be counted. Setting this value to anything other than zero (e.g. 10,000) will stop the background search if it exceeds this limit. Note some pathological queries may find very few hits but still end up looking at everything.

'''MaxConcurrentSearches=#''' Controls the maximum number of searches that can be run concurrently by setting the database pool size. The searches may be on the same or different databases. If a search comes in and the pool is full it will have to wait for another search to finish - this increases the request time. 

Typically if each search is using all the processing cores on a machine then additional searches will run at 1/Nth the speed. If the request time is substantially larger that the search time the request had to wait for resources to become available. When switching between a large number of databases it can be useful to have a larger pool size, the only trade off is keeping file pointers open.  Default: 6

'''Binary Fingerprint Folding''' Arthor uses binary circular fingerprints (ECFP4/radius=2) for similarity. When creating an ATFP index you can specify how large to make your fingerprints. Circular fingerprints are sparser than path based fingerprints (e.g. Daylight) and so can be folded smaller without too much degradation in performance. Folding can significantly reduce the footprint size of a database and improve search speeds. A 256-bit fingerprint takes up 1/4 of the space of 1024-bit and can therefore be traversed 4x faster.

This is more important for very large databases with billions of compounds, in such instances a minor drop in precision is likely tolerable as ultimately all that happens is some hits may swap places in the hit list.

==Virtual Memory==
In addition to modifying the arthor.cfg file, virtual memory can also be used to make queries faster. There can still be More information can be found in pages 10-16 in the Arthor Documentation.

==Setting up Round Table==
This is a new feature in Arthor 3.0 and is currently beta (January 2020). See section 2.4 in the manual
As explained in the manual, "Round Table allows you to serve and split chemical searches across multiple host machines. The implementation provides a lightweight proxy that forwards requests to other Arthor host servers that do the actual search. Communication is done using the existing Web APIs.

Since Arthor requires CentOS 7, as of January 2020 we have 6 servers that are capable of running Arthor with Round Table.

===Setting up Host Server===
If we want to add machines to the Round Table, for example 'nun' and 'samekh', we need to edit their arthor.cfg file so that when our Local Machine passes commands these secondary servers know to perform the search they are given.

$ cat arthor.cfg
MaxThreadsPerSearch=4
AutomaticIndex=false
DATADIR=<Directory where smiles are located>

We then run the jar server on each of these host machines containing data on any available port.

java -jar /nfs/ex9/work/xyz/psql/arthor-3.3-centos7/java/arthor.jar --httpPort <port>

For our local machine, the arthor.cfg file will look different.

$ cat arthor.cfg
[RoundTable]
RemoteClient=http://skynet:<port number where jar server is running>/
RemoteClient=http://hal:<port number where jar server is running>/

Please refer to Section 2 in the RoundTable Documentation file (pages 6-8) for more useful information on configuration.

Then run the following command on n-1-136:

java -jar /nfs/ex9/work/xyz/psql/arthor-3.3-centos7/java/arthor.jar --httpPort <port>

===Public Arthor===
{| class="wikitable"
|-
! CentOS 7 Machine
! Port
! Total Files Size
! Arthor Install Location
! Round Table Data Directory
! Active
|-
| samekh
| 10.20.0.41:8000
| 2.4TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/public_arthor/
| active
|-
| nun
| 10.20.0.40:8000
| 2.4TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/public_arthor/
| active
|-
| n-9-22
| 10.20.9.22:8000
| 2.4TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /export/db4/public_arthor/
| active
|-
|}

===Arthor Round Table Head===
{| class="wikitable"
|-
! CentOS 7 Machine
! Port
! Database
! Arthor Install Location
! Round Table Data Directory
! Active
|-
| samekh
| 10.20.0.41:8080
| Enamine_REAL_Q2-2020-All-13B
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
| nun
| 10.20.0.40:8080
| Enamine_REAL_Q2-2020-All-41B
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
|}

===Arthor Round Table Nodes===
{| class="wikitable"
|-
! CentOS 7 Machine
! Port
! Database
! Total Files Size
! Arthor Install Location
! Round Table Data Directory
!Active
|-
| samekh
| 10.20.0.41:8008
| Enamine_REAL_Q2-2020-All-13B (26 slices)
| 4.5TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
| nun
| 10.20.0.40:8008
| Enamine_REAL_Space_June_2020_S41B (aa-ae, 5 slices), Enamine_REAL_Space_June_2020_M41B (af-am, 8 slices), zinc22_2d (H04~H25, 22 slices)
| 5.6TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
| n-1-17
| 10.20.1.17:8008
| Enamine_REAL_Space_June_2020_M41B (aa-ae, 5 slices), zinc22_2d (H25~H29, 4 slices)
| 3.7TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
| n-5-32
| 10.20.5.32:8008
| Enamine_REAL_Space_June_2020_M41B (an~az, 13 slices), zinc22_2d (H30, 1 slice)
| 5.6TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
| n-5-33
| 10.20.5.33:8008
| Enamine_REAL_Space_June_2020_M41B (ba-bl, 12 slices)
| 5.3TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
|}

===Arthor Local 8081 (Datasets all local to samekh/nun)===
{| class="wikitable"
|-
! CentOS 7 Machine
! Port
! Database
! Total Files Size
! Arthor Install Location
! Round Table Data Directory
!Active
|-
| samekh
| 10.20.0.41:8081
| Enamine_REAL_Q2-2020-All-13B (26 slices)
| 4.5TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_local_8081/
| active
|-
| nun
| 10.20.0.40:8081
| Enamine_REAL_Space_June_2020_S41B (aa-ae, 5 slices), Enamine_REAL_Space_June_2020_M41B (aa-an, 14 slices)
| 4.3TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_local_8081/
| active
|-
|}

Arthor Documentation for Future Developer

2021-01-07T00:33:48Z

Jgutie11: /* Setting environment variables for TomCat Server */

Written by Jennifer Young on December 16, 2019. Last edited January 05, 2021

==Install and Set Up on TomCat (Method 1)==
Arthor ran on n-1-136, which runs CentOS Linux release 7.7.1908 (Core). You can check the version of CentOS with the following command
cat /etc/centos-release

Check your current version of Java with the following command:
java -version

On n-1-136 we are running openjdk version "1.8.0_222", OpenJDK Runtime Environment (build 1.8.0_222-b10), and OpenJDK 64-Bit Server VM (build 25.222-b10, mixed mode)
If Java is not installed, install it using yum

==See this wiki page for more detailed information about installing Tomcat on our cluster==
http://wiki.docking.org/index.php/Tomcat_Installation

==Open port for Arthor==
In order for Arthor to be usable in the browser, the port you wish to run it on must be opened.
https://www.thegeekdiary.com/how-to-open-a-ports-in-centos-rhel-7/

===Step 1: Check Port Status===
Check that the port is not open and that Apache is not showing that port.
netstat -na | grep <port number you are checking>

lsof -i -P |grep http

===Step 2: Check Port Status in IP Tables===
iptables-save | grep <port number you are checking>

I skipped Step 3 from the guide, because there was a lot of information in the /etc/services file and I didn't want to edit it and break something.

===Step 4: Open Firewall Ports===
I did not include the zone=public section because the stand-alone servers are usually used for private instances of Arthor and SmallWorld.
Run as root.
firewall-cmd --add-port=<port number you are adding>/tcp --permanent

You need to reload the firewall after a change is made.
firewall-cmd --reload

===Step 5: Check that port is working===
To check that the port is active, run.
iptables -nL

You should see something along the lines of:
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:<port number you're adding> ctstate NEW,UNTRACKED

==How to run standalone Arthor instance==

===Step 1: Use or start a bash shell===
You can check your default shell using
echo $SHELL

If your default shell is csh, use
bash
to start a new bash shell in the current terminal window. Note that echo $SHELL will show you your default shell regardless of the current shell.

===Step 2: Set your environment variables===
export ARTHOR_DIR=/opt/nextmove/arthor/arthor-3.3-centos7
export PATH=$ARTHOR_DIR/bin/:$PATH

Make sure the ARTHOR_DIR variable is set to the directory for the latest version of Arthor or whichever version you would like to test.
The PATH environment variable is needed if you wish to use the Arthor tools from the command line

===Step 3: Run the arthor-server.jar===
java -jar /opt/nextmove/arthor/arthor-3.3-centos7/java/arthor.jar --httpPort <your httpPort>

==Setting environment variables for an Arthor Server==
Set the environment variables in the setenv.sh file. Note: Be sure to edit the file in the directory corresponding to the latest version of TomCat. As of December 2019, we are running 9.0.27 on n-1-136.

vim /opt/tomcat/apache-tomcat-9.0.27/bin/setenv.sh

Add the line below to the setenv.sh file above, or substitute the path to wherever you currently store the arthor.cfg file
export ARTHOR_CONFIG=/usr/local/tomcat/arthor.cfg

Here is an example of the arthor.cfg file:
BinDir=/opt/nextmove/arthor/arthor-3.3-centos7/bin
DataDir=/local2/arthor_local_8081/
MaxConcurrentSearches=6
MaxThreadsPerSearch=8
AutomaticIndex=false
AsyncHitCountMax=1000000
Resolver=https://sw.docking.org/util/smi2mol?smi=%s

'''Important parts of the arthor.cfg file'''

'''BinDir''': is the location of the Arthor command line binaries. These are used to generate the Arthor index files and to perform searches directly on n-1-136. An example of this would be using atdbgrep for substructure search.

'''DataDir''': This is the directory where the Arthor data files live. Location where the index files will be created and loaded from.

'''MaxConcurrentSearches''': Controls the maximum number of searches that can be run concurrently by setting the database pool size. When switching between a large number of databases it can be useful to have a larger pool size, the only trade off is keeping file pointers open.

'''MaxThreadsPerSearch''': The number of threads to use for both ATDB and ATFP searches

Set '''AutomaticIndex''' to false if you don't want new smiles files added to the data directory to be indexed automatically

'''AsyncHitCountMax''': The upper-bound for the number of hits to retrieve in background searches.

'''Resolver''': Using Smallworld API, allows input box to take in a SMILE format and automatically draw on the board.

==Background==
Before working with Arthor, it is recommended that you familiarize yourself with the Arthor documentation. Some useful pages to look at include 3-5, 22-25 and 33-39. Of course, reading everything would be the best!

==Checking Disk Space Usage==
Before building arthor indexes, it's always a good thing to check what percent of the memory is being used. Try to be cautious with how much memory you have left, and make sure to check while building indexes to make sure that you have enough space. To check, run the following command:

df -h /<directory with disc>

==Building Large Databases==
At the moment, we are building databases of size 500M molecules by merging smile files. There are multiple methods of trying to create large databases, one being merging based off of the same H?? prefix and stopping once the database reaches > 500M molecules (or whatever upperbound you want to use). Here is some python code that simulates this merging process. Essentially the program takes all of the .smi files within an input directory, sorts them lexiographically, and begins merging these .smi files together in order until the size reaches > 500M molecules.

Feel free to modify it if you think a better method exists.

import subprocess
import sys
import os

from os import listdir
from os.path import isfile, join

mypath = "<Path to directory holding .smi files>"
onlyfiles = [f for f in listdir(mypath) if isfile(join(mypath, f))]
onlyfiles.sort()

create_fp = True
cur_mols = 0
lower_bound = 500000000
upper_bound = 600000000
files_to_merge = []

def merge_files(f_t_m):
arr = f_t_m[0].split(".")
arr2 = f_t_m[len(f_t_m) - 1].split(".")
file_name_merge = (arr[0] + "_" + arr2[0] + ".smi")
print ("File being created: " + file_name_merge)

for file in f_t_m:
tmp = file.split(".")
process = subprocess.Popen("cat " + join(mypath, file) + " >> " + file_name_merge, shell=True)
process.wait()

for file in onlyfiles:
arr = file.split(".")

if (arr[len(arr) - 1] == "smi"):
print("Working with " + file)
mol = sum(1 for line in open(join(mypath, file)))
print(file, mol, cur_mols)

if (cur_mols + mol > lower_bound):
if (cur_mols + mol < upper_bound):
files_to_merge.append(file)
merge_files(files_to_merge)
cur_mols = 0
files_to_merge.clear()
else:
merge_files(files_to_merge)
files_to_merge.clear()
files_to_merge.append(file)
merge_files(files_to_merge)
cur_mols = 0
files_to_merge.clear()
else:
cur_mols += mol
files_to_merge.append(file)

if (len(files_to_merge) != 0):
merge_files(files_to_merge)

==Building Arthor Indexes==
Once you've merged the .smi files together, it's time to start building the databases themselves. To do this we use the command

smi2atdb -j 0 -p <The .smi file> <The .atdb>

The flag "-j 0" enables parallel generation and utilizes all available processors to generate the .atdb file. The "-p" flag stores the offset position in the ATDB file. Since we're building indexes for the Web Application, you must use the "-p" flag when building indexes. Please note that the name of the .smi file should also be the name of the .atdb file. That way, the Web Application knows to use these files together and correctly display the required images. Refer to pages 33-34 in the Arthor documentation for more information.

If there are too many large .smi files and you do not want to manually build each .atdb file, you can use this python script which takes all of the .smi files in the current directory and converts them to .atdb files. Make sure to modify mypath to the directory containing the .smi files. You can change the variable "create_fp" to false if you don't want to create .atdb.fp files (refer to page 9 in the Arthor documentation).

import subprocess
import sys
import os

from os import listdir
from os.path import isfile, join

mypath = "<Path containing the .smi files"
onlyfiles = [f for f in listdir(mypath) if isfile(join(mypath, f))]

create_fp = True

for file in onlyfiles:
arr = file.split(".")

if (arr[len(arr) - 1] == "smi"):
process = subprocess.Popen("/nfs/ex9/work/xyz/psql/arthor-3.3-centos7/bin/smi2atdb -j 0 -p {0} {1}.atdb".format(join(mypath, file), arr[0]), shell=True)
process.wait()

print("SUCCESS! {0}.atdb file was created!".format(arr[0]))

if (create_fp):
process = subprocess.Popen("/nfs/ex9/work/xyz/psql/arthor-3.3-centos7/bin/atdb2fp -j 0 {0}.atdb".format(arr[0]), shell=True)
process.wait()

print("SUCCESS! {0}.atdb.fp file was created!".format(arr[0]))

==Uploading Indexes to the Web Application==
One can upload indexes to the Web Application by changing the "DATADIR" variable in the arthor.cfg file to the directory holding the .atdb files. This is already set up on n-1-136 and n-5-34.

==Further Arthor Optimizations==
The following edits can be made to the arthor.cfg to optimize substructure search queries. More information can be found in pages 6-8 in the Arthor Documentation file.

'''NodeAffinity NUMA''': optimized flag, pin processing to specific CPU sets to where the data is located in memory. There is a small start-up cost and is most useful for long running services (see Non-Uniform Memory Access (NUMA))

'''AsyncHitCountAllowed=true|false''' After fetching a page from a substructure or formula search the server will spin off a background process to count the total number of hits. This can be resource intensive for large databases and may not be desirable for servers under heavy load and may not even be needed.

'''AsyncHitCountMax=#''' The upper-bound for the number of hits to retrieve in background searches. If very generic queries are issued (e.g. benzene or methane) hundred’s of millions of hits may be counted. Setting this value to anything other than zero (e.g. 10,000) will stop the background search if it exceeds this limit. Note some pathological queries may find very few hits but still end up looking at everything.

'''MaxConcurrentSearches=#''' Controls the maximum number of searches that can be run concurrently by setting the database pool size. The searches may be on the same or different databases. If a search comes in and the pool is full it will have to wait for another search to finish - this increases the request time. 

Typically if each search is using all the processing cores on a machine then additional searches will run at 1/Nth the speed. If the request time is substantially larger that the search time the request had to wait for resources to become available. When switching between a large number of databases it can be useful to have a larger pool size, the only trade off is keeping file pointers open.  Default: 6

'''Binary Fingerprint Folding''' Arthor uses binary circular fingerprints (ECFP4/radius=2) for similarity. When creating an ATFP index you can specify how large to make your fingerprints. Circular fingerprints are sparser than path based fingerprints (e.g. Daylight) and so can be folded smaller without too much degradation in performance. Folding can significantly reduce the footprint size of a database and improve search speeds. A 256-bit fingerprint takes up 1/4 of the space of 1024-bit and can therefore be traversed 4x faster.

This is more important for very large databases with billions of compounds, in such instances a minor drop in precision is likely tolerable as ultimately all that happens is some hits may swap places in the hit list.

==Virtual Memory==
In addition to modifying the arthor.cfg file, virtual memory can also be used to make queries faster. There can still be More information can be found in pages 10-16 in the Arthor Documentation.

==Setting up Round Table==
This is a new feature in Arthor 3.0 and is currently beta (January 2020). See section 2.4 in the manual
As explained in the manual, "Round Table allows you to serve and split chemical searches across multiple host machines. The implementation provides a lightweight proxy that forwards requests to other Arthor host servers that do the actual search. Communication is done using the existing Web APIs.

Since Arthor requires CentOS 7, as of January 2020 we have 6 servers that are capable of running Arthor with Round Table.

===Setting up Host Server===
If we want to add machines to the Round Table, for example 'nun' and 'samekh', we need to edit their arthor.cfg file so that when our Local Machine passes commands these secondary servers know to perform the search they are given.

$ cat arthor.cfg
MaxThreadsPerSearch=4
AutomaticIndex=false
DATADIR=<Directory where smiles are located>

We then run the jar server on each of these host machines containing data on any available port.

java -jar /nfs/ex9/work/xyz/psql/arthor-3.3-centos7/java/arthor.jar --httpPort <port>

For our local machine, the arthor.cfg file will look different.

$ cat arthor.cfg
[RoundTable]
RemoteClient=http://skynet:<port number where jar server is running>/
RemoteClient=http://hal:<port number where jar server is running>/

Please refer to Section 2 in the RoundTable Documentation file (pages 6-8) for more useful information on configuration.

Then run the following command on n-1-136:

java -jar /nfs/ex9/work/xyz/psql/arthor-3.3-centos7/java/arthor.jar --httpPort <port>

===Public Arthor===
{| class="wikitable"
|-
! CentOS 7 Machine
! Port
! Total Files Size
! Arthor Install Location
! Round Table Data Directory
! Active
|-
| samekh
| 10.20.0.41:8000
| 2.4TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/public_arthor/
| active
|-
| nun
| 10.20.0.40:8000
| 2.4TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/public_arthor/
| active
|-
| n-9-22
| 10.20.9.22:8000
| 2.4TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /export/db4/public_arthor/
| active
|-
|}

===Arthor Round Table Head===
{| class="wikitable"
|-
! CentOS 7 Machine
! Port
! Database
! Arthor Install Location
! Round Table Data Directory
! Active
|-
| samekh
| 10.20.0.41:8080
| Enamine_REAL_Q2-2020-All-13B
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
| nun
| 10.20.0.40:8080
| Enamine_REAL_Q2-2020-All-41B
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
|}

===Arthor Round Table Nodes===
{| class="wikitable"
|-
! CentOS 7 Machine
! Port
! Database
! Total Files Size
! Arthor Install Location
! Round Table Data Directory
!Active
|-
| samekh
| 10.20.0.41:8008
| Enamine_REAL_Q2-2020-All-13B (26 slices)
| 4.5TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
| nun
| 10.20.0.40:8008
| Enamine_REAL_Space_June_2020_S41B (aa-ae, 5 slices), Enamine_REAL_Space_June_2020_M41B (af-am, 8 slices), zinc22_2d (H04~H25, 22 slices)
| 5.6TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
| n-1-17
| 10.20.1.17:8008
| Enamine_REAL_Space_June_2020_M41B (aa-ae, 5 slices), zinc22_2d (H25~H29, 4 slices)
| 3.7TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
| n-5-32
| 10.20.5.32:8008
| Enamine_REAL_Space_June_2020_M41B (an~az, 13 slices), zinc22_2d (H30, 1 slice)
| 5.6TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
| n-5-33
| 10.20.5.33:8008
| Enamine_REAL_Space_June_2020_M41B (ba-bl, 12 slices)
| 5.3TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
|}

===Arthor Local 8081 (Datasets all local to samekh/nun)===
{| class="wikitable"
|-
! CentOS 7 Machine
! Port
! Database
! Total Files Size
! Arthor Install Location
! Round Table Data Directory
!Active
|-
| samekh
| 10.20.0.41:8081
| Enamine_REAL_Q2-2020-All-13B (26 slices)
| 4.5TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_local_8081/
| active
|-
| nun
| 10.20.0.40:8081
| Enamine_REAL_Space_June_2020_S41B (aa-ae, 5 slices), Enamine_REAL_Space_June_2020_M41B (aa-an, 14 slices)
| 4.3TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_local_8081/
| active
|-
|}

Arthor Documentation for Future Developer

2021-01-06T23:55:05Z

Jgutie11: /* How to run standalone Arthor instance */

Written by Jennifer Young on December 16, 2019. Last edited January 05, 2021

==Install and Set Up on TomCat (Method 1)==
Arthor ran on n-1-136, which runs CentOS Linux release 7.7.1908 (Core). You can check the version of CentOS with the following command
cat /etc/centos-release

Check your current version of Java with the following command:
java -version

On n-1-136 we are running openjdk version "1.8.0_222", OpenJDK Runtime Environment (build 1.8.0_222-b10), and OpenJDK 64-Bit Server VM (build 25.222-b10, mixed mode)
If Java is not installed, install it using yum

==See this wiki page for more detailed information about installing Tomcat on our cluster==
http://wiki.docking.org/index.php/Tomcat_Installation

==Open port for Arthor==
In order for Arthor to be usable in the browser, the port you wish to run it on must be opened.
https://www.thegeekdiary.com/how-to-open-a-ports-in-centos-rhel-7/

===Step 1: Check Port Status===
Check that the port is not open and that Apache is not showing that port.
netstat -na | grep <port number you are checking>

lsof -i -P |grep http

===Step 2: Check Port Status in IP Tables===
iptables-save | grep <port number you are checking>

I skipped Step 3 from the guide, because there was a lot of information in the /etc/services file and I didn't want to edit it and break something.

===Step 4: Open Firewall Ports===
I did not include the zone=public section because the stand-alone servers are usually used for private instances of Arthor and SmallWorld.
Run as root.
firewall-cmd --add-port=<port number you are adding>/tcp --permanent

You need to reload the firewall after a change is made.
firewall-cmd --reload

===Step 5: Check that port is working===
To check that the port is active, run.
iptables -nL

You should see something along the lines of:
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:<port number you're adding> ctstate NEW,UNTRACKED

==How to run standalone Arthor instance==

===Step 1: Use or start a bash shell===
You can check your default shell using
echo $SHELL

If your default shell is csh, use
bash
to start a new bash shell in the current terminal window. Note that echo $SHELL will show you your default shell regardless of the current shell.

===Step 2: Set your environment variables===
export ARTHOR_DIR=/opt/nextmove/arthor/arthor-3.3-centos7
export PATH=$ARTHOR_DIR/bin/:$PATH

Make sure the ARTHOR_DIR variable is set to the directory for the latest version of Arthor or whichever version you would like to test.
The PATH environment variable is needed if you wish to use the Arthor tools from the command line

===Step 3: Run the arthor-server.jar===
java -jar /opt/nextmove/arthor/arthor-3.3-centos7/java/arthor.jar --httpPort <your httpPort>

==Setting environment variables for TomCat Server==
Set the environment variables in the setenv.sh file. Note: Be sure to edit the file in the directory corresponding to the latest version of TomCat. As of December 2019, we are running 9.0.27 on n-1-136.

vim /opt/tomcat/apache-tomcat-9.0.27/bin/setenv.sh

Add the line below to the setenv.sh file above, or substitute the path to wherever you currently store the arthor.cfg file
export ARTHOR_CONFIG=/usr/local/tomcat/arthor.cfg

Here is an example of the arthor.cfg file:
# Arthor generated config file
BINDIR=/opt/nextmove/arthor/arthor-2.1.2-centos7/bin
DATADIR=/usr/local/tomcat/arthor_data
STAGEDIR=/usr/local/arthor_data/stage
NTHREADS=64 .
NODEAFFINITY=true
SearchAsYouDraw=true
AutomaticIndex=true
DEPICTION=./depict/bot/svg?w=%w&h=%h&svgunits=px&smi=%s&zoom=0.8&sma=%m&smalim=1
RESOLVER=

'''Important parts of the arthor.cfg file'''

'''BINDIR''' is the location of the Arthor command line binaries. These are used to generate the Arthor index files and to perform searches directly on n-1-136. An example of this would be using atdbgrep for substructure search.

'''DATADIR''' This is the directory where the Arthor data files live. Location where the index files will be created and loaded from.

'''STAGEDIR''' Location where the index files will be built before being moved into the DATADIR.

'''NTHREADS''' The number of threads to use for both ATDB and ATFP searches

Set '''AutomaticIndex''' to false if you don't want new smiles files added to the data directory to be indexed automatically

==Background==
Before working with Arthor, it is recommended that you familiarize yourself with the Arthor documentation. Some useful pages to look at include 3-5, 22-25 and 33-39. Of course, reading everything would be the best!

==Checking Disk Space Usage==
Before building arthor indexes, it's always a good thing to check what percent of the memory is being used. Try to be cautious with how much memory you have left, and make sure to check while building indexes to make sure that you have enough space. To check, run the following command:

df -h /<directory with disc>

==Building Large Databases==
At the moment, we are building databases of size 500M molecules by merging smile files. There are multiple methods of trying to create large databases, one being merging based off of the same H?? prefix and stopping once the database reaches > 500M molecules (or whatever upperbound you want to use). Here is some python code that simulates this merging process. Essentially the program takes all of the .smi files within an input directory, sorts them lexiographically, and begins merging these .smi files together in order until the size reaches > 500M molecules.

Feel free to modify it if you think a better method exists.

import subprocess
import sys
import os

from os import listdir
from os.path import isfile, join

mypath = "<Path to directory holding .smi files>"
onlyfiles = [f for f in listdir(mypath) if isfile(join(mypath, f))]
onlyfiles.sort()

create_fp = True
cur_mols = 0
lower_bound = 500000000
upper_bound = 600000000
files_to_merge = []

def merge_files(f_t_m):
arr = f_t_m[0].split(".")
arr2 = f_t_m[len(f_t_m) - 1].split(".")
file_name_merge = (arr[0] + "_" + arr2[0] + ".smi")
print ("File being created: " + file_name_merge)

for file in f_t_m:
tmp = file.split(".")
process = subprocess.Popen("cat " + join(mypath, file) + " >> " + file_name_merge, shell=True)
process.wait()

for file in onlyfiles:
arr = file.split(".")

if (arr[len(arr) - 1] == "smi"):
print("Working with " + file)
mol = sum(1 for line in open(join(mypath, file)))
print(file, mol, cur_mols)

if (cur_mols + mol > lower_bound):
if (cur_mols + mol < upper_bound):
files_to_merge.append(file)
merge_files(files_to_merge)
cur_mols = 0
files_to_merge.clear()
else:
merge_files(files_to_merge)
files_to_merge.clear()
files_to_merge.append(file)
merge_files(files_to_merge)
cur_mols = 0
files_to_merge.clear()
else:
cur_mols += mol
files_to_merge.append(file)

if (len(files_to_merge) != 0):
merge_files(files_to_merge)

==Building Arthor Indexes==
Once you've merged the .smi files together, it's time to start building the databases themselves. To do this we use the command

smi2atdb -j 0 -p <The .smi file> <The .atdb>

The flag "-j 0" enables parallel generation and utilizes all available processors to generate the .atdb file. The "-p" flag stores the offset position in the ATDB file. Since we're building indexes for the Web Application, you must use the "-p" flag when building indexes. Please note that the name of the .smi file should also be the name of the .atdb file. That way, the Web Application knows to use these files together and correctly display the required images. Refer to pages 33-34 in the Arthor documentation for more information.

If there are too many large .smi files and you do not want to manually build each .atdb file, you can use this python script which takes all of the .smi files in the current directory and converts them to .atdb files. Make sure to modify mypath to the directory containing the .smi files. You can change the variable "create_fp" to false if you don't want to create .atdb.fp files (refer to page 9 in the Arthor documentation).

import subprocess
import sys
import os

from os import listdir
from os.path import isfile, join

mypath = "<Path containing the .smi files"
onlyfiles = [f for f in listdir(mypath) if isfile(join(mypath, f))]

create_fp = True

for file in onlyfiles:
arr = file.split(".")

if (arr[len(arr) - 1] == "smi"):
process = subprocess.Popen("/nfs/ex9/work/xyz/psql/arthor-3.3-centos7/bin/smi2atdb -j 0 -p {0} {1}.atdb".format(join(mypath, file), arr[0]), shell=True)
process.wait()

print("SUCCESS! {0}.atdb file was created!".format(arr[0]))

if (create_fp):
process = subprocess.Popen("/nfs/ex9/work/xyz/psql/arthor-3.3-centos7/bin/atdb2fp -j 0 {0}.atdb".format(arr[0]), shell=True)
process.wait()

print("SUCCESS! {0}.atdb.fp file was created!".format(arr[0]))

==Uploading Indexes to the Web Application==
One can upload indexes to the Web Application by changing the "DATADIR" variable in the arthor.cfg file to the directory holding the .atdb files. This is already set up on n-1-136 and n-5-34.

==Further Arthor Optimizations==
The following edits can be made to the arthor.cfg to optimize substructure search queries. More information can be found in pages 6-8 in the Arthor Documentation file.

'''NodeAffinity NUMA''': optimized flag, pin processing to specific CPU sets to where the data is located in memory. There is a small start-up cost and is most useful for long running services (see Non-Uniform Memory Access (NUMA))

'''AsyncHitCountAllowed=true|false''' After fetching a page from a substructure or formula search the server will spin off a background process to count the total number of hits. This can be resource intensive for large databases and may not be desirable for servers under heavy load and may not even be needed.

'''AsyncHitCountMax=#''' The upper-bound for the number of hits to retrieve in background searches. If very generic queries are issued (e.g. benzene or methane) hundred’s of millions of hits may be counted. Setting this value to anything other than zero (e.g. 10,000) will stop the background search if it exceeds this limit. Note some pathological queries may find very few hits but still end up looking at everything.

'''MaxConcurrentSearches=#''' Controls the maximum number of searches that can be run concurrently by setting the database pool size. The searches may be on the same or different databases. If a search comes in and the pool is full it will have to wait for another search to finish - this increases the request time. 

Typically if each search is using all the processing cores on a machine then additional searches will run at 1/Nth the speed. If the request time is substantially larger that the search time the request had to wait for resources to become available. When switching between a large number of databases it can be useful to have a larger pool size, the only trade off is keeping file pointers open.  Default: 6

'''Binary Fingerprint Folding''' Arthor uses binary circular fingerprints (ECFP4/radius=2) for similarity. When creating an ATFP index you can specify how large to make your fingerprints. Circular fingerprints are sparser than path based fingerprints (e.g. Daylight) and so can be folded smaller without too much degradation in performance. Folding can significantly reduce the footprint size of a database and improve search speeds. A 256-bit fingerprint takes up 1/4 of the space of 1024-bit and can therefore be traversed 4x faster.

This is more important for very large databases with billions of compounds, in such instances a minor drop in precision is likely tolerable as ultimately all that happens is some hits may swap places in the hit list.

==Virtual Memory==
In addition to modifying the arthor.cfg file, virtual memory can also be used to make queries faster. There can still be More information can be found in pages 10-16 in the Arthor Documentation.

==Setting up Round Table==
This is a new feature in Arthor 3.0 and is currently beta (January 2020). See section 2.4 in the manual
As explained in the manual, "Round Table allows you to serve and split chemical searches across multiple host machines. The implementation provides a lightweight proxy that forwards requests to other Arthor host servers that do the actual search. Communication is done using the existing Web APIs.

Since Arthor requires CentOS 7, as of January 2020 we have 6 servers that are capable of running Arthor with Round Table.

===Setting up Host Server===
If we want to add machines to the Round Table, for example 'nun' and 'samekh', we need to edit their arthor.cfg file so that when our Local Machine passes commands these secondary servers know to perform the search they are given.

$ cat arthor.cfg
MaxThreadsPerSearch=4
AutomaticIndex=false
DATADIR=<Directory where smiles are located>

We then run the jar server on each of these host machines containing data on any available port.

java -jar /nfs/ex9/work/xyz/psql/arthor-3.3-centos7/java/arthor.jar --httpPort <port>

For our local machine, the arthor.cfg file will look different.

$ cat arthor.cfg
[RoundTable]
RemoteClient=http://skynet:<port number where jar server is running>/
RemoteClient=http://hal:<port number where jar server is running>/

Please refer to Section 2 in the RoundTable Documentation file (pages 6-8) for more useful information on configuration.

Then run the following command on n-1-136:

java -jar /nfs/ex9/work/xyz/psql/arthor-3.3-centos7/java/arthor.jar --httpPort <port>

===Public Arthor===
{| class="wikitable"
|-
! CentOS 7 Machine
! Port
! Total Files Size
! Arthor Install Location
! Round Table Data Directory
! Active
|-
| samekh
| 10.20.0.41:8000
| 2.4TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/public_arthor/
| active
|-
| nun
| 10.20.0.40:8000
| 2.4TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/public_arthor/
| active
|-
| n-9-22
| 10.20.9.22:8000
| 2.4TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /export/db4/public_arthor/
| active
|-
|}

===Arthor Round Table Head===
{| class="wikitable"
|-
! CentOS 7 Machine
! Port
! Database
! Arthor Install Location
! Round Table Data Directory
! Active
|-
| samekh
| 10.20.0.41:8080
| Enamine_REAL_Q2-2020-All-13B
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
| nun
| 10.20.0.40:8080
| Enamine_REAL_Q2-2020-All-41B
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
|}

===Arthor Round Table Nodes===
{| class="wikitable"
|-
! CentOS 7 Machine
! Port
! Database
! Total Files Size
! Arthor Install Location
! Round Table Data Directory
!Active
|-
| samekh
| 10.20.0.41:8008
| Enamine_REAL_Q2-2020-All-13B (26 slices)
| 4.5TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
| nun
| 10.20.0.40:8008
| Enamine_REAL_Space_June_2020_S41B (aa-ae, 5 slices), Enamine_REAL_Space_June_2020_M41B (af-am, 8 slices), zinc22_2d (H04~H25, 22 slices)
| 5.6TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
| n-1-17
| 10.20.1.17:8008
| Enamine_REAL_Space_June_2020_M41B (aa-ae, 5 slices), zinc22_2d (H25~H29, 4 slices)
| 3.7TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
| n-5-32
| 10.20.5.32:8008
| Enamine_REAL_Space_June_2020_M41B (an~az, 13 slices), zinc22_2d (H30, 1 slice)
| 5.6TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
| n-5-33
| 10.20.5.33:8008
| Enamine_REAL_Space_June_2020_M41B (ba-bl, 12 slices)
| 5.3TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
|}

===Arthor Local 8081 (Datasets all local to samekh/nun)===
{| class="wikitable"
|-
! CentOS 7 Machine
! Port
! Database
! Total Files Size
! Arthor Install Location
! Round Table Data Directory
!Active
|-
| samekh
| 10.20.0.41:8081
| Enamine_REAL_Q2-2020-All-13B (26 slices)
| 4.5TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_local_8081/
| active
|-
| nun
| 10.20.0.40:8081
| Enamine_REAL_Space_June_2020_S41B (aa-ae, 5 slices), Enamine_REAL_Space_June_2020_M41B (aa-an, 14 slices)
| 4.3TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_local_8081/
| active
|-
|}

Arthor Documentation for Future Developer

2021-01-06T22:58:03Z

Jgutie11: /* Arthor Round Table Nodes */

Written by Jennifer Young on December 16, 2019. Last edited January 05, 2021

==Install and Set Up on TomCat (Method 1)==
Arthor ran on n-1-136, which runs CentOS Linux release 7.7.1908 (Core). You can check the version of CentOS with the following command
cat /etc/centos-release

Check your current version of Java with the following command:
java -version

On n-1-136 we are running openjdk version "1.8.0_222", OpenJDK Runtime Environment (build 1.8.0_222-b10), and OpenJDK 64-Bit Server VM (build 25.222-b10, mixed mode)
If Java is not installed, install it using yum

==See this wiki page for more detailed information about installing Tomcat on our cluster==
http://wiki.docking.org/index.php/Tomcat_Installation

==Open port for Arthor==
In order for Arthor to be usable in the browser, the port you wish to run it on must be opened.
https://www.thegeekdiary.com/how-to-open-a-ports-in-centos-rhel-7/

===Step 1: Check Port Status===
Check that the port is not open and that Apache is not showing that port.
netstat -na | grep <port number you are checking>

lsof -i -P |grep http

===Step 2: Check Port Status in IP Tables===
iptables-save | grep <port number you are checking>

I skipped Step 3 from the guide, because there was a lot of information in the /etc/services file and I didn't want to edit it and break something.

===Step 4: Open Firewall Ports===
I did not include the zone=public section because the stand-alone servers are usually used for private instances of Arthor and SmallWorld.
Run as root.
firewall-cmd --add-port=<port number you are adding>/tcp --permanent

You need to reload the firewall after a change is made.
firewall-cmd --reload

===Step 5: Check that port is working===
To check that the port is active, run.
iptables -nL

You should see something along the lines of:
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:<port number you're adding> ctstate NEW,UNTRACKED

==How to run standalone Arthor instance==

===Step 1: Use or start a bash shell===
You can check your default shell using
echo $SHELL

If your default shell is csh, use
bash
to start a new bash shell in the current terminal window. Note that echo $SHELL will show you your default shell regardless of the current shell.

===Step 2: Set your environment variables===
export ARTHOR_DIR=/opt/nextmove/arthor/arthor-3.1-centos7
export PATH=$ARTHOR_DIR/bin/:$PATH

Make sure the ARTHOR_DIR variable is set to the directory for the latest version of Arthor or whichever version you would like to test.
The PATH environment variable is needed if you wish to use the Arthor tools from the command line

===Step 3: Run the arthor-server.jar===
java -jar /opt/nextmove/arthor/arthor-3.0-rt-beta-linux/java/arthor-server.jar --httpPort <your httpPort>

==Setting environment variables for TomCat Server==
Set the environment variables in the setenv.sh file. Note: Be sure to edit the file in the directory corresponding to the latest version of TomCat. As of December 2019, we are running 9.0.27 on n-1-136.

vim /opt/tomcat/apache-tomcat-9.0.27/bin/setenv.sh

Add the line below to the setenv.sh file above, or substitute the path to wherever you currently store the arthor.cfg file
export ARTHOR_CONFIG=/usr/local/tomcat/arthor.cfg

Here is an example of the arthor.cfg file:
# Arthor generated config file
BINDIR=/opt/nextmove/arthor/arthor-2.1.2-centos7/bin
DATADIR=/usr/local/tomcat/arthor_data
STAGEDIR=/usr/local/arthor_data/stage
NTHREADS=64 .
NODEAFFINITY=true
SearchAsYouDraw=true
AutomaticIndex=true
DEPICTION=./depict/bot/svg?w=%w&h=%h&svgunits=px&smi=%s&zoom=0.8&sma=%m&smalim=1
RESOLVER=

'''Important parts of the arthor.cfg file'''

'''BINDIR''' is the location of the Arthor command line binaries. These are used to generate the Arthor index files and to perform searches directly on n-1-136. An example of this would be using atdbgrep for substructure search.

'''DATADIR''' This is the directory where the Arthor data files live. Location where the index files will be created and loaded from.

'''STAGEDIR''' Location where the index files will be built before being moved into the DATADIR.

'''NTHREADS''' The number of threads to use for both ATDB and ATFP searches

Set '''AutomaticIndex''' to false if you don't want new smiles files added to the data directory to be indexed automatically

==Background==
Before working with Arthor, it is recommended that you familiarize yourself with the Arthor documentation. Some useful pages to look at include 3-5, 22-25 and 33-39. Of course, reading everything would be the best!

==Checking Disk Space Usage==
Before building arthor indexes, it's always a good thing to check what percent of the memory is being used. Try to be cautious with how much memory you have left, and make sure to check while building indexes to make sure that you have enough space. To check, run the following command:

df -h /<directory with disc>

==Building Large Databases==
At the moment, we are building databases of size 500M molecules by merging smile files. There are multiple methods of trying to create large databases, one being merging based off of the same H?? prefix and stopping once the database reaches > 500M molecules (or whatever upperbound you want to use). Here is some python code that simulates this merging process. Essentially the program takes all of the .smi files within an input directory, sorts them lexiographically, and begins merging these .smi files together in order until the size reaches > 500M molecules.

Feel free to modify it if you think a better method exists.

import subprocess
import sys
import os

from os import listdir
from os.path import isfile, join

mypath = "<Path to directory holding .smi files>"
onlyfiles = [f for f in listdir(mypath) if isfile(join(mypath, f))]
onlyfiles.sort()

create_fp = True
cur_mols = 0
lower_bound = 500000000
upper_bound = 600000000
files_to_merge = []

def merge_files(f_t_m):
arr = f_t_m[0].split(".")
arr2 = f_t_m[len(f_t_m) - 1].split(".")
file_name_merge = (arr[0] + "_" + arr2[0] + ".smi")
print ("File being created: " + file_name_merge)

for file in f_t_m:
tmp = file.split(".")
process = subprocess.Popen("cat " + join(mypath, file) + " >> " + file_name_merge, shell=True)
process.wait()

for file in onlyfiles:
arr = file.split(".")

if (arr[len(arr) - 1] == "smi"):
print("Working with " + file)
mol = sum(1 for line in open(join(mypath, file)))
print(file, mol, cur_mols)

if (cur_mols + mol > lower_bound):
if (cur_mols + mol < upper_bound):
files_to_merge.append(file)
merge_files(files_to_merge)
cur_mols = 0
files_to_merge.clear()
else:
merge_files(files_to_merge)
files_to_merge.clear()
files_to_merge.append(file)
merge_files(files_to_merge)
cur_mols = 0
files_to_merge.clear()
else:
cur_mols += mol
files_to_merge.append(file)

if (len(files_to_merge) != 0):
merge_files(files_to_merge)

==Building Arthor Indexes==
Once you've merged the .smi files together, it's time to start building the databases themselves. To do this we use the command

smi2atdb -j 0 -p <The .smi file> <The .atdb>

The flag "-j 0" enables parallel generation and utilizes all available processors to generate the .atdb file. The "-p" flag stores the offset position in the ATDB file. Since we're building indexes for the Web Application, you must use the "-p" flag when building indexes. Please note that the name of the .smi file should also be the name of the .atdb file. That way, the Web Application knows to use these files together and correctly display the required images. Refer to pages 33-34 in the Arthor documentation for more information.

If there are too many large .smi files and you do not want to manually build each .atdb file, you can use this python script which takes all of the .smi files in the current directory and converts them to .atdb files. Make sure to modify mypath to the directory containing the .smi files. You can change the variable "create_fp" to false if you don't want to create .atdb.fp files (refer to page 9 in the Arthor documentation).

import subprocess
import sys
import os

from os import listdir
from os.path import isfile, join

mypath = "<Path containing the .smi files"
onlyfiles = [f for f in listdir(mypath) if isfile(join(mypath, f))]

create_fp = True

for file in onlyfiles:
arr = file.split(".")

if (arr[len(arr) - 1] == "smi"):
process = subprocess.Popen("/nfs/ex9/work/xyz/psql/arthor-3.3-centos7/bin/smi2atdb -j 0 -p {0} {1}.atdb".format(join(mypath, file), arr[0]), shell=True)
process.wait()

print("SUCCESS! {0}.atdb file was created!".format(arr[0]))

if (create_fp):
process = subprocess.Popen("/nfs/ex9/work/xyz/psql/arthor-3.3-centos7/bin/atdb2fp -j 0 {0}.atdb".format(arr[0]), shell=True)
process.wait()

print("SUCCESS! {0}.atdb.fp file was created!".format(arr[0]))

==Uploading Indexes to the Web Application==
One can upload indexes to the Web Application by changing the "DATADIR" variable in the arthor.cfg file to the directory holding the .atdb files. This is already set up on n-1-136 and n-5-34.

==Further Arthor Optimizations==
The following edits can be made to the arthor.cfg to optimize substructure search queries. More information can be found in pages 6-8 in the Arthor Documentation file.

'''NodeAffinity NUMA''': optimized flag, pin processing to specific CPU sets to where the data is located in memory. There is a small start-up cost and is most useful for long running services (see Non-Uniform Memory Access (NUMA))

'''AsyncHitCountAllowed=true|false''' After fetching a page from a substructure or formula search the server will spin off a background process to count the total number of hits. This can be resource intensive for large databases and may not be desirable for servers under heavy load and may not even be needed.

'''AsyncHitCountMax=#''' The upper-bound for the number of hits to retrieve in background searches. If very generic queries are issued (e.g. benzene or methane) hundred’s of millions of hits may be counted. Setting this value to anything other than zero (e.g. 10,000) will stop the background search if it exceeds this limit. Note some pathological queries may find very few hits but still end up looking at everything.

'''MaxConcurrentSearches=#''' Controls the maximum number of searches that can be run concurrently by setting the database pool size. The searches may be on the same or different databases. If a search comes in and the pool is full it will have to wait for another search to finish - this increases the request time. 

Typically if each search is using all the processing cores on a machine then additional searches will run at 1/Nth the speed. If the request time is substantially larger that the search time the request had to wait for resources to become available. When switching between a large number of databases it can be useful to have a larger pool size, the only trade off is keeping file pointers open.  Default: 6

'''Binary Fingerprint Folding''' Arthor uses binary circular fingerprints (ECFP4/radius=2) for similarity. When creating an ATFP index you can specify how large to make your fingerprints. Circular fingerprints are sparser than path based fingerprints (e.g. Daylight) and so can be folded smaller without too much degradation in performance. Folding can significantly reduce the footprint size of a database and improve search speeds. A 256-bit fingerprint takes up 1/4 of the space of 1024-bit and can therefore be traversed 4x faster.

This is more important for very large databases with billions of compounds, in such instances a minor drop in precision is likely tolerable as ultimately all that happens is some hits may swap places in the hit list.

==Virtual Memory==
In addition to modifying the arthor.cfg file, virtual memory can also be used to make queries faster. There can still be More information can be found in pages 10-16 in the Arthor Documentation.

==Setting up Round Table==
This is a new feature in Arthor 3.0 and is currently beta (January 2020). See section 2.4 in the manual
As explained in the manual, "Round Table allows you to serve and split chemical searches across multiple host machines. The implementation provides a lightweight proxy that forwards requests to other Arthor host servers that do the actual search. Communication is done using the existing Web APIs.

Since Arthor requires CentOS 7, as of January 2020 we have 6 servers that are capable of running Arthor with Round Table.

===Setting up Host Server===
If we want to add machines to the Round Table, for example 'nun' and 'samekh', we need to edit their arthor.cfg file so that when our Local Machine passes commands these secondary servers know to perform the search they are given.

$ cat arthor.cfg
MaxThreadsPerSearch=4
AutomaticIndex=false
DATADIR=<Directory where smiles are located>

We then run the jar server on each of these host machines containing data on any available port.

java -jar /nfs/ex9/work/xyz/psql/arthor-3.3-centos7/java/arthor.jar --httpPort <port>

For our local machine, the arthor.cfg file will look different.

$ cat arthor.cfg
[RoundTable]
RemoteClient=http://skynet:<port number where jar server is running>/
RemoteClient=http://hal:<port number where jar server is running>/

Please refer to Section 2 in the RoundTable Documentation file (pages 6-8) for more useful information on configuration.

Then run the following command on n-1-136:

java -jar /nfs/ex9/work/xyz/psql/arthor-3.3-centos7/java/arthor.jar --httpPort <port>

===Public Arthor===
{| class="wikitable"
|-
! CentOS 7 Machine
! Port
! Total Files Size
! Arthor Install Location
! Round Table Data Directory
! Active
|-
| samekh
| 10.20.0.41:8000
| 2.4TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/public_arthor/
| active
|-
| nun
| 10.20.0.40:8000
| 2.4TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/public_arthor/
| active
|-
| n-9-22
| 10.20.9.22:8000
| 2.4TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /export/db4/public_arthor/
| active
|-
|}

===Arthor Round Table Head===
{| class="wikitable"
|-
! CentOS 7 Machine
! Port
! Database
! Arthor Install Location
! Round Table Data Directory
! Active
|-
| samekh
| 10.20.0.41:8080
| Enamine_REAL_Q2-2020-All-13B
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
| nun
| 10.20.0.40:8080
| Enamine_REAL_Q2-2020-All-41B
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
|}

===Arthor Round Table Nodes===
{| class="wikitable"
|-
! CentOS 7 Machine
! Port
! Database
! Total Files Size
! Arthor Install Location
! Round Table Data Directory
!Active
|-
| samekh
| 10.20.0.41:8008
| Enamine_REAL_Q2-2020-All-13B (26 slices)
| 4.5TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
| nun
| 10.20.0.40:8008
| Enamine_REAL_Space_June_2020_S41B (aa-ae, 5 slices), Enamine_REAL_Space_June_2020_M41B (af-am, 8 slices), zinc22_2d (H04~H25, 22 slices)
| 5.6TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
| n-1-17
| 10.20.1.17:8008
| Enamine_REAL_Space_June_2020_M41B (aa-ae, 5 slices), zinc22_2d (H25~H29, 4 slices)
| 3.7TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
| n-5-32
| 10.20.5.32:8008
| Enamine_REAL_Space_June_2020_M41B (an~az, 13 slices), zinc22_2d (H30, 1 slice)
| 5.6TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
| n-5-33
| 10.20.5.33:8008
| Enamine_REAL_Space_June_2020_M41B (ba-bl, 12 slices)
| 5.3TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
|}

===Arthor Local 8081 (Datasets all local to samekh/nun)===
{| class="wikitable"
|-
! CentOS 7 Machine
! Port
! Database
! Total Files Size
! Arthor Install Location
! Round Table Data Directory
!Active
|-
| samekh
| 10.20.0.41:8081
| Enamine_REAL_Q2-2020-All-13B (26 slices)
| 4.5TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_local_8081/
| active
|-
| nun
| 10.20.0.40:8081
| Enamine_REAL_Space_June_2020_S41B (aa-ae, 5 slices), Enamine_REAL_Space_June_2020_M41B (aa-an, 14 slices)
| 4.3TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_local_8081/
| active
|-
|}

Arthor Documentation for Future Developer

2021-01-06T22:52:40Z

Jgutie11: /* Arthor (local 8081) */

Written by Jennifer Young on December 16, 2019. Last edited January 05, 2021

==Install and Set Up on TomCat (Method 1)==
Arthor ran on n-1-136, which runs CentOS Linux release 7.7.1908 (Core). You can check the version of CentOS with the following command
cat /etc/centos-release

Check your current version of Java with the following command:
java -version

On n-1-136 we are running openjdk version "1.8.0_222", OpenJDK Runtime Environment (build 1.8.0_222-b10), and OpenJDK 64-Bit Server VM (build 25.222-b10, mixed mode)
If Java is not installed, install it using yum

==See this wiki page for more detailed information about installing Tomcat on our cluster==
http://wiki.docking.org/index.php/Tomcat_Installation

==Open port for Arthor==
In order for Arthor to be usable in the browser, the port you wish to run it on must be opened.
https://www.thegeekdiary.com/how-to-open-a-ports-in-centos-rhel-7/

===Step 1: Check Port Status===
Check that the port is not open and that Apache is not showing that port.
netstat -na | grep <port number you are checking>

lsof -i -P |grep http

===Step 2: Check Port Status in IP Tables===
iptables-save | grep <port number you are checking>

I skipped Step 3 from the guide, because there was a lot of information in the /etc/services file and I didn't want to edit it and break something.

===Step 4: Open Firewall Ports===
I did not include the zone=public section because the stand-alone servers are usually used for private instances of Arthor and SmallWorld.
Run as root.
firewall-cmd --add-port=<port number you are adding>/tcp --permanent

You need to reload the firewall after a change is made.
firewall-cmd --reload

===Step 5: Check that port is working===
To check that the port is active, run.
iptables -nL

You should see something along the lines of:
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:<port number you're adding> ctstate NEW,UNTRACKED

==How to run standalone Arthor instance==

===Step 1: Use or start a bash shell===
You can check your default shell using
echo $SHELL

If your default shell is csh, use
bash
to start a new bash shell in the current terminal window. Note that echo $SHELL will show you your default shell regardless of the current shell.

===Step 2: Set your environment variables===
export ARTHOR_DIR=/opt/nextmove/arthor/arthor-3.1-centos7
export PATH=$ARTHOR_DIR/bin/:$PATH

Make sure the ARTHOR_DIR variable is set to the directory for the latest version of Arthor or whichever version you would like to test.
The PATH environment variable is needed if you wish to use the Arthor tools from the command line

===Step 3: Run the arthor-server.jar===
java -jar /opt/nextmove/arthor/arthor-3.0-rt-beta-linux/java/arthor-server.jar --httpPort <your httpPort>

==Setting environment variables for TomCat Server==
Set the environment variables in the setenv.sh file. Note: Be sure to edit the file in the directory corresponding to the latest version of TomCat. As of December 2019, we are running 9.0.27 on n-1-136.

vim /opt/tomcat/apache-tomcat-9.0.27/bin/setenv.sh

Add the line below to the setenv.sh file above, or substitute the path to wherever you currently store the arthor.cfg file
export ARTHOR_CONFIG=/usr/local/tomcat/arthor.cfg

Here is an example of the arthor.cfg file:
# Arthor generated config file
BINDIR=/opt/nextmove/arthor/arthor-2.1.2-centos7/bin
DATADIR=/usr/local/tomcat/arthor_data
STAGEDIR=/usr/local/arthor_data/stage
NTHREADS=64 .
NODEAFFINITY=true
SearchAsYouDraw=true
AutomaticIndex=true
DEPICTION=./depict/bot/svg?w=%w&h=%h&svgunits=px&smi=%s&zoom=0.8&sma=%m&smalim=1
RESOLVER=

'''Important parts of the arthor.cfg file'''

'''BINDIR''' is the location of the Arthor command line binaries. These are used to generate the Arthor index files and to perform searches directly on n-1-136. An example of this would be using atdbgrep for substructure search.

'''DATADIR''' This is the directory where the Arthor data files live. Location where the index files will be created and loaded from.

'''STAGEDIR''' Location where the index files will be built before being moved into the DATADIR.

'''NTHREADS''' The number of threads to use for both ATDB and ATFP searches

Set '''AutomaticIndex''' to false if you don't want new smiles files added to the data directory to be indexed automatically

==Background==
Before working with Arthor, it is recommended that you familiarize yourself with the Arthor documentation. Some useful pages to look at include 3-5, 22-25 and 33-39. Of course, reading everything would be the best!

==Checking Disk Space Usage==
Before building arthor indexes, it's always a good thing to check what percent of the memory is being used. Try to be cautious with how much memory you have left, and make sure to check while building indexes to make sure that you have enough space. To check, run the following command:

df -h /<directory with disc>

==Building Large Databases==
At the moment, we are building databases of size 500M molecules by merging smile files. There are multiple methods of trying to create large databases, one being merging based off of the same H?? prefix and stopping once the database reaches > 500M molecules (or whatever upperbound you want to use). Here is some python code that simulates this merging process. Essentially the program takes all of the .smi files within an input directory, sorts them lexiographically, and begins merging these .smi files together in order until the size reaches > 500M molecules.

Feel free to modify it if you think a better method exists.

import subprocess
import sys
import os

from os import listdir
from os.path import isfile, join

mypath = "<Path to directory holding .smi files>"
onlyfiles = [f for f in listdir(mypath) if isfile(join(mypath, f))]
onlyfiles.sort()

create_fp = True
cur_mols = 0
lower_bound = 500000000
upper_bound = 600000000
files_to_merge = []

def merge_files(f_t_m):
arr = f_t_m[0].split(".")
arr2 = f_t_m[len(f_t_m) - 1].split(".")
file_name_merge = (arr[0] + "_" + arr2[0] + ".smi")
print ("File being created: " + file_name_merge)

for file in f_t_m:
tmp = file.split(".")
process = subprocess.Popen("cat " + join(mypath, file) + " >> " + file_name_merge, shell=True)
process.wait()

for file in onlyfiles:
arr = file.split(".")

if (arr[len(arr) - 1] == "smi"):
print("Working with " + file)
mol = sum(1 for line in open(join(mypath, file)))
print(file, mol, cur_mols)

if (cur_mols + mol > lower_bound):
if (cur_mols + mol < upper_bound):
files_to_merge.append(file)
merge_files(files_to_merge)
cur_mols = 0
files_to_merge.clear()
else:
merge_files(files_to_merge)
files_to_merge.clear()
files_to_merge.append(file)
merge_files(files_to_merge)
cur_mols = 0
files_to_merge.clear()
else:
cur_mols += mol
files_to_merge.append(file)

if (len(files_to_merge) != 0):
merge_files(files_to_merge)

==Building Arthor Indexes==
Once you've merged the .smi files together, it's time to start building the databases themselves. To do this we use the command

smi2atdb -j 0 -p <The .smi file> <The .atdb>

The flag "-j 0" enables parallel generation and utilizes all available processors to generate the .atdb file. The "-p" flag stores the offset position in the ATDB file. Since we're building indexes for the Web Application, you must use the "-p" flag when building indexes. Please note that the name of the .smi file should also be the name of the .atdb file. That way, the Web Application knows to use these files together and correctly display the required images. Refer to pages 33-34 in the Arthor documentation for more information.

If there are too many large .smi files and you do not want to manually build each .atdb file, you can use this python script which takes all of the .smi files in the current directory and converts them to .atdb files. Make sure to modify mypath to the directory containing the .smi files. You can change the variable "create_fp" to false if you don't want to create .atdb.fp files (refer to page 9 in the Arthor documentation).

import subprocess
import sys
import os

from os import listdir
from os.path import isfile, join

mypath = "<Path containing the .smi files"
onlyfiles = [f for f in listdir(mypath) if isfile(join(mypath, f))]

create_fp = True

for file in onlyfiles:
arr = file.split(".")

if (arr[len(arr) - 1] == "smi"):
process = subprocess.Popen("/nfs/ex9/work/xyz/psql/arthor-3.3-centos7/bin/smi2atdb -j 0 -p {0} {1}.atdb".format(join(mypath, file), arr[0]), shell=True)
process.wait()

print("SUCCESS! {0}.atdb file was created!".format(arr[0]))

if (create_fp):
process = subprocess.Popen("/nfs/ex9/work/xyz/psql/arthor-3.3-centos7/bin/atdb2fp -j 0 {0}.atdb".format(arr[0]), shell=True)
process.wait()

print("SUCCESS! {0}.atdb.fp file was created!".format(arr[0]))

==Uploading Indexes to the Web Application==
One can upload indexes to the Web Application by changing the "DATADIR" variable in the arthor.cfg file to the directory holding the .atdb files. This is already set up on n-1-136 and n-5-34.

==Further Arthor Optimizations==
The following edits can be made to the arthor.cfg to optimize substructure search queries. More information can be found in pages 6-8 in the Arthor Documentation file.

'''NodeAffinity NUMA''': optimized flag, pin processing to specific CPU sets to where the data is located in memory. There is a small start-up cost and is most useful for long running services (see Non-Uniform Memory Access (NUMA))

'''AsyncHitCountAllowed=true|false''' After fetching a page from a substructure or formula search the server will spin off a background process to count the total number of hits. This can be resource intensive for large databases and may not be desirable for servers under heavy load and may not even be needed.

'''AsyncHitCountMax=#''' The upper-bound for the number of hits to retrieve in background searches. If very generic queries are issued (e.g. benzene or methane) hundred’s of millions of hits may be counted. Setting this value to anything other than zero (e.g. 10,000) will stop the background search if it exceeds this limit. Note some pathological queries may find very few hits but still end up looking at everything.

'''MaxConcurrentSearches=#''' Controls the maximum number of searches that can be run concurrently by setting the database pool size. The searches may be on the same or different databases. If a search comes in and the pool is full it will have to wait for another search to finish - this increases the request time. 

Typically if each search is using all the processing cores on a machine then additional searches will run at 1/Nth the speed. If the request time is substantially larger that the search time the request had to wait for resources to become available. When switching between a large number of databases it can be useful to have a larger pool size, the only trade off is keeping file pointers open.  Default: 6

'''Binary Fingerprint Folding''' Arthor uses binary circular fingerprints (ECFP4/radius=2) for similarity. When creating an ATFP index you can specify how large to make your fingerprints. Circular fingerprints are sparser than path based fingerprints (e.g. Daylight) and so can be folded smaller without too much degradation in performance. Folding can significantly reduce the footprint size of a database and improve search speeds. A 256-bit fingerprint takes up 1/4 of the space of 1024-bit and can therefore be traversed 4x faster.

This is more important for very large databases with billions of compounds, in such instances a minor drop in precision is likely tolerable as ultimately all that happens is some hits may swap places in the hit list.

==Virtual Memory==
In addition to modifying the arthor.cfg file, virtual memory can also be used to make queries faster. There can still be More information can be found in pages 10-16 in the Arthor Documentation.

==Setting up Round Table==
This is a new feature in Arthor 3.0 and is currently beta (January 2020). See section 2.4 in the manual
As explained in the manual, "Round Table allows you to serve and split chemical searches across multiple host machines. The implementation provides a lightweight proxy that forwards requests to other Arthor host servers that do the actual search. Communication is done using the existing Web APIs.

Since Arthor requires CentOS 7, as of January 2020 we have 6 servers that are capable of running Arthor with Round Table.

===Setting up Host Server===
If we want to add machines to the Round Table, for example 'nun' and 'samekh', we need to edit their arthor.cfg file so that when our Local Machine passes commands these secondary servers know to perform the search they are given.

$ cat arthor.cfg
MaxThreadsPerSearch=4
AutomaticIndex=false
DATADIR=<Directory where smiles are located>

We then run the jar server on each of these host machines containing data on any available port.

java -jar /nfs/ex9/work/xyz/psql/arthor-3.3-centos7/java/arthor.jar --httpPort <port>

For our local machine, the arthor.cfg file will look different.

$ cat arthor.cfg
[RoundTable]
RemoteClient=http://skynet:<port number where jar server is running>/
RemoteClient=http://hal:<port number where jar server is running>/

Please refer to Section 2 in the RoundTable Documentation file (pages 6-8) for more useful information on configuration.

Then run the following command on n-1-136:

java -jar /nfs/ex9/work/xyz/psql/arthor-3.3-centos7/java/arthor.jar --httpPort <port>

===Public Arthor===
{| class="wikitable"
|-
! CentOS 7 Machine
! Port
! Total Files Size
! Arthor Install Location
! Round Table Data Directory
! Active
|-
| samekh
| 10.20.0.41:8000
| 2.4TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/public_arthor/
| active
|-
| nun
| 10.20.0.40:8000
| 2.4TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/public_arthor/
| active
|-
| n-9-22
| 10.20.9.22:8000
| 2.4TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /export/db4/public_arthor/
| active
|-
|}

===Arthor Round Table Head===
{| class="wikitable"
|-
! CentOS 7 Machine
! Port
! Database
! Arthor Install Location
! Round Table Data Directory
! Active
|-
| samekh
| 10.20.0.41:8080
| Enamine_REAL_Q2-2020-All-13B
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
| nun
| 10.20.0.40:8080
| Enamine_REAL_Q2-2020-All-41B
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
|}

===Arthor Round Table Nodes===
{| class="wikitable"
|-
! CentOS 7 Machine
! Port
! Database
! Total Files Size
! Arthor Install Location
! Round Table Data Directory
!Active
|-
| samekh
| 10.20.0.41:8008
| Enamine_REAL_Q2-2020-All-13B (26 slices)
| 4.5TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
| nun
| 10.20.0.40:8008
| Enamine_REAL_Space_June_2020_S41B (aa-ae, 5 slices), Enamine_REAL_Space_June_2020_M41B (af-am, 8 slices), zinc22_2d (H04~H25, 22 slices)
| 5.6TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
| n-1-17
| 10.20.1.17:8008
| Enamine_REAL_Space_June_2020_M41B (aa-ae, 5 slices), zinc22_2d (H25~H29, 4 slices)
| 3.7TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
| n-5-32
| 10.20.5.32:8008
| Enamine_REAL_Space_June_2020_M41B (an~az, 13 slices)
| 5.6TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
| n-5-33
| 10.20.5.33:8008
| Enamine_REAL_Space_June_2020_M41B (ba-bl, 12 slices)
| 5.3TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
|}

===Arthor Local 8081 (Datasets all local to samekh/nun)===
{| class="wikitable"
|-
! CentOS 7 Machine
! Port
! Database
! Total Files Size
! Arthor Install Location
! Round Table Data Directory
!Active
|-
| samekh
| 10.20.0.41:8081
| Enamine_REAL_Q2-2020-All-13B (26 slices)
| 4.5TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_local_8081/
| active
|-
| nun
| 10.20.0.40:8081
| Enamine_REAL_Space_June_2020_S41B (aa-ae, 5 slices), Enamine_REAL_Space_June_2020_M41B (aa-an, 14 slices)
| 4.3TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_local_8081/
| active
|-
|}

Arthor Documentation for Future Developer

2021-01-06T21:06:38Z

Jgutie11: /* Arthor Round Table Nodes */

Written by Jennifer Young on December 16, 2019. Last edited January 05, 2021

==Install and Set Up on TomCat (Method 1)==
Arthor ran on n-1-136, which runs CentOS Linux release 7.7.1908 (Core). You can check the version of CentOS with the following command
cat /etc/centos-release

Check your current version of Java with the following command:
java -version

On n-1-136 we are running openjdk version "1.8.0_222", OpenJDK Runtime Environment (build 1.8.0_222-b10), and OpenJDK 64-Bit Server VM (build 25.222-b10, mixed mode)
If Java is not installed, install it using yum

==See this wiki page for more detailed information about installing Tomcat on our cluster==
http://wiki.docking.org/index.php/Tomcat_Installation

==Open port for Arthor==
In order for Arthor to be usable in the browser, the port you wish to run it on must be opened.
https://www.thegeekdiary.com/how-to-open-a-ports-in-centos-rhel-7/

===Step 1: Check Port Status===
Check that the port is not open and that Apache is not showing that port.
netstat -na | grep <port number you are checking>

lsof -i -P |grep http

===Step 2: Check Port Status in IP Tables===
iptables-save | grep <port number you are checking>

I skipped Step 3 from the guide, because there was a lot of information in the /etc/services file and I didn't want to edit it and break something.

===Step 4: Open Firewall Ports===
I did not include the zone=public section because the stand-alone servers are usually used for private instances of Arthor and SmallWorld.
Run as root.
firewall-cmd --add-port=<port number you are adding>/tcp --permanent

You need to reload the firewall after a change is made.
firewall-cmd --reload

===Step 5: Check that port is working===
To check that the port is active, run.
iptables -nL

You should see something along the lines of:
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:<port number you're adding> ctstate NEW,UNTRACKED

==How to run standalone Arthor instance==

===Step 1: Use or start a bash shell===
You can check your default shell using
echo $SHELL

If your default shell is csh, use
bash
to start a new bash shell in the current terminal window. Note that echo $SHELL will show you your default shell regardless of the current shell.

===Step 2: Set your environment variables===
export ARTHOR_DIR=/opt/nextmove/arthor/arthor-3.1-centos7
export PATH=$ARTHOR_DIR/bin/:$PATH

Make sure the ARTHOR_DIR variable is set to the directory for the latest version of Arthor or whichever version you would like to test.
The PATH environment variable is needed if you wish to use the Arthor tools from the command line

===Step 3: Run the arthor-server.jar===
java -jar /opt/nextmove/arthor/arthor-3.0-rt-beta-linux/java/arthor-server.jar --httpPort <your httpPort>

==Setting environment variables for TomCat Server==
Set the environment variables in the setenv.sh file. Note: Be sure to edit the file in the directory corresponding to the latest version of TomCat. As of December 2019, we are running 9.0.27 on n-1-136.

vim /opt/tomcat/apache-tomcat-9.0.27/bin/setenv.sh

Add the line below to the setenv.sh file above, or substitute the path to wherever you currently store the arthor.cfg file
export ARTHOR_CONFIG=/usr/local/tomcat/arthor.cfg

Here is an example of the arthor.cfg file:
# Arthor generated config file
BINDIR=/opt/nextmove/arthor/arthor-2.1.2-centos7/bin
DATADIR=/usr/local/tomcat/arthor_data
STAGEDIR=/usr/local/arthor_data/stage
NTHREADS=64 .
NODEAFFINITY=true
SearchAsYouDraw=true
AutomaticIndex=true
DEPICTION=./depict/bot/svg?w=%w&h=%h&svgunits=px&smi=%s&zoom=0.8&sma=%m&smalim=1
RESOLVER=

'''Important parts of the arthor.cfg file'''

'''BINDIR''' is the location of the Arthor command line binaries. These are used to generate the Arthor index files and to perform searches directly on n-1-136. An example of this would be using atdbgrep for substructure search.

'''DATADIR''' This is the directory where the Arthor data files live. Location where the index files will be created and loaded from.

'''STAGEDIR''' Location where the index files will be built before being moved into the DATADIR.

'''NTHREADS''' The number of threads to use for both ATDB and ATFP searches

Set '''AutomaticIndex''' to false if you don't want new smiles files added to the data directory to be indexed automatically

==Background==
Before working with Arthor, it is recommended that you familiarize yourself with the Arthor documentation. Some useful pages to look at include 3-5, 22-25 and 33-39. Of course, reading everything would be the best!

==Checking Disk Space Usage==
Before building arthor indexes, it's always a good thing to check what percent of the memory is being used. Try to be cautious with how much memory you have left, and make sure to check while building indexes to make sure that you have enough space. To check, run the following command:

df -h /<directory with disc>

==Building Large Databases==
At the moment, we are building databases of size 500M molecules by merging smile files. There are multiple methods of trying to create large databases, one being merging based off of the same H?? prefix and stopping once the database reaches > 500M molecules (or whatever upperbound you want to use). Here is some python code that simulates this merging process. Essentially the program takes all of the .smi files within an input directory, sorts them lexiographically, and begins merging these .smi files together in order until the size reaches > 500M molecules.

Feel free to modify it if you think a better method exists.

import subprocess
import sys
import os

from os import listdir
from os.path import isfile, join

mypath = "<Path to directory holding .smi files>"
onlyfiles = [f for f in listdir(mypath) if isfile(join(mypath, f))]
onlyfiles.sort()

create_fp = True
cur_mols = 0
lower_bound = 500000000
upper_bound = 600000000
files_to_merge = []

def merge_files(f_t_m):
arr = f_t_m[0].split(".")
arr2 = f_t_m[len(f_t_m) - 1].split(".")
file_name_merge = (arr[0] + "_" + arr2[0] + ".smi")
print ("File being created: " + file_name_merge)

for file in f_t_m:
tmp = file.split(".")
process = subprocess.Popen("cat " + join(mypath, file) + " >> " + file_name_merge, shell=True)
process.wait()

for file in onlyfiles:
arr = file.split(".")

if (arr[len(arr) - 1] == "smi"):
print("Working with " + file)
mol = sum(1 for line in open(join(mypath, file)))
print(file, mol, cur_mols)

if (cur_mols + mol > lower_bound):
if (cur_mols + mol < upper_bound):
files_to_merge.append(file)
merge_files(files_to_merge)
cur_mols = 0
files_to_merge.clear()
else:
merge_files(files_to_merge)
files_to_merge.clear()
files_to_merge.append(file)
merge_files(files_to_merge)
cur_mols = 0
files_to_merge.clear()
else:
cur_mols += mol
files_to_merge.append(file)

if (len(files_to_merge) != 0):
merge_files(files_to_merge)

==Building Arthor Indexes==
Once you've merged the .smi files together, it's time to start building the databases themselves. To do this we use the command

smi2atdb -j 0 -p <The .smi file> <The .atdb>

The flag "-j 0" enables parallel generation and utilizes all available processors to generate the .atdb file. The "-p" flag stores the offset position in the ATDB file. Since we're building indexes for the Web Application, you must use the "-p" flag when building indexes. Please note that the name of the .smi file should also be the name of the .atdb file. That way, the Web Application knows to use these files together and correctly display the required images. Refer to pages 33-34 in the Arthor documentation for more information.

If there are too many large .smi files and you do not want to manually build each .atdb file, you can use this python script which takes all of the .smi files in the current directory and converts them to .atdb files. Make sure to modify mypath to the directory containing the .smi files. You can change the variable "create_fp" to false if you don't want to create .atdb.fp files (refer to page 9 in the Arthor documentation).

import subprocess
import sys
import os

from os import listdir
from os.path import isfile, join

mypath = "<Path containing the .smi files"
onlyfiles = [f for f in listdir(mypath) if isfile(join(mypath, f))]

create_fp = True

for file in onlyfiles:
arr = file.split(".")

if (arr[len(arr) - 1] == "smi"):
process = subprocess.Popen("/nfs/ex9/work/xyz/psql/arthor-3.3-centos7/bin/smi2atdb -j 0 -p {0} {1}.atdb".format(join(mypath, file), arr[0]), shell=True)
process.wait()

print("SUCCESS! {0}.atdb file was created!".format(arr[0]))

if (create_fp):
process = subprocess.Popen("/nfs/ex9/work/xyz/psql/arthor-3.3-centos7/bin/atdb2fp -j 0 {0}.atdb".format(arr[0]), shell=True)
process.wait()

print("SUCCESS! {0}.atdb.fp file was created!".format(arr[0]))

==Uploading Indexes to the Web Application==
One can upload indexes to the Web Application by changing the "DATADIR" variable in the arthor.cfg file to the directory holding the .atdb files. This is already set up on n-1-136 and n-5-34.

==Further Arthor Optimizations==
The following edits can be made to the arthor.cfg to optimize substructure search queries. More information can be found in pages 6-8 in the Arthor Documentation file.

'''NodeAffinity NUMA''': optimized flag, pin processing to specific CPU sets to where the data is located in memory. There is a small start-up cost and is most useful for long running services (see Non-Uniform Memory Access (NUMA))

'''AsyncHitCountAllowed=true|false''' After fetching a page from a substructure or formula search the server will spin off a background process to count the total number of hits. This can be resource intensive for large databases and may not be desirable for servers under heavy load and may not even be needed.

'''AsyncHitCountMax=#''' The upper-bound for the number of hits to retrieve in background searches. If very generic queries are issued (e.g. benzene or methane) hundred’s of millions of hits may be counted. Setting this value to anything other than zero (e.g. 10,000) will stop the background search if it exceeds this limit. Note some pathological queries may find very few hits but still end up looking at everything.

'''MaxConcurrentSearches=#''' Controls the maximum number of searches that can be run concurrently by setting the database pool size. The searches may be on the same or different databases. If a search comes in and the pool is full it will have to wait for another search to finish - this increases the request time. 

Typically if each search is using all the processing cores on a machine then additional searches will run at 1/Nth the speed. If the request time is substantially larger that the search time the request had to wait for resources to become available. When switching between a large number of databases it can be useful to have a larger pool size, the only trade off is keeping file pointers open.  Default: 6

'''Binary Fingerprint Folding''' Arthor uses binary circular fingerprints (ECFP4/radius=2) for similarity. When creating an ATFP index you can specify how large to make your fingerprints. Circular fingerprints are sparser than path based fingerprints (e.g. Daylight) and so can be folded smaller without too much degradation in performance. Folding can significantly reduce the footprint size of a database and improve search speeds. A 256-bit fingerprint takes up 1/4 of the space of 1024-bit and can therefore be traversed 4x faster.

This is more important for very large databases with billions of compounds, in such instances a minor drop in precision is likely tolerable as ultimately all that happens is some hits may swap places in the hit list.

==Virtual Memory==
In addition to modifying the arthor.cfg file, virtual memory can also be used to make queries faster. There can still be More information can be found in pages 10-16 in the Arthor Documentation.

==Setting up Round Table==
This is a new feature in Arthor 3.0 and is currently beta (January 2020). See section 2.4 in the manual
As explained in the manual, "Round Table allows you to serve and split chemical searches across multiple host machines. The implementation provides a lightweight proxy that forwards requests to other Arthor host servers that do the actual search. Communication is done using the existing Web APIs.

Since Arthor requires CentOS 7, as of January 2020 we have 6 servers that are capable of running Arthor with Round Table.

===Setting up Host Server===
If we want to add machines to the Round Table, for example 'nun' and 'samekh', we need to edit their arthor.cfg file so that when our Local Machine passes commands these secondary servers know to perform the search they are given.

$ cat arthor.cfg
MaxThreadsPerSearch=4
AutomaticIndex=false
DATADIR=<Directory where smiles are located>

We then run the jar server on each of these host machines containing data on any available port.

java -jar /nfs/ex9/work/xyz/psql/arthor-3.3-centos7/java/arthor.jar --httpPort <port>

For our local machine, the arthor.cfg file will look different.

$ cat arthor.cfg
[RoundTable]
RemoteClient=http://skynet:<port number where jar server is running>/
RemoteClient=http://hal:<port number where jar server is running>/

Please refer to Section 2 in the RoundTable Documentation file (pages 6-8) for more useful information on configuration.

Then run the following command on n-1-136:

java -jar /nfs/ex9/work/xyz/psql/arthor-3.3-centos7/java/arthor.jar --httpPort <port>

===Public Arthor===
{| class="wikitable"
|-
! CentOS 7 Machine
! Port
! Total Files Size
! Arthor Install Location
! Round Table Data Directory
! Active
|-
| samekh
| 10.20.0.41:8000
| 2.4TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/public_arthor/
| active
|-
| nun
| 10.20.0.40:8000
| 2.4TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/public_arthor/
| active
|-
| n-9-22
| 10.20.9.22:8000
| 2.4TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /export/db4/public_arthor/
| active
|-
|}

===Arthor Round Table Head===
{| class="wikitable"
|-
! CentOS 7 Machine
! Port
! Database
! Arthor Install Location
! Round Table Data Directory
! Active
|-
| samekh
| 10.20.0.41:8080
| Enamine_REAL_Q2-2020-All-13B
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
| nun
| 10.20.0.40:8080
| Enamine_REAL_Q2-2020-All-41B
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
|}

===Arthor Round Table Nodes===
{| class="wikitable"
|-
! CentOS 7 Machine
! Port
! Database
! Total Files Size
! Arthor Install Location
! Round Table Data Directory
!Active
|-
| samekh
| 10.20.0.41:8008
| Enamine_REAL_Q2-2020-All-13B (26 slices)
| 4.5TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
| nun
| 10.20.0.40:8008
| Enamine_REAL_Space_June_2020_S41B (aa-ae, 5 slices), Enamine_REAL_Space_June_2020_M41B (af-am, 8 slices), zinc22_2d (H04~H25, 22 slices)
| 5.6TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
| n-1-17
| 10.20.1.17:8008
| Enamine_REAL_Space_June_2020_M41B (aa-ae, 5 slices), zinc22_2d (H25~H29, 4 slices)
| 3.7TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
| n-5-32
| 10.20.5.32:8008
| Enamine_REAL_Space_June_2020_M41B (an~az, 13 slices)
| 5.6TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
| n-5-33
| 10.20.5.33:8008
| Enamine_REAL_Space_June_2020_M41B (ba-bl, 12 slices)
| 5.3TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
|}

===Arthor (local 8081)===
{| class="wikitable"
|-
! CentOS 7 Machine
! Port
! Database
! Total Files Size
! Arthor Install Location
! Round Table Data Directory
!Active
|-
| samekh
| 10.20.0.41:8081
| Enamine_REAL_Q2-2020-All-13B (26 slices)
| 4.5TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_local_8081/
| active
|-
| nun
| 10.20.0.40:8081
| Enamine_REAL_Space_June_2020_S41B (aa-ae, 5 slices), Enamine_REAL_Space_June_2020_M41B (aa-ar)
| 738GB (soft linked) + 1.9TB = 2.6TB total
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_local_8081/
| active
|-
|}

Arthor Documentation for Future Developer

2021-01-06T20:19:51Z

Jgutie11:

Written by Jennifer Young on December 16, 2019. Last edited January 05, 2021

==Install and Set Up on TomCat (Method 1)==
Arthor ran on n-1-136, which runs CentOS Linux release 7.7.1908 (Core). You can check the version of CentOS with the following command
cat /etc/centos-release

Check your current version of Java with the following command:
java -version

On n-1-136 we are running openjdk version "1.8.0_222", OpenJDK Runtime Environment (build 1.8.0_222-b10), and OpenJDK 64-Bit Server VM (build 25.222-b10, mixed mode)
If Java is not installed, install it using yum

==See this wiki page for more detailed information about installing Tomcat on our cluster==
http://wiki.docking.org/index.php/Tomcat_Installation

==Open port for Arthor==
In order for Arthor to be usable in the browser, the port you wish to run it on must be opened.
https://www.thegeekdiary.com/how-to-open-a-ports-in-centos-rhel-7/

===Step 1: Check Port Status===
Check that the port is not open and that Apache is not showing that port.
netstat -na | grep <port number you are checking>

lsof -i -P |grep http

===Step 2: Check Port Status in IP Tables===
iptables-save | grep <port number you are checking>

I skipped Step 3 from the guide, because there was a lot of information in the /etc/services file and I didn't want to edit it and break something.

===Step 4: Open Firewall Ports===
I did not include the zone=public section because the stand-alone servers are usually used for private instances of Arthor and SmallWorld.
Run as root.
firewall-cmd --add-port=<port number you are adding>/tcp --permanent

You need to reload the firewall after a change is made.
firewall-cmd --reload

===Step 5: Check that port is working===
To check that the port is active, run.
iptables -nL

You should see something along the lines of:
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:<port number you're adding> ctstate NEW,UNTRACKED

==How to run standalone Arthor instance==

===Step 1: Use or start a bash shell===
You can check your default shell using
echo $SHELL

If your default shell is csh, use
bash
to start a new bash shell in the current terminal window. Note that echo $SHELL will show you your default shell regardless of the current shell.

===Step 2: Set your environment variables===
export ARTHOR_DIR=/opt/nextmove/arthor/arthor-3.1-centos7
export PATH=$ARTHOR_DIR/bin/:$PATH

Make sure the ARTHOR_DIR variable is set to the directory for the latest version of Arthor or whichever version you would like to test.
The PATH environment variable is needed if you wish to use the Arthor tools from the command line

===Step 3: Run the arthor-server.jar===
java -jar /opt/nextmove/arthor/arthor-3.0-rt-beta-linux/java/arthor-server.jar --httpPort <your httpPort>

==Setting environment variables for TomCat Server==
Set the environment variables in the setenv.sh file. Note: Be sure to edit the file in the directory corresponding to the latest version of TomCat. As of December 2019, we are running 9.0.27 on n-1-136.

vim /opt/tomcat/apache-tomcat-9.0.27/bin/setenv.sh

Add the line below to the setenv.sh file above, or substitute the path to wherever you currently store the arthor.cfg file
export ARTHOR_CONFIG=/usr/local/tomcat/arthor.cfg

Here is an example of the arthor.cfg file:
# Arthor generated config file
BINDIR=/opt/nextmove/arthor/arthor-2.1.2-centos7/bin
DATADIR=/usr/local/tomcat/arthor_data
STAGEDIR=/usr/local/arthor_data/stage
NTHREADS=64 .
NODEAFFINITY=true
SearchAsYouDraw=true
AutomaticIndex=true
DEPICTION=./depict/bot/svg?w=%w&h=%h&svgunits=px&smi=%s&zoom=0.8&sma=%m&smalim=1
RESOLVER=

'''Important parts of the arthor.cfg file'''

'''BINDIR''' is the location of the Arthor command line binaries. These are used to generate the Arthor index files and to perform searches directly on n-1-136. An example of this would be using atdbgrep for substructure search.

'''DATADIR''' This is the directory where the Arthor data files live. Location where the index files will be created and loaded from.

'''STAGEDIR''' Location where the index files will be built before being moved into the DATADIR.

'''NTHREADS''' The number of threads to use for both ATDB and ATFP searches

Set '''AutomaticIndex''' to false if you don't want new smiles files added to the data directory to be indexed automatically

==Background==
Before working with Arthor, it is recommended that you familiarize yourself with the Arthor documentation. Some useful pages to look at include 3-5, 22-25 and 33-39. Of course, reading everything would be the best!

==Checking Disk Space Usage==
Before building arthor indexes, it's always a good thing to check what percent of the memory is being used. Try to be cautious with how much memory you have left, and make sure to check while building indexes to make sure that you have enough space. To check, run the following command:

df -h /<directory with disc>

==Building Large Databases==
At the moment, we are building databases of size 500M molecules by merging smile files. There are multiple methods of trying to create large databases, one being merging based off of the same H?? prefix and stopping once the database reaches > 500M molecules (or whatever upperbound you want to use). Here is some python code that simulates this merging process. Essentially the program takes all of the .smi files within an input directory, sorts them lexiographically, and begins merging these .smi files together in order until the size reaches > 500M molecules.

Feel free to modify it if you think a better method exists.

import subprocess
import sys
import os

from os import listdir
from os.path import isfile, join

mypath = "<Path to directory holding .smi files>"
onlyfiles = [f for f in listdir(mypath) if isfile(join(mypath, f))]
onlyfiles.sort()

create_fp = True
cur_mols = 0
lower_bound = 500000000
upper_bound = 600000000
files_to_merge = []

def merge_files(f_t_m):
arr = f_t_m[0].split(".")
arr2 = f_t_m[len(f_t_m) - 1].split(".")
file_name_merge = (arr[0] + "_" + arr2[0] + ".smi")
print ("File being created: " + file_name_merge)

for file in f_t_m:
tmp = file.split(".")
process = subprocess.Popen("cat " + join(mypath, file) + " >> " + file_name_merge, shell=True)
process.wait()

for file in onlyfiles:
arr = file.split(".")

if (arr[len(arr) - 1] == "smi"):
print("Working with " + file)
mol = sum(1 for line in open(join(mypath, file)))
print(file, mol, cur_mols)

if (cur_mols + mol > lower_bound):
if (cur_mols + mol < upper_bound):
files_to_merge.append(file)
merge_files(files_to_merge)
cur_mols = 0
files_to_merge.clear()
else:
merge_files(files_to_merge)
files_to_merge.clear()
files_to_merge.append(file)
merge_files(files_to_merge)
cur_mols = 0
files_to_merge.clear()
else:
cur_mols += mol
files_to_merge.append(file)

if (len(files_to_merge) != 0):
merge_files(files_to_merge)

==Building Arthor Indexes==
Once you've merged the .smi files together, it's time to start building the databases themselves. To do this we use the command

smi2atdb -j 0 -p <The .smi file> <The .atdb>

The flag "-j 0" enables parallel generation and utilizes all available processors to generate the .atdb file. The "-p" flag stores the offset position in the ATDB file. Since we're building indexes for the Web Application, you must use the "-p" flag when building indexes. Please note that the name of the .smi file should also be the name of the .atdb file. That way, the Web Application knows to use these files together and correctly display the required images. Refer to pages 33-34 in the Arthor documentation for more information.

If there are too many large .smi files and you do not want to manually build each .atdb file, you can use this python script which takes all of the .smi files in the current directory and converts them to .atdb files. Make sure to modify mypath to the directory containing the .smi files. You can change the variable "create_fp" to false if you don't want to create .atdb.fp files (refer to page 9 in the Arthor documentation).

import subprocess
import sys
import os

from os import listdir
from os.path import isfile, join

mypath = "<Path containing the .smi files"
onlyfiles = [f for f in listdir(mypath) if isfile(join(mypath, f))]

create_fp = True

for file in onlyfiles:
arr = file.split(".")

if (arr[len(arr) - 1] == "smi"):
process = subprocess.Popen("/nfs/ex9/work/xyz/psql/arthor-3.3-centos7/bin/smi2atdb -j 0 -p {0} {1}.atdb".format(join(mypath, file), arr[0]), shell=True)
process.wait()

print("SUCCESS! {0}.atdb file was created!".format(arr[0]))

if (create_fp):
process = subprocess.Popen("/nfs/ex9/work/xyz/psql/arthor-3.3-centos7/bin/atdb2fp -j 0 {0}.atdb".format(arr[0]), shell=True)
process.wait()

print("SUCCESS! {0}.atdb.fp file was created!".format(arr[0]))

==Uploading Indexes to the Web Application==
One can upload indexes to the Web Application by changing the "DATADIR" variable in the arthor.cfg file to the directory holding the .atdb files. This is already set up on n-1-136 and n-5-34.

==Further Arthor Optimizations==
The following edits can be made to the arthor.cfg to optimize substructure search queries. More information can be found in pages 6-8 in the Arthor Documentation file.

'''NodeAffinity NUMA''': optimized flag, pin processing to specific CPU sets to where the data is located in memory. There is a small start-up cost and is most useful for long running services (see Non-Uniform Memory Access (NUMA))

'''AsyncHitCountAllowed=true|false''' After fetching a page from a substructure or formula search the server will spin off a background process to count the total number of hits. This can be resource intensive for large databases and may not be desirable for servers under heavy load and may not even be needed.

'''AsyncHitCountMax=#''' The upper-bound for the number of hits to retrieve in background searches. If very generic queries are issued (e.g. benzene or methane) hundred’s of millions of hits may be counted. Setting this value to anything other than zero (e.g. 10,000) will stop the background search if it exceeds this limit. Note some pathological queries may find very few hits but still end up looking at everything.

'''MaxConcurrentSearches=#''' Controls the maximum number of searches that can be run concurrently by setting the database pool size. The searches may be on the same or different databases. If a search comes in and the pool is full it will have to wait for another search to finish - this increases the request time. 

Typically if each search is using all the processing cores on a machine then additional searches will run at 1/Nth the speed. If the request time is substantially larger that the search time the request had to wait for resources to become available. When switching between a large number of databases it can be useful to have a larger pool size, the only trade off is keeping file pointers open.  Default: 6

'''Binary Fingerprint Folding''' Arthor uses binary circular fingerprints (ECFP4/radius=2) for similarity. When creating an ATFP index you can specify how large to make your fingerprints. Circular fingerprints are sparser than path based fingerprints (e.g. Daylight) and so can be folded smaller without too much degradation in performance. Folding can significantly reduce the footprint size of a database and improve search speeds. A 256-bit fingerprint takes up 1/4 of the space of 1024-bit and can therefore be traversed 4x faster.

This is more important for very large databases with billions of compounds, in such instances a minor drop in precision is likely tolerable as ultimately all that happens is some hits may swap places in the hit list.

==Virtual Memory==
In addition to modifying the arthor.cfg file, virtual memory can also be used to make queries faster. There can still be More information can be found in pages 10-16 in the Arthor Documentation.

==Setting up Round Table==
This is a new feature in Arthor 3.0 and is currently beta (January 2020). See section 2.4 in the manual
As explained in the manual, "Round Table allows you to serve and split chemical searches across multiple host machines. The implementation provides a lightweight proxy that forwards requests to other Arthor host servers that do the actual search. Communication is done using the existing Web APIs.

Since Arthor requires CentOS 7, as of January 2020 we have 6 servers that are capable of running Arthor with Round Table.

===Setting up Host Server===
If we want to add machines to the Round Table, for example 'nun' and 'samekh', we need to edit their arthor.cfg file so that when our Local Machine passes commands these secondary servers know to perform the search they are given.

$ cat arthor.cfg
MaxThreadsPerSearch=4
AutomaticIndex=false
DATADIR=<Directory where smiles are located>

We then run the jar server on each of these host machines containing data on any available port.

java -jar /nfs/ex9/work/xyz/psql/arthor-3.3-centos7/java/arthor.jar --httpPort <port>

For our local machine, the arthor.cfg file will look different.

$ cat arthor.cfg
[RoundTable]
RemoteClient=http://skynet:<port number where jar server is running>/
RemoteClient=http://hal:<port number where jar server is running>/

Please refer to Section 2 in the RoundTable Documentation file (pages 6-8) for more useful information on configuration.

Then run the following command on n-1-136:

java -jar /nfs/ex9/work/xyz/psql/arthor-3.3-centos7/java/arthor.jar --httpPort <port>

===Public Arthor===
{| class="wikitable"
|-
! CentOS 7 Machine
! Port
! Total Files Size
! Arthor Install Location
! Round Table Data Directory
! Active
|-
| samekh
| 10.20.0.41:8000
| 2.4TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/public_arthor/
| active
|-
| nun
| 10.20.0.40:8000
| 2.4TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/public_arthor/
| active
|-
| n-9-22
| 10.20.9.22:8000
| 2.4TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /export/db4/public_arthor/
| active
|-
|}

===Arthor Round Table Head===
{| class="wikitable"
|-
! CentOS 7 Machine
! Port
! Database
! Arthor Install Location
! Round Table Data Directory
! Active
|-
| samekh
| 10.20.0.41:8080
| Enamine_REAL_Q2-2020-All-13B
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
| nun
| 10.20.0.40:8080
| Enamine_REAL_Q2-2020-All-41B
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
|}

===Arthor Round Table Nodes===
{| class="wikitable"
|-
! CentOS 7 Machine
! Port
! Database
! Total Files Size
! Arthor Install Location
! Round Table Data Directory
!Active
|-
| samekh
| 10.20.0.41:8008
| Enamine_REAL_Q2-2020-All-13B (26 slices)
| 4.5TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
| nun
| 10.20.0.40:8008
| Enamine_REAL_Space_June_2020_S41B (aa-ae, 5 slices)
| 738GB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
| n-5-33
| 10.20.5.33:8008
| Enamine_REAL_Space_June_2020_M41B (ba-bl, 12 slices)
| 5.3TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
|}

===Arthor (local 8081)===
{| class="wikitable"
|-
! CentOS 7 Machine
! Port
! Database
! Total Files Size
! Arthor Install Location
! Round Table Data Directory
!Active
|-
| samekh
| 10.20.0.41:8081
| Enamine_REAL_Q2-2020-All-13B (26 slices)
| 4.5TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_local_8081/
| active
|-
| nun
| 10.20.0.40:8081
| Enamine_REAL_Space_June_2020_S41B (aa-ae, 5 slices), Enamine_REAL_Space_June_2020_M41B (aa-ar)
| 738GB (soft linked) + 1.9TB = 2.6TB total
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_local_8081/
| active
|-
|}

Arthor Documentation for Future Developer

2021-01-06T01:44:16Z

Jgutie11: /* Arthor Round Table Nodes */

Written by Jennifer Young on December 16, 2019. Last edited January 05, 2021

==Install and Set Up on TomCat (Method 1)==
Arthor ran on n-1-136, which runs CentOS Linux release 7.7.1908 (Core). You can check the version of CentOS with the following command
cat /etc/centos-release

Check your current version of Java with the following command:
java -version

On n-1-136 we are running openjdk version "1.8.0_222", OpenJDK Runtime Environment (build 1.8.0_222-b10), and OpenJDK 64-Bit Server VM (build 25.222-b10, mixed mode)
If Java is not installed, install it using yum

==See this wiki page for more detailed information about installing Tomcat on our cluster==
http://wiki.docking.org/index.php/Tomcat_Installation

==Open port for Arthor==
In order for Arthor to be usable in the browser, the port you wish to run it on must be opened.
https://www.thegeekdiary.com/how-to-open-a-ports-in-centos-rhel-7/

===Step 1: Check Port Status===
Check that the port is not open and that Apache is not showing that port.
netstat -na | grep <port number you are checking>

lsof -i -P |grep http

===Step 2: Check Port Status in IP Tables===
iptables-save | grep <port number you are checking>

I skipped Step 3 from the guide, because there was a lot of information in the /etc/services file and I didn't want to edit it and break something.

===Step 4: Open Firewall Ports===
I did not include the zone=public section because the stand-alone servers are usually used for private instances of Arthor and SmallWorld.
Run as root.
firewall-cmd --add-port=<port number you are adding>/tcp --permanent

You need to reload the firewall after a change is made.
firewall-cmd --reload

===Step 5: Check that port is working===
To check that the port is active, run.
iptables -nL

You should see something along the lines of:
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:<port number you're adding> ctstate NEW,UNTRACKED

==How to run standalone Arthor instance==

===Step 1: Use or start a bash shell===
You can check your default shell using
echo $SHELL

If your default shell is csh, use
bash
to start a new bash shell in the current terminal window. Note that echo $SHELL will show you your default shell regardless of the current shell.

===Step 2: Set your environment variables===
export ARTHOR_DIR=/opt/nextmove/arthor/arthor-3.1-centos7
export PATH=$ARTHOR_DIR/bin/:$PATH

Make sure the ARTHOR_DIR variable is set to the directory for the latest version of Arthor or whichever version you would like to test.
The PATH environment variable is needed if you wish to use the Arthor tools from the command line

===Step 3: Run the arthor-server.jar===
java -jar /opt/nextmove/arthor/arthor-3.0-rt-beta-linux/java/arthor-server.jar --httpPort <your httpPort>

==Setting environment variables for TomCat Server==
Set the environment variables in the setenv.sh file. Note: Be sure to edit the file in the directory corresponding to the latest version of TomCat. As of December 2019, we are running 9.0.27 on n-1-136.

vim /opt/tomcat/apache-tomcat-9.0.27/bin/setenv.sh

Add the line below to the setenv.sh file above, or substitute the path to wherever you currently store the arthor.cfg file
export ARTHOR_CONFIG=/usr/local/tomcat/arthor.cfg

Here is an example of the arthor.cfg file:
# Arthor generated config file
BINDIR=/opt/nextmove/arthor/arthor-2.1.2-centos7/bin
DATADIR=/usr/local/tomcat/arthor_data
STAGEDIR=/usr/local/arthor_data/stage
NTHREADS=64 .
NODEAFFINITY=true
SearchAsYouDraw=true
AutomaticIndex=true
DEPICTION=./depict/bot/svg?w=%w&h=%h&svgunits=px&smi=%s&zoom=0.8&sma=%m&smalim=1
RESOLVER=

'''Important parts of the arthor.cfg file'''

'''BINDIR''' is the location of the Arthor command line binaries. These are used to generate the Arthor index files and to perform searches directly on n-1-136. An example of this would be using atdbgrep for substructure search.

'''DATADIR''' This is the directory where the Arthor data files live. Location where the index files will be created and loaded from.

'''STAGEDIR''' Location where the index files will be built before being moved into the DATADIR.

'''NTHREADS''' The number of threads to use for both ATDB and ATFP searches

Set '''AutomaticIndex''' to false if you don't want new smiles files added to the data directory to be indexed automatically

==Background==
Before working with Arthor, it is recommended that you familiarize yourself with the Arthor documentation. Some useful pages to look at include 3-5, 22-25 and 33-39. Of course, reading everything would be the best!

==Checking Disk Space Usage==
Before building arthor indexes, it's always a good thing to check what percent of the memory is being used. Try to be cautious with how much memory you have left, and make sure to check while building indexes to make sure that you have enough space. To check, run the following command:

df -h /<directory with disc>

==Building Large Databases==
At the moment, we are building databases of size 500M molecules by merging smile files. There are multiple methods of trying to create large databases, one being merging based off of the same H?? prefix and stopping once the database reaches > 500M molecules (or whatever upperbound you want to use). Here is some python code that simulates this merging process. Essentially the program takes all of the .smi files within an input directory, sorts them lexiographically, and begins merging these .smi files together in order until the size reaches > 500M molecules.

Feel free to modify it if you think a better method exists.

import subprocess
import sys
import os

from os import listdir
from os.path import isfile, join

mypath = "<Path to directory holding .smi files>"
onlyfiles = [f for f in listdir(mypath) if isfile(join(mypath, f))]
onlyfiles.sort()

create_fp = True
cur_mols = 0
lower_bound = 500000000
upper_bound = 600000000
files_to_merge = []

def merge_files(f_t_m):
arr = f_t_m[0].split(".")
arr2 = f_t_m[len(f_t_m) - 1].split(".")
file_name_merge = (arr[0] + "_" + arr2[0] + ".smi")
print ("File being created: " + file_name_merge)

for file in f_t_m:
tmp = file.split(".")
process = subprocess.Popen("cat " + join(mypath, file) + " >> " + file_name_merge, shell=True)
process.wait()

for file in onlyfiles:
arr = file.split(".")

if (arr[len(arr) - 1] == "smi"):
print("Working with " + file)
mol = sum(1 for line in open(join(mypath, file)))
print(file, mol, cur_mols)

if (cur_mols + mol > lower_bound):
if (cur_mols + mol < upper_bound):
files_to_merge.append(file)
merge_files(files_to_merge)
cur_mols = 0
files_to_merge.clear()
else:
merge_files(files_to_merge)
files_to_merge.clear()
files_to_merge.append(file)
merge_files(files_to_merge)
cur_mols = 0
files_to_merge.clear()
else:
cur_mols += mol
files_to_merge.append(file)

if (len(files_to_merge) != 0):
merge_files(files_to_merge)

==Building Arthor Indexes==
Once you've merged the .smi files together, it's time to start building the databases themselves. To do this we use the command

smi2atdb -j 0 -p <The .smi file> <The .atdb>

The flag "-j 0" enables parallel generation and utilizes all available processors to generate the .atdb file. The "-p" flag stores the offset position in the ATDB file. Since we're building indexes for the Web Application, you must use the "-p" flag when building indexes. Please note that the name of the .smi file should also be the name of the .atdb file. That way, the Web Application knows to use these files together and correctly display the required images. Refer to pages 33-34 in the Arthor documentation for more information.

If there are too many large .smi files and you do not want to manually build each .atdb file, you can use this python script which takes all of the .smi files in the current directory and converts them to .atdb files. Make sure to modify mypath to the directory containing the .smi files. You can change the variable "create_fp" to false if you don't want to create .atdb.fp files (refer to page 9 in the Arthor documentation).

import subprocess
import sys
import os

from os import listdir
from os.path import isfile, join

mypath = "<Path containing the .smi files"
onlyfiles = [f for f in listdir(mypath) if isfile(join(mypath, f))]

create_fp = True

for file in onlyfiles:
arr = file.split(".")

if (arr[len(arr) - 1] == "smi"):
process = subprocess.Popen("/nfs/ex9/work/xyz/psql/arthor-3.3-centos7/bin/smi2atdb -j 0 -p {0} {1}.atdb".format(join(mypath, file), arr[0]), shell=True)
process.wait()

print("SUCCESS! {0}.atdb file was created!".format(arr[0]))

if (create_fp):
process = subprocess.Popen("/nfs/ex9/work/xyz/psql/arthor-3.3-centos7/bin/atdb2fp -j 0 {0}.atdb".format(arr[0]), shell=True)
process.wait()

print("SUCCESS! {0}.atdb.fp file was created!".format(arr[0]))

==Uploading Indexes to the Web Application==
One can upload indexes to the Web Application by changing the "DATADIR" variable in the arthor.cfg file to the directory holding the .atdb files. This is already set up on n-1-136 and n-5-34.

==Further Arthor Optimizations==
The following edits can be made to the arthor.cfg to optimize substructure search queries. More information can be found in pages 6-8 in the Arthor Documentation file.

'''NodeAffinity NUMA''': optimized flag, pin processing to specific CPU sets to where the data is located in memory. There is a small start-up cost and is most useful for long running services (see Non-Uniform Memory Access (NUMA))

'''AsyncHitCountAllowed=true|false''' After fetching a page from a substructure or formula search the server will spin off a background process to count the total number of hits. This can be resource intensive for large databases and may not be desirable for servers under heavy load and may not even be needed.

'''AsyncHitCountMax=#''' The upper-bound for the number of hits to retrieve in background searches. If very generic queries are issued (e.g. benzene or methane) hundred’s of millions of hits may be counted. Setting this value to anything other than zero (e.g. 10,000) will stop the background search if it exceeds this limit. Note some pathological queries may find very few hits but still end up looking at everything.

'''MaxConcurrentSearches=#''' Controls the maximum number of searches that can be run concurrently by setting the database pool size. The searches may be on the same or different databases. If a search comes in and the pool is full it will have to wait for another search to finish - this increases the request time. 

Typically if each search is using all the processing cores on a machine then additional searches will run at 1/Nth the speed. If the request time is substantially larger that the search time the request had to wait for resources to become available. When switching between a large number of databases it can be useful to have a larger pool size, the only trade off is keeping file pointers open.  Default: 6

'''Binary Fingerprint Folding''' Arthor uses binary circular fingerprints (ECFP4/radius=2) for similarity. When creating an ATFP index you can specify how large to make your fingerprints. Circular fingerprints are sparser than path based fingerprints (e.g. Daylight) and so can be folded smaller without too much degradation in performance. Folding can significantly reduce the footprint size of a database and improve search speeds. A 256-bit fingerprint takes up 1/4 of the space of 1024-bit and can therefore be traversed 4x faster.

This is more important for very large databases with billions of compounds, in such instances a minor drop in precision is likely tolerable as ultimately all that happens is some hits may swap places in the hit list.

==Virtual Memory==
In addition to modifying the arthor.cfg file, virtual memory can also be used to make queries faster. There can still be More information can be found in pages 10-16 in the Arthor Documentation.

==Setting up Round Table==
This is a new feature in Arthor 3.0 and is currently beta (January 2020). See section 2.4 in the manual
As explained in the manual, "Round Table allows you to serve and split chemical searches across multiple host machines. The implementation provides a lightweight proxy that forwards requests to other Arthor host servers that do the actual search. Communication is done using the existing Web APIs.

Since Arthor requires CentOS 7, as of January 2020 we have 6 servers that are capable of running Arthor with Round Table.

===Setting up Host Server===
If we want to add machines to the Round Table, for example 'nun' and 'samekh', we need to edit their arthor.cfg file so that when our Local Machine passes commands these secondary servers know to perform the search they are given.

$ cat arthor.cfg
MaxThreadsPerSearch=4
AutomaticIndex=false
DATADIR=<Directory where smiles are located>

We then run the jar server on each of these host machines containing data on any available port.

java -jar /nfs/ex9/work/xyz/psql/arthor-3.3-centos7/java/arthor.jar --httpPort <port>

For our local machine, the arthor.cfg file will look different.

$ cat arthor.cfg
[RoundTable]
RemoteClient=http://skynet:<port number where jar server is running>/
RemoteClient=http://hal:<port number where jar server is running>/

Please refer to Section 2 in the RoundTable Documentation file (pages 6-8) for more useful information on configuration.

Then run the following command on n-1-136:

java -jar /nfs/ex9/work/xyz/psql/arthor-3.3-centos7/java/arthor.jar --httpPort <port>

===Public Arthor===
{| class="wikitable"
|-
! CentOS 7 Machine
! Port
! Arthor Install Location
! Round Table Data Directory
! Active
|-
| samekh
| 10.20.0.41:8000
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/public_arthor/
| active
|-
| nun
| 10.20.0.40:8000
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/public_arthor/
| active
|-
| n-9-22
| 10.20.9.22:8000
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /export/db4/public_arthor/
| active
|-
|}

===Arthor Round Table Head===
{| class="wikitable"
|-
! CentOS 7 Machine
! Port
! Database
! Arthor Install Location
! Round Table Data Directory
! Active
|-
| samekh
| 10.20.0.41:8080
| Enamine_REAL_Q2-2020-All-13B
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
| nun
| 10.20.0.40:8080
| Enamine_REAL_Q2-2020-All-41B
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
|}

===Arthor Round Table Nodes===
{| class="wikitable"
|-
! CentOS 7 Machine
! Port
! Database
! Total Files Size
! Arthor Install Location
! Round Table Data Directory
!Active
|-
| samekh
| 10.20.0.41:8008
| Enamine_REAL_Q2-2020-M-13B (aa-ax, 24 slices), Enamine_REAL_Q2-2020-S-13B (aa-ab, 2 slices)
| 3.5TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
| nun
| 10.20.0.40:8008
| Enamine_REAL_Space_June_2020_S41B (aa-ae, 5 slices)
| 738GB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
| n-5-33
| 10.20.5.33:8008
| Enamine_REAL_Space_June_2020_M41B (ba-bl, 12 slices)
| 5.3TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
|}

===Arthor (local 8081)===
{| class="wikitable"
|-
! CentOS 7 Machine
! Port
! Database
! Total Files Size
! Arthor Install Location
! Round Table Data Directory
!Active
|-
| samekh
| 10.20.0.41:8081
| Enamine_REAL_Q2-2020-All-13B (26 slices)
| 2.5TB (soft linked) + 2.0TB = 4.5TB total
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_local_8081/
| active
|-
| nun
| 10.20.0.40:8081
| Enamine_REAL_Space_June_2020_S41B (aa-ae, 5 slices), Enamine_REAL_Space_June_2020_M41B (aa-ar)
| 738GB (soft linked) + 1.9TB = 2.6TB total
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_local_8081/
| active
|-
|}

Arthor Documentation for Future Developer

2021-01-06T01:43:15Z

Jgutie11: /* Arthor Round Table Nodes */

Written by Jennifer Young on December 16, 2019. Last edited January 05, 2021

==Install and Set Up on TomCat (Method 1)==
Arthor ran on n-1-136, which runs CentOS Linux release 7.7.1908 (Core). You can check the version of CentOS with the following command
cat /etc/centos-release

Check your current version of Java with the following command:
java -version

On n-1-136 we are running openjdk version "1.8.0_222", OpenJDK Runtime Environment (build 1.8.0_222-b10), and OpenJDK 64-Bit Server VM (build 25.222-b10, mixed mode)
If Java is not installed, install it using yum

==See this wiki page for more detailed information about installing Tomcat on our cluster==
http://wiki.docking.org/index.php/Tomcat_Installation

==Open port for Arthor==
In order for Arthor to be usable in the browser, the port you wish to run it on must be opened.
https://www.thegeekdiary.com/how-to-open-a-ports-in-centos-rhel-7/

===Step 1: Check Port Status===
Check that the port is not open and that Apache is not showing that port.
netstat -na | grep <port number you are checking>

lsof -i -P |grep http

===Step 2: Check Port Status in IP Tables===
iptables-save | grep <port number you are checking>

I skipped Step 3 from the guide, because there was a lot of information in the /etc/services file and I didn't want to edit it and break something.

===Step 4: Open Firewall Ports===
I did not include the zone=public section because the stand-alone servers are usually used for private instances of Arthor and SmallWorld.
Run as root.
firewall-cmd --add-port=<port number you are adding>/tcp --permanent

You need to reload the firewall after a change is made.
firewall-cmd --reload

===Step 5: Check that port is working===
To check that the port is active, run.
iptables -nL

You should see something along the lines of:
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:<port number you're adding> ctstate NEW,UNTRACKED

==How to run standalone Arthor instance==

===Step 1: Use or start a bash shell===
You can check your default shell using
echo $SHELL

If your default shell is csh, use
bash
to start a new bash shell in the current terminal window. Note that echo $SHELL will show you your default shell regardless of the current shell.

===Step 2: Set your environment variables===
export ARTHOR_DIR=/opt/nextmove/arthor/arthor-3.1-centos7
export PATH=$ARTHOR_DIR/bin/:$PATH

Make sure the ARTHOR_DIR variable is set to the directory for the latest version of Arthor or whichever version you would like to test.
The PATH environment variable is needed if you wish to use the Arthor tools from the command line

===Step 3: Run the arthor-server.jar===
java -jar /opt/nextmove/arthor/arthor-3.0-rt-beta-linux/java/arthor-server.jar --httpPort <your httpPort>

==Setting environment variables for TomCat Server==
Set the environment variables in the setenv.sh file. Note: Be sure to edit the file in the directory corresponding to the latest version of TomCat. As of December 2019, we are running 9.0.27 on n-1-136.

vim /opt/tomcat/apache-tomcat-9.0.27/bin/setenv.sh

Add the line below to the setenv.sh file above, or substitute the path to wherever you currently store the arthor.cfg file
export ARTHOR_CONFIG=/usr/local/tomcat/arthor.cfg

Here is an example of the arthor.cfg file:
# Arthor generated config file
BINDIR=/opt/nextmove/arthor/arthor-2.1.2-centos7/bin
DATADIR=/usr/local/tomcat/arthor_data
STAGEDIR=/usr/local/arthor_data/stage
NTHREADS=64 .
NODEAFFINITY=true
SearchAsYouDraw=true
AutomaticIndex=true
DEPICTION=./depict/bot/svg?w=%w&h=%h&svgunits=px&smi=%s&zoom=0.8&sma=%m&smalim=1
RESOLVER=

'''Important parts of the arthor.cfg file'''

'''BINDIR''' is the location of the Arthor command line binaries. These are used to generate the Arthor index files and to perform searches directly on n-1-136. An example of this would be using atdbgrep for substructure search.

'''DATADIR''' This is the directory where the Arthor data files live. Location where the index files will be created and loaded from.

'''STAGEDIR''' Location where the index files will be built before being moved into the DATADIR.

'''NTHREADS''' The number of threads to use for both ATDB and ATFP searches

Set '''AutomaticIndex''' to false if you don't want new smiles files added to the data directory to be indexed automatically

==Background==
Before working with Arthor, it is recommended that you familiarize yourself with the Arthor documentation. Some useful pages to look at include 3-5, 22-25 and 33-39. Of course, reading everything would be the best!

==Checking Disk Space Usage==
Before building arthor indexes, it's always a good thing to check what percent of the memory is being used. Try to be cautious with how much memory you have left, and make sure to check while building indexes to make sure that you have enough space. To check, run the following command:

df -h /<directory with disc>

==Building Large Databases==
At the moment, we are building databases of size 500M molecules by merging smile files. There are multiple methods of trying to create large databases, one being merging based off of the same H?? prefix and stopping once the database reaches > 500M molecules (or whatever upperbound you want to use). Here is some python code that simulates this merging process. Essentially the program takes all of the .smi files within an input directory, sorts them lexiographically, and begins merging these .smi files together in order until the size reaches > 500M molecules.

Feel free to modify it if you think a better method exists.

import subprocess
import sys
import os

from os import listdir
from os.path import isfile, join

mypath = "<Path to directory holding .smi files>"
onlyfiles = [f for f in listdir(mypath) if isfile(join(mypath, f))]
onlyfiles.sort()

create_fp = True
cur_mols = 0
lower_bound = 500000000
upper_bound = 600000000
files_to_merge = []

def merge_files(f_t_m):
arr = f_t_m[0].split(".")
arr2 = f_t_m[len(f_t_m) - 1].split(".")
file_name_merge = (arr[0] + "_" + arr2[0] + ".smi")
print ("File being created: " + file_name_merge)

for file in f_t_m:
tmp = file.split(".")
process = subprocess.Popen("cat " + join(mypath, file) + " >> " + file_name_merge, shell=True)
process.wait()

for file in onlyfiles:
arr = file.split(".")

if (arr[len(arr) - 1] == "smi"):
print("Working with " + file)
mol = sum(1 for line in open(join(mypath, file)))
print(file, mol, cur_mols)

if (cur_mols + mol > lower_bound):
if (cur_mols + mol < upper_bound):
files_to_merge.append(file)
merge_files(files_to_merge)
cur_mols = 0
files_to_merge.clear()
else:
merge_files(files_to_merge)
files_to_merge.clear()
files_to_merge.append(file)
merge_files(files_to_merge)
cur_mols = 0
files_to_merge.clear()
else:
cur_mols += mol
files_to_merge.append(file)

if (len(files_to_merge) != 0):
merge_files(files_to_merge)

==Building Arthor Indexes==
Once you've merged the .smi files together, it's time to start building the databases themselves. To do this we use the command

smi2atdb -j 0 -p <The .smi file> <The .atdb>

The flag "-j 0" enables parallel generation and utilizes all available processors to generate the .atdb file. The "-p" flag stores the offset position in the ATDB file. Since we're building indexes for the Web Application, you must use the "-p" flag when building indexes. Please note that the name of the .smi file should also be the name of the .atdb file. That way, the Web Application knows to use these files together and correctly display the required images. Refer to pages 33-34 in the Arthor documentation for more information.

If there are too many large .smi files and you do not want to manually build each .atdb file, you can use this python script which takes all of the .smi files in the current directory and converts them to .atdb files. Make sure to modify mypath to the directory containing the .smi files. You can change the variable "create_fp" to false if you don't want to create .atdb.fp files (refer to page 9 in the Arthor documentation).

import subprocess
import sys
import os

from os import listdir
from os.path import isfile, join

mypath = "<Path containing the .smi files"
onlyfiles = [f for f in listdir(mypath) if isfile(join(mypath, f))]

create_fp = True

for file in onlyfiles:
arr = file.split(".")

if (arr[len(arr) - 1] == "smi"):
process = subprocess.Popen("/nfs/ex9/work/xyz/psql/arthor-3.3-centos7/bin/smi2atdb -j 0 -p {0} {1}.atdb".format(join(mypath, file), arr[0]), shell=True)
process.wait()

print("SUCCESS! {0}.atdb file was created!".format(arr[0]))

if (create_fp):
process = subprocess.Popen("/nfs/ex9/work/xyz/psql/arthor-3.3-centos7/bin/atdb2fp -j 0 {0}.atdb".format(arr[0]), shell=True)
process.wait()

print("SUCCESS! {0}.atdb.fp file was created!".format(arr[0]))

==Uploading Indexes to the Web Application==
One can upload indexes to the Web Application by changing the "DATADIR" variable in the arthor.cfg file to the directory holding the .atdb files. This is already set up on n-1-136 and n-5-34.

==Further Arthor Optimizations==
The following edits can be made to the arthor.cfg to optimize substructure search queries. More information can be found in pages 6-8 in the Arthor Documentation file.

'''NodeAffinity NUMA''': optimized flag, pin processing to specific CPU sets to where the data is located in memory. There is a small start-up cost and is most useful for long running services (see Non-Uniform Memory Access (NUMA))

'''AsyncHitCountAllowed=true|false''' After fetching a page from a substructure or formula search the server will spin off a background process to count the total number of hits. This can be resource intensive for large databases and may not be desirable for servers under heavy load and may not even be needed.

'''AsyncHitCountMax=#''' The upper-bound for the number of hits to retrieve in background searches. If very generic queries are issued (e.g. benzene or methane) hundred’s of millions of hits may be counted. Setting this value to anything other than zero (e.g. 10,000) will stop the background search if it exceeds this limit. Note some pathological queries may find very few hits but still end up looking at everything.

'''MaxConcurrentSearches=#''' Controls the maximum number of searches that can be run concurrently by setting the database pool size. The searches may be on the same or different databases. If a search comes in and the pool is full it will have to wait for another search to finish - this increases the request time. 

Typically if each search is using all the processing cores on a machine then additional searches will run at 1/Nth the speed. If the request time is substantially larger that the search time the request had to wait for resources to become available. When switching between a large number of databases it can be useful to have a larger pool size, the only trade off is keeping file pointers open.  Default: 6

'''Binary Fingerprint Folding''' Arthor uses binary circular fingerprints (ECFP4/radius=2) for similarity. When creating an ATFP index you can specify how large to make your fingerprints. Circular fingerprints are sparser than path based fingerprints (e.g. Daylight) and so can be folded smaller without too much degradation in performance. Folding can significantly reduce the footprint size of a database and improve search speeds. A 256-bit fingerprint takes up 1/4 of the space of 1024-bit and can therefore be traversed 4x faster.

This is more important for very large databases with billions of compounds, in such instances a minor drop in precision is likely tolerable as ultimately all that happens is some hits may swap places in the hit list.

==Virtual Memory==
In addition to modifying the arthor.cfg file, virtual memory can also be used to make queries faster. There can still be More information can be found in pages 10-16 in the Arthor Documentation.

==Setting up Round Table==
This is a new feature in Arthor 3.0 and is currently beta (January 2020). See section 2.4 in the manual
As explained in the manual, "Round Table allows you to serve and split chemical searches across multiple host machines. The implementation provides a lightweight proxy that forwards requests to other Arthor host servers that do the actual search. Communication is done using the existing Web APIs.

Since Arthor requires CentOS 7, as of January 2020 we have 6 servers that are capable of running Arthor with Round Table.

===Setting up Host Server===
If we want to add machines to the Round Table, for example 'nun' and 'samekh', we need to edit their arthor.cfg file so that when our Local Machine passes commands these secondary servers know to perform the search they are given.

$ cat arthor.cfg
MaxThreadsPerSearch=4
AutomaticIndex=false
DATADIR=<Directory where smiles are located>

We then run the jar server on each of these host machines containing data on any available port.

java -jar /nfs/ex9/work/xyz/psql/arthor-3.3-centos7/java/arthor.jar --httpPort <port>

For our local machine, the arthor.cfg file will look different.

$ cat arthor.cfg
[RoundTable]
RemoteClient=http://skynet:<port number where jar server is running>/
RemoteClient=http://hal:<port number where jar server is running>/

Please refer to Section 2 in the RoundTable Documentation file (pages 6-8) for more useful information on configuration.

Then run the following command on n-1-136:

java -jar /nfs/ex9/work/xyz/psql/arthor-3.3-centos7/java/arthor.jar --httpPort <port>

===Public Arthor===
{| class="wikitable"
|-
! CentOS 7 Machine
! Port
! Arthor Install Location
! Round Table Data Directory
! Active
|-
| samekh
| 10.20.0.41:8000
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/public_arthor/
| active
|-
| nun
| 10.20.0.40:8000
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/public_arthor/
| active
|-
| n-9-22
| 10.20.9.22:8000
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /export/db4/public_arthor/
| active
|-
|}

===Arthor Round Table Head===
{| class="wikitable"
|-
! CentOS 7 Machine
! Port
! Database
! Arthor Install Location
! Round Table Data Directory
! Active
|-
| samekh
| 10.20.0.41:8080
| Enamine_REAL_Q2-2020-All-13B
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
| nun
| 10.20.0.40:8080
| Enamine_REAL_Q2-2020-All-41B
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
|}

===Arthor Round Table Nodes===
{| class="wikitable"
|-
! CentOS 7 Machine
! Port
! Database
! Total Files Size
! Arthor Install Location
! Round Table Data Directory
!Active
|-
| samekh
| 10.20.0.41:8008
| Enamine_REAL_Q2-2020-M-13B (aa-ax, 24 slices), Enamine_REAL_Q2-2020-S-13B (aa-ab, 2 slices)
| 3.5TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
| nun
| 10.20.0.40:8008
| Enamine_REAL_Space_June_2020_S41B (aa-ae, 5 slices)
| 738GB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
| n-1-17
| 10.20.1.17:8008
| Enamine_REAL_Space_June_2020_M41B (aa-am, 14 slices)
| 4.3TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
| n-5-32
| 10.20.5.32:8008
| Enamine_REAL_Space_June_2020_M41B (an-az, 13 slices)
| 5.0TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
| n-5-33
| 10.20.5.33:8008
| Enamine_REAL_Space_June_2020_M41B (ba-bl, 12 slices)
| 5.3TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
| n-1-20
| 10.20.1.20:8008
| 12, 15, 16 ,27, 27, 30, 36
| 897GB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| not active
|-
| n-5-35
| 10.20.5.35:8008
| 2, 3, 8, 10, 34, 44_results, 44_results2, all-zinc-xad, in-stock-40, on-demand
| 1.5TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| not active
|-
|}

===Arthor (local 8081)===
{| class="wikitable"
|-
! CentOS 7 Machine
! Port
! Database
! Total Files Size
! Arthor Install Location
! Round Table Data Directory
!Active
|-
| samekh
| 10.20.0.41:8081
| Enamine_REAL_Q2-2020-All-13B (26 slices)
| 2.5TB (soft linked) + 2.0TB = 4.5TB total
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_local_8081/
| active
|-
| nun
| 10.20.0.40:8081
| Enamine_REAL_Space_June_2020_S41B (aa-ae, 5 slices), Enamine_REAL_Space_June_2020_M41B (aa-ar)
| 738GB (soft linked) + 1.9TB = 2.6TB total
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_local_8081/
| active
|-
|}

Arthor Documentation for Future Developer

2021-01-06T01:26:42Z

Jgutie11:

Written by Jennifer Young on December 16, 2019. Last edited January 05, 2021

==Install and Set Up on TomCat (Method 1)==
Arthor ran on n-1-136, which runs CentOS Linux release 7.7.1908 (Core). You can check the version of CentOS with the following command
cat /etc/centos-release

Check your current version of Java with the following command:
java -version

On n-1-136 we are running openjdk version "1.8.0_222", OpenJDK Runtime Environment (build 1.8.0_222-b10), and OpenJDK 64-Bit Server VM (build 25.222-b10, mixed mode)
If Java is not installed, install it using yum

==See this wiki page for more detailed information about installing Tomcat on our cluster==
http://wiki.docking.org/index.php/Tomcat_Installation

==Open port for Arthor==
In order for Arthor to be usable in the browser, the port you wish to run it on must be opened.
https://www.thegeekdiary.com/how-to-open-a-ports-in-centos-rhel-7/

===Step 1: Check Port Status===
Check that the port is not open and that Apache is not showing that port.
netstat -na | grep <port number you are checking>

lsof -i -P |grep http

===Step 2: Check Port Status in IP Tables===
iptables-save | grep <port number you are checking>

I skipped Step 3 from the guide, because there was a lot of information in the /etc/services file and I didn't want to edit it and break something.

===Step 4: Open Firewall Ports===
I did not include the zone=public section because the stand-alone servers are usually used for private instances of Arthor and SmallWorld.
Run as root.
firewall-cmd --add-port=<port number you are adding>/tcp --permanent

You need to reload the firewall after a change is made.
firewall-cmd --reload

===Step 5: Check that port is working===
To check that the port is active, run.
iptables -nL

You should see something along the lines of:
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:<port number you're adding> ctstate NEW,UNTRACKED

==How to run standalone Arthor instance==

===Step 1: Use or start a bash shell===
You can check your default shell using
echo $SHELL

If your default shell is csh, use
bash
to start a new bash shell in the current terminal window. Note that echo $SHELL will show you your default shell regardless of the current shell.

===Step 2: Set your environment variables===
export ARTHOR_DIR=/opt/nextmove/arthor/arthor-3.1-centos7
export PATH=$ARTHOR_DIR/bin/:$PATH

Make sure the ARTHOR_DIR variable is set to the directory for the latest version of Arthor or whichever version you would like to test.
The PATH environment variable is needed if you wish to use the Arthor tools from the command line

===Step 3: Run the arthor-server.jar===
java -jar /opt/nextmove/arthor/arthor-3.0-rt-beta-linux/java/arthor-server.jar --httpPort <your httpPort>

==Setting environment variables for TomCat Server==
Set the environment variables in the setenv.sh file. Note: Be sure to edit the file in the directory corresponding to the latest version of TomCat. As of December 2019, we are running 9.0.27 on n-1-136.

vim /opt/tomcat/apache-tomcat-9.0.27/bin/setenv.sh

Add the line below to the setenv.sh file above, or substitute the path to wherever you currently store the arthor.cfg file
export ARTHOR_CONFIG=/usr/local/tomcat/arthor.cfg

Here is an example of the arthor.cfg file:
# Arthor generated config file
BINDIR=/opt/nextmove/arthor/arthor-2.1.2-centos7/bin
DATADIR=/usr/local/tomcat/arthor_data
STAGEDIR=/usr/local/arthor_data/stage
NTHREADS=64 .
NODEAFFINITY=true
SearchAsYouDraw=true
AutomaticIndex=true
DEPICTION=./depict/bot/svg?w=%w&h=%h&svgunits=px&smi=%s&zoom=0.8&sma=%m&smalim=1
RESOLVER=

'''Important parts of the arthor.cfg file'''

'''BINDIR''' is the location of the Arthor command line binaries. These are used to generate the Arthor index files and to perform searches directly on n-1-136. An example of this would be using atdbgrep for substructure search.

'''DATADIR''' This is the directory where the Arthor data files live. Location where the index files will be created and loaded from.

'''STAGEDIR''' Location where the index files will be built before being moved into the DATADIR.

'''NTHREADS''' The number of threads to use for both ATDB and ATFP searches

Set '''AutomaticIndex''' to false if you don't want new smiles files added to the data directory to be indexed automatically

==Background==
Before working with Arthor, it is recommended that you familiarize yourself with the Arthor documentation. Some useful pages to look at include 3-5, 22-25 and 33-39. Of course, reading everything would be the best!

==Checking Disk Space Usage==
Before building arthor indexes, it's always a good thing to check what percent of the memory is being used. Try to be cautious with how much memory you have left, and make sure to check while building indexes to make sure that you have enough space. To check, run the following command:

df -h /<directory with disc>

==Building Large Databases==
At the moment, we are building databases of size 500M molecules by merging smile files. There are multiple methods of trying to create large databases, one being merging based off of the same H?? prefix and stopping once the database reaches > 500M molecules (or whatever upperbound you want to use). Here is some python code that simulates this merging process. Essentially the program takes all of the .smi files within an input directory, sorts them lexiographically, and begins merging these .smi files together in order until the size reaches > 500M molecules.

Feel free to modify it if you think a better method exists.

import subprocess
import sys
import os

from os import listdir
from os.path import isfile, join

mypath = "<Path to directory holding .smi files>"
onlyfiles = [f for f in listdir(mypath) if isfile(join(mypath, f))]
onlyfiles.sort()

create_fp = True
cur_mols = 0
lower_bound = 500000000
upper_bound = 600000000
files_to_merge = []

def merge_files(f_t_m):
arr = f_t_m[0].split(".")
arr2 = f_t_m[len(f_t_m) - 1].split(".")
file_name_merge = (arr[0] + "_" + arr2[0] + ".smi")
print ("File being created: " + file_name_merge)

for file in f_t_m:
tmp = file.split(".")
process = subprocess.Popen("cat " + join(mypath, file) + " >> " + file_name_merge, shell=True)
process.wait()

for file in onlyfiles:
arr = file.split(".")

if (arr[len(arr) - 1] == "smi"):
print("Working with " + file)
mol = sum(1 for line in open(join(mypath, file)))
print(file, mol, cur_mols)

if (cur_mols + mol > lower_bound):
if (cur_mols + mol < upper_bound):
files_to_merge.append(file)
merge_files(files_to_merge)
cur_mols = 0
files_to_merge.clear()
else:
merge_files(files_to_merge)
files_to_merge.clear()
files_to_merge.append(file)
merge_files(files_to_merge)
cur_mols = 0
files_to_merge.clear()
else:
cur_mols += mol
files_to_merge.append(file)

if (len(files_to_merge) != 0):
merge_files(files_to_merge)

==Building Arthor Indexes==
Once you've merged the .smi files together, it's time to start building the databases themselves. To do this we use the command

smi2atdb -j 0 -p <The .smi file> <The .atdb>

The flag "-j 0" enables parallel generation and utilizes all available processors to generate the .atdb file. The "-p" flag stores the offset position in the ATDB file. Since we're building indexes for the Web Application, you must use the "-p" flag when building indexes. Please note that the name of the .smi file should also be the name of the .atdb file. That way, the Web Application knows to use these files together and correctly display the required images. Refer to pages 33-34 in the Arthor documentation for more information.

If there are too many large .smi files and you do not want to manually build each .atdb file, you can use this python script which takes all of the .smi files in the current directory and converts them to .atdb files. Make sure to modify mypath to the directory containing the .smi files. You can change the variable "create_fp" to false if you don't want to create .atdb.fp files (refer to page 9 in the Arthor documentation).

import subprocess
import sys
import os

from os import listdir
from os.path import isfile, join

mypath = "<Path containing the .smi files"
onlyfiles = [f for f in listdir(mypath) if isfile(join(mypath, f))]

create_fp = True

for file in onlyfiles:
arr = file.split(".")

if (arr[len(arr) - 1] == "smi"):
process = subprocess.Popen("/nfs/ex9/work/xyz/psql/arthor-3.3-centos7/bin/smi2atdb -j 0 -p {0} {1}.atdb".format(join(mypath, file), arr[0]), shell=True)
process.wait()

print("SUCCESS! {0}.atdb file was created!".format(arr[0]))

if (create_fp):
process = subprocess.Popen("/nfs/ex9/work/xyz/psql/arthor-3.3-centos7/bin/atdb2fp -j 0 {0}.atdb".format(arr[0]), shell=True)
process.wait()

print("SUCCESS! {0}.atdb.fp file was created!".format(arr[0]))

==Uploading Indexes to the Web Application==
One can upload indexes to the Web Application by changing the "DATADIR" variable in the arthor.cfg file to the directory holding the .atdb files. This is already set up on n-1-136 and n-5-34.

==Further Arthor Optimizations==
The following edits can be made to the arthor.cfg to optimize substructure search queries. More information can be found in pages 6-8 in the Arthor Documentation file.

'''NodeAffinity NUMA''': optimized flag, pin processing to specific CPU sets to where the data is located in memory. There is a small start-up cost and is most useful for long running services (see Non-Uniform Memory Access (NUMA))

'''AsyncHitCountAllowed=true|false''' After fetching a page from a substructure or formula search the server will spin off a background process to count the total number of hits. This can be resource intensive for large databases and may not be desirable for servers under heavy load and may not even be needed.

'''AsyncHitCountMax=#''' The upper-bound for the number of hits to retrieve in background searches. If very generic queries are issued (e.g. benzene or methane) hundred’s of millions of hits may be counted. Setting this value to anything other than zero (e.g. 10,000) will stop the background search if it exceeds this limit. Note some pathological queries may find very few hits but still end up looking at everything.

'''MaxConcurrentSearches=#''' Controls the maximum number of searches that can be run concurrently by setting the database pool size. The searches may be on the same or different databases. If a search comes in and the pool is full it will have to wait for another search to finish - this increases the request time. 

Typically if each search is using all the processing cores on a machine then additional searches will run at 1/Nth the speed. If the request time is substantially larger that the search time the request had to wait for resources to become available. When switching between a large number of databases it can be useful to have a larger pool size, the only trade off is keeping file pointers open.  Default: 6

'''Binary Fingerprint Folding''' Arthor uses binary circular fingerprints (ECFP4/radius=2) for similarity. When creating an ATFP index you can specify how large to make your fingerprints. Circular fingerprints are sparser than path based fingerprints (e.g. Daylight) and so can be folded smaller without too much degradation in performance. Folding can significantly reduce the footprint size of a database and improve search speeds. A 256-bit fingerprint takes up 1/4 of the space of 1024-bit and can therefore be traversed 4x faster.

This is more important for very large databases with billions of compounds, in such instances a minor drop in precision is likely tolerable as ultimately all that happens is some hits may swap places in the hit list.

==Virtual Memory==
In addition to modifying the arthor.cfg file, virtual memory can also be used to make queries faster. There can still be More information can be found in pages 10-16 in the Arthor Documentation.

==Setting up Round Table==
This is a new feature in Arthor 3.0 and is currently beta (January 2020). See section 2.4 in the manual
As explained in the manual, "Round Table allows you to serve and split chemical searches across multiple host machines. The implementation provides a lightweight proxy that forwards requests to other Arthor host servers that do the actual search. Communication is done using the existing Web APIs.

Since Arthor requires CentOS 7, as of January 2020 we have 6 servers that are capable of running Arthor with Round Table.

===Setting up Host Server===
If we want to add machines to the Round Table, for example 'nun' and 'samekh', we need to edit their arthor.cfg file so that when our Local Machine passes commands these secondary servers know to perform the search they are given.

$ cat arthor.cfg
MaxThreadsPerSearch=4
AutomaticIndex=false
DATADIR=<Directory where smiles are located>

We then run the jar server on each of these host machines containing data on any available port.

java -jar /nfs/ex9/work/xyz/psql/arthor-3.3-centos7/java/arthor.jar --httpPort <port>

For our local machine, the arthor.cfg file will look different.

$ cat arthor.cfg
[RoundTable]
RemoteClient=http://skynet:<port number where jar server is running>/
RemoteClient=http://hal:<port number where jar server is running>/

Please refer to Section 2 in the RoundTable Documentation file (pages 6-8) for more useful information on configuration.

Then run the following command on n-1-136:

java -jar /nfs/ex9/work/xyz/psql/arthor-3.3-centos7/java/arthor.jar --httpPort <port>

===Public Arthor===
{| class="wikitable"
|-
! CentOS 7 Machine
! Port
! Arthor Install Location
! Round Table Data Directory
! Active
|-
| samekh
| 10.20.0.41:8000
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/public_arthor/
| active
|-
| nun
| 10.20.0.40:8000
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/public_arthor/
| active
|-
| n-9-22
| 10.20.9.22:8000
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /export/db4/public_arthor/
| active
|-
|}

===Arthor Round Table Head===
{| class="wikitable"
|-
! CentOS 7 Machine
! Port
! Database
! Arthor Install Location
! Round Table Data Directory
! Active
|-
| samekh
| 10.20.0.41:8080
| Enamine_REAL_Q2-2020-All-13B
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
| nun
| 10.20.0.40:8080
| Enamine_REAL_Q2-2020-All-41B
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
|}

===Arthor Round Table Nodes===
{| class="wikitable"
|-
! CentOS 7 Machine
! Port
! Database
! Total Files Size
! Arthor Install Location
! Round Table Data Directory
!Active
|-
| samekh
| 10.20.0.41:8008
| Enamine_REAL_Q2-2020-M-13B (am-ax, 12 slices), Enamine_REAL_Q2-2020-S-13B (aa-ab, 2 slices)
| 2.5TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
| nun
| 10.20.0.40:8008
| Enamine_REAL_Space_June_2020_S41B (aa-ae, 5 slices)
| 738GB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
| n-1-16
| 10.20.1.16:8008
| Enamine_REAL_Q2-2020-M-13B (aa-al, 12 slices)
| 2.0TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
| n-1-17
| 10.20.1.17:8008
| Enamine_REAL_Space_June_2020_M41B (aa-am, 14 slices)
| 4.3TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
| n-5-32
| 10.20.5.32:8008
| Enamine_REAL_Space_June_2020_M41B (an-az, 13 slices)
| 5.0TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
| n-5-33
| 10.20.5.33:8008
| Enamine_REAL_Space_June_2020_M41B (ba-bl, 12 slices)
| 5.3TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
| qof
| 10.20.9.29:8008
| 18, 25, 37, 45, 5
| 173GB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /export/ex9/work/btingle/auto_atdb/
| not active
|-
| lamed
| 10.20.9.15: 8008
| 17, 29, 40, 6
| 512GB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /export/ex6/work/btingle/auto_atdb
| not active
|-
| n-1-20
| 10.20.1.20:8008
| 12, 15, 16 ,27, 27, 30, 36
| 897GB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| not active
|-
| n-5-34
| 10.20.5.34:8008
| 9, 19, 21, 28, 31, 42, all-zinc-xab, all-zinc-xafn in-stock
| 875GB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| not active
|-
| n-5-35
| 10.20.5.35:8008
| 2, 3, 8, 10, 34, 44_results, 44_results2, all-zinc-xad, in-stock-40, on-demand
| 1.5TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| not active
|-
|}

===Arthor (local 8081)===
{| class="wikitable"
|-
! CentOS 7 Machine
! Port
! Database
! Total Files Size
! Arthor Install Location
! Round Table Data Directory
!Active
|-
| samekh
| 10.20.0.41:8081
| Enamine_REAL_Q2-2020-All-13B (26 slices)
| 2.5TB (soft linked) + 2.0TB = 4.5TB total
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_local_8081/
| active
|-
| nun
| 10.20.0.40:8081
| Enamine_REAL_Space_June_2020_S41B (aa-ae, 5 slices), Enamine_REAL_Space_June_2020_M41B (aa-ar)
| 738GB (soft linked) + 1.9TB = 2.6TB total
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_local_8081/
| active
|-
|}

Arthor Documentation for Future Developer

2021-01-06T01:18:28Z

Jgutie11: /* Arthor Round Table Head */

Written by Jennifer Young on December 16, 2019. Last edited January 30, 2020

==Install and Set Up on TomCat==
Arthor currently runs on n-1-136, which runs CentOS Linux release 7.7.1908 (Core). You can check the version of CentOS with the following command
cat /etc/centos-release

Check your current version of Java with the following command:
java -version

On n-1-136 we are running openjdk version "1.8.0_222", OpenJDK Runtime Environment (build 1.8.0_222-b10), and OpenJDK 64-Bit Server VM (build 25.222-b10, mixed mode)
If Java is not installed, install it using yum

==See this wiki page for more detailed information about installing Tomcat on our cluster==
http://wiki.docking.org/index.php/Tomcat_Installation

==Open port for Arthor==
In order for Arthor to be usable in the browser, the port you wish to run it on must be opened.
https://www.thegeekdiary.com/how-to-open-a-ports-in-centos-rhel-7/

===Step 1: Check Port Status===
Check that the port is not open and that Apache is not showing that port.
netstat -na | grep <port number you are checking>

lsof -i -P |grep http

===Step 2: Check Port Status in IP Tables===
iptables-save | grep <port number you are checking>

I skipped Step 3 from the guide, because there was a lot of information in the /etc/services file and I didn't want to edit it and break something.

===Step 4: Open Firewall Ports===
I did not include the zone=public section because the stand-alone servers are usually used for private instances of Arthor and SmallWorld.
Run as root.
firewall-cmd --add-port=<port number you are adding>/tcp --permanent

You need to reload the firewall after a change is made.
firewall-cmd --reload

===Step 5: Check that port is working===
To check that the port is active, run.
iptables -nL

You should see something along the lines of:
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:<port number you're adding> ctstate NEW,UNTRACKED

==How to run standalone Arthor instance==

===Step 1: Use or start a bash shell===
You can check your default shell using
echo $SHELL

If your default shell is csh, use
bash
to start a new bash shell in the current terminal window. Note that echo $SHELL will show you your default shell regardless of the current shell.

===Step 2: Set your environment variables===
export ARTHOR_DIR=/opt/nextmove/arthor/arthor-3.1-centos7
export PATH=$ARTHOR_DIR/bin/:$PATH

Make sure the ARTHOR_DIR variable is set to the directory for the latest version of Arthor or whichever version you would like to test.
The PATH environment variable is needed if you wish to use the Arthor tools from the command line

===Step 3: Run the arthor-server.jar===
java -jar /opt/nextmove/arthor/arthor-3.0-rt-beta-linux/java/arthor-server.jar --httpPort <your httpPort>

==Setting environment variables for TomCat Server==
Set the environment variables in the setenv.sh file. Note: Be sure to edit the file in the directory corresponding to the latest version of TomCat. As of December 2019, we are running 9.0.27 on n-1-136.

vim /opt/tomcat/apache-tomcat-9.0.27/bin/setenv.sh

Add the line below to the setenv.sh file above, or substitute the path to wherever you currently store the arthor.cfg file
export ARTHOR_CONFIG=/usr/local/tomcat/arthor.cfg

Here is an example of the arthor.cfg file:
# Arthor generated config file
BINDIR=/opt/nextmove/arthor/arthor-2.1.2-centos7/bin
DATADIR=/usr/local/tomcat/arthor_data
STAGEDIR=/usr/local/arthor_data/stage
NTHREADS=64 .
NODEAFFINITY=true
SearchAsYouDraw=true
AutomaticIndex=true
DEPICTION=./depict/bot/svg?w=%w&h=%h&svgunits=px&smi=%s&zoom=0.8&sma=%m&smalim=1
RESOLVER=

'''Important parts of the arthor.cfg file'''

'''BINDIR''' is the location of the Arthor command line binaries. These are used to generate the Arthor index files and to perform searches directly on n-1-136. An example of this would be using atdbgrep for substructure search.

'''DATADIR''' This is the directory where the Arthor data files live. Location where the index files will be created and loaded from.

'''STAGEDIR''' Location where the index files will be built before being moved into the DATADIR.

'''NTHREADS''' The number of threads to use for both ATDB and ATFP searches

Set '''AutomaticIndex''' to false if you don't want new smiles files added to the data directory to be indexed automatically

==Background==
Before working with Arthor, it is recommended that you familiarize yourself with the Arthor documentation. Some useful pages to look at include 3-5, 22-25 and 33-39. Of course, reading everything would be the best!

==Checking Memory Usage==
Before building arthor indexes, it's always a good thing to check what percent of the memory is being used. Try to be cautious with how much memory you have left, and make sure to check while building indexes to make sure that you have enough space. To check, run the following command:

df -h /<directory with disc>

==Building Large Databases==
At the moment, we are building databases of size 500M molecules by merging smile files. There are multiple methods of trying to create large databases, one being merging based off of the same H?? prefix and stopping once the database reaches > 500M molecules (or whatever upperbound you want to use). Here is some python code that simulates this merging process. Essentially the program takes all of the .smi files within an input directory, sorts them lexiographically, and begins merging these .smi files together in order until the size reaches > 500M molecules.

Feel free to modify it if you think a better method exists.

import subprocess
import sys
import os

from os import listdir
from os.path import isfile, join

mypath = "<Path to directory holding .smi files>"
onlyfiles = [f for f in listdir(mypath) if isfile(join(mypath, f))]
onlyfiles.sort()

create_fp = True
cur_mols = 0
lower_bound = 500000000
upper_bound = 600000000
files_to_merge = []

def merge_files(f_t_m):
arr = f_t_m[0].split(".")
arr2 = f_t_m[len(f_t_m) - 1].split(".")
file_name_merge = (arr[0] + "_" + arr2[0] + ".smi")
print ("File being created: " + file_name_merge)

for file in f_t_m:
tmp = file.split(".")
process = subprocess.Popen("cat " + join(mypath, file) + " >> " + file_name_merge, shell=True)
process.wait()

for file in onlyfiles:
arr = file.split(".")

if (arr[len(arr) - 1] == "smi"):
print("Working with " + file)
mol = sum(1 for line in open(join(mypath, file)))
print(file, mol, cur_mols)

if (cur_mols + mol > lower_bound):
if (cur_mols + mol < upper_bound):
files_to_merge.append(file)
merge_files(files_to_merge)
cur_mols = 0
files_to_merge.clear()
else:
merge_files(files_to_merge)
files_to_merge.clear()
files_to_merge.append(file)
merge_files(files_to_merge)
cur_mols = 0
files_to_merge.clear()
else:
cur_mols += mol
files_to_merge.append(file)

if (len(files_to_merge) != 0):
merge_files(files_to_merge)

==Building Arthor Indexes==
Once you've merged the .smi files together, it's time to start building the databases themselves. To do this we use the command

smi2atdb -j 0 -p <The .smi file> <The .atdb>

The flag "-j 0" enables parallel generation and utilizes all available processors to generate the .atdb file. The "-p" flag stores the offset position in the ATDB file. Since we're building indexes for the Web Application, you must use the "-p" flag when building indexes. Please note that the name of the .smi file should also be the name of the .atdb file. That way, the Web Application knows to use these files together and correctly display the required images. Refer to pages 33-34 in the Arthor documentation for more information.

If there are too many large .smi files and you do not want to manually build each .atdb file, you can use this python script which takes all of the .smi files in the current directory and converts them to .atdb files. Make sure to modify mypath to the directory containing the .smi files. You can change the variable "create_fp" to false if you don't want to create .atdb.fp files (refer to page 9 in the Arthor documentation).

import subprocess
import sys
import os

from os import listdir
from os.path import isfile, join

mypath = "<Path containing the .smi files"
onlyfiles = [f for f in listdir(mypath) if isfile(join(mypath, f))]

create_fp = True

for file in onlyfiles:
arr = file.split(".")

if (arr[len(arr) - 1] == "smi"):
process = subprocess.Popen("/nfs/ex9/work/xyz/psql/arthor-3.3-centos7/bin/smi2atdb -j 0 -p {0} {1}.atdb".format(join(mypath, file), arr[0]), shell=True)
process.wait()

print("SUCCESS! {0}.atdb file was created!".format(arr[0]))

if (create_fp):
process = subprocess.Popen("/nfs/ex9/work/xyz/psql/arthor-3.3-centos7/bin/atdb2fp -j 0 {0}.atdb".format(arr[0]), shell=True)
process.wait()

print("SUCCESS! {0}.atdb.fp file was created!".format(arr[0]))

==Uploading Indexes to the Web Application==
One can upload indexes to the Web Application by changing the "DATADIR" variable in the arthor.cfg file to the directory holding the .atdb files. This is already set up on n-1-136 and n-5-34.

==Further Arthor Optimizations==
The following edits can be made to the arthor.cfg to optimize substructure search queries. More information can be found in pages 6-8 in the Arthor Documentation file.

'''NodeAffinity NUMA''': optimized flag, pin processing to specific CPU sets to where the data is located in memory. There is a small start-up cost and is most useful for long running services (see Non-Uniform Memory Access (NUMA))

'''AsyncHitCountAllowed=true|false''' After fetching a page from a substructure or formula search the server will spin off a background process to count the total number of hits. This can be resource intensive for large databases and may not be desirable for servers under heavy load and may not even be needed.

'''AsyncHitCountMax=#''' The upper-bound for the number of hits to retrieve in background searches. If very generic queries are issued (e.g. benzene or methane) hundred’s of millions of hits may be counted. Setting this value to anything other than zero (e.g. 10,000) will stop the background search if it exceeds this limit. Note some pathological queries may find very few hits but still end up looking at everything.

'''MaxConcurrentSearches=#''' Controls the maximum number of searches that can be run concurrently by setting the database pool size. The searches may be on the same or different databases. If a search comes in and the pool is full it will have to wait for another search to finish - this increases the request time. 

Typically if each search is using all the processing cores on a machine then additional searches will run at 1/Nth the speed. If the request time is substantially larger that the search time the request had to wait for resources to become available. When switching between a large number of databases it can be useful to have a larger pool size, the only trade off is keeping file pointers open.  Default: 6

'''Binary Fingerprint Folding''' Arthor uses binary circular fingerprints (ECFP4/radius=2) for similarity. When creating an ATFP index you can specify how large to make your fingerprints. Circular fingerprints are sparser than path based fingerprints (e.g. Daylight) and so can be folded smaller without too much degradation in performance. Folding can significantly reduce the footprint size of a database and improve search speeds. A 256-bit fingerprint takes up 1/4 of the space of 1024-bit and can therefore be traversed 4x faster.

This is more important for very large databases with billions of compounds, in such instances a minor drop in precision is likely tolerable as ultimately all that happens is some hits may swap places in the hit list.

==Virtual Memory==
In addition to modifying the arthor.cfg file, virtual memory can also be used to make queries faster. There can still be More information can be found in pages 10-16 in the Arthor Documentation.

==Setting up Round Table==
This is a new feature in Arthor 3.0 and is currently beta (January 2020). See section 2.4 in the manual
As explained in the manual, "Round Table allows you to serve and split chemical searches across multiple host machines. The implementation provides a lightweight proxy that forwards requests to other Arthor host servers that do the actual search. Communication is done using the existing Web APIs.

Since Arthor requires CentOS 7, as of January 2020 we have 6 servers that are capable of running Arthor with Round Table.

===Setting up Host Server===
If we want to add machines to the Round Table, for example 'nun' and 'samekh', we need to edit their arthor.cfg file so that when our Local Machine passes commands these secondary servers know to perform the search they are given.

$ cat arthor.cfg
MaxThreadsPerSearch=4
AutomaticIndex=false
DATADIR=<Directory where smiles are located>

We then run the jar server on each of these host machines containing data on any available port.

java -jar /nfs/ex9/work/xyz/psql/arthor-3.3-centos7/java/arthor.jar --httpPort <port>

For our local machine, the arthor.cfg file will look different.

$ cat arthor.cfg
[RoundTable]
RemoteClient=http://skynet:<port number where jar server is running>/
RemoteClient=http://hal:<port number where jar server is running>/

Please refer to Section 2 in the RoundTable Documentation file (pages 6-8) for more useful information on configuration.

Then run the following command on n-1-136:

java -jar /nfs/ex9/work/xyz/psql/arthor-3.3-centos7/java/arthor.jar --httpPort <port>

===Arthor Round Table Head===
{| class="wikitable"
|-
! CentOS 7 Machine
! Port
! Database
! Arthor Install Location
! Round Table Data Directory
! Active
|-
| samekh
| 10.20.0.41:8080
| Enamine_REAL_Q2-2020-All-13B
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
| nun
| 10.20.0.40:8080
| Enamine_REAL_Q2-2020-All-41B
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
|}

===Arthor Round Table Nodes===
{| class="wikitable"
|-
! CentOS 7 Machine
! Port
! Database
! Total Files Size
! Arthor Install Location
! Round Table Data Directory
!Active
|-
| samekh
| 10.20.0.41:8008
| Enamine_REAL_Q2-2020-M-13B (am-ax, 12 slices), Enamine_REAL_Q2-2020-S-13B (aa-ab, 2 slices)
| 2.5TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
| nun
| 10.20.0.40:8008
| Enamine_REAL_Space_June_2020_S41B (aa-ae, 5 slices)
| 738GB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
| n-1-16
| 10.20.1.16:8008
| Enamine_REAL_Q2-2020-M-13B (aa-al, 12 slices)
| 2.0TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
| n-1-17
| 10.20.1.17:8008
| Enamine_REAL_Space_June_2020_M41B (aa-am, 14 slices)
| 4.3TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
| n-5-32
| 10.20.5.32:8008
| Enamine_REAL_Space_June_2020_M41B (an-az, 13 slices)
| 5.0TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
| n-5-33
| 10.20.5.33:8008
| Enamine_REAL_Space_June_2020_M41B (ba-bl, 12 slices)
| 5.3TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| active
|-
| qof
| 10.20.9.29:8008
| 18, 25, 37, 45, 5
| 173GB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /export/ex9/work/btingle/auto_atdb/
| not active
|-
| lamed
| 10.20.9.15: 8008
| 17, 29, 40, 6
| 512GB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /export/ex6/work/btingle/auto_atdb
| not active
|-
| n-1-20
| 10.20.1.20:8008
| 12, 15, 16 ,27, 27, 30, 36
| 897GB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| not active
|-
| n-5-34
| 10.20.5.34:8008
| 9, 19, 21, 28, 31, 42, all-zinc-xab, all-zinc-xafn in-stock
| 875GB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| not active
|-
| n-5-35
| 10.20.5.35:8008
| 2, 3, 8, 10, 34, 44_results, 44_results2, all-zinc-xad, in-stock-40, on-demand
| 1.5TB
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_database/
| not active
|-
|}

===Arthor (local 8081)===
{| class="wikitable"
|-
! CentOS 7 Machine
! Port
! Database
! Total Files Size
! Arthor Install Location
! Round Table Data Directory
!Active
|-
| samekh
| 10.20.0.41:8081
| Enamine_REAL_Q2-2020-All-13B (26 slices)
| 2.5TB (soft linked) + 2.0TB = 4.5TB total
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_local_8081/
| active
|-
| nun
| 10.20.0.40:8081
| Enamine_REAL_Space_June_2020_S41B (aa-ae, 5 slices), Enamine_REAL_Space_June_2020_M41B (aa-ar)
| 738GB (soft linked) + 1.9TB = 2.6TB total
| /opt/nextmove/arthor/arthor-3.3-centos7/
| /local2/arthor_local_8081/
| active
|-
|}

Manage Lab Websites

2021-01-05T20:51:59Z

Jgutie11:

== List of Websites ==
Last updated on 01/05/2021
{| class="wikitable"
!Website
!Machine:Port
!Run on
!Hosted in (httpd/conf.d)
!Working
|-
|Amis: https://amis.docking.org/
|
|
|Files2 (169.230.75.3)
|Yes
|-
|Arthor: https://arthor.docking.org/
|nun, samekh, n-9-22; Port 8000
|Screen
|Files2 (169.230.75.3)
|Yes
|-
|BKSLab: http://www.bkslab.org/
|gimel:5002
|Supervisord
|Tau (169.230.26.43)
|Yes
|-
|Blaster: https://blaster.docking.org/
|
|
|Tau (169.230.26.43, can not be moved to files2)
|Yes
|-
|Cartblanche: https://cartblanche.docking.org/
|gimel:5066-5067
|
|Files2 (169.230.75.3)
|Yes
|-
|Covalent: http://covalent.docking.org/
|
|
|Tau (169.230.26.43, can not be moved to files2)
|Yes
|-
|DSF: https://dsf.docking.org/
|
|
|Tau (169.230.26.43)
|No (wrong website)
|-
|Duc: https://duc.docking.org/
|
|
|Tau (169.230.26.43)
|No
|-
|Dud: https://dud.docking.org/
|
|
|Tau (169.230.26.43, can not be moved to files2)
|Yes
|-
|Dude18: http://dude18.docking.org/
|
|
|Tau (169.230.26.43)
|No
|-
|Dude: http://dude.docking.org/
|
|
|Tau (169.230.26.43, can not be moved to files2)
|Yes
|-
|Dudez: http://dudez.docking.org/
|
|
|Files2 (169.230.75.3)
|Yes
|-
|Excipients: http://excipients.docking.org/
|gimel:8093
|
|Tau (169.230.26.43)
|Yes
|-
|Gitlab: https://gitlab.docking.org/
|
|
|Files2 (169.230.75.3)
|No
|-
|HG: https://hg.docking.org/
|
|
|Files2 (169.230.75.3)
|No
|-
|IrwinLab: http://irwinlab.compbio.ucsf.edu/
|gimel:5004
|Supervisord
|Tau (169.230.26.43)
|Yes
|-
|Khanh: http://khanh.docking.org/
|
|
|Tau (169.230.26.43)
|Yes
|-
|Metabolite: http://metabolite.docking.org/
|
|
|Tau (169.230.26.43, can not be moved to files2)
|showing wrong website
|-
|Deepchemworkshop: http://deepchemworkshop.docking.org/
|
|
|Tau (169.230.26.43)
|Yes
|-
|Prices: http://prices.docking.org/
|gimel:5022
|
|Tau (169.230.26.43)
|No
|-
|Psicquic: http://psicquic.docking.org/
|
|
|Tau (169.230.26.43)
|Yes (broken???)
|-
|Reactor: http://reactor.docking.org/
|
|
|Tau (169.230.26.43)
|Yes
|-
|Transporters: http://transporters.ucsf.bkslab.org/
|
|
|Tau (169.230.26.43)
|Yes
|-
|Sea16: http://sea16.docking.org/
|gimel:8086
|
|Files2 (169.230.75.3)
|Yes
|-
|SEC: http://sec.docking.org/
|
|
|Tau (169.230.26.43)
|Yes
|-
|Smallworld Public: http://sw.docking.org/
|abacus:5020
|screen
|Files2 (169.230.75.3)
|Yes
|-
|Smallworld Private: http://swp.docking.org/
|abacus:8080
|screen
|Files2 (169.230.75.3)
|Yes
|-
|Smallworld Super Private: http://swc.docking.org/
|abacus:5099
|screen
|Files2 (169.230.75.3)
|Yes
|-
|Stats: http://stats.docking.org/
|
|
|Tau (169.230.26.43, can not be moved to files2)
|No
|-
|Symp: http://symp.docking.org/
|
|
|Tau (169.230.26.43)
|No
|-
|TLDR http://tldr.docking.org/
|gimel:5011
|Supervisord
|Tau (169.230.26.43)
|Yes
|-
|Tool-Selector: http://tool-selector.ucsf.bkslab.org/
|
|
|Tau (169.230.26.43)
|Yes
|-
|Transportal: http://transportal.docking.org/
|n-9-22 : port 8123
|screen
|Tau (169.230.26.43)
|Yes
|-
|Upload: http://upload.docking.org/
|
|
|Tau (169.230.26.43)
|Yes
|-
|Zinc12: http://zinc12.docking.org/
|
|
|Tau (169.230.26.43, can not be moved to files2)
|Yes
|-
|-
|Zinc15: http://zinc.docking.org/
|
|
|Files2 (169.230.75.3)
|Yes
|-
|Zinc20: http://zinc20.docking.org/
|
|
|Files2 (169.230.75.3)
|Yes
|-
|}

== Restart Instructions after UPS==
=== For websites running on Screen ===
* Become khtang on gimel
$ screen -ls

21546.SEA (Detached)
23528.transportal (Detached)
22877.conference (Detached)
16132.upload (Detached)
24786.oeb_lib_building (Detached)
4796.ShopAppMol (Detached)
19162.FASTROCS-server (Detached)

$ screen -r <screen_id>

==== SEA ====
Become www on gimel2
$ cd /nfs/soft/www/apps/seadev/
On bash shell
$ source tools/anaconda2/bin/activate sea16
$ cd work/seaware/seaware-academic
$ make SEAserver-stop-devel
$ make all
$ make SEAserver-start-devel

==== Transportal ====
$ cd /mnt/nfs/soft/www/apps/transportal/src/transportal
$ source venv/bin/activate.csh
$ python manage.py runserver 0.0.0.0:8123

=== For websites running on Supervisord ===
* Become root on server the website runs on
$ supervisorctl status
$ supervisorctl restart <name>
i.e
$ supervisorctl restart bks-lab
$ supervisorctl restart tools18

[[Category : Khanh]]