DISI - User contributions [en]

Python

2019-08-16T20:02:30Z

Benrwong:

Python is probably the single most useful programming language you can learn today. Most of our new code is written in Python. The interface to many programs we use such as [[PyMol]], [[Chimera]], [[Omega]] and [[rdkit]] are in python. There are times to swim upstream and go against the crowd, but with Python in 2014, '''this is not one of those times'''.

There are two version streams, 2.7.x and 3.7.x. As of February 2014, our software uses exclusively 2.7.x and will not work with 3.x.x nor with 2.5.x or earlier versions.

Modules required: numpy, scipy, biopython, psycopg2, MySQL-python, sh, matplotlib

Python 2.7.6 is currently installed in nfs-soft:/mnt/nfs/python/python-2.7.6
Python 3.7.0 is currently installed in nfs:soft:/mnt/nfs/python/python-3.7.0

===Installing a new version of Python onto nfs-soft for general cluster usage===

The below commands were used to install the latest version of Python onto nfs-soft.

Note #1: in command 5, the --prefix option notes that which directory to install the Python files. This is necessary because otherwise, Python will get installed in /usr/local/bin and not in an nfs export directory where it can be accessed by other hosts in the cluster. 
Note #2: In command 6, use 'make altinstall' instead of 'make install'. This prevents the new installation from overwriting the system installation of Python which is crucial for basic processes (the package manager, yum, needs the original installation of Python to be unchanged)

1) [s_bwong1@bet ~]$ cd /export/soft/python/build
2) [s_bwong1@bet build]$ sudo wget https://www.python.org/ftp/python/3.7.0/Python-3.7.0.tar.xz
3) [s_bwong1@bet build]$ sudo unxz Python-3.7.0.tar.xz
4) [s_bwong1@bet build]$ sudo tar -xvf Python-3.7.0.tar
5) [s_bwong1@bet build]$ #sudo ./configure --prefix=/export/soft/python/python-3.7.0 --enable-optimizations --with-ensurepip=install
6) [s_bwong1@bet build]$ sudo make && sudo make altinstall

===Environment Variables for using Python3.7===

.bash_profile
export PATH="/nfs/soft/python/python-3.7.0/bin:$PATH"
export LD_LIBRARY_PATH=/mnt/nfs/soft/openssl/openssl-1.1.1a/lib/

[[Category:Software]]
[[Category:Developer]]
[[Category:Free]]

Schrodinger

2019-05-20T20:08:04Z

Benrwong: /* Troubleshooting Schrodinger Issues */ Added section about multi-process jobs

SCHRODINGER - getting it running

= Get a License File: =
Get an email about Schrodinger license keys ready for retrieval. 
Click the link that follows: "please use this form to generate the license file:" 

In the License Retrieval Assistant, make sure you have the following information for the respective categories: 
Host ID: 0015605f526c 
Machine Name: nis.compbio.ucsf.edu 
FLEXIm Server Port: 2700 

= Debugging: =
Cluster 0, all schrodinger files are located locally on nfshead2:/raid3 but the commands below should be executed on nis as user tdemers.

Make sure that the variable $LM_LICENSE_FILE has port@same_exact_server_name_as_in_license_file. The license.dat file must contain:

SERVER nis.compbio.ucsf.edu 0015605f526c 27000
VENDOR SCHROD PORT=53000

Make sure the port is open in iptables
source /raid3/software/schrodinger/current.sh
Try some combination of the following:

$SCHRODINGER/licadmin STAT -c $SCHRODINGER/license.dat
$SCHRODINGER/licadmin REREAD -l $SCHRODINGER/lmgrd.log -c $SCHRODINGER/license.dat
$SCHRODINGER/licadmin SERVERDOWN
$SCHRODINGER/licadmin SERVERUP -l $SCHRODINGER/lmgrd.log -c $SCHRODINGER/license.dat

= Installing Schrodinger on Cluster 0 =
First you need to go to the website and download the software. You should end up with two files: Schrodinger Worflow … .zip and Schrodinger Suites …..tar
scp both these files to the server, to the schrodinger directory.
On the server, in the schrodinger directory mkdir MonthYear. cd into that directory Untar the tar file and run the INSTALL script. At the end you’ll see something like this:

<nowiki>*) Licensing</nowiki>
You will need one or more licenses before you can run the
software you have just installed. 
Please note the following information, which you will need in
order to generate a license key: 
Host ID: 001e0bd543b8
Machine name: nfshead2.bkslab.org 
If you are not performing this installation on your license
server, you will need the output of: 
$SCHRODINGER/machid -hostid

= Installing Schrodinger 2019 on Cluster 2 =
===Install===
https://www.schrodinger.com/downloads/releases

Select the Linux 64-bit version. Download it to your local computer first. Then scp the tarball over the nfs-soft in the appropriate directory. Extract the tarball and you'll get a bunch of smaller tarfiles.

# ls
Schrodinger_Suites_2019-1_Linux-x86_64.tar
# tar -xvf Schrodinger_Suites_2019-1_Linux-x86_64.tar
Schrodinger_Suites_2019-1_Linux-x86_64/canvas-v3.9-Linux-x86_64.tar.gz
Schrodinger_Suites_2019-1_Linux-x86_64/mcpro-v5.3-Linux-x86_64.tar.gz
Schrodinger_Suites_2019-1_Linux-x86_64/desmond-v5.7-Linux-x86_64.tar.gz
Schrodinger_Suites_2019-1_Linux-x86_64/INSTALL
.
.
.
Schrodinger_Suites_2019-1_Linux-x86_64/CHECKSUM.md5

https://www.schrodinger.com/license-installation-instructions

We do not need to untar these individually. The INSTALL script takes care of nearly everything. All we have to do is set the path of where we want the installed programs to go to.

[root@bet ~]# export SCHRODINGER=/export/soft/schrodinger/2019-1/
[root@bet ~]# ./INSTALL

The install script will ask you where you're running your license server. We run the license server on the same server as the installation server so tell the software that it will run on 27008@bet

===Set Environment Files===

Notice we set the SCHROD_LICENSE_FILE as '27008@bet'. We do not use the FQDN. This is because the desktops are on the public network (compbio.ucsf.edu) while the cluster is on a private network (cluster.ucsf.bkslab.org). If we use the FQDN, the desktops may recognize the domain but not the cluster and vice versa. Therefore, we will reference the license server as simply 'bet'

env.sh
#!/bin/bash
export SCHRODINGER="/nfs/soft/schrodinger/2019-1"
export SCHRODINGER_THIRDPARTY="$SCHRODINGER/thirdparty"
export SCHRODINGER_PDB="$SCHRODINGER_THIRDPARTY/database/pdb"
export SCHRODINGER_UTILITIES="$SCHRODINGER/utilities"
export SCHRODINGER_RCP="scp"
export SCHRODINGER_RSH="ssh"
export PSP_BLASTDB="$SCHRODINGER_THIRDPARTY/database/blast/"
export PSP_BLAST_DATA="$SCHRODINGER_THIRDPARTY/bin/Linux-x86/blast/data/"
export PSP_BLAST_DIR="$SCHRODINGER_THIRDPARTY/bin/Linux-x86/blast/"
export SCHROD_LICENSE_FILE="27008@bet"
export LM_LICENSE_FILE="27008@bet"
export PATH="${SCHRODINGER}:${SCHRODINGER_UTILITIES}:${PATH}:${SCHRODINGER_THIRDPARTY}/desmond_to_trj"

env.csh
#!/bin/csh
setenv SCHRODINGER "/mnt/nfs/soft/schrodinger/2019-1"
setenv SCHRODINGER_THIRDPARTY "$SCHRODINGER/thirdparty"
setenv SCHRODINGER_PDB "$SCHRODINGER_THIRDPARTY/database/pdb"
setenv SCHRODINGER_UTILITIES "$SCHRODINGER/utilities"
setenv SCHRODINGER_RCP "scp"
setenv SCHRODINGER_RSH "ssh"
setenv PSP_BLASTDB "$SCHRODINGER_THIRDPARTY/database/blast/"
setenv PSP_BLAST_DATA "$SCHRODINGER_THIRDPARTY/bin/Linux-x86/blast/data/"
setenv PSP_BLAST_DIR "$SCHRODINGER_THIRDPARTY/bin/Linux-x86/blast/"
setenv SCHROD_LICENSE_FILE "27008@bet"
setenv PATH "${SCHRODINGER}:${SCHRODINGER_UTILITIES}:${PATH}:${SCHRODINGER_THIRDPARTY}/desmond_to_trj"

===Licensing===

Edit the license file line that contains 'SERVER'. For Server, we will put 'this_host' instead of the hostname. This way, the license server will be recognized by any of its DNS hostnames regardless of different domains.
SERVER this_host 80c16e65897d 27008

===Schrodinger Hosts & Queue Config Files===

The schrodinger.hosts file exists within the schrodinger current installation directory. schrodinger.hosts contains the list of queues available for schrodinger to use. The first host entry should just be a localhost entry to allow users to run Schrodinger on their local machine. Other host entries will contain information such as what queue to use, how many processors are available, what GPUs exist, if parallelization is enabled, etc.

schrodinger.hosts file
Name: gimel-sge
host: gimel
queue: SGE
qargs: -q gpu.q -pe local %NPROC% -l gpu=1
tmpdir: /scratch
processors: 32
gpgpu: 0, nvidia
gpgpu: 1, nvidia
gpgpu: 2, nvidia
gpgpu: 3, nvidia
parallel: 1

Name: gimel2-sge
host: gimel2
queue: SGE
qargs: -q gpu.q -pe local %NPROC% -l gpu=1
tmpdir: /scratch
processors: 32
gpgpu: 0, nvidia
gpgpu: 1, nvidia
gpgpu: 2, nvidia
gpgpu: 3, nvidia
parallel: 1

name: gimel2-n923q
host: gimel2
queue: SGE
qargs: -q n-9-23.q -pe local %NPROC%
tmpdir: /scratch
processors: 80
parallel: 1

Since we use opengrid engine, we must configure the queue config file that exists for SGE. This file is located in the $SCHRODINGER/queues/SGE/config.

QPATH=/usr/bin/
QPROFILE=/nfs/ge/ucsf.bks/cell/common/settings.sh
QSUB=qsub
QDEL=qdel
QSTAT=qstat
LICENSE_CHECKING=yes

===Troubleshooting: D-Bus Errors===
We had a period where our jobs were dying upon submission. We would get this strange error message:

process 23478: arguments to dbus_move_error() were incorrect, assertion "(dest) == NULL || !dbus_error_is_set ((dest))" failed in file dbus-errors.c line 278.
This is normally a bug in some application using the D-Bus library.
D-Bus not built with -rdynamic so unable to print a backtrace
Fatal Python error: Aborted

It turns out, this was due to SELinux being on. As a temporary workaround, I have disabled SELinux on hosts that were experiencing this issue. We'll need to dig deeper in /var/log/audit/audit.log to diagnose what was wrong.

===Troubleshooting: All processes go onto the same GPU===
When we submit GPU jobs via Maestro/Desmond, we can choose the number of GPUs we use in the run. However, when we first did this while declaring that we wanted four GPUs to be used in a process, Schrodinger would allocate the four separate processes all on the same GPU. To address this, we have to log into the GPU nodes and set the GPUs into exclusive mode. This means that no more than one process would run on a GPU at a time.

$ nvidia-smi -c 3

Found on this webpage: https://www.schrodinger.com/kb/1834

===Troubleshooting: Multi-process jobs only finishes a single process===

Ligprep jobs get sent to a node to begin. We've been sending ligprep jobs that would utilize six additional parallel processes. These parallel processes would be spawned as six sub-jobs. Unfortunately, when we first tried, only the head process would spawn but non of the sub-jobs would get submitted. This happened because of the way Schrodinger tries to spawn additional subprocesses. The head job would run on a compute node and then try to contact an SGE submit host (gimel,gimel2) via SSH. If you do not have passwordless SSH enabled, the job would fail to spawn sub-jobs. What you need to do is create an ssh-key in your home directory that would solely be used when an SSH connection is initialized between a compute node and gimel/gimel2. Since your home directory is NFS-mounted across all nodes on the cluster, you only need to create an ssh-key and append the public key to your authorized_keys file under .ssh.

$ ssh-keygen (follow steps and don't make a password)
$ cat ~/.ssh/<new key>.pub >> ~/.ssh/authorized_keys
$ vi ~/.ssh/config
Host gimel gimel2
IdentityFile ~/.ssh/compute_to_gimel

This way, the process on the compute node can successfully contact the SGE submission hosts and spawn additional subprocesses.

[[Category:Sysadmin]]

2019-04-08T19:44:14Z

Benrwong:

On March 7th of 2019, our SSL certificates for Puppet/Foreman and TLS LDAP all reached their five year expiration period. This has caused Puppet and Foreman to stop working and our authentication server cannot be accessed. I cannot create new users that are recognized by the cluster. And users cannot login to machines that they have previously never logged into. You can imagine the trouble this has caused. As of March 22, I have not solved this yet but I will be documenting my attempts to fix this.

===Notes===
I got a great deal of clues based on the configuration files located in the following areas:
/etc/foreman
/etc/foreman/ssl (check the README file here. In our cluster, our Puppet certs and Foreman certs are different. You need to generate Foreman certs separately)
/etc/puppet
/etc/puppet/puppet.conf (tells you the puppet master and puppet client FQDNs for the puppetmaster)
/etc/httpd/conf.d
/etc/httpd/conf.d/05-foreman-ssl.conf (tells you what SSL certificates files foreman looks at)
/etc/httpd/conf.d/25-puppet.conf (tells you what certificate name that the puppetmaster uses)

===Procedure to Renew SSL Certificates on Puppet Master===
First of all, stop httpd. Puppet master and Foreman run via httpd
[root@alpha ~]# service httpd stop

Get subject of original certificate
[root@alpha ~]# openssl x509 -in /var/lib/puppet/ssl/ca/ca_crt.pem -noout -subject
subject= /CN=Puppet CA: alpha.ucsf.bkslab.org

Get serial of original certificate
[root@alpha ~]# openssl x509 -in /var/lib/puppet/ssl/ca/ca_crt.pem -noout -serial
serial=01

Extract info from puppetmaster cert
[root@alpha ~]# openssl x509 -in /var/lib/puppet/ssl/certs/puppetmaster.cluster.ucsf.bkslab.org.pem -text -noout \
-certopt no_subject,no_header,no_version,no_serial,no_signame,no_validity,no_subject,no_issuer,no_pubkey,no_sigdump,no_aux

Create directories to stage new valid periods (I determined puppetmaster.cluster.ucsf.bkslab.org.pem was the .pem file we need because certname in [master] section of /etc/puppet/puppet.conf has this name)
[root@alpha ~]# mkdir /root/puppet_renewal
[root@alpha ~]# cd /root/puppet_renewal
[root@alpha ~]# mkdir /root/puppet_renewal/ca
[root@alpha ~]# mkdir /root/puppet_renewal/puppetmaster
[root@alpha ~]# mkdir /root/puppet_renewal/puppetmaster/private_keys
[root@alpha ~]# mkdir /root/puppet_renewal/puppetmaster/certs

Copy the existing certificate authority's key and the puppetmaster's private key (Only certificates expire. Private keys do not so they can be reused).
[root@alpha ~]# cp /var/lib/puppet/ssl/ca/ca_key.pem /root/puppet_renewal/ca
[root@alpha ~]# cp /var/lib/puppet/ssl/private_keys/mypuppetmaster.cluster.ucsf.bkslab.org.pem /root/puppet_renewal/puppetmaster/private_keys

Create an openssl configuration file
[root@alpha ~]# vi puppet_renewal/renewpuppet.cnf
''renewpuppet.cnf''
[ v3_ca ]
basicConstraints= CA:TRUE
subjectKeyIdentifier= hash
# authorityKeyIdentifier= keyid:always,issuer:always
keyUsage = critical, cRLSign, keyCertSign
nsComment = 'Puppet Ruby/OpenSSL Internal Certificate'

[ v3 ]
basicConstraints= CA:FALSE
subjectKeyIdentifier= hash
nsComment = 'Puppet Ruby/OpenSSL Internal Certificate'
keyUsage = critical, digitalSignature, keyEncipherment
extendedKeyUsage = critical, serverAuth, clientAuth
subjectAltName = @alt_names

[ alt_names ]
DNS.1 = puppet
DNS.2 = puppet.ucsf.bkslab.org
DNS.3 = puppet.cluster.ucsf.bkslab.org
DNS.4 = puppetmaster
DNS.5 = puppetmaster.ucsf.bkslab.org
DNS.6 = puppetmaster.cluster.ucsf.bkslab.org
DNS.7 = alpha
DNS.8 = alpha.ucsf.bkslab.org
DNS.9 = alpha.cluster.ucsf.bkslab.org
DNS.10 = alpha.compbio.ucsf.edu

Create certificate signing request with existing files
[root@alpha ~]# openssl req -key /root/puppet_renewal/ca/ca_key.pem -new -batch -subj "/CN=Puppet CA: alpha.ucsf.bkslab.org" -out /root/puppet_renewal/ca/ca_new.csr

Create a new CA certificate
[root@alpha ~]#openssl x509 -req -days 3650 -in /root/puppet_renewal/ca/ca_new.csr -signkey /root/puppet_renewal/ca/ca_key.pem \
-out /root/puppet_renewal/ca/ca_crt.pem -extfile /root/puppet_renewal/renewpuppet.cnf -extensions v3_ca -set_serial 1
Signature ok
subject=/CN=Puppet CA: alpha.ucsf.bkslab.org
Getting Private key

Get serial number for your existing CA
[root@alpha puppet_renewal]# echo $((0x`cat /var/lib/puppet/ssl/ca/serial`))
451

Create new certificate signing request with puppet server's key
[root@alpha ~]# openssl req -key /root/puppet_renewal/puppetmaster/private_keys/puppetmaster.cluster.ucsf.bkslab.org.pem -new \
-batch -subj "/CN=alpha.ucsf.bkslab.org" -out /root/puppet_renewal/mypuppetmaster.csr

Create new puppet master's certificate
[root@alpha ~]# openssl x509 -extfile /root/puppet_renewal/renewpuppet.cnf -extensions v3 -req -days 1825 -in /root/puppet_renewal/mypuppetmaster.csr \
-CA /root/puppet_renewal/ca/ca_crt.pem -CAkey /root/puppet_renewal/ca/ca_key.pem -CAcreateserial \
-out /root/puppet_renewal/puppetmaster/certs/puppetmaster.cluster.ucsf.bkslab.org.pem -sha256 -set_serial 451

Replace Puppet's ca_crt.pem, ca.pem, and puppetmaster.pem
[root@alpha ~]# cp puppet_renewal/ca/ca_crt.pem /var/lib/puppet/ssl/ca/ca_crt.pem
[root@alpha ~]# puppet_renewal/ca/ca_crt.pem /var/lib/puppet/ssl/certs/ca.pem
[root@alpha ~]# cp puppet_renewal/puppetmaster/certs/puppetmaster.cluster.ucsf.bkslab.org.pem /var/lib/puppet/ssl/certs/puppetmaster.cluster.ucsf.bkslab.org.pem

Restart httpd
# If this fails, a mistake was made. Check /var/log/httpd.
[root@alpha ~]# service httpd start
Starting httpd: [Wed Mar 27 12:28:24 2019] [warn] module passenger_module is already loaded, skipping
[ OK ]

===Renewing Foreman's certificates (in progress)===
Investigate /etc/foreman/ssl/README
cluster2 on foreman

Follow the commands in the README file while in the /etc/foreman/ssl directory. Only do this step after you've done the Puppet certificates in the previous section.
The foreman related commands here rely on the ca.pem generated by the previous puppet commands.

#openssl genrsa -aes128 -out foreman.key 2048
openssl genrsa -out foreman.key 2048
openssl req -new -out foreman.csr -key foreman.key -config foreman.cnf
openssl req -text -in foreman.csr -noout
openssl x509 -req -in foreman.csr -CA /var/lib/puppet/ssl/ca/ca_crt.pem -CAkey /var/lib/puppet/ssl/ca/ca_key.pem -CAserial /var/lib/puppet/ssl/ca/serial -out foreman.crt -days 1730

openssl rsa -in foreman.key -out foreman.key-unlocked

ln -s foreman.key-unlocked key.pem
ln -s foreman.crt cert.pem

Restart httpd when this is done.

===Further Reading===
A big thanks to these two blogs for pointing me in the right direction:
Sean the Sysadmin: http://www.scrosby.com/2017/06/renewing-puppet-ca-and-puppet-master.html
Flying Circus: https://blog.flyingcircus.io/2017/09/01/how-to-renew-puppet-ca-and-server-certificates-in-place/
[[Category:Sysadmin]]

Renewing Puppet Certificates

2019-04-08T19:41:13Z

Benrwong: /* Renewing Foreman's certificates (in progress) */

Renewing Puppet Certificates

2019-04-08T19:38:46Z

Benrwong: /* Notes */

Install Miniconda

2019-03-29T21:20:54Z

Renewing Puppet Certificates

2019-03-25T19:10:24Z

Benrwong: Changing directory names

Renewing Puppet Certificates

2019-03-22T22:19:54Z

Benrwong:

Renewing Puppet Certificates

2019-03-22T21:31:34Z

Benrwong: