Cartblanche22 Build Instructions: Difference between revisions

From DISI
Jump to navigation Jump to search
(Created page with "== Local environment == * Clone repo Git clone https://gitlab.docking.org/chinzo/molecule-shopping-cart Cd molecule-shopping-cart * Create conda environment with python 3...")
 
(18 intermediate revisions by the same user not shown)
Line 1: Line 1:
== Local environment ==
== Development environment inside cluster==


* Clone repo
* Clone repo
Line 7: Line 7:
   conda env create -f environment.yml
   conda env create -f environment.yml
   conda activate cartblanche
   conda activate cartblanche
* Open ssh tunnel to machines, sshuttle is a good tool to avoid having to open the tunnel to each tin machine individually. This does not work for vendor lookup at the moment since antimony machines have the alias stored and not the actual IP
  sudo sshuttle --dns -v -vvvv -r username@gimel.ucsf.bkslab.org:22 -x gimel.ucsf.bkslab.org:22 0/0


* Start application
* Start application
   ./boot-test.sh
   ./boot-test.sh


== Deployment Information ==
'''Gitlab Deployment Stages'''


== Deployment Information ==
Production builds are distributed among 3 machines
Live site runs on n-9-22:5067. Pushing to master branch will deploy here.
    - epyc2
Dev server runs on n-1-17:5068. Pushing to the dev branch will deploy to this server.  
    - n-9-38
    - n-1-18
 
The first stage will build/update the current docker image to the gitlab server running on n-9-22. On the web interface, this is in the cartblanche repo under 'Packages & Registries'.
 
The second stage tests the new container and runs tests for
    - ZincID Search
    - Supplier Code Search
    - SMILES Search


Useful Commands (Use these in the respective server to debug):
Finally, the docker image is pulled in each of the three servers and restarted.
docker restart molecule-shopping-cart (restart the docker container)
docker logs molecule-shopping-cart (view app logs, only stdout)
docker exec -it molecule-shopping-cart /bin/bash (open a shell inside of the docker container, useful for making quick minimal changes without having to redeploy)


To view error logs (stderr)
'''Useful Commands''' (Use these in the respective server to debug):
docker exec -it molecule-shopping-cart /bin/bash
  sudo docker restart cartblanche22 (restart the docker container)
cat std.log
  sudo docker logs cartblanche22 (view app logs)


Some troubleshooting tips:
'''Some troubleshooting tips:'''
Cartblanche relies on other outside services (smallworld, arthur, databases)  so it’s good to check the status of those before looking into the code.
Cartblanche relies on other outside services (smallworld, arthur, databases)  so it’s good to check the status of those before looking into the code.
For smallworld, check the status of api calls, or run a sample query.  
 
For arthor, check the status of api class, or run a sample query.
For smallworld, check the status of api calls, or [https://sw.docking.org/search/view/?hlid=19842&draw=6&columns%5B0%5D%5Bdata%5D=0&columns%5B0%5D%5Bname%5D=alignment&columns%5B0%5D%5Bsearchable%5D=true&columns%5B0%5D%5Borderable%5D=false&columns%5B0%5D%5Bsearch%5D%5Bvalue%5D=&columns%5B0%5D%5Bsearch%5D%5Bregex%5D=false&columns%5B1%5D%5Bdata%5D=1&columns%5B1%5D%5Bname%5D=dist&columns%5B1%5D%5Bsearchable%5D=true&columns%5B1%5D%5Borderable%5D=true&columns%5B1%5D%5Bsearch%5D%5Bvalue%5D=0-10&columns%5B1%5D%5Bsearch%5D%5Bregex%5D=false&columns%5B2%5D%5Bdata%5D=2&columns%5B2%5D%5Bname%5D=ecfp4&columns%5B2%5D%5Bsearchable%5D=true&columns%5B2%5D%5Borderable%5D=true&columns%5B2%5D%5Bsearch%5D%5Bvalue%5D=&columns%5B2%5D%5Bsearch%5D%5Bregex%5D=false&columns%5B3%5D%5Bdata%5D=3&columns%5B3%5D%5Bname%5D=daylight&columns%5B3%5D%5Bsearchable%5D=true&columns%5B3%5D%5Borderable%5D=true&columns%5B3%5D%5Bsearch%5D%5Bvalue%5D=&columns%5B3%5D%5Bsearch%5D%5Bregex%5D=false&columns%5B4%5D%5Bdata%5D=4&columns%5B4%5D%5Bname%5D=topodist&columns%5B4%5D%5Bsearchable%5D=true&columns%5B4%5D%5Borderable%5D=true&columns%5B4%5D%5Bsearch%5D%5Bvalue%5D=0-4&columns%5B4%5D%5Bsearch%5D%5Bregex%5D=false&columns%5B5%5D%5Bdata%5D=5&columns%5B5%5D%5Bname%5D=mces&columns%5B5%5D%5Bsearchable%5D=true&columns%5B5%5D%5Borderable%5D=true&columns%5B5%5D%5Bsearch%5D%5Bvalue%5D=&columns%5B5%5D%5Bsearch%5D%5Bregex%5D=false&columns%5B6%5D%5Bdata%5D=6&columns%5B6%5D%5Bname%5D=tdn&columns%5B6%5D%5Bsearchable%5D=true&columns%5B6%5D%5Borderable%5D=true&columns%5B6%5D%5Bsearch%5D%5Bvalue%5D=0-4&columns%5B6%5D%5Bsearch%5D%5Bregex%5D=false&columns%5B7%5D%5Bdata%5D=7&columns%5B7%5D%5Bname%5D=tup&columns%5B7%5D%5Bsearchable%5D=true&columns%5B7%5D%5Borderable%5D=true&columns%5B7%5D%5Bsearch%5D%5Bvalue%5D=0-4&columns%5B7%5D%5Bsearch%5D%5Bregex%5D=false&columns%5B8%5D%5Bdata%5D=8&columns%5B8%5D%5Bname%5D=rdn&columns%5B8%5D%5Bsearchable%5D=true&columns%5B8%5D%5Borderable%5D=true&columns%5B8%5D%5Bsearch%5D%5Bvalue%5D=0-4&columns%5B8%5D%5Bsearch%5D%5Bregex%5D=false&columns%5B9%5D%5Bdata%5D=9&columns%5B9%5D%5Bname%5D=rup&columns%5B9%5D%5Bsearchable%5D=true&columns%5B9%5D%5Borderable%5D=true&columns%5B9%5D%5Bsearch%5D%5Bvalue%5D=0-4&columns%5B9%5D%5Bsearch%5D%5Bregex%5D=false&columns%5B10%5D%5Bdata%5D=10&columns%5B10%5D%5Bname%5D=ldn&columns%5B10%5D%5Bsearchable%5D=true&columns%5B10%5D%5Borderable%5D=true&columns%5B10%5D%5Bsearch%5D%5Bvalue%5D=0-4&columns%5B10%5D%5Bsearch%5D%5Bregex%5D=false&columns%5B11%5D%5Bdata%5D=11&columns%5B11%5D%5Bname%5D=lup&columns%5B11%5D%5Bsearchable%5D=true&columns%5B11%5D%5Borderable%5D=true&columns%5B11%5D%5Bsearch%5D%5Bvalue%5D=0-4&columns%5B11%5D%5Bsearch%5D%5Bregex%5D=false&columns%5B12%5D%5Bdata%5D=12&columns%5B12%5D%5Bname%5D=mut&columns%5B12%5D%5Bsearchable%5D=true&columns%5B12%5D%5Borderable%5D=true&columns%5B12%5D%5Bsearch%5D%5Bvalue%5D=&columns%5B12%5D%5Bsearch%5D%5Bregex%5D=false&columns%5B13%5D%5Bdata%5D=13&columns%5B13%5D%5Bname%5D=maj&columns%5B13%5D%5Bsearchable%5D=true&columns%5B13%5D%5Borderable%5D=true&columns%5B13%5D%5Bsearch%5D%5Bvalue%5D=0-4&columns%5B13%5D%5Bsearch%5D%5Bregex%5D=false&columns%5B14%5D%5Bdata%5D=14&columns%5B14%5D%5Bname%5D=min&columns%5B14%5D%5Bsearchable%5D=true&columns%5B14%5D%5Borderable%5D=true&columns%5B14%5D%5Bsearch%5D%5Bvalue%5D=0-4&columns%5B14%5D%5Bsearch%5D%5Bregex%5D=false&columns%5B15%5D%5Bdata%5D=15&columns%5B15%5D%5Bname%5D=hyb&columns%5B15%5D%5Bsearchable%5D=true&columns%5B15%5D%5Borderable%5D=true&columns%5B15%5D%5Bsearch%5D%5Bvalue%5D=0-4&columns%5B15%5D%5Bsearch%5D%5Bregex%5D=false&columns%5B16%5D%5Bdata%5D=16&columns%5B16%5D%5Bname%5D=sub&columns%5B16%5D%5Bsearchable%5D=true&columns%5B16%5D%5Borderable%5D=true&columns%5B16%5D%5Bsearch%5D%5Bvalue%5D=0-4&columns%5B16%5D%5Bsearch%5D%5Bregex%5D=false&order%5B0%5D%5Bcolumn%5D=0&order%5B0%5D%5Bdir%5D=asc&start=0&length=24&search%5Bvalue%5D=&search%5Bregex%5D=false&_=1654029670029 run a sample query. ]
For search errors, the docker app logs are very clear as to where the data is being searched.
 
For arthor, check the status of api class, or [https://arthor.docking.org//dt/ZINC-All-19Q4-1.4B/search?query=CC&type=Substructure&draw=2&start=0&length=80&flags=6144&columns%5B0%5D%5Bdata%5D=0&columns%5B0%5D%5Bname%5D=&columns%5B0%5D%5Bsearchable%5D=true&columns%5B0%5D%5Borderable%5D=false&columns%5B0%5D%5Bsearch%5D%5Bvalue%5D=&columns%5B0%5D%5Bsearch%5D%5Bregex%5D=false&columns%5B1%5D%5Bdata%5D=1&columns%5B1%5D%5Bname%5D=alignment&columns%5B1%5D%5Bsearchable%5D=true&columns%5B1%5D%5Borderable%5D=false&columns%5B1%5D%5Bsearch%5D%5Bvalue%5D=&columns%5B1%5D%5Bsearch%5D%5Bregex%5D=false&columns%5B2%5D%5Bdata%5D=2&columns%5B2%5D%5Bname%5D=sim&columns%5B2%5D%5Bsearchable%5D=true&columns%5B2%5D%5Borderable%5D=false&columns%5B2%5D%5Bsearch%5D%5Bvalue%5D=&columns%5B2%5D%5Bsearch%5D%5Bregex%5D=false run a sample query].
 
 
For search errors (zincid, smiles, vendor), the docker app logs are usually very informative.
 
It is also useful to see the status of the redis and redis-sentinel services on each of those machines.
 
There are three redis databases, one main and two backups. If the main db goes down, one of the backups becomes main. When the old main goes back up, it becomes a backup database.


== Deployment Process ==
== Deployment Process ==
Line 40: Line 52:
== Setting up Auto-Deployment ==
== Setting up Auto-Deployment ==


To set up an environment on a new machine, docker and gitlab-runner will need to be installed
* Follow the instructions on setting up a new gitlab runner, [https://gitlab.docking.org/admin/runners found here.]
Follow the instructions on setting up a new gitlab runner, found here https://gitlab.docking.org/admin/runners
 
Register the runner with the credentials provided in the link above
* Register the runner with the credentials provided in the link above.
Set ‘shell’ as the executor for the runner when prompted
 
After configuration, the gitlab-runner config found in /etc/gitlab-runner/config.toml should look something like this:  
* Set ‘shell’ as the executor for the runner when prompted.
 
* After configuration, the gitlab-runner config found in /etc/gitlab-runner/config.toml should look something like this:  


   oncurrent = 1
   oncurrent = 1
Line 69: Line 83:
Return to https://gitlab.docking.org/admin/runners, where the new runner will be displayed. Edit the runner’s tags to describe what the runner will be deploying. For example, the test server has the tag ‘cartblanche-dev’ and the production server has ‘cartblanche22’.  
Return to https://gitlab.docking.org/admin/runners, where the new runner will be displayed. Edit the runner’s tags to describe what the runner will be deploying. For example, the test server has the tag ‘cartblanche-dev’ and the production server has ‘cartblanche22’.  
Once the tag is set, a new job will need to be added to the gitlab deployment script to deploy the correct branch to the correct machine. https://gitlab.docking.org/chinzo/molecule-shopping-cart/-/blob/master/.gitlab-ci.yml  
Once the tag is set, a new job will need to be added to the gitlab deployment script to deploy the correct branch to the correct machine. https://gitlab.docking.org/chinzo/molecule-shopping-cart/-/blob/master/.gitlab-ci.yml  
A sample job will look like so
 
A sample job will look like so:


   services:
   services:
     - redis:latest
     - redis:latest
   deploy-dev job:
   '''deploy-dev''' job:
       only:
       '''only:
           - dev
           - dev'''
       stage: deploy
       stage: deploy
       services:
       services:
           - redis:latest
           - redis:latest
       tags:
       '''tags:
           - cartblanche-dev
           - cartblanche-dev'''
       script:
       script:
           - docker build -t "${CI_PROJECT_NAME}:${CI_COMMIT_REF_NAME}-0.1.${CI_JOB_ID}" .
           - docker login ${GITLAB_REGISTRY_URL} -u ${GITLAB_REGISTRY_USER} -p ${GITLAB_REGISTRY_PASS}
          - docker pull ${GITLAB_REGISTRY_URL}/mar/cartblanche22
           - docker stop ${CI_PROJECT_NAME}
           - docker stop ${CI_PROJECT_NAME}
          - docker rm ${CI_PROJECT_NAME}
          - docker stop redis
          - docker rm redis
           - docker run --net cartblanche --name redis -d redis
           - docker run --net cartblanche --name redis -d redis
           - docker run --net cartblanche -v /nfs/db2/smallworld_anon_21Q4:/nfs/db3/private_smallworld_4th_gen/anon
           - docker run --net cartblanche -v /nfs/db2/smallworld_anon_21Q4:/nfs/db3/private_smallworld_4th_gen/anon
Line 92: Line 105:
                     -v /nfs/db3/private_smallworld_4th_gen/smallworld.cfg:/nfs/db3/private_smallworld_4th_gen/smallworld.cfg
                     -v /nfs/db3/private_smallworld_4th_gen/smallworld.cfg:/nfs/db3/private_smallworld_4th_gen/smallworld.cfg
                     -v /nfs/db3/private_smallworld_4th_gen/maps:/nfs/db3/private_smallworld_4th_gen/maps
                     -v /nfs/db3/private_smallworld_4th_gen/maps:/nfs/db3/private_smallworld_4th_gen/maps
                     --name ${CI_PROJECT_NAME} -d -p 0.0.0.0:5068:5000 ${CI_PROJECT_NAME}:${CI_COMMIT_REF_NAME}-0.1.${CI_JOB_ID}
                     --name ${CI_PROJECT_NAME} -d -p '''0.0.0.0:5068''':5000 ${CI_PROJECT_NAME}:${CI_COMMIT_REF_NAME}-0.1.${CI_JOB_ID}
 
* The name ‘deploy-dev’ refers to the job name.
 
* 'only: -dev' refers to the script only deploying the dev branch. More branches can be added to deploy pushes to different branches to the same machine.
 
* 'tags: -cartblanche-dev' refers to the job tags, which determines which machine will take care of that deployment job. This can be changed to the tag that was set up earlier. You can also add multiple tags to deploy the same branch (or branches) to multiple machines at the same time.


The name ‘deploy-dev’ refers to the job name. The only: -dev refers to ‘only use this script to deploy the dev branch. More branches can be added to deploy pushes to different branches to the same machine. The tags: -cartblanche-dev refers to the job tags, which determines which machine will take care of that deployment job. This can be changed to the tag that was set up earlier. You can also add multiple tags to deploy the same branch (or branches) to multiple machines at the same time.
* 0.0.0.0:5068 refers to the port that points to the application.

Revision as of 18:49, 16 November 2023

Development environment inside cluster

  • Clone repo
 Git clone https://gitlab.docking.org/chinzo/molecule-shopping-cart
 Cd molecule-shopping-cart
  • Create conda environment with python 3.7, install dependencies
 conda env create -f environment.yml
 conda activate cartblanche
  • Start application
 ./boot-test.sh

Deployment Information

Gitlab Deployment Stages

Production builds are distributed among 3 machines

   - epyc2
   - n-9-38
   - n-1-18

The first stage will build/update the current docker image to the gitlab server running on n-9-22. On the web interface, this is in the cartblanche repo under 'Packages & Registries'.

The second stage tests the new container and runs tests for

   - ZincID Search
   - Supplier Code Search
   - SMILES Search

Finally, the docker image is pulled in each of the three servers and restarted.

Useful Commands (Use these in the respective server to debug):

 sudo docker restart cartblanche22 (restart the docker container)
 sudo docker logs cartblanche22 (view app logs)

Some troubleshooting tips: Cartblanche relies on other outside services (smallworld, arthur, databases) so it’s good to check the status of those before looking into the code.

For smallworld, check the status of api calls, or run a sample query.

For arthor, check the status of api class, or run a sample query.


For search errors (zincid, smiles, vendor), the docker app logs are usually very informative.

It is also useful to see the status of the redis and redis-sentinel services on each of those machines.

There are three redis databases, one main and two backups. If the main db goes down, one of the backups becomes main. When the old main goes back up, it becomes a backup database.

Deployment Process

Deployment is as simple as pushing changes to the respective branch. A docker image will be created with the new version of the code, which will then replace the current version. To redeploy manually, go to https://gitlab.docking.org/chinzo/molecule-shopping-cart/-/pipelines/new

Setting up Auto-Deployment

  • Follow the instructions on setting up a new gitlab runner, found here.
  • Register the runner with the credentials provided in the link above.
  • Set ‘shell’ as the executor for the runner when prompted.
  • After configuration, the gitlab-runner config found in /etc/gitlab-runner/config.toml should look something like this:
 oncurrent = 1
 check_interval = 0
 [session_server]
   session_timeout = 1800
 runners
   name = "cartblanche n-1-17"
   url = "http://gitlab.docking.org/"
   token = "Lvp2X3zRFAN_JQuVZK3y"
   privileged = true
   executor = "shell"
   builds_dir = "/home/cartblanche"
   [runners.custom_build_dir]
   [runners.docker]
     privileged = true
     disable_cache = false
     cache_dir = "cache"
    [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]
    [runners.cache.azure]

Return to https://gitlab.docking.org/admin/runners, where the new runner will be displayed. Edit the runner’s tags to describe what the runner will be deploying. For example, the test server has the tag ‘cartblanche-dev’ and the production server has ‘cartblanche22’. Once the tag is set, a new job will need to be added to the gitlab deployment script to deploy the correct branch to the correct machine. https://gitlab.docking.org/chinzo/molecule-shopping-cart/-/blob/master/.gitlab-ci.yml

A sample job will look like so:

 services:
   - redis:latest
 deploy-dev job:
     only:
         - dev
     stage: deploy
     services:
         - redis:latest
     tags:
         - cartblanche-dev
     script:
         - docker login ${GITLAB_REGISTRY_URL} -u ${GITLAB_REGISTRY_USER} -p ${GITLAB_REGISTRY_PASS}
         - docker pull ${GITLAB_REGISTRY_URL}/mar/cartblanche22
         - docker stop ${CI_PROJECT_NAME}
         - docker run --net cartblanche --name redis -d redis
         - docker run --net cartblanche -v /nfs/db2/smallworld_anon_21Q4:/nfs/db3/private_smallworld_4th_gen/anon
                    -v /export/db4/smallworld-java:/export/db4/smallworld-java
                    -v /nfs/db3/private_smallworld_4th_gen/smallworld.cfg:/nfs/db3/private_smallworld_4th_gen/smallworld.cfg
                    -v /nfs/db3/private_smallworld_4th_gen/maps:/nfs/db3/private_smallworld_4th_gen/maps
                    --name ${CI_PROJECT_NAME} -d -p 0.0.0.0:5068:5000 ${CI_PROJECT_NAME}:${CI_COMMIT_REF_NAME}-0.1.${CI_JOB_ID}
  • The name ‘deploy-dev’ refers to the job name.
  • 'only: -dev' refers to the script only deploying the dev branch. More branches can be added to deploy pushes to different branches to the same machine.
  • 'tags: -cartblanche-dev' refers to the job tags, which determines which machine will take care of that deployment job. This can be changed to the tag that was set up earlier. You can also add multiple tags to deploy the same branch (or branches) to multiple machines at the same time.
  • 0.0.0.0:5068 refers to the port that points to the application.