AWS:Set up account

Part 1 of 3 of the AWS Docking tutorial.

Next Tutorial: Docking_Submission_On_AWS

Installation

Docker is required to run the aws-setup scripts. https://www.docker.com/get-started/

An Amazon AWS account is also required, with payment attached. https://aws.amazon.com/premiumsupport/knowledge-center/create-and-activate-aws-account/

On a linux/mac/windows computer with docker or docker desktop installed, run the following commands in a terminal:

docker pull btingle/aws-setup
docker run -v /var/run/docker.sock:/var/run/docker.sock --rm -it btingle/aws-setup

Explanation of arguments:

• -v /var/run/docker.sock:/var/run/docker.sock Allows the container to use your system's Docker
• --rm Cleans up the container once you've exited
• -it Runs the container interactively

It may be necessary to give the container additional privileges. When you enter the image, test this with the following command:

root@f54f423d64b1:/home/awsuser# docker ps

If you get a permission denied error, exit the container and run again with the --privileged option enabled:

docker run --privileged -v /var/run/docker.sock:/var/run/docker.sock -it btingle/aws-setup

If you're using a remote docker instance through the DOCKER_HOST environment variable, for example on windows WSL, you can use the following script in place of 'docker run':

host=$(basename $DOCKER_HOST | cut -d':' -f1)
port=$(basename $DOCKER_HOST | cut -d':' -f2)
prot=$(dirname $DOCKER_HOST)

if [ "$host" = "localhost" ] || [ "$host" == "127.0.0.1" ]; then
	host=host.docker.internal
fi

# essentially we are just forwarding the DOCKER_HOST information to the container (making sure to use host.docker.internal if DOCKER_HOST is localhost)
docker run --env DOCKER_HOST=$prot//$host:$port -it btingle/awsdock-setup

Container Environment

The container uses the ubuntu distribution. Some utilities such as curl and vi are installed so you can download files and edit them. You can also install whatever software you like using "apt install", e.g "apt install git".

If you have files you'd like to access from the container, you can link them in using the docker "-v" option. By default we link the docker socket using this option ("-v /var/run/docker.sock:/var/run/docker.sock"), but you can link any number of directories or files in this manner. For example, if you would like the contents of the "/tmp" directory on your local machine to be available under "/temp" in the docker image, you would add the following option to your "docker run" command: "-v /tmp:/temp", for a final command of:

docker run -v /tmp:/temp -v /var/run/docker.sock:/var/run/docker.sock -it btingle/aws-setup:latest

If you're an advanced user and you'd like to create your own version of the aws-setup image with certain software preinstalled, you can request us for access to the aws-setup repository, which contains the scripts and Dockerfile we use to set up the docker image. You can also build your own image using our aws-setup image as a base.

Quickstart - Creating your first AWS docking environment

Setup

Credentials & Region

When you enter the docker image, you will be in /home/awsuser. There should be two directories in front of you, aws-setup and awsdock. We start off by going into the aws-setup directory and configuring our AWS credentials. (This needs to be done every time you log in to the container)

root@f54f423d64b1:/home/awsuser# cd aws-setup
root@f54f423d64b1:/home/awsuser# aws configure

You'll now be prompted to enter your AWS access key ID & AWS secret access key. If you already know what these are you can enter them and move on. If you don't know what your AWS secret key and access key are, follow this tutorial: https://aws.amazon.com/blogs/security/wheres-my-secret-access-key/. Make sure to save your keys somewhere safe that you will remember!!

Next, you'll be prompted on which region you would like to use. These will be overwritten when using the quickstart script, but it will be helpful to educate yourself on what AWS regions are for the future. More info on regions & region codes here: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html Our lab's molecule data repository (zinc3d) is located in us-east-1, so this region will be the most economic to run jobs in.

Feel free to leave output format blank, or set it to "json".

S3 Bucket

An S3 bucket is a virtual hard drive that your AWS resources can access from anywhere. You will need to create one on your account prior to creating your AWS environment. Follow the amazon tutorial on how to do this:

Make sure to create this bucket in the us-east-1 region.

First time setup

If it is your first time setting up an environment on your AWS account, you will need to run initialize-aws-batch.bash. This script only needs to be run once per account.

root@f54f423d64b1:/home/awsuser/aws-setup# bash initialize-aws-batch.bash

You should see this script spit out a bunch of JSON text. If you accidentally run this script when it has already been run before, you may see a bunch of errors along the lines of: "Service role name <blank> has been taken in this account". Don't worry about these, they don't mean anything.

Environment Creation

root@f54f423d64b1:/home/awsuser/aws-setup# bash create-aws-batch-env.bash /home/awsuser/awsdock/aws-setup-configs/awsdock_quickstart.config

The quickstart configuration will name your environment "dockenv-us-east-1". This name serves as the unique identifier for this environment, you'll refer to it later when submitting jobs.

Attach the bucket you created to the environment. Don't qualify this with the s3:// path, just the plain name.

What bucket would you like to attach to this environment? mybucket

Set MAX_CPUS for your environment to desired value. This parameter refers to the maximum number of jobs that can be run in parallel. You should set this at or below the suggested value- this value is derived from the AWS imposed resource limit. You can learn more about resource limits and how to increase them at this page: Docking_Submission_On_AWS#Resource_Limits

How many CPUS would you like to allocate to this environment at maximum? [suggested: 128]: 

Set BID_PERCENTAGE for your environment to desired value. See section below for more explanation of this parameter, it can potentially save you money. If you're not sure, keep the default.

What is your bid percentage threshold for spot instances? See the docs for more info on this parameter. [default: 100]: 100

Bid Percentage

In order to use resources efficiently, our AWS environment uses AWS spot instances to buy compute resources. AWS spot instances basically allow us to purchase compute resources for a fraction of the price, with the caveat that service may be interrupted at any time. Our AWS docking image allows us to take advantage of this service by saving progress whenever the instance is about to be interrupted. The bid percentage parameter indicates what % of the on-demand price we are willing to pay for compute resources. If left at 100, the scheduler will pay the on-demand price for compute resources if no spot instances are available.