AWS:Set up account

This page pertains to the scripts created by Benjamin Tingle to set up an environment for running DOCK workloads in AWS.

WIP: Not complete, just saving progress

Getting Started

Requirements:

• Docker (https://www.docker.com/)
• Git

Things to pull:

git pull https://github.com/docking-org/awsdock-scripts.git
docker pull btingle/awsdock-setup

All scripts for setting up docking with AWS are run in a docker container. (If you need to run bare-metal contact me @ ben@tingle.org)

Running with Docker Desktop for Windows

In cmd/powershell

1. docker pull btingle/awsdock-setup

2. docker run -v /var/run/docker.sock:/var/run/docker.sock -it btingle/awsdock-setup

Running with Docker Desktop + WSL2

On the docker desktop window, go to Settings->General and enable "Expose daemon on tcp://localhost:2375 without TLS"

Now in WSL2:

1. Install docker client (if not already installed)

2. export DOCKER_HOST=tcp://localhost:2375

3. docker pull btingle/awsdock-setup

4. bash run_docker.bash

Running with Linux/Mac

1. sudo docker pull btingle/awsdock-setup:latest

2. bash run_docker.bash

If you've set DOCKER_HOST already (i.e you are using a remote docker instance) the run_docker.bash script will still work.

Running the Scripts

When you enter the docker image, you will be in /tmp. cd to the /tmp/aws-setup directory to view all the scripts at your disposal.

aws configure

Each time you enter the awsdock-setup docker image, run "aws configure". Enter your AWS access key and secret key, which can be retrieved from your AWS account (instructions). Region and output format can be set if desired, but will be overwritten by our scripts.

initialize-aws-batch.bash

Usage:
  bash initialize-aws-batch-env.bash

A one-off script that is run before setting up environments. If it is your first time creating an awsdock environment on your aws account, run this script once. This will set up policies and roles on your AWS account that are needed across all batch environments.

create-aws-batch-env.bash

Usage:
  bash create-aws-batch-env.bash <config>

Main script for setting up awsdock environments. Should be run for each region you wish to run docking jobs in. You may also want to create separate environments for different versions of DOCK or alternate datasets. Depending on the configuration, this script can be interactive or automated by configuration variables. For more information on configuration variables for this script, see the "configuration" section of this page. Running this script with the included "awsdock.config" configuration file will provide an interactive experience for setting up your first awsdock environment.

Environment Creation Steps

Step {0} is for setting up the environment's NAME and REGION. These two properties in combination serve as the unique identifier for your batch environment.

• Your environment's NAME can be set by the ENV_NAME configuration variable, otherwise set interactively.
• Likewise, the REGION can be set by the ENV_AWS_REGION configuration variable, otherwise set interactively.

Step {1} sets up the profile/role that will be used in this environment. This part will always be non-interactive, so this step is mostly here as a debug tool/progress marker.

Step {2} sets up bucket policies for the environment. For example, if you would like to pull DB2 data from bkslab's zinc3d bucket, you would set that up here.

• You can set bucket policies using the ENV_BUCKET_CONFIGS configuration variable. Multiple buckets+policies can be included in this variable, for example:
  • ENV_BUCKET_POLICIES="zinc3d:input mybucket:output" will set up bkslab's zinc3d bucket as the input source for the environment and mybucket as the output destination.
  • ENV_BUCKET_POLICIES="zinc3d:input prompt" will set up zinc3d as the input source, and one other bucket policy through interactive prompt. This is the default setting of awsdock.config.
  • ENV_BUCKET_POLICIES="zinc3d:input mybucket1:input,output mybucket2:output" sets up 3 policies- indicating that zinc3d can be used for input, mybucket1 can be used for input/output, and mybucket2 can be used for just output.
  • ENV_BUCKET_POLICIES="prompt+" Will set up as many bucket policies as desired through interactive prompt.

Step {3} sets up the docker image to be used in this environment. The name of the image is set by the JOB_IMAGE config variable.

• In awsdock.config, this is bkslab's btingle/awsdock image. This image is compatible with our script for submitting DOCK jobs to AWS.
• JOB_IMAGE can be set to any image you so desire, but keep in mind that our DOCK job submission/collection scripts are specialized to the input/output format our awsdock image uses.

Step {4} creates all the AWS components needed to run jobs with AWS batch, including Job Queue, Compute Environment, and Job definition. The MAX_CPUS and BID_PERCENTAGE parameters are used/set during this step. The RETRY_STRATEGY and JOB_JSON_CONFIG parameters will also be used.

• MAX_CPUS is the maximum number of virtual cpus (vCPUs) allocated at any one time by AWS batch for this environment. The default JOB_JSON_CONFIG assigns one vcpu to one job, so a MAX_CPUS value of 100 means that a maximum of 100 jobs can run at any one given time.
• BID_PERCENTAGE is a value between (0, 100) and tells AWS what % of the on-demand price you are willing to pay for compute resources. A lower BID_PERCENTAGE value will result in better economy for your jobs, but also lower potential allocation.
  • A BID_PERCENTAGE value of 100 will still save money over the on-demand price in many cases, but if demand is high it will pay full price.
  • A BID_PERCENTAGE value of 50 will always save money, however during periods of high demand you may struggle to allocate machines for your jobs.
• RETRY_STRATEGY tells aws batch how many times to retry a failed job before giving up. By default awsdock.config tells the job to retry 5 times before giving up. Because we use the economical "spot fleet" strategy, jobs may fail because the machine they are on was allocated to a higher paying customer.
  • Our awsdock image will save progress when spot fleet re-allocates the machine it is running on. This way no compute time is wasted.
• JOB_JSON_CONFIG is a json structure specifying the resource parameters jobs will use. These values can be changed, for example if you would like jobs to have more memory you can change the "memory" attribute to a higher value.
  • If you are using our awsdock image, you should not need to change any of these parameters.

Example: Setting up your first awsdock environment

Prerequisites:

• An AWS S3 bucket
  • Note: buckets can be created using the aws cli within the aws-setup docker image

aws s3api create-bucket --bucket <bucket name>

1. Set up your aws credentials

aws configure

2. Initialize your account

bash /tmp/aws-setup/initialize-aws-batch.bash

3. Run the environment setup script

bash /tmp/aws-setup/create-aws-batch-env.bash /tmp/aws-setup/awsdock.config

4. Enter the desired name for your environment

What would you like this environment to be called? [default: "dockenv"]: <your env name here>

5. Enter the aws region for your environment

Which region is this environment based in? [default: us-west-1]: <your region here>

6. Attach the bucket you created to the environment

What bucket would you like to attach to this environment? <your bucket>

7. Set the I/O policy for your bucket to input,output.

Which action(s) would you like to perform on this bucket? Choose from input/output, if using multiple separate by commas. input,output

8. Confirm that you will use the btingle/dockaws image for this environment.

What docker image should this environment use? [default: btingle/dockaws:latest]: btingle/dockaws:latest

9. Set MAX_CPUS for your environment to desired value. If you will be running jobs at as large a scale as possible, give this a high value, e.g 9999.

How many CPUS would you like to allocate to this environment at maximum? [default: None]:

10. Set BID_PERCENTAGE for your environment to desired value. See "Environment Creation Steps" for more explanation of this parameter.

What is your bid percentage threshold for spot instances? See the docs for more info on this parameter. [default: 100]: <your value>

11. All done!

Bid Percentage

In order to use resources efficiently, our AWS environment uses AWS spot instances to buy compute resources. AWS spot instances basically allow us to purchase compute resources for a fraction of the price, with the caveat that service may be interrupted at any time. Our AWS docking image allows us to take advantage of this service by saving progress whenever the instance is about to be interrupted. The bid percentage parameter indicates what % of the on-demand price we are willing to pay for compute resources. If left at 100, the scheduler will pay the on-demand price for compute resources if no spot instances are available.