AWS Auto Scaling

From DISI
Revision as of 13:51, 2 June 2020 by Dudenko (talk | contribs)
Jump to navigation Jump to search
The printable version is no longer supported and may have rendering errors. Please update your browser bookmarks and please use the default browser print function instead.

A step-by-step instruction how to create a slurm cluster on AWS with auto-scaling possibility


a) upgrade your pip: pip install --upgrade pip

b) install aws client: pip install awscli

c) pip install aws-parallelcluster


d) prepare your aws_access_key_id and aws_secret_access_key.
Those can be found in "My Security Credentials -> Access keys (access key ID and secret access key)" section.
If you haven't got it yet, press "Create New Access Key" and follow the instructions.

 aws configure
 Access Key ID [None]: _YOUR_ACCESS_KEY_ID_
 AWS Secret Access Key [None]: _YOUR_SECRET_ACCESS_KEY_
 this will be stored in .aws/credentials

e) parallel cluster configuration NB: We use AWS Region us-east-1, which corresponds to N.Virginia.
You are welcome to re-consider this choice.

 pcluster configure
 Allowed values for AWS Region ID:
 1. ap-northeast-1
 2. ap-northeast-2
 3. ap-south-1
 4. ap-southeast-1
 5. ap-southeast-2
 6. ca-central-1
 7. eu-central-1
 8. eu-north-1
 9. eu-west-1
 10. eu-west-2
 11. eu-west-3
 12. sa-east-1
 13. us-east-1
 14. us-east-2
 15. us-west-1
 16. us-west-2
 AWS Region ID [us-east-1]:

Network & Security -> Key Pairs -> Create New Pair

 Allowed values for EC2 Key Pair Name:
 1. EC2_v1
 EC2 Key Pair Name [EC2_v1]:
 Allowed values for Scheduler:
 1. sge
 2. torque
 3. slurm
 4. awsbatch
 Scheduler [slurm]:
 Allowed values for Scheduler:
 1. sge
 2. torque
 3. slurm
 4. awsbatch
 Scheduler [slurm]:
 Minimum cluster size (instances) [0]: 
 Maximum cluster size (instances) [10]: 
 Master instance type [t2.micro]: 
 Compute instance type [t2.micro]:
 Automate VPC creation? (y/n) [n]: 
 Allowed values for VPC ID:
 1. vpc-579d8e2d | 0 subnets inside
 VPC ID [vpc-579d8e2d]: 
 Allowed values for Network Configuration:
 1. Master in a public subnet and compute fleet in a private subnet
 2. Master and compute fleet in the same public subnet
 Network Configuration [Master in a public subnet and compute fleet in a private subnet]: 1


The config file is ready and stored in ~/.parallelcluster/config
You may revise it and edit, if needed.

To create a cluster on AWS, do pcluster create -c ~/.parallelcluster/config UCSFbeta


Useful links: