View source for AWS Auto Scaling

A step-by-step instruction how to create a slurm cluster on AWS with auto-scaling possibility


a) upgrade your pip:
pip install --upgrade pip

b) install aws client:
pip install awscli

c) pip install aws-parallelcluster


d) prepare your aws_access_key_id and aws_secret_access_key.<br>
Those can be found in "My Security Credentials -> Access keys (access key ID and secret access key)" section.<br>
If you haven't got it yet, press "Create New Access Key" and follow the instructions.

  aws configure
  Access Key ID [None]: _YOUR_ACCESS_KEY_ID_
  AWS Secret Access Key [None]: _YOUR_SECRET_ACCESS_KEY_
  this will be stored in .aws/credentials

e) parallel cluster configuration
NB: We use AWS Region us-east-1, which corresponds to N.Virginia.<br>
You are welcome to re-consider this choice. 

  pcluster configure

Allowed values for AWS Region ID:
1. ap-northeast-1
2. ap-northeast-2
3. ap-south-1
4. ap-southeast-1
5. ap-southeast-2
6. ca-central-1
7. eu-central-1
8. eu-north-1
9. eu-west-1
10. eu-west-2
11. eu-west-3
12. sa-east-1
13. us-east-1
14. us-east-2
15. us-west-1
16. us-west-2
AWS Region ID [us-east-1]:

Network & Security -> Key Pairs -> Create New Pair
Allowed values for EC2 Key Pair Name:
1. EC2_v1
EC2 Key Pair Name [EC2_v1]:
Allowed values for Scheduler:
1. sge
2. torque
3. slurm
4. awsbatch
Scheduler [slurm]:
Allowed values for Scheduler:
1. sge
2. torque
3. slurm
4. awsbatch
Scheduler [slurm]:
Minimum cluster size (instances) [0]: 
Maximum cluster size (instances) [10]: 
Master instance type [t2.micro]: 
Compute instance type [t2.micro]:
Automate VPC creation? (y/n) [n]: 
Allowed values for VPC ID:
1. vpc-579d8e2d | 0 subnets inside
VPC ID [vpc-579d8e2d]: 
Allowed values for Network Configuration:
1. Master in a public subnet and compute fleet in a private subnet
2. Master and compute fleet in the same public subnet
Network Configuration [Master in a public subnet and compute fleet in a private subnet]: 1



'''Useful links:'''
* https://support.lumerical.com/hc/en-us/articles/360034082194-AWS-Use-Case-Multi-node-clusters-with-launch-templates
* https://github.com/aws/aws-parallelcluster
* https://aws-parallelcluster.readthedocs.io/en/latest/
* https://docs.aws.amazon.com/parallelcluster/index.html
* https://docs.aws.amazon.com/autoscaling/ec2/userguide/what-is-amazon-ec2-auto-scaling.html