Difference between revisions of "AWS Auto Scaling"
From DISI
m |
m |
||
Line 26: | Line 26: | ||
pcluster configure | pcluster configure | ||
− | Allowed values for AWS Region ID: | + | Allowed values for AWS Region ID: |
− | 1. ap-northeast-1 | + | 1. ap-northeast-1 |
− | 2. ap-northeast-2 | + | 2. ap-northeast-2 |
− | 3. ap-south-1 | + | 3. ap-south-1 |
− | 4. ap-southeast-1 | + | 4. ap-southeast-1 |
− | 5. ap-southeast-2 | + | 5. ap-southeast-2 |
− | 6. ca-central-1 | + | 6. ca-central-1 |
− | 7. eu-central-1 | + | 7. eu-central-1 |
− | 8. eu-north-1 | + | 8. eu-north-1 |
− | 9. eu-west-1 | + | 9. eu-west-1 |
− | 10. eu-west-2 | + | 10. eu-west-2 |
− | 11. eu-west-3 | + | 11. eu-west-3 |
− | 12. sa-east-1 | + | 12. sa-east-1 |
− | 13. us-east-1 | + | 13. us-east-1 |
− | 14. us-east-2 | + | 14. us-east-2 |
− | 15. us-west-1 | + | 15. us-west-1 |
− | 16. us-west-2 | + | 16. us-west-2 |
− | AWS Region ID [us-east-1]: | + | AWS Region ID [us-east-1]: |
Network & Security -> Key Pairs -> Create New Pair | Network & Security -> Key Pairs -> Create New Pair | ||
− | Allowed values for EC2 Key Pair Name: | + | |
− | 1. EC2_v1 | + | Allowed values for EC2 Key Pair Name: |
− | EC2 Key Pair Name [EC2_v1]: | + | 1. EC2_v1 |
− | Allowed values for Scheduler: | + | EC2 Key Pair Name [EC2_v1]: |
− | 1. sge | + | Allowed values for Scheduler: |
− | 2. torque | + | 1. sge |
− | 3. slurm | + | 2. torque |
− | 4. awsbatch | + | 3. slurm |
− | Scheduler [slurm]: | + | 4. awsbatch |
− | Allowed values for Scheduler: | + | Scheduler [slurm]: |
− | 1. sge | + | Allowed values for Scheduler: |
− | 2. torque | + | 1. sge |
− | 3. slurm | + | 2. torque |
− | 4. awsbatch | + | 3. slurm |
− | Scheduler [slurm]: | + | 4. awsbatch |
− | Minimum cluster size (instances) [0]: | + | Scheduler [slurm]: |
− | Maximum cluster size (instances) [10]: | + | Minimum cluster size (instances) [0]: |
− | Master instance type [t2.micro]: | + | Maximum cluster size (instances) [10]: |
− | Compute instance type [t2.micro]: | + | Master instance type [t2.micro]: |
− | Automate VPC creation? (y/n) [n]: | + | Compute instance type [t2.micro]: |
− | Allowed values for VPC ID: | + | Automate VPC creation? (y/n) [n]: |
− | 1. vpc-579d8e2d | 0 subnets inside | + | Allowed values for VPC ID: |
− | VPC ID [vpc-579d8e2d]: | + | 1. vpc-579d8e2d | 0 subnets inside |
− | Allowed values for Network Configuration: | + | VPC ID [vpc-579d8e2d]: |
− | 1. Master in a public subnet and compute fleet in a private subnet | + | Allowed values for Network Configuration: |
− | 2. Master and compute fleet in the same public subnet | + | 1. Master in a public subnet and compute fleet in a private subnet |
− | Network Configuration [Master in a public subnet and compute fleet in a private subnet]: 1 | + | 2. Master and compute fleet in the same public subnet |
+ | Network Configuration [Master in a public subnet and compute fleet in a private subnet]: 1 | ||
Revision as of 05:46, 2 June 2020
A step-by-step instruction how to create a slurm cluster on AWS with auto-scaling possibility
a) upgrade your pip:
pip install --upgrade pip
b) install aws client: pip install awscli
c) pip install aws-parallelcluster
d) prepare your aws_access_key_id and aws_secret_access_key.
Those can be found in "My Security Credentials -> Access keys (access key ID and secret access key)" section.
If you haven't got it yet, press "Create New Access Key" and follow the instructions.
aws configure Access Key ID [None]: _YOUR_ACCESS_KEY_ID_ AWS Secret Access Key [None]: _YOUR_SECRET_ACCESS_KEY_ this will be stored in .aws/credentials
e) parallel cluster configuration
NB: We use AWS Region us-east-1, which corresponds to N.Virginia.
You are welcome to re-consider this choice.
pcluster configure
Allowed values for AWS Region ID: 1. ap-northeast-1 2. ap-northeast-2 3. ap-south-1 4. ap-southeast-1 5. ap-southeast-2 6. ca-central-1 7. eu-central-1 8. eu-north-1 9. eu-west-1 10. eu-west-2 11. eu-west-3 12. sa-east-1 13. us-east-1 14. us-east-2 15. us-west-1 16. us-west-2 AWS Region ID [us-east-1]:
Network & Security -> Key Pairs -> Create New Pair
Allowed values for EC2 Key Pair Name: 1. EC2_v1 EC2 Key Pair Name [EC2_v1]: Allowed values for Scheduler: 1. sge 2. torque 3. slurm 4. awsbatch Scheduler [slurm]: Allowed values for Scheduler: 1. sge 2. torque 3. slurm 4. awsbatch Scheduler [slurm]: Minimum cluster size (instances) [0]: Maximum cluster size (instances) [10]: Master instance type [t2.micro]: Compute instance type [t2.micro]: Automate VPC creation? (y/n) [n]: Allowed values for VPC ID: 1. vpc-579d8e2d | 0 subnets inside VPC ID [vpc-579d8e2d]: Allowed values for Network Configuration: 1. Master in a public subnet and compute fleet in a private subnet 2. Master and compute fleet in the same public subnet Network Configuration [Master in a public subnet and compute fleet in a private subnet]: 1
Useful links:
- https://support.lumerical.com/hc/en-us/articles/360034082194-AWS-Use-Case-Multi-node-clusters-with-launch-templates
- https://github.com/aws/aws-parallelcluster
- https://aws-parallelcluster.readthedocs.io/en/latest/
- https://docs.aws.amazon.com/parallelcluster/index.html
- https://docs.aws.amazon.com/autoscaling/ec2/userguide/what-is-amazon-ec2-auto-scaling.html