Using AWS Setup For Cloud Computation - Revision history

Frodo: asdf

2022-08-09T16:38:37Z

asdf

← Older revision		Revision as of 16:38, 9 August 2022
Line 64:		Line 64:
	# We are using the bkslab/awsdock:latest docker image as our software environment		# We are using the bkslab/awsdock:latest docker image as our software environment
	# This cluster is allowed to read from the zinc3d data bucket, and is allowed to read and write to the my_stuff data bucket.		# This cluster is allowed to read from the zinc3d data bucket, and is allowed to read and write to the my_stuff data bucket.


			[[Category:AWS]]

Btingle: /* Translating Cluster Concepts to AWS Cloud Concepts */

2022-08-03T22:40:35Z

Translating Cluster Concepts to AWS Cloud Concepts

← Older revision		Revision as of 22:40, 3 August 2022
Line 35:		Line 35:
	! Concept !! Description		! Concept !! Description
	\|-		\|-
	\| Regions \|\| ~~Logical Divisions~~ of the AWS ~~Network based on geographic location~~. Resources in one region are typically not visible to those in other regions.		\| Regions \|\| Geographic divisions of the AWS network. Resources in one region are typically not visible to those in other regions.
	\|}		\|}

Btingle: /* Translating Cluster Concepts to AWS Cloud Concepts */

2022-08-03T22:40:08Z

Translating Cluster Concepts to AWS Cloud Concepts

← Older revision		Revision as of 22:40, 3 August 2022
Line 35:		Line 35:
	! Concept !! Description		! Concept !! Description
	\|-		\|-
	\| Regions \|\| Logical Divisions of the AWS Network based on geographic location. Resources ~~created~~ in one region are typically not visible to those in other regions.		\| Regions \|\| Logical Divisions of the AWS Network based on geographic location. Resources in one region are typically not visible to those in other regions.
	\|}		\|}

Btingle: /* Introduction to the aws-setup platform */

2022-08-03T22:33:24Z

Introduction to the aws-setup platform

← Older revision		Revision as of 22:33, 3 August 2022
Line 9:		Line 9:
	If attempting to set up your own cluster environment on the cloud is like rowing a paddleboat into a raging storm, then cluster-in-the-cloud is like taking a luxury cruise into a tsunami.		If attempting to set up your own cluster environment on the cloud is like rowing a paddleboat into a raging storm, then cluster-in-the-cloud is like taking a luxury cruise into a tsunami.

	aws-setup, on the other hand, is like a submarine. Sure, there's no hot tub or open bar, but it will get you to where you want to go. What aws-setup promises is tight integration with AWS systems, robustness, efficiency, and relative simplicity (compared to setting things up oneself).		aws-setup, on the other hand, is like a submarine. Sure, there's no hot tub or open bar, but it will get you to where you want to go. What aws-setup promises is tight integration with AWS systems, robustness, efficiency, and relative simplicity of use (compared to setting things up oneself).

	= Translating Cluster Concepts to AWS Cloud Concepts =		= Translating Cluster Concepts to AWS Cloud Concepts =

Btingle: /* Batch Environments in aws-setup */

2022-08-03T21:34:02Z

Batch Environments in aws-setup

← Older revision		Revision as of 21:34, 3 August 2022
Line 40:		Line 40:
	= Batch Environments in aws-setup =		= Batch Environments in aws-setup =

	The core concept in aws-setup is that of the "batch environment". A batch environment is the interface used to run a particular computation or set of computations. Each environment created with aws-setup has three core properties:		The core concept in aws-setup is that of the "batch environment". A batch environment is the interface used to spin up a cluster and run a particular computation or set of computations. Each environment created with aws-setup has three core properties:

	# <Region>		# <Region>

Btingle at 21:33, 3 August 2022

2022-08-03T21:33:29Z

@@ Line 1: / Line 1: @@
-= Introduction to the AWS Setup platform =
+= Introduction to the aws-setup platform =
 The concept of "the cloud" as a platform for computation is somewhat mystified. Scientists talk about running their calculations "in the cloud", but to what end? How does one even begin to use the cloud? We are broadly familiar with the academic cluster environment, but the transition from that environment to cloud computation is not often discussed in detail.
@@ Line 30: / Line 30: @@
 | Package Managers/Lmod || Containers || Used to create a particular software environment for scripts/pipelines
 |}

Btingle: Created page with "= Introduction to the AWS Setup platform = The concept of "the cloud" as a platform for computation is somewhat mystified. Scientists talk about running their calculations "i..."

2022-08-03T20:48:38Z

Created page with "= Introduction to the AWS Setup platform = The concept of "the cloud" as a platform for computation is somewhat mystified. Scientists talk about running their calculations "i..."

New page

= Introduction to the AWS Setup platform =

The concept of "the cloud" as a platform for computation is somewhat mystified. Scientists talk about running their calculations "in the cloud", but to what end? How does one even begin to use the cloud? We are broadly familiar with the academic cluster environment, but the transition from that environment to cloud computation is not often discussed in detail.

Enter aws-setup: a collection of scripts for configuring a virtual cluster environment in the AWS cloud. The power of these scripts lies in the deep knowledge they encapsulate of AWS systems.

There exist other solutions for easy creation of a cluster environment in the cloud, for example cluster-in-the-cloud, which allows one to create a SLURM-managed cluster on any of the major cloud providers' systems. The power of this is in the simplicity of transition- one can take a script written for use in an academic cluster and port it straight to the cloud! There are pitfalls to this approach- in attempting to cover up all the complexity of cloud systems, there is a risk of that complexity emerging in undesirable ways.

If attempting to set up your own cluster environment on the cloud is like rowing a paddleboat into a raging storm, then cluster-in-the-cloud is like taking a luxury cruise into a tsunami.

aws-setup, on the other hand, is like a submarine. Sure, there's no hot tub or open bar, but it will get you to where you want to go. What aws-setup promises is tight integration with AWS systems, robustness, efficiency, and relative simplicity (compared to setting things up oneself).

= Translating Cluster Concepts to AWS Cloud Concepts =

{| class="wikitable"
|+ Understanding AWS in relation to a traditional cluster
|-
! Traditional Cluster Concept !! Corresponding AWS Concept !! Shared Properties
|-
| NFS || S3 Bucket || Storage Accessible from anywhere
|-
| CPU || vCPU || One Unit of computing power
|-
| Node || EC2 Instance || Machine (virtual or otherwise) with CPUs, memory, disk, etc.
|-
| Users/Groups || Roles, Policies, IAM Users || Security structures for limiting access to resources
|-
| Slurm/SGE || AWS Batch || Platform for scheduling compute jobs
|-
| Package Managers/Lmod || Containers || Used to create a particular software environment for scripts/pipelines
|}