Docking in AWS With DOCK 3.8 - Revision history

Btingle at 23:16, 2 December 2021

2021-12-02T23:16:55Z

← Older revision		Revision as of 23:16, 2 December 2021
Line 183:		Line 183:
	<nowiki>		<nowiki>
	aws iam put-role-policy --role-name ecsInstanceRole --policy-name s3policy --policy-document file://s3policy.json</nowiki>		aws iam put-role-policy --role-name ecsInstanceRole --policy-name s3policy --policy-document file://s3policy.json</nowiki>

			[[Category:AWS]]

Btingle: /* Misc: Configuring Launch Template with Dockerhub credentials */

2021-06-02T19:47:33Z

Misc: Configuring Launch Template with Dockerhub credentials

← Older revision		Revision as of 19:47, 2 June 2021
Line 125:		Line 125:
	</nowiki>		</nowiki>

	In the above example, we use aws secretsmanager to fetch our dockerhub password. This is accomplished by adding the AWSSecretsManager:GetSecret permission (for our dockerhub password secret arn) to our ecsInstance role. Alternatively, you can hard-code the authentication details into the launch template.		In the above example, we use aws secretsmanager to fetch our dockerhub password. This is accomplished by adding the AWSSecretsManager:GetSecret permission for our dockerhub password secret arn to our ecsInstance role. Alternatively, you can hard-code the authentication details into the launch template.

	For more information on authentication methods, try the following links:		For more information on authentication methods, try the following links:

Btingle: /* Optional: AWS credentials */

2021-05-15T21:19:53Z

Optional: AWS credentials

← Older revision		Revision as of 21:19, 15 May 2021
Line 86:		Line 86:
	If you are submitting within aws batch, you may want to consider adding s3 permissions to your ecsInstanceRole (or whatever role you are using for aws batch instances) in lieu of an explicit access key.		If you are submitting within aws batch, you may want to consider adding s3 permissions to your ecsInstanceRole (or whatever role you are using for aws batch instances) in lieu of an explicit access key.

	If you do not have your aws access key & secret key in aws secretsmanager, (under the names "aws_access_key" and "aws_secret_key" respectively) you will also want to export a non-null value for NO_ACCESS_KEY.		In the current configuration, any access keys supplied via environmental parameters is used to fetch access keys from aws secretsmanager, which will be used to fetch/put data in s3. If you do not have your aws access key & secret key in aws secretsmanager, (under the names "aws_access_key" and "aws_secret_key" respectively) you will also want to export a non-null value for NO_ACCESS_KEY.

	=== Restartability & Spot Instances ===		=== Restartability & Spot Instances ===

Btingle: /* Optional: AWS credentials */

2021-05-15T21:15:23Z

Optional: AWS credentials

← Older revision		Revision as of 21:15, 15 May 2021
Line 85:		Line 85:

	If you are submitting within aws batch, you may want to consider adding s3 permissions to your ecsInstanceRole (or whatever role you are using for aws batch instances) in lieu of an explicit access key.		If you are submitting within aws batch, you may want to consider adding s3 permissions to your ecsInstanceRole (or whatever role you are using for aws batch instances) in lieu of an explicit access key.

			If you do not have your aws access key & secret key in aws secretsmanager, (under the names "aws_access_key" and "aws_secret_key" respectively) you will also want to export a non-null value for NO_ACCESS_KEY.

	=== Restartability & Spot Instances ===		=== Restartability & Spot Instances ===

Btingle: /* S3_INPUT_LOCATION */

2021-05-14T18:56:55Z

S3_INPUT_LOCATION

← Older revision		Revision as of 18:56, 14 May 2021
Line 17:		Line 17:
	If S3_INPUT_LOCATION is a directory:		If S3_INPUT_LOCATION is a directory:

	Input will be taken from the AWS_BATCH_JOB_ARRAY_INDEX'th file listed under the directory		Input will be taken from the AWS_BATCH_JOB_ARRAY_INDEX'th file (0-indexed, because that's how aws happens to do it) listed under the directory

	<nowiki>		<nowiki>
	AWS_BATCH_JOB_ARRAY_INDEX=3		AWS_BATCH_JOB_ARRAY_INDEX=2
	S3_INPUT_LOCATION=s3://mytestbucket/input		S3_INPUT_LOCATION=s3://mytestbucket/input

Line 33:		Line 33:
	If S3_INPUT_LOCATION is a file:		If S3_INPUT_LOCATION is a file:

	Input will be downloaded from the AWS_BATCH_JOB_ARRAY_INDEX'th line in the file		Input will be downloaded from the AWS_BATCH_JOB_ARRAY_INDEX'th (again, 0-indexed) line in the file

	<nowiki>		<nowiki>
	AWS_BATCH_JOB_ARRAY_INDEX=2		AWS_BATCH_JOB_ARRAY_INDEX=1
	S3_INPUT_LOCATION=s3://mytestbucket/input/inputlist_batch1.txt		S3_INPUT_LOCATION=s3://mytestbucket/input/inputlist_batch1.txt
	### inputlist_batch1.txt		### inputlist_batch1.txt

Btingle: /* Misc: Configuring Launch Template with Dockerhub credentials */

2021-05-14T18:54:06Z

Misc: Configuring Launch Template with Dockerhub credentials

← Older revision		Revision as of 18:54, 14 May 2021
Line 110:		Line 110:

	#!/bin/bash		#!/bin/bash
			yum install -y aws-cli # make sure the aws cli is installed, otherwise this next step won't work
	docker_email=ben@tingle.org		docker_email=ben@tingle.org
	docker_user=btingle		docker_user=btingle

Btingle: /* Misc: Configuring Launch Template with Dockerhub credentials */

2021-05-14T18:53:18Z

Misc: Configuring Launch Template with Dockerhub credentials

← Older revision		Revision as of 18:53, 14 May 2021
Line 114:		Line 114:
	export AWS_DEFAULT_REGION=us-west-1		export AWS_DEFAULT_REGION=us-west-1
	docker_auth=$(aws secretsmanager get-secret-value --secret-id dockerhub_pass \| grep "SecretString" \| cut -d':' -f2 \| sed 's/\"//g' \| sed 's/,//g' \| tail -c +2 \| sed 's/ *$//g')		docker_auth=$(aws secretsmanager get-secret-value --secret-id dockerhub_pass \| grep "SecretString" \| cut -d':' -f2 \| sed 's/\"//g' \| sed 's/,//g' \| tail -c +2 \| sed 's/ *$//g')
	# aws secretsmanager returns a json ~~formatted string~~, this convoluted pipe exists to format it such that only the plain text password is retrieved		# aws secretsmanager returns a json object, this convoluted pipe exists to format it such that only the plain text password is retrieved

	echo ECS_ENGINE_AUTH_TYPE=docker >> /etc/ecs/ecs.config		echo ECS_ENGINE_AUTH_TYPE=docker >> /etc/ecs/ecs.config

Btingle: /* Misc: Configuring Launch Template with Dockerhub credentials */

2021-05-14T18:51:49Z

Misc: Configuring Launch Template with Dockerhub credentials

← Older revision		Revision as of 18:51, 14 May 2021
Line 113:		Line 113:
	docker_user=btingle		docker_user=btingle
	export AWS_DEFAULT_REGION=us-west-1		export AWS_DEFAULT_REGION=us-west-1
	docker_auth=$(aws secretsmanager get-secret-value --secret-id dockerhub_pass \| grep "SecretString" \| cut -d':' -f2 \| sed 's/\"//g' \| sed 's/,//g' \| tail -c +2)		docker_auth=$(aws secretsmanager get-secret-value --secret-id dockerhub_pass \| grep "SecretString" \| cut -d':' -f2 \| sed 's/\"//g' \| sed 's/,//g' \| tail -c +2 \| sed 's/ *$//g')
			# aws secretsmanager returns a json formatted string, this convoluted pipe exists to format it such that only the plain text password is retrieved

	echo ECS_ENGINE_AUTH_TYPE=docker >> /etc/ecs/ecs.config		echo ECS_ENGINE_AUTH_TYPE=docker >> /etc/ecs/ecs.config
	echo ECS_ENGINE_AUTH_DATA={\"https://index.docker.io/v1/\":{\"~~auth~~\":\"${docker_auth}\"\,\"email\":\"${docker_email}\"\,\"username\":\"${docker_user}\"}} >> /etc/ecs/ecs.config		echo ECS_ENGINE_AUTH_DATA={\"https://index.docker.io/v1/\":{\"password\":\"${docker_auth}\"\,\"email\":\"${docker_email}\"\,\"username\":\"${docker_user}\"}} >> /etc/ecs/ecs.config

	--==DOCKERHUBINIT==--		--==DOCKERHUBINIT==--

Btingle: /* Misc: Configuring Launch Template with Dockerhub credentials */

2021-05-14T18:48:48Z

Misc: Configuring Launch Template with Dockerhub credentials

← Older revision		Revision as of 18:48, 14 May 2021
Line 116:		Line 116:

	echo ECS_ENGINE_AUTH_TYPE=docker >> /etc/ecs/ecs.config		echo ECS_ENGINE_AUTH_TYPE=docker >> /etc/ecs/ecs.config
	echo ECS_ENGINE_AUTH_DATA={\"https://index.docker.io/v1/\":{\"auth\":\"${docker_auth}\",\"email\":\"${docker_email}\",\"username\":\"${docker_user}\"}} >> /etc/ecs/ecs.config		echo ECS_ENGINE_AUTH_DATA={\"https://index.docker.io/v1/\":{\"auth\":\"${docker_auth}\"\,\"email\":\"${docker_email}\"\,\"username\":\"${docker_user}\"}} >> /etc/ecs/ecs.config

	--==DOCKERHUBINIT==--		--==DOCKERHUBINIT==--

Btingle at 18:27, 14 May 2021

2021-05-14T18:27:27Z

@@ Line 70: / Line 70: @@
 description: The array index of the current job. Automatically supplied when submitting via aws batch.
 === Optional: AWS credentials ===
@@ Line 126: / Line 130: @@
 https://aws.amazon.com/blogs/containers/authenticating-with-docker-hub-for-aws-container-services/