Shrink the ECS cluster to one t3a.medium #255

Dependency

• Move the four production services to bridge networking #254

Overview

We need to replace the cluster's two m5.large instances with one t3a.medium, because once every service uses bridge networking nothing needs the second host or the larger size. This is the step that saves the money, about $115/month.

Action Items

The capacity provider has managed draining disabled, so terminating an instance without draining it first kills its tasks. Drain manually every time.

• PR: in terraform/platform/compute.tf, change instance_type on aws_launch_template.ecs (line 44 when written — instance_type = "m5.large") to "t3a.medium", and desired_capacity on aws_autoscaling_group.ecs (line 66) from 2 to 1. Keep min_size = 1 and max_size = 2; the second slot is what makes an AMI replacement possible without an outage.
  • desired_capacity is in ignore_changes, so the edit only keeps the code honest; the actual scale-in is the manual steps below.
• Same PR: in terraform/modules/container/main.tf, replace the two spread strategies in placement_strategies (line 127) with a single binpack on memory. Spread existed only because ENI slots were scarce; its comment says so and should go with it.
• Run terraform validate and terraform fmt -check -diff locally; confirm in the CI plan that the services and ASG update in place.
• After the PR merges, swap the hosts one at a time:
  1. Set one m5.large container instance to DRAINING and wait until its tasks are running on the other. Then aws autoscaling terminate-instance-in-auto-scaling-group --instance-id <id> --should-decrement-desired-capacity.
  2. aws autoscaling set-desired-capacity --auto-scaling-group-name ecs-incubator-prod --desired-capacity 2. The new instance launches as a t3a.medium from launch template version $Latest. Wait until it registers with the cluster.
  3. Drain the remaining m5.large, wait for every task to land on the t3a.medium, then terminate it with --should-decrement-desired-capacity.
• After the swap, confirm: one container instance, type t3a.medium, HttpPutResponseHopLimit: 1; all ten services 1/1 with healthy targets; and record its registered and remaining memory in a comment:

  aws ecs describe-container-instances --cluster incubator-prod \
    --container-instances $(aws ecs list-container-instances --cluster incubator-prod --query 'containerInstanceArns[]' --output text) \
    --query 'containerInstances[].{ec2:ec2InstanceId,tasks:runningTasksCount,registered:registeredResources[?name==`MEMORY`].integerValue,remaining:remainingResources[?name==`MEMORY`].integerValue}'
  

  Remaining memory should be about 900 MB. That figure is the cluster's real headroom and belongs in the documentation in the next child.
• A week after the swap, check the instance's CPUCreditBalance is flat or rising, and that no task was stopped for running out of memory.

Resources/Instructions

• terraform/platform/compute.tf and terraform/modules/container/main.tf. Line numbers were accurate when written and may drift.
• The launch template's AMI (ami-036428f37186903ce, AL2023 ECS-optimized, x86_64) runs on t3a. The user data's awsvpcTrunking line is harmless on an instance type that does not support trunking.
• T3 unlimited mode is the default. It charges extra only if the 24-hour average stays above the 20%-per-vCPU baseline; measured demand averages about a fifth of that.
• Draining container instances.