Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Public AMIs need automatic cleanup in teleport-stage #52679

Open
camscale opened this issue Mar 3, 2025 · 2 comments
Open

Public AMIs need automatic cleanup in teleport-stage #52679

camscale opened this issue Mar 3, 2025 · 2 comments

Comments

@camscale
Copy link
Contributor

camscale commented Mar 3, 2025

The overnight release build produces 24 public AMI images each night (4 versions, 6 images per version). Some developer builds are also published on occasion. There is a limit of 200 public AMIs in this account and we hit that limit reasonably quickly, leading to overnight build failures during the "publish AMI" jobs.

We should have an automated clean-up that removes older images so that we always have enough capacity to run the overnight build.

@camscale
Copy link
Contributor Author

camscale commented Mar 3, 2025

@fheinecke
Copy link
Contributor

Here are the quick and dirty scripts I use to nuke these every week:

#!/bin/bash

set -o pipefail

echo "Account: $(aws sts get-caller-identity | jq -rc '.Arn')"
read -p "Are you sure (y/n)? " -n 1 -r
echo
if [[ ! ${REPLY} =~ ^[Yy]$ ]]
then
    echo "Did not receive confirmation, aborting."
    exit 1
fi

cleanup-region() {
    region="$1"
    echo "Purging images for ${region}"
    for image in $(aws ec2 describe-images --owners self --region "${region}" | jq -rc '.Images[] | select(.Name | contains("teleport-hardened-base-image-") | not) | {"imageId": .ImageId, "snapshotId": (.BlockDeviceMappings | first | .Ebs.SnapshotId)}'); do
        imageId="$(echo "${image}" | jq -rc '.imageId')"
        snapshotId="$(echo "${image}" | jq -rc '.snapshotId')"
        aws ec2 deregister-image --region "${region}" --image-id "${imageId}"
        aws ec2 delete-snapshot --region "${region}" --snapshot-id "${snapshotId}"
    done
    echo "Region ${region} complete"
}


for region in $(aws account list-regions --region-opt-status-contains ENABLED ENABLING ENABLED_BY_DEFAULT DISABLING | jq -rc '.Regions[].RegionName'); do
    cleanup-region "${region}" &
done

wait < <(jobs -p)

Related: gravitational/shared-workflows#214

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants