Image building with Packer · Lesson 4 of 5 · 5 min read
Pipelines and rollout
An image pipeline
Enlarge diagram
Swipe sideways to see all of it, or tap to open it full screen.
A typical pipeline in GitHub Actions or Jenkins:
- Trigger on changes to the Packer repository, on a schedule (weekly patching), and when the upstream base image is updated.
packer init,fmt -check,validate.packer buildin a dedicated image-build account, with the build role assumed through OIDC.- Test and scan the new AMI; fail the pipeline on errors or critical vulnerabilities.
- Publish: copy to the regions you use (
ami_regions), share with workload accounts (ami_usersorami_org_arns), and record the new AMI ID in a lookup that consumers read, such as an SSM parameter (/images/base-al2023/latest) or the HCP Packer registry's channels. - Promote: dev picks up the latest automatically; staging and production move to a version after it has run cleanly for a while.
If the AMI's snapshots are encrypted with a customer-managed KMS key, the accounts that launch it also need permission to use that key, or launches fail with a confusing Client.InvalidKMSKey.InvalidState or Client.InternalError.
Terraform can then read the image instead of hard-coding IDs:
data "aws_ssm_parameter" "base_ami" {
name = "/images/base-al2023/prod"
}
resource "aws_launch_template" "app" {
image_id = data.aws_ssm_parameter.base_ami.value
# ...
}
Rolling out a new image
Enlarge diagram
Swipe sideways to see all of it, or tap to open it full screen.
- Auto Scaling groups: update the launch template to a new version with the new AMI, then start an instance refresh, which replaces instances in batches while keeping a minimum healthy percentage, with optional checkpoints and automatic rollback on failed health checks or alarms.
- EKS managed node groups: update the node group to the new AMI release or a new launch template version; nodes are cordoned, drained (respecting PodDisruptionBudgets) and replaced.
- Karpenter: the
EC2NodeClassselects AMIs (amiSelectorTerms, for example by alias or by tag/ID). When the selected AMI changes, Karpenter marks existing nodes as drifted and replaces them gradually, within the disruption budgets you set on theNodePool.
Enlarge diagram
Swipe sideways to see all of it, or tap to open it full screen.
Roll out to one environment at a time, watch error rates and node health, and keep the previous AMI available so rollback is just pointing back at it.
Lifecycle and alternatives
- Old images cost money through their EBS snapshots. Deregister AMIs past a retention window (for example, keep the last five plus anything still referenced by a launch template) and delete their snapshots. Deregistering an AMI does not delete its snapshots automatically. EC2's AMI deprecation dates and Recycle Bin rules help.
- EC2 Image Builder is AWS's managed alternative: pipelines, components, tests, distribution and scheduling as AWS resources, with no build servers to run. A good fit for AWS-only teams; Packer is better when you build for several platforms or want the templates in the same workflow as Terraform.
- Bottlerocket is AWS's minimal, container-only OS for EKS nodes: read-only root filesystem, no package manager, updated as a whole image. It reduces how much you need to bake at all.
- Packer, like Terraform, moved to the Business Source Licence in 2023; it remains free to use for building your own images.