Opscraft
Image building with Packer · Lesson 4 of 5 · 5 min read

Pipelines and rollout

An image pipeline

Enlarge diagramTriggercommit, weekly, new base AMIValidateinit, fmt, validateBuildimage account, OIDC roleTest and scanGoss, InspectorPublishcopy to regions, share, SSMparameterPromotedev → staging → prod123456
Figure 1. An image pipeline: build, test, scan, publish, then promote one environment at a time.
Swipe sideways to see all of it, or tap to open it full screen.

A typical pipeline in GitHub Actions or Jenkins:

  1. Trigger on changes to the Packer repository, on a schedule (weekly patching), and when the upstream base image is updated.
  2. packer init, fmt -check, validate.
  3. packer build in a dedicated image-build account, with the build role assumed through OIDC.
  4. Test and scan the new AMI; fail the pipeline on errors or critical vulnerabilities.
  5. Publish: copy to the regions you use (ami_regions), share with workload accounts (ami_users or ami_org_arns), and record the new AMI ID in a lookup that consumers read, such as an SSM parameter (/images/base-al2023/latest) or the HCP Packer registry's channels.
  6. Promote: dev picks up the latest automatically; staging and production move to a version after it has run cleanly for a while.

If the AMI's snapshots are encrypted with a customer-managed KMS key, the accounts that launch it also need permission to use that key, or launches fail with a confusing Client.InvalidKMSKey.InvalidState or Client.InternalError.

Terraform can then read the image instead of hard-coding IDs:

data "aws_ssm_parameter" "base_ami" {
  name = "/images/base-al2023/prod"
}
resource "aws_launch_template" "app" {
  image_id = data.aws_ssm_parameter.base_ami.value
  # ...
}

Rolling out a new image

Enlarge diagramSSM parameternew AMI ID perchannellaunchtemplateversion 8 usesnew AMIinstancerefreshbatches, minhealthy 90%health checksroll back on failure
Figure 2. Rolling out a new AMI: a new launch template version, then an instance refresh replaces servers in batches.
Swipe sideways to see all of it, or tap to open it full screen.
  • Auto Scaling groups: update the launch template to a new version with the new AMI, then start an instance refresh, which replaces instances in batches while keeping a minimum healthy percentage, with optional checkpoints and automatic rollback on failed health checks or alarms.
  • EKS managed node groups: update the node group to the new AMI release or a new launch template version; nodes are cordoned, drained (respecting PodDisruptionBudgets) and replaced.
  • Karpenter: the EC2NodeClass selects AMIs (amiSelectorTerms, for example by alias or by tag/ID). When the selected AMI changes, Karpenter marks existing nodes as drifted and replaces them gradually, within the disruption budgets you set on the NodePool.
Enlarge diagramEC2NodeClassamiSelectorTerms → newAMIdrift detectedold nodes no longer matchreplace graduallycordon, drain, respect PDBs
Figure 3. With Karpenter, changing the AMI selection marks nodes as drifted, and they are replaced within the disruption budget.
Swipe sideways to see all of it, or tap to open it full screen.

Roll out to one environment at a time, watch error rates and node health, and keep the previous AMI available so rollback is just pointing back at it.

Lifecycle and alternatives

  • Old images cost money through their EBS snapshots. Deregister AMIs past a retention window (for example, keep the last five plus anything still referenced by a launch template) and delete their snapshots. Deregistering an AMI does not delete its snapshots automatically. EC2's AMI deprecation dates and Recycle Bin rules help.
  • EC2 Image Builder is AWS's managed alternative: pipelines, components, tests, distribution and scheduling as AWS resources, with no build servers to run. A good fit for AWS-only teams; Packer is better when you build for several platforms or want the templates in the same workflow as Terraform.
  • Bottlerocket is AWS's minimal, container-only OS for EKS nodes: read-only root filesystem, no package manager, updated as a whole image. It reduces how much you need to bake at all.
  • Packer, like Terraform, moved to the Business Source Licence in 2023; it remains free to use for building your own images.