Open Policy Agent · Lesson 5 of 9 · 4 min read
Rolling out policies safely and operating them
A rollout plan that doesn't break teams
- Inventory and agree the policies with stakeholders; write the reason and remediation into each message.
- Deploy the policy engine and constraints via GitOps (Flux) from a policy repo with cluster overlays.
- Start every new constraint in
dryrun(orwarn, so developers see warnings in kubectl output). - Read audit results: how many existing violations, which teams.
- Notify owners with guidance and a deadline; fix the platform's own components first.
- Switch to
denyper namespace/environment gradually — dev first, then staging, then prod. - Keep a documented exception process: excluded namespaces, an exemption label validated by the platform team, or per-resource exceptions with expiry.
Enlarge diagram
Swipe sideways to see all of it, or tap to open it full screen.
If an audit shows 300 violations, prioritise by risk (privileged pods before missing labels), group by owner, automate fixes where possible (e.g. a Kustomize component that adds limits), and set dates.
Availability and failure modes
The admission webhook sits in the path of every matching API write. Design for its failure:
Enlarge diagram
Swipe sideways to see all of it, or tap to open it full screen.
failurePolicy |
If Gatekeeper is down or times out | Trade-off |
|---|---|---|
Ignore (fail-open) |
Requests are allowed | Deploys keep working, but policies can be bypassed during outages |
Fail (fail-closed) |
Requests are rejected | Safer, but an outage of the webhook blocks all deploys |
Gatekeeper's default is fail-open (Ignore) for the validating webhook. For fail-closed, you need:
- 3+ replicas spread across nodes/AZs, with a PDB and priority class.
- A short timeout (e.g. 3 s) so a slow webhook doesn't stall the API server.
- Exempt system namespaces (
kube-system,gatekeeper-system, the CNI's namespace) so the cluster can heal itself — otherwise a broken webhook can block the very pods needed to fix it. - Monitoring and alerts on webhook latency, errors and rejections.
Performance
Each admission request evaluates all matching constraints. Keep Rego efficient (avoid nested loops over large synced data), scope match narrowly (kinds, namespaces), and watch Gatekeeper's metrics (gatekeeper_validation_request_duration_seconds, violation counts).
Managing policies across many clusters
- One policy repo; base templates and constraints with Kustomize overlays per cluster/environment (dryrun in dev, deny in prod, different exemptions).
- Flux reconciles it to every cluster — the same pattern you use for applications.
- Version policies (tags/releases), test in CI (
gator verify,opa test), and promote like code. - Dashboards of audit violations per cluster and team; measure success as violations trending to zero and few blocked deploys with clear messages.