Kubernetes troubleshooting · Lesson 10 of 11 · 3 min read
GitOps (Flux) troubleshooting
When the cluster doesn't match Git, check the Flux chain from source to workload:
Enlarge diagram
Swipe sideways to see all of it, or tap to open it full screen.
flux check # controllers healthy, versions
flux get sources git -A # can Flux fetch the repo? (auth, branch, deploy key)
flux get kustomizations -A # Ready? which revision applied? error message
flux get helmreleases -A # chart install/upgrade failures
flux logs --level=error --all-namespaces
kubectl describe kustomization apps -n flux-system
flux reconcile kustomization apps --with-source # force a sync now
flux diff kustomization apps --path ./clusters/prod # what would change
flux suspend kustomization apps / flux resume kustomization apps
Typical failures:
- Source not ready — Git auth (deploy key rotated), wrong branch/path, SOPS decryption key missing.
- Kustomization failed — invalid YAML/Kustomize build error, a CRD not installed yet (use
dependsOnso CRDs and controllers apply first), validation webhook rejection, immutable field changes (e.g. a Deployment selector or Job spec — needsforce: trueor delete/recreate). - Health checks timing out — Flux waits for workloads to become Ready; a crashing Deployment shows up here first.
- HelmRelease failed — install/upgrade errors, values errors; check remediation/rollback settings.
- Drift — someone
kubectl edited a resource: Flux reverts it on the next reconcile; changes must go through Git. In emergencies,flux suspend, fix, then revert in Git and resume. - Image automation not updating tags — ImageRepository can't list ECR (IRSA), policy (semver/regex) doesn't match tags, push to Git failing.