DevSecOps & secrets management · Lesson 5 of 7 · 4 min read
Secrets management
Principles
- Never in Git (even private) or container images; scan to enforce.
- Centralised store with access control and audit: AWS Secrets Manager, SSM Parameter Store, HashiCorp Vault.
- Short-lived, dynamic where possible (Vault database secrets engine issues per-app credentials with TTLs; IAM roles instead of keys).
- Rotation automated (Secrets Manager rotation Lambdas, Vault leases).
- Least privilege per workload (IRSA/Pod Identity policy scoped to specific secret ARNs and KMS keys).
- Audit every access (CloudTrail, Vault audit device).
Enlarge diagram
Swipe sideways to see all of it, or tap to open it full screen.
Delivering secrets to Kubernetes workloads
| Approach | How | Trade-offs |
|---|---|---|
| External Secrets Operator | ExternalSecret → syncs from Secrets Manager/SSM/Vault into a K8s Secret, refresh interval |
GitOps-friendly; secret lands in etcd (encrypt with KMS); rotation propagates on refresh but apps must reload |
| Secrets Store CSI Driver (+ AWS provider) | Mounts secrets as files directly from the store at pod start; optional sync to K8s Secret | Avoids K8s Secrets if not synced; rotation via polling; pod needs the mount |
| Vault Agent Injector / Vault Secrets Operator | Sidecar renders secrets to files, renews leases; or operator syncs | Dynamic secrets, powerful; Vault to operate |
| SOPS / Sealed Secrets | Encrypted secrets in Git, decrypted in cluster | Simple GitOps; rotation is manual; key management matters |
Kubernetes Secrets are only base64-encoded — not encrypted — unless you enable encryption at rest (EKS: KMS envelope encryption, now default for new clusters) and restrict RBAC get/list on secrets. Prefer mounting secrets as files over env vars (env vars leak into crash dumps, /proc, child processes and logs more easily).
Enlarge diagram
Swipe sideways to see all of it, or tap to open it full screen.
ExternalSecret example
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata: { name: orders-db, namespace: orders }
spec:
refreshInterval: 1h
secretStoreRef: { name: aws-secrets-manager, kind: ClusterSecretStore }
target: { name: orders-db, creationPolicy: Owner }
data:
- secretKey: password
remoteRef: { key: prod/orders/db, property: password }
Secret leak response
Revoke/rotate first (assume compromised the moment it's exposed — bots scrape public GitHub within minutes), then investigate usage in audit logs, purge from Git history only as hygiene (git filter-repo, forks and clones may still have it), and fix the cause (push protection, pre-commit hooks, moving to OIDC/roles).
Enlarge diagram
Swipe sideways to see all of it, or tap to open it full screen.