Opscraft
DevSecOps & secrets management · Lesson 5 of 7 · 4 min read

Secrets management

Principles

  1. Never in Git (even private) or container images; scan to enforce.
  2. Centralised store with access control and audit: AWS Secrets Manager, SSM Parameter Store, HashiCorp Vault.
  3. Short-lived, dynamic where possible (Vault database secrets engine issues per-app credentials with TTLs; IAM roles instead of keys).
  4. Rotation automated (Secrets Manager rotation Lambdas, Vault leases).
  5. Least privilege per workload (IRSA/Pod Identity policy scoped to specific secret ARNs and KMS keys).
  6. Audit every access (CloudTrail, Vault audit device).
Enlarge diagramNever in Gitor images: scannedCentral storeSecrets Manager, VaultShort-liveddynamic credentials, rolesRotatedautomaticallyLeast privilegeone secret ARN per workloadAuditedCloudTrail, Vault audit
Figure 1. Principles for secrets: one central store, short-lived where possible, scoped per workload, and every read audited.
Swipe sideways to see all of it, or tap to open it full screen.

Delivering secrets to Kubernetes workloads

Approach How Trade-offs
External Secrets Operator ExternalSecret → syncs from Secrets Manager/SSM/Vault into a K8s Secret, refresh interval GitOps-friendly; secret lands in etcd (encrypt with KMS); rotation propagates on refresh but apps must reload
Secrets Store CSI Driver (+ AWS provider) Mounts secrets as files directly from the store at pod start; optional sync to K8s Secret Avoids K8s Secrets if not synced; rotation via polling; pod needs the mount
Vault Agent Injector / Vault Secrets Operator Sidecar renders secrets to files, renews leases; or operator syncs Dynamic secrets, powerful; Vault to operate
SOPS / Sealed Secrets Encrypted secrets in Git, decrypted in cluster Simple GitOps; rotation is manual; key management matters

Kubernetes Secrets are only base64-encoded — not encrypted — unless you enable encryption at rest (EKS: KMS envelope encryption, now default for new clusters) and restrict RBAC get/list on secrets. Prefer mounting secrets as files over env vars (env vars leak into crash dumps, /proc, child processes and logs more easily).

Enlarge diagramExternal Secrets OperatorSecrets Store CSI driverSyncs into a Kubernetes SecretRefreshes on an intervalGitOps-friendly: only the reference in GitMounts files at pod startNo Kubernetes Secret neededRotation needs a remount or sync
Figure 2. Two common ways to get secrets into pods on EKS.
Swipe sideways to see all of it, or tap to open it full screen.

ExternalSecret example

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata: { name: orders-db, namespace: orders }
spec:
  refreshInterval: 1h
  secretStoreRef: { name: aws-secrets-manager, kind: ClusterSecretStore }
  target: { name: orders-db, creationPolicy: Owner }
  data:
    - secretKey: password
      remoteRef: { key: prod/orders/db, property: password }

Secret leak response

Revoke/rotate first (assume compromised the moment it's exposed — bots scrape public GitHub within minutes), then investigate usage in audit logs, purge from Git history only as hygiene (git filter-repo, forks and clones may still have it), and fix the cause (push protection, pre-commit hooks, moving to OIDC/roles).

Enlarge diagramRevoke /rotateimmediatelyInvestigateaudit logs: was itused?Clean upattackerresourcesPurge historyhygiene onlyPreventpush protection,OIDC, no statickeys12345
Figure 3. A leaked key: revoke first, because bots find public keys within minutes. Cleaning Git history comes last.
Swipe sideways to see all of it, or tap to open it full screen.