The Kubernetes model: declarative state and control loops
Senior DevOps roles expect you to know Kubernetes inside and out (and if you hold the CKA, interviewers will assume you do). Interviews probe the internals: how a request flows through the API server, what etcd and Raft guarantee, how the scheduler decides, what the kubelet does on a node, how Services and DNS really route traffic, how storage binds, and how RBAC and admission control secure the cluster. This topic is the deep reference; the separate Kubernetes troubleshooting and EKS topics cover day-two operations and AWS specifics.
Objects
Everything in Kubernetes is an object stored by the API server:
apiVersion: apps/v1 # API group/version
kind: Deployment # type
metadata: # identity and metadata
name: orders
namespace: shop
labels: { app: orders, team: checkout }
annotations: { owner: "[email protected]" }
spec: # DESIRED state — written by you
replicas: 3
...
status: # OBSERVED state — written by controllers
readyReplicas: 3
- spec is what you want; status is what the system observed. Controllers work to make status match spec.
- API groups and versions: core (
v1: Pods, Services, ConfigMaps, Secrets, Namespaces, Nodes, PVs/PVCs),apps/v1(Deployments, StatefulSets, DaemonSets, ReplicaSets),batch/v1(Jobs, CronJobs),networking.k8s.io/v1(Ingress, NetworkPolicy),rbac.authorization.k8s.io/v1,autoscaling/v2,policy/v1(PDB), and many more. Versions movev1alpha1 → v1beta1 → v1; beta APIs get removed in later releases, which is why upgrades break old manifests. - Each object has a resourceVersion used for optimistic concurrency (a conflicting update gets HTTP 409) and for watches.
Reconciliation
Kubernetes is a set of controllers, each running a loop:
This level-triggered design is why Kubernetes self-heals: delete a pod managed by a ReplicaSet and the controller notices "2 exist, 3 desired" and creates one. It doesn't matter why state drifted.
Controllers don't poll; they use watches — long-lived streaming connections to the API server that deliver ADDED/MODIFIED/DELETED events. Client libraries (informers) keep a local cache and re-list periodically. Chatty custom controllers that LIST large collections repeatedly are a classic cause of API server load.
Labels, selectors and annotations
- Labels — identifying key/value pairs used for selection: Services select pods, ReplicaSets own pods, NetworkPolicies target pods,
kubectl get pods -l app=orders,tier!=batch. - Selectors — equality (
app=orders) or set-based (env in (staging,prod)). - Annotations — non-identifying metadata for tools and humans (
kubernetes.io/change-cause, ingress controller settings, Prometheus scrape hints).
A Service's selector matches any pod with those labels in the namespace — including pods from a different Deployment that happen to share labels. That's a real outage pattern (a debug pod or a canary with the same labels silently receiving production traffic).
Recommended labels: app.kubernetes.io/name, /instance, /version, /component, /part-of, /managed-by.
Namespaces
A namespace scopes names and is the unit for RBAC, ResourceQuotas, LimitRanges, NetworkPolicies and Pod Security levels. Some resources are cluster-scoped: Nodes, PersistentVolumes, StorageClasses, ClusterRoles, CRDs, Namespaces themselves. Default namespaces: default, kube-system, kube-public, kube-node-lease (node heartbeats).
Ownership, garbage collection and finalizers
- ownerReferences link dependants to owners (Pod → ReplicaSet → Deployment). Deleting an owner cascades:
--cascade=foreground(dependants first),background(default), ororphan. - Finalizers are keys in
metadata.finalizersthat block deletion until a controller cleans up external resources (e.g. the AWS LB controller deleting an ALB,kubernetes.io/pvc-protection). The object gets adeletionTimestampand waits. If the controller is gone, the object (or namespace) hangs in Terminating forever — fix the controller or, as a last resort, remove the finalizer knowingly.