Opscraft
Kubernetes in depth · Lesson 1 of 11 · 6 min read

The Kubernetes model: declarative state and control loops

Senior DevOps roles expect you to know Kubernetes inside and out (and if you hold the CKA, interviewers will assume you do). Interviews probe the internals: how a request flows through the API server, what etcd and Raft guarantee, how the scheduler decides, what the kubelet does on a node, how Services and DNS really route traffic, how storage binds, and how RBAC and admission control secure the cluster. This topic is the deep reference; the separate Kubernetes troubleshooting and EKS topics cover day-two operations and AWS specifics.

Objects

Enlarge diagramCONTROL PLANE (MANAGED BY AWS ON EKS)WORKER NODE 1WORKER NODE 2kube-apiserverthe front dooretcdall cluster stateschedulerpicks a nodecontrollersreconcile loopskubeletruns the podskube-proxyService rulespodpodpodkubeletruns the podskube-proxyService rulespodpodpod
Figure 1. A Kubernetes cluster. The control plane stores and decides; worker nodes run the pods. Everything talks only to the API server.
Swipe sideways to see all of it, or tap to open it full screen.

Everything in Kubernetes is an object stored by the API server:

apiVersion: apps/v1          # API group/version
kind: Deployment              # type
metadata:                     # identity and metadata
  name: orders
  namespace: shop
  labels: { app: orders, team: checkout }
  annotations: { owner: "[email protected]" }
spec:                         # DESIRED state — written by you
  replicas: 3
  ...
status:                       # OBSERVED state — written by controllers
  readyReplicas: 3
  • spec is what you want; status is what the system observed. Controllers work to make status match spec.
  • API groups and versions: core (v1: Pods, Services, ConfigMaps, Secrets, Namespaces, Nodes, PVs/PVCs), apps/v1 (Deployments, StatefulSets, DaemonSets, ReplicaSets), batch/v1 (Jobs, CronJobs), networking.k8s.io/v1 (Ingress, NetworkPolicy), rbac.authorization.k8s.io/v1, autoscaling/v2, policy/v1 (PDB), and many more. Versions move v1alpha1 → v1beta1 → v1; beta APIs get removed in later releases, which is why upgrades break old manifests.
  • Each object has a resourceVersion used for optimistic concurrency (a conflicting update gets HTTP 409) and for watches.
Enlarge diagramspec: written by youreplicas: 3 · image: orders:1.4controllercomparesstatus: written bycontrollersreadyReplicas: 2 → creating1
Figure 2. Every object has a spec (what you want) and a status (what the system sees). Controllers work to make status match spec.
Swipe sideways to see all of it, or tap to open it full screen.

Reconciliation

Kubernetes is a set of controllers, each running a loop:

Enlarge diagramObservewatch the current stateComparewith the desired stateActcreate, delete, updateRepeatforever
Figure 3. The reconciliation loop every controller runs. It does not matter why things drifted; the loop simply drives reality back to the desired state.
Swipe sideways to see all of it, or tap to open it full screen.

This level-triggered design is why Kubernetes self-heals: delete a pod managed by a ReplicaSet and the controller notices "2 exist, 3 desired" and creates one. It doesn't matter why state drifted.

Controllers don't poll; they use watches — long-lived streaming connections to the API server that deliver ADDED/MODIFIED/DELETED events. Client libraries (informers) keep a local cache and re-list periodically. Chatty custom controllers that LIST large collections repeatedly are a classic cause of API server load.

Labels, selectors and annotations

  • Labels — identifying key/value pairs used for selection: Services select pods, ReplicaSets own pods, NetworkPolicies target pods, kubectl get pods -l app=orders,tier!=batch.
  • Selectors — equality (app=orders) or set-based (env in (staging,prod)).
  • Annotations — non-identifying metadata for tools and humans (kubernetes.io/change-cause, ingress controller settings, Prometheus scrape hints).

A Service's selector matches any pod with those labels in the namespace — including pods from a different Deployment that happen to share labels. That's a real outage pattern (a debug pod or a canary with the same labels silently receiving production traffic).

Recommended labels: app.kubernetes.io/name, /instance, /version, /component, /part-of, /managed-by.

Enlarge diagramService ordersselector: app=ordersDeployment ordersselector: app=orderspodapp=orderspodapp=orderspodapp=paymentsnot selected
Figure 4. Labels are how Kubernetes objects find each other. The Service and the Deployment both select pods with app=orders.
Swipe sideways to see all of it, or tap to open it full screen.

Namespaces

A namespace scopes names and is the unit for RBAC, ResourceQuotas, LimitRanges, NetworkPolicies and Pod Security levels. Some resources are cluster-scoped: Nodes, PersistentVolumes, StorageClasses, ClusterRoles, CRDs, Namespaces themselves. Default namespaces: default, kube-system, kube-public, kube-node-lease (node heartbeats).

Ownership, garbage collection and finalizers

  • ownerReferences link dependants to owners (Pod → ReplicaSet → Deployment). Deleting an owner cascades: --cascade=foreground (dependants first), background (default), or orphan.
  • Finalizers are keys in metadata.finalizers that block deletion until a controller cleans up external resources (e.g. the AWS LB controller deleting an ALB, kubernetes.io/pvc-protection). The object gets a deletionTimestamp and waits. If the controller is gone, the object (or namespace) hangs in Terminating forever — fix the controller or, as a last resort, remove the finalizer knowingly.
Enlarge diagramDeploymentordersReplicaSetorders-7d9f… (templatehash)Pod 1ownerReference →RSPod 2ownerReference →RSPod 3ownerReference →RSowns
Figure 5. Ownership: a Deployment owns ReplicaSets, which own Pods. Delete the owner and the dependants go too.
Swipe sideways to see all of it, or tap to open it full screen.