What is different about ML systems
More and more DevOps and platform roles now include running machine-learning workloads: GPU node pools, model serving endpoints, pipelines that retrain models, and applications built on large language models (LLMs). You don't need to be a data scientist, but interviewers expect you to know what is different about ML systems, how models are trained, versioned and served, how GPUs work on Kubernetes, how to monitor models as well as servers, and the security and cost traps. This topic covers those basics for AWS and Kubernetes.
The ML lifecycle
A traditional service is code plus configuration. An ML system is code plus data plus a trained model, and the model's behaviour comes from the data it learned from, not from rules someone wrote. Its life cycle:
- Collect and prepare data: raw data is cleaned, labelled and turned into features (inputs the model uses).
- Train: an algorithm fits model weights to the training data, often on GPUs, for minutes to weeks.
- Evaluate: measure accuracy (or other metrics) on data the model hasn't seen; compare with the current production model.
- Register: store the approved model artifact with its version, metrics and the data and code that produced it.
- Deploy: serve it for real-time predictions, batch jobs or streaming.
- Monitor: watch both system health and model quality; when quality drops because the world changed, retrain.
MLOps applies DevOps practices — version control, automation, CI/CD, monitoring, reproducibility — to this loop.
Why it's harder to operate
- Silent failure: a model can return confident, wrong answers with HTTP 200 and normal latency. Server metrics alone won't catch it.
- Data dependence: a change upstream (a renamed field, a new currency, a different camera) can break a model with no code change at all.
- Expensive hardware: GPUs cost many times more than CPUs, are scarce in some regions and must be kept busy.
- Large artifacts: model files from hundreds of megabytes to hundreds of gigabytes, which affects image sizes, start-up time and storage.
- Non-determinism: retraining on the same data can produce a slightly different model, and LLM outputs vary between calls.
Who does what
| Role | Focus |
|---|---|
| Data scientist | Explores data, chooses and trains models, evaluates them |
| ML engineer | Turns models into production pipelines and services |
| DevOps / platform engineer | GPU infrastructure, clusters, pipelines, serving platform, observability, security, cost |
In interviews, position yourself clearly: you build and run the platform that makes training and serving reliable, repeatable and affordable.