Opscraft
Running AI/ML systems · Lesson 5 of 5 · 3 min read

Interview questions and model answers

"What's different about operating ML systems compared with normal services?"

Behaviour depends on data, not just code, so a model can fail silently — returning wrong answers with healthy latency — when input data changes. Artifacts are large, GPUs are expensive and scarce, and releases need evaluation against data, not just tests. So beyond normal DevOps I'd add data validation, model and data versioning with a registry, model-quality and drift monitoring, and safe release techniques like shadow and canary deployments.

"How would you set up GPU workloads on EKS?"

Dedicated GPU NodePools in Karpenter (or node groups) using EKS accelerated AMIs, the NVIDIA device plugin or GPU Operator, taints on GPU nodes with tolerations on GPU pods, and pods requesting nvidia.com/gpu in limits. Scale from zero with Karpenter, pre-pull large images and cache model weights to cut start-up time, monitor utilisation with DCGM exporter, and use MIG or time-slicing for small models. Spot with checkpointing for training, on-demand or reservations for latency-critical inference.

"How do you deploy a new model version safely?"

Train and evaluate it in a pipeline, register it with metrics and lineage, and only promote it if it beats the current model on the evaluation set. Run it in shadow mode against live traffic, then canary a small share, comparing latency, errors and business metrics, with automatic rollback to the previous registry version if they degrade. Monitoring of drift and quality continues after full rollout.

"What is data drift and how do you detect it?"

Data drift is when the distribution of production inputs moves away from what the model was trained on, for example new customer segments or changed upstream data. Detect it by comparing feature distributions between training and recent production data with statistical tests on a schedule, alongside prediction distribution changes and, when labels arrive, real accuracy. Significant drift triggers investigation and possibly retraining.

"Hosted LLM API or self-hosted model?"

Hosted APIs like Bedrock give the strongest models with no GPU operations and pay-per-token pricing, ideal to start and for moderate volume. Self-hosting an open-weight model with vLLM on GPUs makes sense for strict data-control requirements, heavy customisation, or high steady volume where per-token pricing exceeds the cost of busy GPUs. I'd decide with a cost model at expected volume, latency targets and data requirements.

"What metrics matter for an LLM endpoint?"

Time to first token and tokens per second for user experience, total throughput, error and timeout rates, GPU utilisation and KV-cache memory pressure for capacity, tokens and cost per request and per customer, and quality signals from evaluation sets and user feedback.

"What are the main security risks in an LLM application?"

Prompt injection through user input or retrieved documents, leaking sensitive data through prompts, outputs or logs, retrieval that ignores document permissions, unsafe model files from public hubs, and runaway cost from abuse. Mitigations: treat model output as untrusted, least-privilege tools and permissions, permission-aware retrieval, data minimisation and redaction, models from a trusted registry in safe formats, authentication, rate limits and token quotas.