CHAPTER 27
Model Serving, MLOps and Monitoring
Getting models into production, keeping them fast, and noticing when they decay
Source: PDF pp. 71–73
Core PDF content
PDF page 71Serving patterns p. 71
| Pattern | How it works | Latency / freshness | Use when |
|---|---|---|---|
| Batch prediction | Run the model on all entities on a schedule; store results in a DB or cache | Fast reads; results hours old | Recommendations for email, churn scores, nightly risk scoring |
| Online (real-time) | Request-response inference with fresh features | Tens to hundreds of ms | Search ranking, fraud at checkout, ads, chat |
| Streaming | Consume events, score continuously, emit results to a topic | Seconds | Anomaly detection, monitoring, real-time personalisation |
| Edge / on-device | Run a compressed model on phone, browser or device | No network; works offline; private | Keyboards, camera effects, wake-word detection |
| Latency budget (example 100 ms p99) | Share |
|---|---|
| Network + gateway | ~10 ms |
| Feature fetch (parallel, online store) | ~10-20 ms |
| Model inference (batched, GPU/CPU) | ~20-40 ms |
| Post-processing, business rules, logging (async) | ~5-10 ms |
| Headroom for tail latency and retries | ~20 ms |
PDF page 72Making inference fast and cheap p. 72
| Technique | What it does | Typical effect |
|---|---|---|
| Quantisation (INT8 / FP8 / INT4) | Lower-precision weights and activations | 2-4x less memory, faster; small accuracy loss; calibrate |
| Distillation | Train a small student to mimic a large teacher | Often 2-10x cheaper at similar quality for a task |
| Pruning / sparsity | Remove low-importance weights or heads | Smaller models; needs hardware/kernel support |
| Compilation and kernels | TensorRT, ONNX Runtime, torch.compile, XLA, operator fusion | 1.5-5x throughput from the same weights |
| Dynamic batching | Collect requests for a few ms and run them as one batch | Large GPU throughput gain; adds small latency |
| Cascades / early exit | Cheap model first; only uncertain cases go to the big model | Cuts average cost and latency |
| Caching | Cache predictions for popular inputs; precompute embeddings | Removes model calls entirely for hot keys |
| Right hardware | CPUs for small tree / linear models; GPUs/accelerators for deep models; batch to fill them | Avoid paying for idle GPUs |
Serving frameworks: NVIDIA Triton, TorchServe, TensorFlow Serving, KServe / Seldon on Kubernetes, BentoML, Ray Serve, and managed endpoints (SageMaker, Vertex AI). For LLMs use specialised engines (vLLM, TensorRT-LLM, SGLang; Ch. 29).
Releasing models safely p. 72
| Stage | What happens | Catches |
|---|---|---|
| Offline gate | Candidate must beat the champion on held-out and slice metrics, and pass data/fairness checks | Obvious regressions |
| Shadow | Candidate receives live traffic but its output is only logged and compared | Latency, crashes, large prediction disagreements |
| Canary | 1-5% of real users get the candidate; auto-rollback on guardrail breach | Production-only bugs and business-metric harm |
| A/B test | Randomised experiment on the primary business metric | True causal impact |
| Gradual rollout | Ramp to 100% with monitoring; keep the old version warm for instant rollback | Scale and long-tail issues |
Monitoring: more than uptime p. 72
- System metrics: latency percentiles, throughput, error rate, GPU/CPU utilisation, queue depth, cost per 1,000 predictions.
- Data quality and drift: missing/NaN rates, schema violations, feature distribution shift vs training (PSI: <0.1 stable, 0.1-0.25 moderate, >0.25 significant shift; KS test; Jensen-Shannon divergence). Prediction drift: change in score distribution.
- Model quality: the hard part is label delay (fraud labels arrive after weeks). Use proxy metrics (click-through, overrides, escalation), delayed-label dashboards, periodic human labelling of samples, and slice analysis (region, device, new vs old users).
- Feedback loops and fairness: watch exposure concentration, popularity bias and per-group error rates.
- Retraining policy: on a schedule (simple), on drift/performance triggers (efficient), or continuously (streaming/online learning, riskiest). Every automatic retrain must pass the same evaluation gate as a human-triggered one.
PDF page 73MLOps: automating the loop p. 73
Level 0: manual notebooks. Level 1: automated training pipeline (continuous training). Level 2: CI/CD of the pipeline itself.
| Need | Common tools |
|---|---|
| Orchestration | Airflow, Dagster, Prefect, Kubeflow Pipelines, Argo Workflows, Metaflow, Flyte |
| Tracking and registry | MLflow, Weights & Biases, Neptune, Vertex / SageMaker model registries |
| Features | Feast, Tecton, Databricks / Vertex / SageMaker feature stores |
| Serving | KServe, Seldon, Triton, BentoML, Ray Serve, managed endpoints |
| Monitoring | Evidently, Arize, WhyLabs, Fiddler, Prometheus + Grafana, custom dashboards |
| Data quality / versioning | Great Expectations, Deequ, dbt tests, DVC, lakeFS, Delta Lake / Iceberg |
Review questions for this chapter have not been written and reviewed yet. The chapter text above is the full source content.
Try it
References
System Design & AI System Design: The Complete Guide — 2026 Edition, pp. 71–73; figures 49, 50.
Tip: use ← and → keys to move between chapters.