CHAPTER 25
ML System Design Foundations
What makes ML systems different, and a framework for designing them
Source: PDF pp. 65–67
PDF page 65Why ML systems are different from ordinary software p. 65
In traditional software, behaviour is defined by code. In an ML system, behaviour is defined by code + data + a trained model, and the world the data came from keeps changing. The model is a small box inside a much larger system of data pipelines, feature computation, serving, monitoring and feedback.
| Property | Consequence for system design |
|---|---|
| Data dependency | Quality, freshness and volume of data bound model quality; data pipelines are first-class, versioned and tested |
| Probabilistic outputs | Correctness is statistical; define metrics, thresholds and fallbacks instead of exact-match tests |
| Silent failure | A degraded model keeps returning plausible answers with no exception; monitoring must measure quality, not only uptime |
| Training-serving skew | Features computed differently offline and online give wrong predictions; share feature definitions and code |
| Feedback loops | The model changes what users see, which changes future training data (popularity bias, echo chambers); needs exploration and care |
| Drift | Input distribution (data drift) or the input-output relationship (concept drift) changes, so models decay and need retraining |
| Hidden technical debt | Glue code, pipeline jungles, undeclared consumers and entangled features dominate maintenance cost |
ML development is a loop, not a line: most of the effort lives in steps 2, 3 and 7.
A framework for ML system design p. 65
1. Clarify the business objective and constraints. What decision does the prediction drive? Latency, throughput, cost, privacy, explainability, regulatory limits, and the cost of each error type. 2. Frame it as an ML task. Classification, regression, ranking, retrieval, clustering, sequence generation, anomaly detection, reinforcement/bandit. Decide whether ML is needed at all (a rule or heuristic baseline first). 3. Define metrics. Business metric (revenue, retention), online metric (CTR, conversion, latency), offline proxy (AUC, NDCG, F1), and guardrail metrics (fairness, safety, latency). 4. Data. Sources, volume, labels (explicit, implicit, human, weak), freshness, privacy, bias, how training data is sampled and split. 5. Features. Candidate signals, how and when each is computed (batch vs real-time), storage, point-in-time correctness.
6. Model. Start simple (heuristic, logistic regression, gradient-boosted trees), justify moving to deep models; training scale, loss, tuning. 7. Evaluation. Offline on time-split data and slices; then shadow, canary and A/B test online. 8. Serving and operations. Batch vs online, latency budget, scaling, caching, fallbacks, monitoring, retraining, rollback.
Problem framing examples p. 66
| Product problem | ML formulation | Typical model / signal |
|---|---|---|
| Spam / abuse detection | Binary classification (imbalanced) | GBDT / text models; precision-recall trade-off |
| Feed, video or product recommendations | Two-stage retrieval then ranking (learning-to-rank) | Two-tower retrieval; DNN / GBDT rankers (Ch. 28) |
| ETA, demand forecast, price | Regression / time-series forecasting | GBDT, temporal models; quantile loss for uncertainty |
| Fraud, intrusion | Anomaly detection / imbalanced classification with delayed labels | GBDT + rules + graph features; real-time velocity features |
| Search relevance | Retrieval + learning-to-rank | BM25 + embeddings, cross-encoder reranker |
| Customer support automation | Generative QA with retrieval and tools | LLM + RAG + agent (Ch. 30-31) |
| Content moderation | Multi-label classification (text / image / video) | Fine-tuned encoders / vision models, LLM classifiers + human review |
Choosing metrics p. 66
| Task | Offline metrics | Notes |
|---|---|---|
| Classification | Precision, recall, F1, ROC-AUC, PR-AUC (imbalanced), log-loss, calibration | Pick the threshold from the cost of false positives vs false negatives; accuracy misleads on skewed classes |
| Regression / forecast | MAE, RMSE, MAPE, quantile loss | MAE robust to outliers; MAPE breaks near zero |
| Ranking / recommendation | NDCG@k, MAP, MRR, Recall@k, Hit rate | Position-aware; evaluate on held-out future interactions |
| Generation (text) | Perplexity, BLEU/ROUGE (weak), LLM-as-judge, human preference, task success | See Ch. 33: evaluate with task-specific rubrics |
| Online | CTR, conversion, retention, session length, latency, revenue per user, long-term value | Short-term clicks can hurt long-term value; track guardrails |
- A/B testing: randomise by a stable unit (user), pre-register the primary metric, size the sample for statistical power, run at least one full weekly cycle (novelty effects, weekday patterns), watch for network interference and sample-ratio mismatch.
- Interleaving (mixing results from two rankers in one list) detects ranking differences with far less traffic. Multi-armed bandits trade exploration against exploitation when you cannot wait for fixed-horizon tests.
Data: where most projects are won or lost p. 66
- Labels: human annotation (expensive, needs guidelines and agreement checks), implicit feedback (clicks, purchases: noisy and biased by position), weak supervision / heuristics, and LLM-assisted labelling verified against a human-labelled sample.
- Class imbalance: use class weights, resampling, focal loss, and PR-AUC; calibrate probabilities afterwards.
- Splitting: use time-based splits (train on past, test on future) for anything temporal, and group splits (by user / session) to prevent the same entity appearing in train and test.
- Leakage: features unavailable at prediction time (e.g. "refund issued" in a fraud model) or computed with future data make offline metrics fantasy. Always compute features as of the prediction timestamp.
- Bias and privacy: audit slices (region, demographic, device); minimise and anonymise PII; consider differential privacy and federated learning where required; track data provenance and consent.
- Data versioning and quality: schema checks, null/range/distribution tests on every pipeline run (Great Expectations, Deequ), dataset versions (DVC, lakeFS, Delta/Iceberg time travel) so any model can be reproduced.
PDF page 67Model choice at a glance p. 67
| Family | Use when | Notes |
|---|---|---|
| Heuristics / rules | Always build first as a baseline | Often 80% of the value; keep as a fallback |
| Linear / logistic regression | Small data, need interpretability, very low latency | Strong baseline with good features |
| Gradient-boosted trees (XGBoost, | Tabular data, the default winner | Fast to train and serve; handle missing values |
LightGBM, CatBoost)
| Deep networks (MLP, CNN, RNN) | Images, audio, sequences, very large sparse data | Need more data and compute; embeddings for categorical IDs |
|---|---|---|
| Transformers / LLMs | Text, code, multimodal, generation, few-shot tasks | Pretrained foundation models; adapt via prompting, RAG or fine-tuning (Part IV later chapters) |
| Graph neural networks | Relational structure (social, fraud rings, molecules) | Heavier infrastructure; consider graph features in GBDT first |
| Bandits / RL | Sequential decisions with feedback (exploration needed) | Offline policy evaluation is hard; guard rails essential |
Review questions for this chapter have not been written and reviewed yet. The chapter text above is the full source content.
References
System Design & AI System Design: The Complete Guide — 2026 Edition, pp. 65–67; figures 46.
Tip: use ← and → keys to move between chapters.