Model serving (online)
Hosts a trained (non-LLM) model behind a prediction API.
Problem it solves: Predictions must be fast and versioned.
Where it sits: Called by services; reads the feature store.
When to use
- Ranking, fraud scoring, classification.
Failure modes
- Model drift silently degrades quality.
Metrics to watch
- Prediction latency
- Prediction distribution drift
Alternatives
- Batch scoring
Configuration in Labs (illustrative defaults)
- Instances (count) = 2 — Identical copies behind this logical component. Capacity scales linearly in this model.
- Capacity per instance (req/s) = 1000 — Maximum requests one instance completes per second before it saturates.
- Base service time (ms) = 10 — Processing time per request at zero load. Queueing adds to this as utilisation rises.