Reranker
Cross-encoder that re-scores retrieved candidates for precision.
Problem it solves: ANN retrieval returns noisy top-k.
Where it sits: After retrieval, before the context builder.
When to use
- Quality-sensitive RAG/search.
When not to use
- Strict latency budgets with good first-stage recall.
Failure modes
- Adds tens of ms per request.
Metrics to watch
- Latency
- nDCG / precision@k
Alternatives
- Better first-stage retrieval
Configuration in Labs (illustrative defaults)
- Instances (count) = 2 — Identical copies behind this logical component. Capacity scales linearly in this model.
- Capacity per instance (req/s) = 200 — Maximum requests one instance completes per second before it saturates.
- Base service time (ms) = 40 — Processing time per request at zero load. Queueing adds to this as utilisation rises.