Retrieval service
Embeds the query, searches indexes, filters by permission, returns passages.
Problem it solves: Grounds an LLM in current, private knowledge.
Where it sits: Called by the orchestrator before the LLM.
When to use
- RAG.
When not to use
- Knowledge already inside the prompt.
Failure modes
- Slow vector search stalls the whole answer.
- Wrong ACL filtering.
Metrics to watch
- Retrieval latency
- Hit rate of relevant docs
Alternatives
- Long-context prompting
Configuration in Labs (illustrative defaults)
- Instances (count) = 2 — Identical copies behind this logical component. Capacity scales linearly in this model.
- Capacity per instance (req/s) = 500 — Maximum requests one instance completes per second before it saturates.
- Base service time (ms) = 10 — Processing time per request at zero load. Queueing adds to this as utilisation rises.