Prompt / context builder
Assembles system prompt, history and retrieved context within the token budget.
Problem it solves: Context windows are finite and tokens cost money.
Where it sits: Before the LLM.
When to use
- Every LLM application.
Failure modes
- Context overflow truncates key facts.
Metrics to watch
- Input tokens / request
Configuration in Labs (illustrative defaults)
- Instances (count) = 2 — Identical copies behind this logical component. Capacity scales linearly in this model.
- Capacity per instance (req/s) = 2000 — Maximum requests one instance completes per second before it saturates.
- Base service time (ms) = 5 — Processing time per request at zero load. Queueing adds to this as utilisation rises.