Azure OpenAI
GPT and other OpenAI models hosted in Azure regions.
Problem it solves: Generation is expensive, token-bound and GPU-limited.
Where it sits: Called by an orchestrator or LLM gateway.
When to use
- Generation, summarisation, reasoning, tool selection.
When not to use
- Deterministic lookups or classification a small model handles.
Failure modes
- GPU saturation: queueing makes TTFT spike.
- Long outputs dominate latency.
Metrics to watch
- TTFT (ms)
- Output tokens/s
- Concurrent sequences
- Cost per request (only with stated pricing)
Alternatives
- Smaller model + routing
- Cached responses
Configuration in Labs (illustrative defaults)
- Instances (count) = 2 — Identical copies behind this logical component. Capacity scales linearly in this model.
- Concurrent sequences / instance (requests) = 16 — Requests one replica decodes in parallel (continuous batching).
- Time to first token (ms) = 400 — Prefill latency before the first output token, driven by input length and model size.
- Input tokens / request (tokens) = 1500 — Prompt + retrieved context length. Used for the token-throughput estimate.
- Output tokens / request (tokens) = 300 — Generated tokens; decode time = outputTokens / tokensPerSec.
- Decode speed per sequence (tokens/s) = 50 — Output tokens per second for one in-flight request.
Vendor examples (pattern only): Azure managed service. Simulated as a generic “LLM inference”.