CHAPTER 32

LLM Application Architecture and LLMOps

Gateway, routing, caching, cost, observability and operating LLM products

Source: PDF pp. 86–87

Core PDF content

PDF page 86Reference architecture p. 86

Figure 59: A reference architecture for an LLM product: thin client, policy at the edges, an orchestrator in the middle, and a platform layer for prompts, evals and observability. — Figure 59. A reference architecture for an LLM product: thin client, policy at the edges, an orchestrator in the middle, and a platform layer for prompts, evals and observability. · PDF p. 86

Key components p. 86

Component	Responsibilities
API gateway	AuthN/Z, per-user and per-tenant token-based rate limits and quotas (requests are not equal), abuse detection, request size limits, streaming (SSE / WebSocket) termination
Orchestrator	Builds the prompt (system instructions, memory, retrieved context, tools); manages the context budget; runs the tool/agent loop; retries; chooses model/route; stores conversation state
Guardrails	Input: PII redaction, jailbreak/injection detection, topic and policy checks. Output: schema validation, moderation, groundedness checks, secret/PII leak filters (Ch. 33)
Model gateway	Unified interface across providers and self-hosted models; routing, fallbacks, timeouts, retry with backoff, load-balancing across keys/regions, usage metering
Caches	Exact response cache, semantic cache (embedding similarity above a threshold), provider prompt caching, retrieval cache
Prompt registry	Version-controlled prompts and configs with environments, rollouts, rollbacks and A/B tests; prompts are code
Observability and evals	Traces per request (retrieval, LLM calls, tools), token and cost accounting, quality scores, feedback capture, regression suites in CI

Cost and latency engineering p. 86

Cost per request = input tokens x input price + output tokens x output price (output usually costs several times more). Illustrative numbers (placeholder prices of $3 per 1 M input and $15 per 1 M output tokens; 1 M requests/day, 3,000 input and 500 output tokens each):

Scenario	Cost per request	Per day	Change
Baseline: one large model, no caching	$0.0165	~$16,500	-
+ prompt caching (80% of input tokens cached at 10% price)	~$0.0100	~$10,000	-39%

Scenario	Cost per request	Per day	Change
+ route 70% of traffic to a model ~10x cheaper	~$0.0037	~$3,700	-78%

Lever	Latency	Cost	How
Stream tokens	Perceived (TTFT)	-	SSE; render as tokens arrive
Shorter prompts / retrieve less	Better	Better	Compress context, summarise history, rerank to fewer chunks
Prompt caching	Better TTFT	Much better	Stable prefix first (system prompt, tools, docs), variable content last
Semantic / response cache	Large	Large	Only for repeatable, non-personalised queries; tune similarity threshold; invalidate on content change
Model routing / cascades	Better	Large	Small model by default; escalate on low confidence, complexity classifier or failed validation
Smaller models, distillation,	Better	Large	Fine-tune a small model for the narrow task (Ch. 33)

quantisation

Parallel tool calls / speculative work	Better	Slight worse	Launch independent retrievals and tools concurrently
Cap output length, structured outputs	Better	Better	Ask for concise JSON; set max_tokens; avoid verbose reasoning where not needed
Batch / async APIs for offline work	-	Large (often ~50%)	Evals, enrichment, backfills, embeddings

Rate limits and 429s: providers throttle by requests and tokens. Use queues, token-bucket limiters, exponential backoff with jitter, multi-key/region pools, priority classes (interactive over batch) and graceful degradation (smaller model, shorter answer).
Semantic cache risk: a too-loose threshold returns a confidently wrong cached answer to a different question. Scope caches per tenant, keep personal data out, and measure false-hit rate.
Reliability: multi-provider failover, circuit breakers, per-step timeouts, idempotent retries, and a degraded mode (cached or canned answer) so an upstream outage does not take the product down.
Privacy and compliance: PII redaction before sending to third parties, zero-retention agreements, regional endpoints, tenant isolation of indexes and caches, encryption, audit trails, retention limits, and clear user consent for training use.

Observability and feedback for LLM apps p. 87

Trace every request end to end as spans (retrieval, rerank, each model call, each tool call) with token counts, latency (TTFT, total), cost, model/prompt version and user/session IDs (OpenTelemetry GenAI conventions; tools such as Langfuse, LangSmith, Arize Phoenix, Helicone).
Quality signals: thumbs, edits, regenerations, abandonment, escalations to a human, task completion; automatic scorers (LLM-judge, groundedness, format validity) on sampled traffic; alert on shifts.
Close the loop: failures flow into a review queue, become eval cases and fine-tuning data, and the next release is gated on the grown suite.

Review questions for this chapter have not been written and reviewed yet. The chapter text above is the full source content.

Try it

Open “LLM application & LLMOps” in Labs LLM inference Prompt / context builder Evaluation / monitoring Bedrock Vertex AI — Gemini Azure OpenAI

References

System Design & AI System Design: The Complete Guide — 2026 Edition, pp. 86–87; figures 59.

Ch. 31 Ch. 33: Fine-Tuning, Evaluation and Safety

Tip: use ← and → keys to move between chapters.