CHAPTER 29
LLM Fundamentals and Inference Systems
What a system designer must know about transformers, tokens, GPUs and serving
Source: PDF pp. 77–79
PDF page 77The mental model p. 77
- Tokens, not words. Text is split into sub-word tokens (BPE / SentencePiece). In English ~4 characters or ~0.75 words per token; code, numbers and many non-English languages use more tokens per word. Cost, latency and context limits are all counted in tokens.
- Decoder-only transformers generate autoregressively: each new token requires a forward pass that attends to all previous tokens, then is appended and the process repeats until a stop condition.
- Attention lets each token weigh every earlier token (queries, keys, values). Compute grows ~quadratically with sequence length; memory for cached keys/values grows linearly.
- Context window is the maximum prompt + output tokens. Long contexts are expensive (prefill compute, KV-cache memory) and models use middle-of-context information less reliably than the start and end.
- Mixture-of-Experts (MoE) models route each token to a few "expert" sub-networks: many total parameters, fewer active parameters per token (cheaper compute), but all experts must still sit in GPU memory.
- How models are built: pre-training (next-token prediction on huge corpora) then post-training: supervised fine-tuning on demonstrations, preference optimisation (RLHF / DPO), and for reasoning models, reinforcement learning with verifiable rewards. Reasoning models spend extra "thinking" tokens, trading latency and cost for accuracy.
Controlling generation p. 77
| Parameter | Effect | Guidance |
|---|---|---|
| temperature | Scales randomness of token choice (0 = most deterministic) | 0-0.3 for extraction, classification, code; 0.7-1 for creative text |
| top_p / top_k | Restrict sampling to the most likely tokens (nucleus / k best) | Tune one of temperature or top_p, not both |
| max_tokens, stop | Bound output length and stop at delimiters | Always set; protects cost and latency |
| Structured output | JSON-schema or grammar-constrained decoding guarantees parseable output | Use for any machine-consumed result |
| Tool / function calling | Model emits a structured call that your code executes | Foundation of agents (Ch. 31) |
| seed | Best-effort reproducibility | Not a guarantee across hardware or model updates |
Latency anatomy: prefill and decode p. 77
Compute-bound: big matrix-matrix work, Memory-bandwidth-bound: every step re-reads all weights
GPU cores saturated. and the whole KV cache for one new token.KV cache prompt K/V stored
Total latency = TTFT + (output tokens - 1) x TPOT. The KV cache grows by one entry per token per layer.
| Metric | Meaning | What improves it |
|---|---|---|
| TTFT | Time to first token (queueing + prefill) | Shorter prompts, prefix caching, chunked prefill, more GPUs, routing to idle replicas |
| TPOT / ITL | Time between streamed tokens (~30-100 ms feels fluent) | Smaller/quantised model, speculative decoding, less batching contention, faster memory |
| Throughput | Total tokens/s across all requests | Bigger batches, continuous batching, paged KV cache, quantisation |
| Cost per 1M tokens | Output tokens usually cost several times input tokens | Smaller models, caching, shorter outputs, batch APIs |
Memory arithmetic for inference p. 78
| Model size | Weights fp16/bf16 (2 B/param) | 8-bit | 4-bit |
|---|---|---|---|
| 7-8 B | ~14-16 GB | ~8 GB | ~4-5 GB |
| 13 B | ~26 GB | ~13 GB | ~7 GB |
| 70 B | ~140 GB (2-4 x 80 GB GPUs) | ~70 GB | ~35-40 GB |
| 400 B+ (MoE) | ~800 GB (multi-node / 8+ GPUs) | ~400 GB | ~200 GB |
KV cache per token = 2 (K and V) x layers x KV-heads x head-dim x bytes. A 70 B model with grouped-query attention (80 layers, 8 KV heads, head-dim 128, fp16) needs 2 x 80 x 8 x 128 x 2 B = ~0.31 MB per token, so one 8,000-token conversation holds ~2.5 GB, and 100 concurrent such conversations need ~250 GB on top of the weights. Memory, not compute, usually limits concurrency. A useful bound for decode speed: tokens/s per sequence <= memory bandwidth / bytes read per token. A 16 GB (8 B fp16) model on a ~3 TB/s GPU is capped near ~200 tokens/s per stream; batching serves many streams per weight read.
Inside an inference engine p. 78
| Technique | Idea | Benefit / caveat |
|---|---|---|
| Continuous (in-flight) batching | Admit new requests and retire finished ones at every decode step instead of waiting for a whole batch | Several-fold throughput vs static batching; the default in modern engines |
| PagedAttention | KV cache stored in fixed-size blocks mapped like virtual memory | Near-zero waste, more concurrent sequences, shared prefixes |
| Prefix (prompt) caching | Reuse KV for an identical leading prompt (system prompt, document, few-shot examples) | Cuts TTFT and cost; put static content first, variable content last |
| Speculative decoding | A small draft model proposes several tokens; the large model verifies them in one pass | Often 2-3x faster decode with identical output distribution |
| Quantisation (AWQ, GPTQ, FP8, INT4) | Lower-precision weights (and KV cache) | 2-4x less memory, faster decode; evaluate quality on your task |
| Technique | Idea | Benefit / caveat |
|---|---|---|
| FlashAttention, GQA / MQA | IO-efficient exact attention; fewer KV heads | Faster attention, much smaller KV cache |
| Tensor / pipeline / expert parallelism | Split a model that does not fit on one GPU across GPUs | Tensor parallel needs NVLink inside a node; pipeline across nodes; experts for MoE |
| Disaggregated prefill / decode | Separate GPU pools for compute-bound prefill and memory-bound decode | Protects TPOT from long prompts; more complex |
| Multi-LoRA serving | One base model, many small adapters swapped per request | Cheap per-tenant customisation |
Engines: vLLM, TensorRT-LLM, SGLang, Hugging Face TGI, and llama.cpp / MLC for local and edge. Autoscaling GPUs is slow (loading tens of GB of weights takes minutes), so keep a warm pool, scale on queue depth and KV-cache utilisation rather than CPU, and use fast model-loading paths (local NVMe, weight snapshots).
API or self-hosted? p. 79
| Hosted model API | Self-hosted open-weights model | |
|---|---|---|
| Quality | Frontier capability, fast upgrades | Strong but usually a step behind; excellent for narrow tasks after fine-tuning |
| Cost shape | Pay per token; zero idle cost; price per token falls over time | Fixed GPU cost; cheaper per token only at high, steady utilisation |
| Latency / control | Provider-controlled; shared capacity, rate limits | Full control: batching, caching, co-location, custom kernels |
| Privacy / | Contractual (zero retention, regional endpoints) | Data never leaves your VPC / on-prem |
residency
Operations Minimal GPU fleet, engine upgrades, monitoring, capacity planning
Review questions for this chapter have not been written and reviewed yet. The chapter text above is the full source content.
Try it
References
System Design & AI System Design: The Complete Guide — 2026 Edition, pp. 77–79; figures 53, 54.
Tip: use ← and → keys to move between chapters.