CHAPTER 30
Embeddings, Vector Search and RAG
Grounding language models in your own, current, access-controlled knowledge
Source: PDF pp. 80–82
Core PDF content
PDF page 80Why retrieval-augmented generation (RAG)? p. 80
LLMs know only what was in their training data, hallucinate when unsure, and cannot cite sources. RAG retrieves relevant passages from an external knowledge base at query time and places them in the prompt, so the model answers from evidence. It gives fresh data (no retraining), per-user access control, citations, and lower hallucination, at the price of a retrieval system to build and evaluate.
Embeddings p. 80
- An embedding maps text (or images, audio, code) to a dense vector so that semantically similar items are close together. Retrieval becomes nearest-neighbour search.
- Similarity: cosine (angle), dot product (equivalent if vectors are normalised), or Euclidean. Use the metric the model was trained with. Typical dimensions 384-3072; Matryoshka embeddings can be truncated to shorter vectors with modest loss.
- Choosing a model: check multilingual coverage, domain fit, max input length, dimensionality (storage and speed), latency/cost, and benchmark on your queries (MTEB is a starting point). Domain adaptation or fine-tuning embeddings can lift recall significantly.
- Operational rule: the same model must embed documents and queries; changing the embedding model means re-embedding and re-indexing everything. Version the index and run old and new in parallel during migration.
Approximate nearest-neighbour (ANN) indexes p. 80
query
| Index | How it works | Pros | Cons |
|---|---|---|---|
| Flat (brute force) | Compare with every vector | Exact, no build | O(N): fine under ~1 M vectors / with GPUs |
| IVF (+PQ) | Cluster vectors; search the nearest nprobe clusters; product quantisation compresses vectors to a few bytes | Small memory, fast, scales to billions | Lower recall unless tuned; needs training |
| HNSW | Multi-layer proximity graph with greedy search | Excellent recall-latency; incremental inserts | High memory; slower builds; deletions awkward |
| DiskANN / ScaNN | SSD-resident graph; anisotropic quantisation | Very large corpora on modest RAM | More tuning |
- Sizing: 10 M vectors x 1,536 dims x 4 B = ~61 GB raw (HNSW adds graph overhead of tens of %). 1 B vectors x 768 dims x 4 B = ~3 TB, so use PQ/8-bit/binary quantisation, disk-based indexes and sharding.
- Metadata filtering (tenant, date, ACL) is critical: pre-filtering is exact but can be slow for restrictive filters; post-filtering can return too few results; good engines integrate filters into the graph/cluster traversal.
- Hybrid search: combine BM25 keyword (exact terms, IDs, rare words) with dense vectors (meaning) and merge with Reciprocal Rank Fusion; this beats either alone on most real corpora.
- Products: pgvector (Postgres), Elasticsearch/OpenSearch, Pinecone, Weaviate, Qdrant, Milvus, Vespa, FAISS / ScaNN (libraries). If you already run Postgres or OpenSearch and have < ~10-50 M vectors, start there.
Chunking and indexing decisions p. 81
| Strategy | Description | When to use |
|---|---|---|
| Fixed size + overlap | 300-800 tokens with 10-20% overlap | Baseline for prose; cheap and predictable |
| Structure-aware / recursive | Split on headings, paragraphs, sentences, code blocks, table rows | Documents with clear structure; keeps semantic units intact |
| Parent-child (small-to-big) | Embed small chunks for precise matching; return the larger parent section to the LLM | Precision in retrieval + enough context for generation |
| Semantic chunking | Break where embedding similarity between sentences drops | Long unstructured text; costlier |
| Contextual chunks | Prepend a short LLM-written summary of document context to each chunk before embedding | Chunks that are ambiguous alone ("it increased 3%") |
| Tables / images / code | Extract tables to text/markdown; caption images or use multimodal embeddings; chunk code by function | Mixed enterprise corpora |
- Always store metadata: source URL, title, section, author, timestamps, language, doc type, and access-control info, to enable filtering, citation, freshness weighting and deletion.
- Freshness: incremental ingestion via webhooks or CDC, content hashes to skip unchanged docs, tombstones for deletes, and scheduled re-crawls; version chunks so updates are atomic.
Improving retrieval quality p. 81
- Hybrid search + reranking is the highest-impact upgrade: retrieve ~20-100 candidates cheaply, then rerank with a cross-encoder (reads query and passage together) and keep the top 5-10.
- Query transformation: rewrite with conversation history ("what about its price?" to a standalone query), expand with synonyms, multi-query (several paraphrases, merged), HyDE (embed a hypothetical answer), step-back questions, decomposition of multi-part questions.
- Diversity (MMR) avoids ten near-duplicate chunks; metadata filters and recency boosts focus on the right slice.
- Context assembly: deduplicate, order by relevance (and put the most important evidence at the start or end), compress irrelevant sentences, cap tokens, and label each passage with its source ID for citation.
- Advanced patterns: agentic RAG (the model decides when and what to retrieve, iterates, uses multiple indexes or SQL), multi-hop retrieval, GraphRAG (entity graph and community summaries for corpus-wide questions), self-/corrective RAG (grade retrieved passages and retry), routing between vector, keyword, SQL and web sources.
- Long context vs RAG: stuffing whole documents is simple for small corpora, but costs more per query, slows TTFT, and degrades with distractors. Use RAG for large, changing, permissioned corpora; use long context (with prompt caching) for single-document analysis.
Evaluating RAG p. 82
| Layer | Metrics | How |
|---|---|---|
| Retrieval | Recall@k, hit rate, MRR, nDCG, context precision | Labelled (question, relevant chunk) pairs; synthetic questions generated from chunks then human-checked |
| Generation | Faithfulness / groundedness (every claim supported by context), answer relevance, completeness, citation accuracy | LLM-as-judge with rubrics and spot human audits; frameworks such as RAGAS, TruLens |
| End to end | Task success, user satisfaction, deflection / escalation rate, latency, cost per answer | Golden set (100-500 real questions) in CI; online A/B and feedback |
| Failure mode | Symptom | Fix |
|---|---|---|
| Poor parsing | Garbled tables/PDF text in chunks | Better parsers / layout models, OCR, table extraction |
| Bad chunking | Answer split across chunks, or chunk lacks context | Parent-child, overlap, contextual chunks |
| Retrieval miss | Right document exists but is not retrieved | Hybrid search, query rewrite, tune embeddings, increase k then rerank |
| Noise / lost in the middle | Model ignores relevant passage among many | Rerank, fewer and better chunks, reorder |
| Hallucination despite context | Claims not in sources | Stricter prompts, citations required, verification pass, "say you do not know" |
| Stale or deleted content | Outdated answer | Incremental indexing, TTLs, tombstones |
| Permission leakage | User sees restricted content | Enforce ACL filters inside retrieval, never rely on the prompt |
| Indirect prompt injection | Document text contains instructions to the model | Treat retrieved text as data; sanitise; limit tool powers (Ch. 33) |
Review questions for this chapter have not been written and reviewed yet. The chapter text above is the full source content.
Try it
References
System Design & AI System Design: The Complete Guide — 2026 Edition, pp. 80–82; figures 55, 56.
Tip: use ← and → keys to move between chapters.