AI Case Studies I: Assistant, Enterprise RAG, Coding Assistant

PDF page 91Design 1: a ChatGPT-style assistant platform p. 91

- Free and paid tiers with quotas; conversation history across devices

• Context management: history grows each turn. Keep the last N turns verbatim, summarise older ones, store long-term memory as retrievable facts, and attach files via retrieval, to stay within the context budget and cost. Put the stable prefix (system prompt, tools) first so prefix caching hits.
• Streaming: SSE from the inference engine through the router, chat service and gateway; the UI renders tokens as they arrive. Cancel generation immediately when the client disconnects (frees GPU).
• Routing and tiers: a cheap classifier or heuristic selects small vs large vs reasoning model; priority queues give paid users lower queueing delay; under overload, shed free-tier load or degrade to a smaller model rather than fail.
• GPU efficiency: continuous batching, paged KV cache, prefix-aware load balancing (send a conversation to the replica holding its cached prefix), speculative decoding, quantisation, disaggregated prefill/decode for long prompts.
• Capacity and failure: autoscale on queue depth and KV-cache utilisation; keep warm spare replicas because loading weights is slow; multi-region with regional capacity pools; retry idempotently on a different replica when a worker dies mid-stream (resume or restart).

• Safety and abuse: input and output moderation, per-user rate and token limits, bot and scraping detection, tool sandboxing (no ambient credentials), logging with retention and privacy controls.

TRADE-OFF Larger models give better answers but cost more and respond slower. The product decision is a routing policy: which requests deserve the expensive model. Measure quality vs cost per segment and tune with experiments.

Design 2: enterprise document Q&A (permission-aware RAG) p. 92

minutes - Feedback and analytics for admins

• Permissions are the hard part. Mirror source ACLs (users, groups, shares, inheritance) into index metadata; sync permission changes quickly (a revoked user must lose access within minutes). When groups are huge or ACLs complex, resolve at query time against the source ("late binding") for the final candidate set as a safety net.
• Freshness: webhooks and change feeds from each connector; content hash to skip no-ops; delete tombstones propagate to the index; periodic full reconciliation.
• Quality stack: structure-aware parsing (tables, slides, OCR), contextual chunking, hybrid search + cross-encoder reranker, query rewriting using conversation context, source-aware ranking (recency, authority), and "I could not find this in the documents you can access" as a valid answer.
• Evaluation: golden set of real questions with expected source docs; track retrieval recall@k, faithfulness and citation accuracy in CI; thumbs and "report wrong answer" feed the set. Test permissions with adversarial users.
• Cost and scale: quantise vectors, shard by tenant, cache frequent queries and embeddings, use a small model for rewrite/routing and a larger one for final synthesis.
• Security: prompt-injection from documents is a real threat (a wiki page that says "ignore previous instructions"); treat retrieved text as data, never grant tools based on it, and log which chunks influenced each answer for audit.

PDF page 93Design 3: an AI coding assistant (completion + chat + agent) p. 93

• Fill-in-the-middle (FIM): completion models are trained to predict code between a prefix and a suffix. Context selection (current file around the cursor, imports, definitions of nearby symbols, recently edited files, similar snippets) matters more than model size for relevance.
• Latency engineering: debounce keystrokes, cancel outdated requests, stream partial results, cache the repeated file prefix (prompt caching), use a small quantised model with speculative decoding, place inference near developers, and show suggestions only when confidence is high enough.
• Repository understanding: index the repo with embeddings of chunks plus a symbol/call graph (tree-sitter / language servers) and keyword search (ripgrep). Agents combine semantic search, exact search and file reads; incremental re-indexing on file change; indexes can live locally for privacy.
• Agent mode: plan, edit via diffs (not full-file rewrites), run tests and linters in a sandbox, read failures, iterate within a step budget, show a reviewable diff, require approval for commands that touch the network or delete files.
• Quality and evaluation: offline: pass@k on benchmark tasks and repo-level tasks with tests (SWE-bench-style), regression suites; online: acceptance rate, accepted-characters retained after edit, time saved, latency percentiles, and rate of reverted changes.
• Security and compliance: secret scanning before sending context, exclude files (.env, keys), no training on customer code by default, license-aware filtering of verbatim public code, audit logs; protect against prompt injection in repo files and tool outputs.