AI Case Studies II: Support Agent, Fraud, Voice, Image Generation

PDF page 94Design 4: an AI customer-support agent p. 94

• Triage first: a small, fast model classifies intent, language, sentiment and risk, decides the required authentication level, and routes. Cheap and keeps the expensive agent for the cases that need it.
• Grounding: answers come only from retrieved help-center and policy content with citations; unknown means escalate, never improvise policy.
• Safe actions: tools are narrow (issue_refund(order_id, amount)), run with scoped credentials, require step-up authentication for account changes, and sit behind a policy engine the model cannot override (auto-refund below a limit; above it, request human approval). All calls are idempotent and logged.
• Escalation design: trigger on low confidence, repeated failure, sentiment, high-value or regulated cases, or explicit request. Hand over a structured summary, the conversation, retrieved evidence and a suggested reply so the customer never repeats themselves.
• Memory: customer profile and past tickets retrieved per conversation (with privacy controls).
• Evaluation: replay historical tickets and run simulated customers (LLM-driven personas) against the agent; metrics: resolution/containment rate, policy-violation rate (must be ~0), CSAT, escalation precision, handle time, cost per resolution; human QA on samples; learn from agent edits to drafts.

PDF page 95Design 5: real-time payment fraud detection p. 95

5 decision: approve / step-up / review / decline (< 100 ms) async: event to Kafka; Flink updates velocity counters; labels arrive later (chargebacks, analyst outcomes)

• Features (the main source of lift): velocity (transactions per card/device/IP in 1 min / 1 h / 24 h), behavioural (amount vs history, new merchant/country), device and network (fingerprint, proxy/VPN, emulator), geo (impossible travel), entity risk (merchant, BIN), and graph features (shared devices/addresses across accounts, fraud rings). Real-time counters come from a streaming job into an online store; slower aggregates from batch.
• Model: gradient-boosted trees (fast, strong on tabular) plus rules for known patterns and hard limits; graph embeddings or a GNN as an added signal. Choose thresholds by expected cost (cost of fraud vs cost of false declines and friction), possibly per segment.
• Labels are delayed, biased and scarce: chargebacks arrive 30-90 days later; declined transactions never reveal whether they were fraud (selection bias). Keep a small randomised holdout (approve some borderline cases) to measure true fraud and train unbiased models; use analyst decisions as faster, noisier labels.
• Training and release: time-based splits, PR-AUC and recall at fixed false-positive rate, backtest against past attack waves, shadow mode before enforcement, canary by merchant segment.
• Adversaries adapt: monitor drift and block-rate by segment, retrain frequently, keep fast-path rules for emerging attacks, and add explainability (top contributing features as reason codes) for analysts and regulators.
• Resilience: the service is in the payment path. Define behaviour when it is down (typically fail-open with conservative limits for low-risk traffic, fail-closed for high-risk), use timeouts, replicate across regions, and degrade to rules-only if the model or feature store is slow.

Design 6: a real-time voice assistant p. 95

- Works in apps (WebRTC) and over telephone (SIP, 8 kHz audio)

Stream every stage so latencies overlap, not add                                  "let me check that..." filler

Budget: speech end to first audio ~ 0.6-1.2 s. Each stage must emit partial output as early as possible.

• Turn-taking is the hard UX problem: voice activity detection (VAD) plus an endpointing model decide when the user has finished; too eager cuts people off, too slow feels laggy. Support barge-in: when the user speaks over the assistant, stop TTS immediately, cancel LLM generation and discard buffered audio.
• Streaming everything: ASR emits partial transcripts; the LLM starts as soon as the utterance is final (or speculatively on partials); TTS speaks the first sentence or clause while the rest is still generating.
• Transport: WebRTC (UDP, jitter buffer, echo cancellation, NAT traversal) for apps; telephony via SIP trunks with narrow-band audio (use telephony-tuned ASR). Keep media servers close to users and models.
• Cascaded vs end-to-end: cascaded ASR+LLM+TTS is controllable, observable (you see the text), and tool-friendly; speech-to-speech models cut latency and preserve prosody/emotion but are harder to ground, audit and constrain. Many products use cascaded with optional native-audio models.
• Quality levers: domain vocabulary and biasing for ASR (names, SKUs), short spoken-style answers (no markdown or long lists), natural filler while tools run, confirmation of risky actions by voice, and fallback to text/links for complex information.
• Capacity and cost: each concurrent call holds ASR + TTS streams (GPU/CPU) and LLM decode; autoscale on concurrent sessions, batch across calls, and keep warm pools. Measure cost per minute.
• Evaluation: word error rate, endpointing accuracy, end-of-speech to first-audio latency percentiles, interruption-handling success, task completion, MOS-style voice quality ratings. Safety: consent and disclosure, voice-cloning abuse prevention, call recording policy.

Design 7: a text-to-image generation service p. 96

- Content safety on prompts and outputs; provenance marking

202 + job_id

                                    API gateway                         Prompt safety                                                                  returned at once
Client                               auth, credits,                      classifier, policy,                    Priority job queue
                                      rate limits                        prompt rewrite

Async by design: generation takes seconds, so the API returns a job ID and the client is notified when the image is ready.

• Job model: POST /images returns 202 and a job ID immediately; clients poll, subscribe (SSE/WebSocket) or receive a webhook. Idempotent job IDs make retries safe; store job state (queued, running, done, failed, moderated).
• Scheduling: priority classes (paid, free, batch), per-user concurrency caps, and fair queuing; autoscale GPUs on queue age; spot capacity for batch work. Preemption-safe: a job restarts on another worker if a node dies.
• GPU efficiency: keep models resident (loading multi-GB weights is slow), route jobs to workers that already hold the requested checkpoint or LoRA (model affinity), batch same-size requests, use fewer-step distilled/consistency models, quantisation and compiled kernels, and cache identical (prompt, seed, params) results.
• Pipelines as DAGs: text-to-image, upscaling, inpainting, ControlNet conditioning and face restoration are chained stages; orchestrate them with a workflow engine and reuse intermediate artifacts.
• Safety and provenance: classify prompts before spending GPU time, classify outputs before storing or showing them, block personal-likeness and copyrighted-character abuse per policy, embed invisible watermarks and C2PA content credentials, keep audit logs, and offer reporting flows.
• Delivery: write to object storage, serve through a CDN with signed expiring URLs, generate thumbnails, apply retention and deletion policies.
• Metrics: queue wait and generation latency percentiles, GPU utilisation, cost per image, moderation block rate, user re-generation rate (a proxy for dissatisfaction), prompt-to-accepted-image rate.