CHAPTER 33

Fine-Tuning, Evaluation and Safety

Choosing how to adapt a model, proving it works, and keeping it safe

Source: PDF pp. 88–90

Core PDF content

PDF page 88Prompting, RAG or fine-tuning? p. 88

Combine freely (RAG + fine-tuned model is common). Whatever you choose, measure it with the same eval set before and after.

Method	What changes	Notes
Supervised fine-tuning (SFT)	All weights, trained on (prompt, ideal response) pairs	Most reliable way to teach format/style/behaviour. Quality beats quantity: 1k-50k curated examples often suffice
LoRA / QLoRA (PEFT)	Train small low-rank adapter matrices (W + BA, rank 8-64) on a frozen base; QLoRA also quantises the base to 4-bit	Fine-tune big models on one or few GPUs; adapters are MBs, swappable per tenant (multi-LoRA serving)
Preference optimisation	RLHF (reward model + PPO) or simpler DPO / IPO / KTO on (chosen, rejected) pairs	Shapes tone, helpfulness and refusals; DPO needs no separate reward model
RL with verifiable rewards	Reinforcement learning where correctness is checkable (math, code, tests)	Builds reasoning ability; needs reliable graders
Distillation	Train a small student on a large teacher's outputs	Cheaper, lower-latency model for a narrow task
Continued pre-training	Further next-token training on domain corpora	Heavy; only for deep domain shift (law, biomedical, a language)

Data pipeline: collect real prompts, write or generate ideal responses (strong-model synthetic data reviewed by humans), deduplicate, filter for quality and safety, apply the model's chat template, hold out validation and test sets that mirror production.
Risks: catastrophic forgetting (mix in general data, use low learning rates or LoRA), overfitting to quirks, baking in outdated knowledge (use RAG for facts, fine-tuning for behaviour), and leaking private data into weights.
Operations: track datasets and runs, evaluate against the base model and prompting baseline, version adapters, canary the new model, and keep a rollback.

Technique	Good for	Watch out
Deterministic checks	JSON validity, schema, regex, required citations, tool-call correctness, unit tests on generated code	Narrow, but cheap and exact: use wherever possible
Reference-based metrics (exact	Extraction, classification, short answers	Poor for open-ended text

match, F1, BLEU/ROUGE, embedding similarity)

LLM-as-judge	Rubric scoring (helpfulness, faithfulness, tone), pairwise comparison of outputs	Biases: position, verbosity, self-preference. Use clear rubrics, pairwise + swapped order, reasoning-before-score, and calibrate against human labels
Human evaluation	Ground truth for subjective quality, safety, edge cases	Expensive, slow: sample smartly, measure inter-rater agreement
Public benchmarks (MMLU, etc.)	Coarse model selection	Contaminated and not your task; build your own golden set (100-500 real cases)
Online metrics	Task completion, edits, retention, escalation rate, cost per task	Lagging and noisy; combine with offline suites

Threat (see OWASP Top 10 for LLM apps)	Mitigation
Prompt injection (direct, or indirect via documents, web pages, emails, tool results)	Separate instructions from data; treat external content as untrusted; privilege separation; least-privilege tools; human confirmation for sensitive actions; detect and quarantine suspicious content. No prompt alone is a sufficient defence
Jailbreaks and policy evasion	Layered input/output classifiers, system-prompt hardening, refusal training, red-team suites, monitoring and rapid patching
Insecure output handling	Never execute or render model output unsanitised: escape HTML (XSS), parameterise SQL, validate JSON against schema, sandbox code
Excessive agency	Minimal tool set and scopes, allow-lists, budgets, approval flows, kill switches
Sensitive-data disclosure	PII redaction, access control at retrieval, tenant isolation, output scanning for secrets, no secrets in prompts
Data and model poisoning, supply chain	Vet datasets, models and MCP servers; signed artifacts; monitor for anomalies
Denial of wallet / model DoS	Per-user token budgets, max input/output sizes, rate limits, anomaly alerts, caching
Hallucination	Ground with RAG and citations, constrain outputs, let the model abstain, verify (retrieval check, tool call, second pass), calibrate UI to show uncertainty, human review in high stakes
Bias, fairness, harmful content	Slice-based evals, moderation models, policy-aligned refusals, diverse red-teaming, transparency and appeal paths

Defence in depth: assume any single layer fails. Combine input filters, hardened prompts, constrained tools, output validation, monitoring and human oversight.
Governance: model/data cards, risk assessment per use case, logging for audit, incident response, and compliance with regulations (for example the EU AI Act classifies systems by risk level and imposes obligations accordingly) and privacy laws (GDPR, CCPA).

Review questions for this chapter have not been written and reviewed yet. The chapter text above is the full source content.

System Design & AI System Design: The Complete Guide — 2026 Edition, pp. 88–90; figures 60, 61.

Tip: use ← and → keys to move between chapters.