CHAPTER 21
Design a Web Crawler
Polite, scalable, duplicate-free crawling of the web
Source: PDF p. 59
Core PDF content
| Functional requirements | Non-functional requirements and scale |
|---|---|
| - Start from seed URLs, discover and download pages, extract links, repeat - Store content for indexing / ML - Re-crawl pages by freshness need | - Politeness: obey robots.txt, limit rate per host - Robust: survive spider traps, malformed HTML, slow servers - Scalable: 1 B pages/month, distributed and fault tolerant - Efficient: avoid re-downloading duplicates |
- Support extra content types later (images, PDFs)
| Quantity | Estimate | Design implication |
|---|---|---|
| Fetch rate | 1 B / (30 x 86,400) = ~400 pages/s (peak 2x) | Hundreds of fetch workers with async IO |
| Storage | 1 B x ~100 KB (compressed HTML) = ~100 TB/month | Object store / HDFS, columnar metadata |
| Bandwidth | 400 x 100 KB = 40 MB/s = ~320 Mbps | Modest; DNS and politeness are the real limits |
| URL-seen set | ~10 B URLs x 8 B hash = ~80 GB | Distributed Bloom filter / hash set |
Deep dives p. 59
- URL frontier: (a) priority front queues rank URLs (by PageRank/site importance, change rate, depth); (b) politeness back queues, one per host, ensure at most one in-flight request per host and a minimum delay (Crawl-delay, adaptive to server latency). A mapping table host -> queue -> worker keeps one worker per host. Persist the frontier on disk so a restart does not lose state.
- Distributing the crawl: partition hosts across crawler nodes by hash(host) (consistent hashing) so politeness state is local; nodes forward discovered URLs belonging to other partitions.
- DNS: resolution is a bottleneck; run a local caching resolver, resolve asynchronously and batch.
- Duplicate detection: URL level (normalise: lowercase host, remove fragments and tracking params, sort query parameters, resolve relative links; then Bloom filter + persistent set); content level (checksum for exact copies, SimHash / MinHash for near-duplicates, mirrors and boilerplate).
- Spider traps: infinite calendars, session IDs in URLs, generated paths. Limit URL length and depth, cap pages per domain, detect repeating path patterns, and blacklist offenders.
- Freshness: estimate each page's change rate (history of ETag/Last-Modified/content hash) and schedule re-crawls proportionally, using HEAD or conditional GETs to save bandwidth.
Review questions for this chapter have not been written and reviewed yet. The chapter text above is the full source content.
Try it
References
System Design & AI System Design: The Complete Guide — 2026 Edition, p. 59; figures 42.
Tip: use ← and → keys to move between chapters.