Design Real-Time Analytics (Ad Click Aggregation)

Deep dives p. 63

• Event time and late data: window by when the click happened, not when it arrived. A watermark trails the max event time by a lateness bound (say 5 minutes); windows close once the watermark passes them, and still-later events are routed to a side output or corrected by the batch job.
• Exactly-once effect: Flink checkpoints operator state and Kafka offsets together; sinks are either transactional or idempotent upserts keyed by (ad_id, window_start), so replays after failure overwrite rather than double-count. Deduplicate producer retries with click_id.
• Hot keys / skew: a viral ad overloads one partition. Use salting (ad_id#0..7) for partial aggregation, then a second stage to combine the partials.
• Reconciliation (lambda-style safety net): raw events land in the lake; a daily Spark job recomputes totals from the source of truth and overwrites the OLAP rows, fixing late events, bugs and fraud filtering. This is essential for billing-grade numbers.
• Query layer: an OLAP column store gives fast time-range group-bys; precompute rollups (minute, hour, day) and choose the grain per query range; cache hot dashboards.
• Fraud and quality: filter bots with rules/ML (click velocity, IP reputation), flag but retain suspicious events for audit.
• Operations: monitor consumer lag, watermark delay, checkpoint duration, dedup state size, and compare streaming vs batch totals as a data-quality alarm.

TRADE-OFF Streaming vs batch accuracy: the stream gives results in seconds but depends on lateness bounds; the batch is authoritative but hours old. Show real-time numbers as "provisional" and finalise them after reconciliation.