CHAPTER 8

Messaging, Streaming and Async Processing

Queues, logs, Kafka, event-driven design, and batch vs. stream processing.

Source: Chapter 8 PDF, pp. 1–9

Learning objective

Choose a messaging model, understand how Kafka scales consumers, handle retries and duplicates safely, and distinguish batch from stream processing.

Core PDF content

PDF page 11. Why do we need asynchronous processing? p. 1

Imagine placing an online order. The system must save the order, send a confirmation email, notify the warehouse and update analytics. The customer usually should not need to wait for every background task to finish.

Figure 1: The order is saved and acknowledged while independent workers process follow-up tasks. — Figure 1. The order is saved and acknowledged while independent workers process follow-up tasks. · PDF p. 1

	Benefit	What it means	Example
	Decoupling	Services can scale and deploy independently.	Email worker can be changed without redeploying orders.
	Buffering	Queue absorbs a burst while workers catch up.	Flash-sale orders.
	Reliability	Failed work can be retried or moved aside.	Temporary email provider outage.
	Fan-out	One event can reach multiple consumers.	Analytics and warehouse updates.
Remember: An HTTP 202 Accepted response means work was accepted, not necessarily completed.

PDF page 22. Queue vs. pub/sub vs. distributed log p. 2

These systems all move information between components, but differ in who receives a message and how long it remains available.

Figure 2: Typical models: competing queue workers, independent subscribers, and retained logs read by consumer groups. — Figure 2. Typical models: competing queue workers, independent subscribers, and retained logs read by consumer groups. · PDF p. 2

Model	Consumption	Retention / replay	Examples
Message queue	Workers compete; a message is typically handled by one worker successfully.	Often acknowledged and removed, depending on broker.	SQS, RabbitMQ
Pub/sub	Each subscription receives the event independently.	Depends on broker and subscription type.	SNS, Google Pub/Sub
Distributed log	Each consumer group maintains its own position.	Retained by policy; replay is a core feature.	Kafka, Pulsar, Redpanda

A queue is a good fit for background jobs. Pub/sub fits event fan-out. A distributed log is useful when many independent consumers need to process and replay the same stream. Individual products may support more than one pattern.

3. How Kafka organizes messages p. 2

A Kafka topic is a named stream of events. Topics are split into partitions, which allow data to be distributed across brokers and processed in parallel. Each partition is an ordered log.

Figure 3: Partitions distribute storage and parallelism; each consumer group independently tracks progress. — Figure 3. Partitions distribute storage and parallelism; each consumer group independently tracks progress. · PDF p. 2

PDF page 33. How Kafka organizes messages (continued) p. 3

Choosing a message key If events for the same order must be processed in order, use order_id as a key so those events are routed to the same partition. Ordering is guaranteed within a partition, not across all partitions.

Figure 4: Events for order 101 share a partition; a different order may be assigned to another partition. — Figure 4. Events for order 101 share a partition; a different order may be assigned to another partition. · PDF p. 3

Consumer groups Within a group, partitions are assigned across consumers. Under normal group operation, a partition is assigned to at most one active consumer at a time. The partition count therefore limits parallel consumption for one group. Adding consumers beyond available partitions will not increase parallelism. Different consumer groups can independently read the same topic. Offsets and consumer lag An offset identifies a record position in a partition. Consumers track progress so they can resume after restarting. Committing after processing generally supports at-least-once behavior; committing before processing may support at-most-once behavior but can lose work if the consumer crashes.

Replication and acknowledgements

Figure 5: A partition leader serves normal writes while follower replicas copy its data. — Figure 5. A partition leader serves normal writes while follower replicas copy its data. · PDF p. 3

Producer setting	General behavior
acks=0	Does not wait for a broker acknowledgement.
acks=1	Leader acknowledges the write.
acks=all	Waits for required in-sync replica acknowledgements under the configured policy.

A suitable replication factor and min.insync.replicas policy can reduce the risk of losing acknowledged records during broker failure. Durability depends on configuration and which replicas remain available.

PDF page 44. Delivery guarantees and safe retries p. 4

A message may be processed successfully even if the acknowledgement is lost. The broker or producer may retry, so duplicate processing is a normal case to plan for.

Guarantee	Meaning	Trade-off
At-most-once	The operation is attempted no more than once.	Work may be lost.
At-least-once	Retries continue according to the delivery policy.	Duplicates can occur.
Exactly-once effect	Duplicate delivery does not duplicate the business effect within a defined scope.	Requires appropriate transactions or idempotency.

Kafka supports idempotent producers and transactions that can atomically combine Kafka records and consumed offsets within Kafka transactional scope. External database writes, emails and payments still need their own protection. Idempotent consumer pattern

Figure 6: Store the message ID and business result together where possible; repeated deliveries can return the stored outcome. — Figure 6. Store the message ID and business result together where possible; repeated deliveries can return the stored outcome. · PDF p. 4

For a payment or order creation, a unique idempotency key helps the service recognize retries and return the original result instead of performing the side effect twice. The business update and deduplication record should be committed atomically where possible.

5. Handling failures in messaging p. 4

Pattern	Purpose	Example
Retry with backoff	Retry temporary failures after a delay; add jitter to avoid synchronized retries.	Wait before retrying an overloaded dependency.
Dead-letter queue (DLQ)	Move repeatedly failing messages aside for investigation.	Malformed or poison message.
Competing consumers	Increase processing capacity with more workers.	Many independent email jobs.
Delay / priority queue	Schedule work for later or prioritize selected tasks.	Reminder notifications.
Claim check	Put a large payload in object storage and send its reference.	Large video or report.

PDF page 55. Backpressure: when producers are faster than consumers p. 5

Suppose a system receives 10,000 messages per second but workers can process only 6,000. If these rates continue, the backlog grows by about 4,000 messages per second.

Figure 7: Unbounded backlog increases lag and may exhaust storage; backpressure makes overload visible and actionable. — Figure 7. Unbounded backlog increases lag and may exhaust storage; backpressure makes overload visible and actionable. · PDF p. 5

Possible responses include limiting producer rates, increasing worker capacity, using bounded buffers, deferring non-urgent work or shedding nonessential load. The correct action depends on whether every event must be retained.

6. Event sourcing and CQRS p. 5

Most applications store current state. Event sourcing stores the sequence of events that produced it, such as OrderPlaced, PaymentCaptured and OrderShipped. The current state can be derived by replaying events, with snapshots used to speed recovery.

Figure 8: Commands change the system and append events; projectors build read-optimized models for queries. — Figure 8. Commands change the system and append events; projectors build read-optimized models for queries. · PDF p. 5

Term	Role
Command	Requests a change, such as placing an order.
Event	Records something that happened, such as OrderPlaced.
Projector	Processes events to build or update read models.
Read model	Stores data in a form suited to queries.

CQRS separates the write model from one or more read models. Event sourcing and CQRS can improve auditability and independent scaling, but introduce schema evolution, replay, operational complexity and possible delay between writes and read models.

PDF page 67. Batch vs. stream processing p. 6

Imagine calculating yesterday's sales from a completed file versus detecting suspicious transactions as they happen. These are different processing styles.

Figure 9: Batch processes bounded collections; stream processing works on continuously arriving events. — Figure 9. Batch processes bounded collections; stream processing works on continuously arriving events. · PDF p. 6

Feature	Batch	Stream
Input	A bounded dataset, such as yesterday's files.	A continuous flow of events.
Typical latency	Minutes to hours.	Milliseconds to seconds, depending on the design.
Tools	Spark, Hive, dbt, Airflow.	Flink, Kafka Streams, Spark Structured Streaming.
Use cases	Reports, historical analysis, backfills.	Fraud detection, alerts, real-time features.

Batch and streaming can coexist. Streaming supports quick updates; batch processing remains useful for historical analysis, recomputation and large backfills.

8. Time windows and late events p. 6

Streaming systems often aggregate events over windows, for example orders per five minutes.

Figure 10: Tumbling windows do not overlap; sliding windows overlap; session windows group activity separated by inactivity gaps. — Figure 10. Tumbling windows do not overlap; sliding windows overlap; session windows group activity separated by inactivity gaps. · PDF p. 6

Term	Meaning
Event time	When the event actually occurred.
Processing time	When the system processes the event.
Watermark	A signal of event-time progress used to decide when a window can be considered sufficiently complete.

Watermarks do not guarantee all late events have arrived. Systems need a policy for late data, such as updating previous results, allowing a grace period or dropping events beyond a threshold.

PDF page 79. Recovery, Lambda and Kappa architectures p. 7

Stream processors may maintain state, such as counters or joins. Checkpoints save enough state and progress to resume consistently after a failure. Correct end-to-end effects also depend on the output sink, transaction boundaries and idempotency.

Figure 11: A checkpoint captures a consistent recovery point; after failure, the processor restores state and resumes. — Figure 11. A checkpoint captures a consistent recovery point; after failure, the processor restores state and resumes. · PDF p. 7

Two common architecture patterns

Figure 12: Lambda combines batch and speed paths; Kappa uses a retained log and a streaming pipeline that can replay events. — Figure 12. Lambda combines batch and speed paths; Kappa uses a retained log and a streaming pipeline that can replay events. · PDF p. 7

Architecture	Idea	Trade-off
Lambda	Use batch processing for accurate recomputation plus a speed layer for fast results.	Two paths and results must be reconciled.
Kappa	Use one streaming pipeline and replay retained events to reprocess.	Replay depends on retention and pipeline capabilities; batch jobs can still be needed.

10. Which messaging technology should you choose? p. 7

Requirement	Possible starting point
Simple background job queue	Amazon SQS or RabbitMQ.
High-throughput event stream, replay and multiple consumers	Kafka, Pulsar or Redpanda.
Lightweight, low-latency messaging	NATS or Redis Streams.
Stateful stream processing	Flink, Kafka Streams or Spark Structured Streaming.

These are starting points, not universal rankings. Compare ordering, retention, delivery guarantees, throughput, operations and cost.

PDF page 8Chapter recap p. 8

Concept	Remember
Asynchronous processing	Separates immediate responses from background work.
Queue / pub-sub / log	Differ in consumer model, retention and replay.
Kafka partitions	Provide per-partition ordering and parallelism.
Idempotency	Makes retries safer when deliveries repeat.
Backpressure	Prevents overload from turning into unbounded queues and lag.
Event sourcing / CQRS	Store event history and separate write models from read models.
Batch / stream	Process bounded collections or continuous event flows.

Check your understanding p. 8

1. Why use a queue to send an email after an order?

A. To make the email service synchronous
B. To return a response faster and process the email independently
C. To guarantee email can never fail 2. What limits parallel consumption of a Kafka topic by one consumer group?
A. Number of partitions
B. Number of topics
C. Number of fields per message 3. Why should a message consumer be idempotent?
A. To prevent every network failure
B. To make retries safe when a message is delivered more than once
C. To eliminate message retention

PDF page 9Check your understanding (continued) p. 9

4. Which approach suits calculating yesterday's sales from a large historical dataset?

A. Batch processing
B. WebSockets
C. Short polling 5. What is the purpose of a dead-letter queue?
A. Store repeatedly failing messages for inspection
B. Permanently increase throughput
C. Guarantee exactly-once processing 6. What do watermarks help a stream processor decide?
A. Which database stores raw files
B. How far event-time processing can reasonably progress despite late events
C. Whether every event is guaranteed to have arrived

Answer key and explanations

Q	Answer	Explanation
1	B	The application can hand off work and let a worker process it in the background.
2	A	Partitions are assigned to active consumers within a group and limit parallel consumption.
3	B	Idempotency prevents duplicate delivery from creating duplicate business effects.
4	A	Batch processing works over a bounded collection of records.
5	A	A DLQ separates repeatedly failing messages from normal processing for diagnosis.
6	B	Watermarks track event-time progress; late-event policies are still needed.

Review questions for this chapter have not been written and reviewed yet. The chapter text above is the full source content.

References

Chapter 8 PDF, pp. 1–9; figures 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12.

Ch. 7 Ch. 9: APIs, Rate Limiting and Security

Tip: use ← and → keys to move between chapters.