CHAPTER 4

Scalability and Load Balancing

Spreading work across machines and keeping the front door healthy.

Source: Chapter 4 PDF, pp. 1–8

Learning objective

Understand how load balancers distribute traffic, choose servers, avoid unhealthy instances, and scale application capacity as demand changes.

Core PDF content

PDF page 11. Why do systems need to scale? p. 1

Imagine an online store during a flash sale. If all requests go to one application server, the server may run out of CPU or memory, queues may grow, and users may see slow pages or errors. The solution is often to spread work across multiple servers - but only after identifying the actual bottleneck.

Figure 1: One overloaded server can turn higher demand into slow responses and failures. — Figure 1. One overloaded server can turn higher demand into slow responses and failures. · PDF p. 1

2. What does a load balancer do? p. 1

A load balancer distributes incoming traffic across a pool of servers. It gives clients a stable entry point and can avoid sending new requests to servers that fail health checks.

Figure 2: Multiple application servers share traffic. They may still depend on a common database. — Figure 2. Multiple application servers share traffic. They may still depend on a common database. · PDF p. 1

PDF page 23. What happens when a server fails? p. 2

Load balancers can actively check server health - for example, by requesting a /health endpoint every few seconds. If a server repeatedly fails its checks, the balancer can remove it from the rotation. The exact thresholds depend on the service and its failure behavior.

Figure 3: New requests go to healthy servers; the unhealthy server is temporarily removed. — Figure 3. New requests go to healthy servers; the unhealthy server is temporarily removed. · PDF p. 2

A healthy load-balancing tier p. 2

The load balancer itself must not become a single point of failure. Managed cloud load balancers are generally implemented as distributed services. Other deployments may use redundant appliances, virtual IP failover, or anycast, depending on the design.

Figure 4: Illustrative multi-zone layout; exact high-availability architecture varies by platform. — Figure 4. Illustrative multi-zone layout; exact high-availability architecture varies by platform. · PDF p. 2

A multi-zone application tier can keep serving when one instance or zone fails, provided capacity, data services and routing are also designed for that failure.

PDF page 34. Layer 4 vs. Layer 7 load balancing p. 3

A load balancer can make decisions using connection information or information inside an application request.

Figure 5: L4 routes based on connection details; L7 can inspect application-level details such as an HTTP path. — Figure 5. L4 routes based on connection details; L7 can inspect application-level details such as an HTTP path. · PDF p. 3

Question	Layer 4 (L4)	Layer 7 (L7)
What it sees	IP address, port and transport connection details.	HTTP path, host, headers and other supported request details.
Routing example	Route a connection to an appropriate server pool.	/products to product service; /payments to payment service.
Strength	Efficient connection-level routing; works for many TCP/UDP use cases.	Application-aware rules, TLS termination and content-based routing.
Trade-off	Does not make decisions from HTTP paths.	More inspection and features can add processing and configuration complexity.
Examples	AWS Network Load Balancer, IPVS, TCP-mode HAProxy.	AWS Application Load Balancer, NGINX, Envoy, HTTP-mode HAProxy.

TLS termination, compression, request rewrites, web application firewall (WAF) integration, canary splits and gRPC-aware routing are common L7 features, depending on the product and configuration.

PDF page 45. How does a load balancer choose a server? p. 4

The algorithm affects how evenly traffic spreads. The best choice depends on how different servers are and how long requests remain active.

Figure 6: Simple algorithms rotate, measure load or use a key to choose a destination. — Figure 6. Simple algorithms rotate, measure load or use a key to choose a destination. · PDF p. 4

Algorithm	How it works	Useful when / caution
Round robin	Cycles through servers in order.	Works well when servers are similar and request costs are similar.
Weighted round robin	Sends traffic in proportion to configured weights.	Useful for mixed server capacity or gradual rollouts.
Least connections	Prefers the server with fewer active connections.	Useful for long-lived or uneven connections; connection count is not always a perfect load measure.
Least response time / EWMA	Prefers destinations with better recent response-time estimates.	Can adapt to slow nodes; measurements can be noisy or biased by traffic.
IP / header hash	Hashes a client IP or key to select a destination.	Can create sticky behavior, but changing pool size may remap many clients.
Consistent hashing	Uses a hash ring to limit remapping when the pool changes.	Useful for caches and partitioned state; balancing can still need tuning.
Power of two choices	Picks two candidates and selects the less loaded.	A simple way to achieve good load distribution with little coordination.

Operational details that matter p. 4

Connection draining: Stop sending new work to a server being removed or upgraded, while allowing in-flight requests to finish where possible. Slow start: Gradually increase traffic to a new instance so cold caches or JIT warm-up do not overwhelm it.

Figure 7: A controlled replacement protects ongoing requests and warms up new capacity. — Figure 7. A controlled replacement protects ongoing requests and warms up new capacity. · PDF p. 4

PDF page 56. Sticky sessions: convenience with a cost p. 5

Sticky sessions attempt to route the same user to the same server, often using a cookie or a hashing rule. This may help legacy applications that keep session data locally, but it can lead to uneven load and lost session state when a server fails.

7. Consistent hashing: keeping keys in place p. 5

Imagine a distributed cache storing millions of product records on several servers. A simple rule such as hash(key) mod N can send many keys to different servers when N changes. That can create a surge of cache misses and extra load on the database.

Figure 8: Conceptual hash ring. Each key maps to a server based on its position and the next server clockwise. — Figure 8. Conceptual hash ring. Each key maps to a server based on its position and the next server clockwise. · PDF p. 5

With consistent hashing, servers and keys are mapped onto a circular hash space. A key belongs to the next server clockwise. Adding or removing a server typically moves only a portion of the keys, rather than remapping almost all keys.

Technique	What changes when a server is added?
hash(key) mod N	Many keys may be reassigned because the modulus changes.
Consistent hashing	Only keys in the affected part of the ring normally move.

Used in some distributed caches, Dynamo-style data stores and sharded systems. Rendezvous (highest-random-weight) hashing and jump consistent hash are alternative techniques for related mapping problems.

PDF page 68. Autoscaling: changing capacity as demand changes p. 6

A load balancer distributes traffic across the instances that exist. Autoscaling changes how many instances are running. It can add capacity during a spike and remove capacity when demand falls.

Figure 9: A typical scaling loop measures demand, adds instances when needed, checks health and then routes traffic. — Figure 9. A typical scaling loop measures demand, adds instances when needed, checks health and then routes traffic. · PDF p. 6

Approach	How it works	Example
Reactive scaling	Adds or removes instances based on live metrics.	Scale out when request count per instance or queue depth exceeds a target.
Scheduled / predictive scaling	Adds capacity before a known demand increase.	Warm up before a flash sale or ticket release.

Metrics that may drive scaling p. 6

CPU or memory utilization
Requests per instance or observed latency
Queue depth / age of the oldest message
Application-specific metrics and expected event traffic

Capacity planning needs headroom p. 6

For latency-sensitive systems, a target such as 50-70% utilization can be a starting assumption, not a universal rule. Plan for traffic variability and the loss of an instance or zone. N+1 capacity means the remaining system can handle the workload after one capacity unit is lost; N+2 plans for two. Scale out quickly enough for rising demand and scale in more cautiously. Cooldowns and health checks help avoid repeated add/remove cycles (often called scaling flapping).

PDF page 79. Service discovery: how components find each other p. 7

In a dynamic system, instances can start, stop or change addresses. Calling services need a reliable way to find healthy destinations without keeping a stale list of machine addresses.

Figure 10: A discovery mechanism or stable endpoint helps a caller locate currently available instances. — Figure 10. A discovery mechanism or stable endpoint helps a caller locate currently available instances. · PDF p. 7

Approach	How it works	Trade-off
Server-side discovery	The caller uses a stable load balancer or virtual IP; the routing layer selects an instance.	Centralizes routing but adds a layer that must be operated reliably.
Client-side discovery	The client gets instance information from a registry and chooses a destination.	Requires discovery logic, refresh and load-balancing behavior in client libraries.

Registries and platform tools p. 7

Consul, etcd, Eureka and ZooKeeper are examples of technologies used for service discovery or coordination. In Kubernetes, Service objects and cluster DNS cover many common needs. A service mesh such as Istio or Linkerd can add traffic management and service-to-service security, but also introduces operational complexity.

10. Global load balancing p. 7

When users are spread across countries, global routing can direct them toward a suitable healthy region. GeoDNS uses DNS answers that may depend on location or policy; anycast advertises the same IP from multiple locations and lets network routing choose a path.

Figure 11: Global routing may steer users to nearby healthy regions and send traffic elsewhere during an outage. — Figure 11. Global routing may steer users to nearby healthy regions and send traffic elsewhere during an outage. · PDF p. 7

DNS caching, data replication and regional dependencies influence how quickly failover works. Routing to another region is only useful if that region can safely serve the required data and workload.

PDF page 8Chapter recap p. 8

A load balancer spreads requests and can stop routing new traffic to unhealthy instances.
L4 routes using connection details; L7 can route using HTTP paths, hosts and headers.
The balancing algorithm should fit the traffic, request duration and server capacity.
Shared session state or suitable tokens help keep application servers stateless.
Consistent hashing reduces how many keys move when a server pool changes.
Autoscaling changes capacity, but only helps when the tier being scaled is the bottleneck.
Service discovery and global routing help callers find instances and healthy regions.

11. Check your understanding p. 8

Choose the best answer for each question, then review the answer key.

1. Why use a load balancer?

A. To store all application data
B. To distribute traffic across servers
C. To eliminate every possible failure

2. /payments must go to the payment service and /products to the product service. Which fits?

A. Layer 4 routing only
B. Layer 7 routing
C. DNS TTL alone

3. Why can locally stored sessions be a problem?

A. Users always generate too many requests
B. Server failure may lose session state and sticky routing can unbalance load
C. Health checks become unnecessary

4. What is the main benefit of consistent hashing?

A. It eliminates database failures
B. It reduces key remapping when the pool changes
C. It guarantees equal load for every key

5. App CPU is low, but the database is saturated. What should you do?

A. Add app servers immediately
B. Investigate the database bottleneck
C. Increase the number of load balancers only

Answer key and explanations

B. A load balancer distributes traffic and can route around unhealthy instances.
B. Layer 7 can inspect HTTP paths and route requests to different destinations.
B. Shared session storage or suitable tokens let requests move between healthy instances.
B. Consistent hashing reduces remapping, although extra techniques may be needed to balance load.
B. Scaling a tier that is not the bottleneck may not improve overall capacity.

Review questions for this chapter have not been written and reviewed yet. The chapter text above is the full source content.

Try it

Open “Three-tier web app” in Labs Load balancer Reverse proxy Application / API service Elastic Load Balancing (ALB/NLB)EC2 (Auto Scaling)Cloud Load Balancing Compute Engine (MIG)Application Gateway / Load Balancer Virtual Machine Scale Sets App Service

References

Chapter 4 PDF, pp. 1–8; figures 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11.

Ch. 3 Ch. 5: Caching and Content Delivery

Tip: use ← and → keys to move between chapters.