Observability and Operations

How to understand system behavior, detect problems before users report them, and recover safely when things go wrong.

1. What is observability? p. 1

Imagine your shopping application suddenly becomes slow. Customers complain that orders take too long, but the application is still running. Is the database slow? Are requests waiting in a queue? Is a payment provider failing? Or did the latest deployment introduce a bug? Observability is the ability to understand a system's internal behavior by examining the information it produces.

Operations teams use this information to investigate issues, assess releases, plan capacity and improve reliability.

What you will learn p. 1

PDF page 22. The three pillars (and a fourth signal) p. 2

service=order-service
level=ERROR
request_id=req-1042
message=payment timeout

3. Connect logs, metrics and traces p. 2

Propagate trace context across services, commonly using the W3C traceparent header. Include trace and request IDs in logs so an operator can move between a trace and its related records.

Why it matters: Correlation turns a search across many unrelated records into an investigation of one request.

PDF page 34. The observability pipeline p. 3

Instrumentation creates telemetry, collection transports it, and observability tools make it searchable. OpenTelemetry is a commonly used instrumentation and telemetry framework.

5. What should you measure? p. 3

Start with measurements that show whether users are affected or the system is approaching a limit.

PDF page 46. RED, USE and business metrics p. 4

RED helps explain request behavior. USE helps find resource problems in CPUs, disks, network interfaces and connection pools.

Measure user and business outcomes too p. 4

A service may return HTTP 200 while customers still cannot complete checkout. Technical metrics alone may miss this.

• Orders completed per minute.
• Sign-ups completed.
• Payments successfully processed.
• Video playback started successfully.

Use synthetic probes to test controlled user journeys, and real-user monitoring to observe actual client experiences.

Goal: Know whether the system is technically healthy and whether users can accomplish what they came to do.

7. How should alerts work? p. 4

High CPU utilization alone does not necessarily mean an incident. Alert when user-impacting symptoms or reliability targets are at risk.

Every page should be urgent, actionable and supported by a runbook. Noisy alerts lead to alert fatigue.

PDF page 58. Error-budget burn rate p. 5

An SLO defines a reliability target, and the error budget is its allowed unreliability. Burn rate measures how quickly that budget is being consumed compared with the sustainable rate.

Illustrative alert thresholds from this chapter:

• Fast burn: Page if the 1-hour burn rate exceeds 14× and a 5-minute window confirms the problem.
• Slow burn: Create a ticket for a burn rate around 6× over 6 hours.

These are examples, not universal defaults. Choose thresholds based on the SLO, budget and operational needs.

9. Incident response: when an alert fires p. 5

The goal is to reduce customer impact quickly, restore service and learn what happened.

Mitigate before investigating every detail. If a recent release caused an outage, rolling it back may restore service faster than debugging every code path while users remain affected.

Common mitigation options p. 5

• Roll back a deployment.
• Fail over to a healthy region.
• Disable a feature using a feature flag.
• Reduce incoming work or disable nonessential functionality.

An incident commander coordinates the response, assigns responsibilities and keeps communication clear. After service is stable, investigate root causes and contributing factors.

PDF page 610. Blameless postmortems p. 6

A postmortem records what happened, why the system allowed it, how the team responded and what should change. It focuses on improving systems and processes, not finding someone to blame.

Action items should have owners and clear completion criteria. A postmortem without follow-through provides limited value.

11. Apply the concepts p. 6

Chapter recap p. 6

• Metrics show trends, logs record events, and traces follow requests across services.
• Shared request and trace IDs make investigation easier.
• Latency, traffic, errors and saturation are the four golden signals.
• RED measures request-driven services; USE helps assess resource health.
• Business metrics and user-side monitoring catch problems infrastructure metrics may miss.
• Alerts should focus on user impact and SLO risk; incidents need clear ownership and rapid mitigation.
• Blameless postmortems turn incidents into measurable improvements.

PDF page 7Check your understanding p. 7

1. Which signal is most useful for following one request across several services?

1. Metric

2. Trace

3. CPU gauge

2. Which four measurements make up the golden signals?

1. CPU, memory, disk, network

2. Latency, traffic, errors, saturation

3. Deployments, users, revenue, logs

3. Why should user_id generally not be used as a metric label for every request?

1. It makes metrics impossible to calculate

2. It can create excessive high-cardinality time series and increase cost

3. It prevents the application from logging errors

4. What should an on-call team usually prioritize during an active outage?

1. Restore service and mitigate customer impact

2. Complete the postmortem before acting

3. Collect every possible log before making any change

5. What is the purpose of a blameless postmortem?

1. Identify one person responsible

2. Document contributing factors and track improvements

3. Avoid recording incidents

Answers and explanations p. 7