Observability at Massive Scale

There is a particular kind of silence that every on-call engineer dreads. Not the silence of a quiet night - the silence of a system that has stopped talking. No logs. No traces. No metrics. Just a wall of nothing, and somewhere out there, users staring at a spinner that will never resolve.

This guide exists because of that silence. It is a complete walkthrough of observability for production distributed systems, starting from absolute first principles and building up to the kind of multi-layered, burn-rate-driven alerting strategy that catches incidents before they become catastrophes. By the end, you will have a mental model for why every piece of observability infrastructure exists, and how to build it properly the first time.

Part I: What Observability Actually Is

Observability is not dashboards. Observability is not alerts. Observability is the property of a system that lets you understand its internal state from its external outputs. Dashboards and alerts are tools; observability is the capability they give you.

More precisely: observability can be framed as the ability to understand and explain any state your system can get into, no matter how strange or novel. A system is observable not because it has a Grafana dashboard, but because an engineer can look at its outputs and reason backward to its internal state.

A good telemetry setup is always focused on:

• User-Centric SLOs and Error Budgets - measuring what customers actually experience, not what servers feel
• Multiple Sources of Verifiability - no single telemetry type is authoritative; they corroborate each other
• High Signal-to-Noise Ratio - engineers and SREs should not drown in false positives
• Ease of Navigation - overly complex graphs and convoluted error messages do not help during an incident call
• Comprehensiveness - covers all important areas and pain points of the system
• Extensibility - requirements change, features get added; the design must grow with them
• Determinism - observability instrumentation should not show deviating outcomes from a simple dependency bump

The Three Pillars

Every observability system rests on three pillars, each answering a different class of question.

Metrics - "What is happening right now, quantitatively?"

A metric is a number over time. It answers questions like: What is our error rate right now? How many requests did we serve in the last minute? How full is our thread pool? Metrics are cheap to store, cheap to query, and cheap to alert on. They give you the 30,000-foot view.

The key metric primitives are:

At massive scale, metrics must be aggregated and labeled carefully to avoid exploding cardinality. Labels like tenant_id, shard_id, model_id, region, tenant_tier, and model_version are all legitimate - but combining too many unbounded labels on a single metric will overwhelm your metrics backend and explode costs.

Logs - "What happened in this specific request?"

A log is a record of a single event. It answers questions like: Why did request abc-123 fail? What was the system state when the error occurred? Which user triggered this code path?

Logs are rich but voluminous with a lot of noise. The observability strategy should be to drive alerts from metrics, and use logs as supporting detail - not the other way around. Aggregating logs for alerting is unreliable because stack traces are often unique and do not aggregate cleanly.

Traces - "How did a request travel through the system?"

A trace follows a single request across every service it touched. Where metrics tell you the aggregate picture and logs tell you about individual events, traces give you the causal chain. They answer: We know request abc-123 was slow. Was it slow because of the database query, the Redis write, the downstream API, or the network hop between services?

OpenTelemetry has become the standard for distributed tracing. A trace is built from spans - each span represents one unit of work in one service, with a start time, duration, and parent span ID. Together, they form a tree: Request → Controller → DB Query (2.1s) → Cache Write (0.05s) → Response.

Beyond these three, there are two more signals that deserve explicit mention:

• Events - deployments, feature flag changes, schema migrations, pipeline runs. They provide the temporal context that makes anomalies interpretable.
• Profiles & Config Data - CPU and memory profiles identify thread deadlock and hot-path issues. Configuration state matters because a secure shard may behave differently than a standard one, with different failure modes.

The goal for multi-modality is to make it trivial for an engineer to pivot from an SLO breach → trace → logs → recent deployment and find the fault.

The Monitoring Pyramid

These pillars support a four-level monitoring pyramid:

Level 1: Business KPIs & SLO Burn Rate    ← Are users succeeding?
Level 2: Application Metrics & Errors     ← Is the app healthy?
Level 3: Infrastructure Health Metrics    ← Are the resources OK?
Level 4: Synthetic Probes                 ← Can users reach us at all?

This pyramid has a crucial implication for alerting strategy: alerts should fire from top to bottom. An SLO burn rate alert means users are already impacted. A thread exhaustion alert means users will be impacted soon. The pyramid tells you the severity of every signal before you even look at the data.

Part II: The Four Golden Signals

Google's Site Reliability Engineering team distilled years of production experience into four metrics that, together, describe the health of almost any service. Every other metric is a derivative or a diagnostic tool. These four are universal.

Reading the Shape of Latency

Not all latency problems look alike. The shape of the degradation tells you the category of problem before you read a single log line.

Spike in p95/p99 only - external pressure

p50 stays flat while p95/p99 spike. Investigate: downstream rate limiting, circuit breaker opens, a dependency throwing exceptions.

Sudden jump across all percentiles - deployment regression

p50, p95, and p99 all step up at the same moment. Correlates with a release or config change. Also caused by upstream errors bleeding latency into your service.

Gradual creep over time - resource saturation

The system is slowly choking. Thread pool congestion as a slow downstream call holds threads, database connection pool exhaustion, consumer lag accumulating. This pattern is insidious because no single moment looks alarming - only the trend does.

Part III: SLOs, SLIs, and Error Budgets

Service Level Objectives (SLOs) are the glue between business expectations and telemetry. Two examples that illustrate the difference:

• "AI Issue Summarization returns a useful summary within 5 seconds for 99% of valid requests over 28 days" - a capability SLO, measuring what users care about
• "LLM Gateway 5xx errors must be under 0.1% of requests per day" - an operational metric, measuring infrastructure health

The SLO should always be capability-first. Operational metrics support SLOs; they do not replace them.

A Service Level Indicator (SLI) is the actual measurement that feeds an SLO - typically the ratio of good events to total events over a time window.

Error Budget is the decision currency. If your SLO is 99.5% reliability over 30 days, you have 0.5% to spend on failure - 216 minutes of allowed downtime per month. Every minute you are degraded beyond your SLO, you are spending from that budget. When the budget hits zero, you have broken your commitment.

Error budgets answer the organizational question: how much failure can we spend on velocity - deploys, experiments, risky changes - before we must bias towards stability? Teams with healthy error budgets can ship aggressively. Teams with depleted budgets must slow down and harden.

A mature observability system should have at least three SLOs monitoring different dimensions: Reliability, Latency, and Quality - with comprehensive documentation on what is being measured and how effectiveness is defined.

Part IV: The Burn Rate Model - The Heart of Modern Alerting

This is the most important section of this guide. If you only internalize one concept, make it this one.

The naive approach to error rate alerting says: "Alert me if error rate exceeds 1%." This sounds reasonable. It is not. There are three fundamental problems:

Noise at low volume. One error out of 50 requests is a 2% error rate. Should that page an on-call engineer? Almost certainly not. It might be a transient network blip, a single misbehaving client, a flaky integration test.

Deafness at high volume. 500 errors out of 100,000 requests is a 0.5% error rate. No page fires. But 500 real users just had their requests fail. That is not acceptable silence.

No budget awareness. The raw error rate tells you nothing about whether this rate is sustainable for your monthly reliability target.

Burn Rate

Burn rate measures how fast you are spending your error budget relative to the steady-state allowed rate:

Burn Rate = Observed Error Rate / Budget Error Rate

If your SLO is 99.5%, your budget error rate is 0.5%. If your observed error rate is 2.5%:

Burn Rate = 2.5% / 0.5% = 5×

You are burning your error budget five times faster than allowed. At this rate, you will exhaust 30 days of error budget in 6 days.

This is the insight that makes burn rate so powerful: it translates abstract error rates into concrete, time-bounded impact. A burn rate of 9× means you are burning through a full month of error budget in roughly 3.3 days.

The Multi-Window Dual Detector

Different failure modes burn budget at different speeds. A single detection window cannot catch all of them. The solution is a dual-window detector for each burn rate tier:

• Steep (9×): A severe incident. A bad deployment causing 18% errors. The short window fires in 4 minutes - fast enough to catch it before most users are impacted.
• Moderate (5×): Significant degradation. Intermittent failures causing 10% errors. Detected in about 20 minutes.
• Slow (1×): A slow, silent leak. A 2% error rate seeping for hours, slowly eroding your monthly budget without triggering any threshold-based alert.

The dual window requirement is the key insight: both the long and short windows must fire simultaneously. Long window alone delays alerting on sudden spikes. Short window alone generates noise on brief blips. Only when both agree is the signal trustworthy.

The Low-Volume Guard

A minimum failure threshold - typically 5 actual failures - gates the burn rate detector. Without it, 1 error out of 3 requests is a 33% error rate that fires the STEEP detector. The minimum threshold ensures the detector only fires when there is statistically meaningful evidence of user impact.

Thresholds by SLO Target

A 90% SLO has more raw budget to spend, so lower multipliers detect meaningful erosion.

Part V: The System Architecture and Observability Layers

At scale, the end-to-end path for a single user action typically traverses five layers:

Each layer has its own observability concerns, but traces and consistent IDs must run through all of them. Without end-to-end trace propagation, incidents in one layer generate symptoms in another, and root cause identification degrades into guesswork and log-tailing.

Service Mesh and OpenTelemetry as Cross-Cutting Fabric

Service meshes (Envoy, Istio) and OpenTelemetry are cross-cutting observability fabrics. The mesh can emit per-route RPS, latency, and errors without any application code changes, and enforces mTLS, retries, and rate limiting. OpenTelemetry provides a vendor-neutral way to emit traces, metrics, and logs from any language.

A mature architecture assumes:

• All HTTP/gRPC traffic between services flows through the service proxy, which exposes standard per-route metrics and adds standardized tracing headers (B3 or W3C)
• All services use OpenTelemetry SDKs to create spans for critical flows, attach semantic attributes (tenant, shard, feature flag, deployment ID), and export via OTLP to a shared collector

This combination allows the observability platform to provide end-to-end traces that cross infrastructure boundaries - mesh, services, queues, databases, third parties - and correlate them with metrics and logs.

Part VI: Layer-by-Layer Observability

Hardware and OS

Hardware and OS metrics are necessary but never sufficient. CPU, memory, disk, and network saturation can cause user-visible issues, but they are also frequently noisy.

Key metrics: CPU utilization, run queue length, context switches; memory usage, page faults, OOM kills; disk IOPS, latency, queue length; network bytes in/out, dropped packet rates; file descriptors, thread count.

In multi-tenant clusters, a single hot tenant or runaway job can consume shard resources. Per-pod and per-tenant metrics are necessary to attribute pain correctly rather than treating infrastructure as a black box.

Infrastructure and Platform: Kubernetes and Service Mesh

Infra metrics tell you whether the environment can run your app reliably:

• Pod and container health - restart count, crash loops, OOM kills, readiness/liveness probe failures
• Scheduling and autoscaling - pending pods, failed scheduling reasons, HPA target vs actual, cluster autoscaler events
• Mesh metrics - per-route RPS, p95/p99 latency, 4xx/5xx rates, retry counts, circuit breaker opens

The nuance: distinguish infra-induced failures from app or dependency failures. If the mesh shows 5xx from a dependency but the app is healthy, it is a dependency or network issue. If pods crash-loop due to OOM but SLOs are green because autoscaling compensates, it is a capacity risk, not yet user pain.

Product, Service, and Feature Layer

This is the most important layer for SLOs - it measures what users actually experience. The RED model (popularized by teams like Atlassian) covers:

• Rate - RPS per endpoint, per capability, broken down by tenant tier and region
• Errors - non-2xx responses per endpoint, classified by cause (client error, app error, dependency error)
• Duration - p50/p90/p99 latency per endpoint and per capability

On top of RED, business and product metrics matter: feature adoption, user task completion rates, conversation success rates. These are the signals that connect infrastructure health to actual business outcomes.

Synthetic Monitoring deserves special mention here. For critical user journeys - login, search, AI-powered actions - synthetic checks (pollinators, synthetic E2E tests) should run continuously and feed into availability SLOs. They are the only detection layer that fires when real traffic disappears.

Shards, Regions, and Global Services

At massive scale, global behavior is rarely uniform. Hot shards, mega-tenants, or regional anomalies cause partial outages that look fine in aggregate.

Key metrics: per-shard request volume, latency, error rate, CPU/memory, cache hit rate; shard balance metrics (distribution of tenants, detection of hot shards); multi-region replication lag and failover status; global services (auth, billing, routing) with their own SLIs and SLOs.

A global SLO can look green while a specific shard or region is red. Per-shard SLOs are essential. For critical enterprise or compliance customers, per-tenant SLOs may be required to meet contractual SLAs.

Databases, Caches, and Storage

Data systems often become the long pole in latency and reliability.

Key metrics: DB QPS, p95/p99 query latency, slow query count, lock wait times, replication lag, connection pool utilization; cache hit rate, eviction rate, memory usage; object store GET/PUT latency, error rates, throttling events.

High latency in a database is more often due to poorly indexed queries or N+1 patterns than under-provisioning. Observability must include query-level attribution - best surfaced via DB-specific tooling correlated with service traces. Database alerts should usually be attached to existing capability SLO alerts as enrichment, not standalone pages.

Queues and Async Systems

Queues protect user flows from synchronous bottlenecks but can hide latent failures and amplify backlogs.

Key metrics: queue depth per queue; age of oldest message (detects stuck consumers even when depth is low); consumer throughput and lag in messages/sec; DLQ volume categorized by error type.

Alert when the backlog grows rapidly and the age of oldest message exceeds the threshold tied to the user-visible SLA. Alert when DLQ volume spikes, especially with novel error types. Fan-out workflows and end-to-end latency tracking require special attention - a user-enqueued job might wait in 3 queues before producing output.

Third-Party APIs

Third-party APIs - connectors, payment providers, identity providers, email services - are a common source of partial outages. They have their own SLAs and rate limits.

Key metrics: availability and p95/p99 latency per endpoint; rate limit and quota usage; error classification; circuit breaker state and retry behavior.

Observability should tell you not just that a third party is down, but whether fallbacks are working. Metrics for "served via degraded path" are important - SLO calculations should not silently exclude vendor downtime.

Part VII: Alert Design and Signal Quality

The biggest failure mode at scale is alert fatigue. Dozens of component alerts firing for every incident leads to normalization of deviance and muted pages. Alerts must be both a reliable indicator of user pain and actionable.

A mature alerting architecture uses two stages:

1. Symptom-level SLO burn alerts:

• Metric: capability SLI (good/bad events)
• Alert: multi-window burn rate (2h/6h) above thresholds corresponding to "budget exhausted in X hours"
• Route to: on-call engineer immediately

2. Cause-level alerts:

• Triggered when infra or dependency metrics cross thresholds AND coincide temporally with a symptom alert
• Examples: DB replication lag spike, provider 5xx spike, circuit breaker opens >1%
• Action: enrich the existing incident; route specialist teams as needed - not a separate page

Dashboard Design

Good dashboards tell a story. They should:

• Start from SLOs and golden signals, not from low-level metrics. The first panels answer "are users impacted?" and "which capability, region, or tenant tier?"
• Align to runbooks and ownership: overlay deployment events, config changes, feature flag flips as vertical markers on every time-series chart
• Be easy to navigate during an incident call - overly complex graphs help no one at 2am

Every time-series chart should show deployment events as vertical lines. The most common question when investigating an anomaly is "did this correlate with a deployment?" Without that context, every investigation starts with a manual cross-reference between the metrics timeline and the deployment history.

Part VIII: Deployment and Release Observability

CI/CD Pipeline Metrics

CI/CD is itself a critical capability. Slow or flaky pipelines reduce velocity and increase risk. Observability for CI/CD includes:

• Pipeline Health - build success rate, average and p95/p99 duration per step, test flakiness
• Resource Usage - agent CPU/memory, queue times, concurrency
• Change Correlation - mapping deploys and feature flag flips to production incidents

OpenTelemetry traces for pipelines (per run, per stage, per step) can be exported via webhooks and OTLP to existing observability backends.

Canary and Progressive Delivery

Safe rollout patterns - canaries, blue-green, shard-wise releases - all require per-cohort metrics. Compare canary vs. baseline on:

• Capability SLOs (success rate, p50/p90/p99 latency)
• Infrastructure resource metrics
• Cost and quality metrics

Progressive rollout steps (5% → 25% → 50% → 100%) should be gated on SLO deltas, with automated rollback when the delta exceeds a threshold.

Rollback Criteria

Rollback decisions should be data-driven and automated where possible:

• SLO burn rate above threshold shortly after a deploy
• Significant regression in cost or quality metrics for key cohorts
• Infrastructure instability (pod crash loops, error spikes) directly correlated with a release

Observability closes the loop: deploys emit events into the telemetry stream; SLO and metrics respond; CI/CD and incident management consume those signals to roll forward, roll back, or refine pipelines.

Part IX: Scaling Nuances

Tail Latency at Scale

Tail latency (p95/p99/p99.9) becomes more important than averages at scale. A small percentage of slow requests can hurt many users and trigger timeouts across services due to cascading effects.

• Coordination overhead - cross-region traffic, distributed locks - can dominate response times
• Background noise (minor error rates, transient spikes) increases simply due to volume; alert thresholds must calibrate to real user tolerance, not naive zero-error ideals

Hot Spots and Noisy Tenants

Per-tenant resource usage and performance visibility is critical for at least the top-N tenants or special cohorts:

• Per-tenant metrics and traces for resource attribution
• Per-tenant SLOs for critical clients (large enterprises, compliance-tier customers) to meet contractual SLAs
• Protection mechanisms: rate limiting, bulkheads, and load shedding, with metrics for "requests dropped or throttled per tenant"

A tenant hogging all resources on a multi-tenant shard is often not caught by aggregate SLOs until the shard's other tenants start complaining.

Cost Observability (FinOps)

At massive scale, cost is as critical as latency and reliability, especially for data platforms and LLM-backed features. Cost observability means:

• Per-feature and per-tenant dashboards covering compute, storage, data pipeline costs, and external API charges
• Cost SLOs and anomaly detection: a feature that is functionally healthy may be generating 10× more spend than expected
• Attribution pipelines that connect usage metrics to billing data

Part X: Instrumentation Strategy

Metric Naming and Cardinality

Consistent naming and constrained label sets are make-or-break at scale. Unbounded labels will overwhelm your metrics backend.

• Metric names should follow: <domain>.<service>.<metric_name>
• Labels should capture environment, region, tenant tier, shard, and operation type
• High-cardinality items like tenant_id, user_id, and trace_id belong in logs and traces - not in metrics labels

env=prod, region=us-east-1, shard=sh-1, tier=enterprise, endpoint=/api/v1/search

Structured Logging

Logs should be structured JSON with standard fields on every line:

Request Identity: requestId, traceId, spanId, transactionId

Context: partitionId, shardId, serviceId, userId, tenantId

Outcome: httpStatus, circuitBreakerStatus, featureFlagName, featureFlagStatus

Performance: latencyMs, operationCount

The principle: every meaningful dimension - identity, context, outcome, and performance - should be available for filtering in your log search tool. When you are investigating an incident at 2am, you want to type tenantId=acme AND httpStatus=500 AND circuitBreakerStatus=OPEN and get the exact log lines that explain the failure. Avoid logging PII or user-generated content unless explicitly required and scoped.

Consistent Tagging

Consistent tagging is essential for slice-and-dice explorability across all telemetry types:

• Environment - dev, stg, prod
• Region - us-east-1, eu-central-1
• Tenant - hashed ID for logs/traces, or complete resource identifier
• Shard - shard-id
• Feature flag / experiment - current flag state and variant

Part XI: Caveats and Failure Modes

Even well-designed observability systems have failure modes. Here are the ones that bite teams most frequently.

C1: Synthetic traffic in production metrics. Synthetic monitoring generates synthetic traffic. If it is not filtered, it inflates your request rate metrics and dilutes your error rate. Always tag synthetic requests at the source and filter them from SLO calculations.

C2: Long-running and async tasks. Batch jobs and async workflows have latency profiles 10–100× longer than synchronous requests. Applying your API's p95 latency alert to a batch job expected to run for 5 minutes floods you with false positives. Maintain separate SLOs with appropriate thresholds for different traffic classes.

C3: Missing metrics are not success signals. If your metrics pipeline breaks, error rates drop to zero and latency metrics flatline. This looks exactly like a healthy system. Zero traffic detection exists precisely because production engineers have been fooled by the silence of a broken observability pipeline into thinking a catastrophic outage was a quiet night.

C4: The right tags make metrics explorable. A counter that just counts errors is useful. A counter tagged with endpoint_path, http_method, http_status, tenant_id, shard_id, and feature_flag_variant is an investigation instrument. The difference between a 15-minute diagnosis and a 2-hour one is usually tag coverage.

C5: SLO breaches must always lead to Post Incident Reviews. Even if automatically mitigated. Systematic issues - bad thresholds, silent failure modes, natural growth getting rate-limited - only surface through the review process.

Part XII: Implementation Checklist

Building a production-grade observability system is a project, not an afternoon task. Here is the ordered implementation path:

1. Instrument your code - add counters and timers to every meaningful code path: HTTP endpoints, database queries, cache operations, queue handlers. This is the foundation everything else depends on.
2. Add tags to every metric - at minimum: endpoint path, HTTP method, HTTP status, tenant tier, service ID, region.
3. Build domain dashboards - one dashboard per concern: endpoint reliability, latency profiles, infrastructure health, queue depth, third-party status.
4. Add deployment markers to every dashboard - vertical lines for every deploy, feature flag flip, and config change.
5. Implement zero traffic detection - protects against complete failure modes that generate no other signals.
6. Configure burn rate SLOs - start with your most critical endpoints. Define the SLO target, configure the three burn rate tiers with dual-window detectors, set minimum failure thresholds.
7. Add infrastructure saturation alerts - thread pool utilization, connection pool saturation, pod crash loops. These are the early warning layer that fires before burn rate does.
8. Implement synthetic monitoring - scheduled E2E probes that validate the full user flow.
9. Set up error tracking - burn rate alerting tells you that errors are happening; error tracking tells you what they are, grouped and deduplicated so you see 3 error patterns instead of 10,000 individual stack traces.
10. Test your alerting - deliberately degrade your service in a non-production environment and verify the right alerts fire at the right severity. An alert that fires during an actual incident for the first time is an untested alert.

Closing: The Philosophy Behind the Practice

Observability is not a compliance requirement or a box-checking exercise. It is the operational expression of taking your users' experience seriously.

Every alert threshold represents a judgment call: at what point does this metric signal user impact? Every SLO target is a public commitment: we will be reliable at least this often. Every structured log field is preparation for a future investigation you cannot yet anticipate.

The burn rate model, the multi-window dual detector, the layered detection architecture - these are not academic constructs. They are the product of engineers getting paged at 2am, learning from what they missed, and encoding that knowledge into their systems.

Build observability as if you will be the one on call. Because you probably will be.

Key Takeaways:

1. Observability is a system property, not a tooling choice - dashboards and alerts are the tools, not the goal.
2. Drive alerts from capability SLOs, not from raw infrastructure thresholds.
3. Burn rate alerting catches every failure mode - steep sudden incidents and slow silent leaks - that threshold alerting misses.
4. Consistent tagging and metric naming discipline are make-or-break at scale; unbounded cardinality breaks your metrics backend.
5. Missing metrics look exactly like a healthy system - zero traffic detection is not optional.
6. The layered detection architecture exists because no single signal catches every failure mode; synthetic probes, infrastructure saturation, and SLO burn rate fire at different incident velocities.

Continue to Part 2 for observability specific to LLM-backed systems - token tracking, streaming health, agentic metrics, and multi-provider resilience.

Further Reading

The concepts in this post draw from a body of work developed primarily at Google, then extended by the broader SRE and observability community. These are the books worth reading in roughly this order:

Foundations

• Site Reliability Engineering - Beyer, Jones, Petoff, Murphy (Google / O'Reilly, 2016). Free online. The source of error budgets, the four golden signals, and the philosophical case for SLO-driven operations. Chapters 4, 6, and 13 are the ones most relevant to this post.

• The Site Reliability Workbook - Beyer et al. (O'Reilly, 2018). Free online. The practical companion to the SRE book. More concrete on SLO implementation, burn rate alerting, and on-call practices.

• Distributed Systems Observability - Cindy Sridharan (O'Reilly, 2018). Short (under 100 pages) but dense. The clearest treatment of the three pillars and why they are complementary rather than substitutes.

Going Deeper

• Observability Engineering - Charity Majors, Liz Fong-Jones, George Miranda (O'Reilly, 2022). Argues that structured, high-cardinality events are the foundation of observability - not metrics and logs. Pushes back on parts of the SRE book's model. Worth reading as a counterpoint, not just a supplement.

• Systems Performance: Enterprise and the Cloud - Brendan Gregg (Addison-Wesley, 2nd ed. 2020). The definitive book on performance analysis methodology. Introduces the USE Method (Utilization, Saturation, Errors) for resource analysis. Dense, long, and worth every page if you work at the infrastructure layer.

Adjacent but Relevant

• Database Reliability Engineering - Laine Campbell and Charity Majors (O'Reilly, 2017). Applies SRE principles specifically to data systems. Relevant to the databases and caches section of this post.

• Implementing Service Level Objectives - Alex Hidalgo (O'Reilly, 2020). The most thorough treatment of SLO design - how to pick SLIs, set targets, use error budgets as a policy instrument, and build organizational buy-in. If the SLO sections of this post resonated, this book is the natural next step.