Practical guide to production visibility: structured logs, distributed tracing, metrics, SLOs, and alerting that respects humans.
If I were instrumenting a production system today, what would I standardize before the first serious incident forced me to?
This playbook is about:
- structured logging that survives an incident;
- distributed tracing across services and queues;
- metrics that describe users, not just machines;
- SLOs and error budgets as working agreements;
- alerting that respects the humans on call;
- dashboards that answer questions instead of decorating walls;
- correlation from the edge to the last background job.
- Observability Playbook
These repositories form one playbook suite:
- AI-Assisted Engineering Playbook — agent workflows, guardrails, and quality control for AI-heavy teams
- API Design Playbook — versioning, pagination, idempotency, error contracts, and webhooks
- Auth & Identity Playbook — sessions, tokens, OAuth, and identity boundaries across the stack
- Backend Architecture Playbook — APIs, boundaries, OpenAPI, persistence, and errors
- Best of JavaScript — curated JS/TS tooling and stack defaults
- Caching Playbook — HTTP, CDN, and application caches; consistency and invalidation
- Code Review Playbook — PR quality, ownership, and review culture
- CTO Playbook — org design, hiring, technical strategy, budgets, and due diligence
- DevOps Delivery Playbook — CI/CD, environments, rollout safety, and observability
- Engineering Lead Playbook — standards, RFCs, and technical leadership habits
- Frontend Architecture Playbook — React structure, performance, and consuming API contracts
- Git Collaboration Playbook — branching, stacked PRs, merge queues, and CI collaboration at scale
- Marketing and SEO Playbook — growth, SEO, experimentation, and marketing surfaces
- Messaging & Async Playbook — queues, events, outbox, idempotent consumers, and retries
- Monorepo Architecture Playbook — workspaces, package boundaries, and shared code at scale
- Node.js Runtime & Performance Playbook — event loop, streams, memory, and production Node performance
- Observability Playbook — logs, traces, metrics, SLOs, and production visibility
- React Cross-Platform Playbook — shared React UI and logic across web and native with TypeScript
- Software Design Playbook — separation of concerns, composition, and module boundaries
- State Management Playbook — client state architecture, MobX, and choosing a state layer
- Styling Architecture Playbook — type-safe styling, design tokens, and theming at scale
- Sysadmin Operations Playbook — servers, backups, DNS, TLS, identity, and the self-hosted ops stack
- Testing Strategy Playbook — unit, integration, contract, E2E, and CI-friendly test layers
Observability is not a wall of dashboards. It is being able to ask new questions of a running system without shipping new code.
Most observability work is not about collecting more data. It is about:
- instrumenting for the incident you have not had yet;
- one consistent way to correlate a request across everything it touched;
- separating "a human should look at this" from background noise;
- measuring what users feel, not just what servers do;
- paying only for signals someone actually queries.
Dashboards answer yesterday's questions. Instrumentation lets you ask tomorrow's.
If I were standing up observability for a product team today, I would start with:
- Logs: structured JSON from day one, one shared schema across services
- Traces: OpenTelemetry SDK with W3C trace context propagation
- Correlation:
request_idand trace id on every log line and every error response — the same convention that already runs through the rest of this suite - Metrics: RED per service (rate, errors, duration), histograms for latency
- Alerts: symptom-based only, driven by SLO burn rather than static thresholds
- Dashboards: one per service, with a drill-down path to traces
None of these require exotic tooling. All of them require deciding before the incident, not during it.
Levels are a contract with the reader, not decoration:
error— a human should look at this, ideally soonwarn— degraded but coping; worth a trend line, not a pageinfo— state changes you would want when reconstructing an incidentdebug— development only; off in production
If error fires a hundred times an hour and nobody looks, the level is lying.
One wide, structured event per request beats twenty scattered lines.
Emit a single canonical log line when the request completes: route, status, duration, request_id, trace id, user tier, feature flags, downstream call counts. Everything you would want in one row when the pager goes off.
{"level":"info","msg":"request completed","request_id":"req_8fZk21","trace_id":"4bf92f35","route":"POST /orders","status":201,"duration_ms":142}Scattered incremental logging is how you end up grepping five lines to reconstruct one request.
- secrets, tokens, passwords — absolutely and without exception
- PII — allowlist fields at the logging edge; a denylist always misses one
- full request or response bodies "just in case"
The allowlist lives in one place, at the boundary where logs are emitted. Every service inventing its own redaction is how leaks happen.
High-volume paths do not need every success recorded:
- keep every error and every slow request;
- sample the healthy majority at a rate you can afford;
- make the sample rate a config value, not a code change.
OpenTelemetry SDK plus auto-instrumentation is the floor, not the finish line. Auto-instrumentation gets you HTTP, database, and client spans for free; manual spans mark the operations your business actually cares about.
OpenTelemetry exists because two competing projects — OpenTracing and OpenCensus — merged in 2019 rather than splitting the ecosystem further. That history explains both of its properties: the reason it is the safe long-term bet (there is no rival standard left to lose to), and the reason its surface has seams, with signals reaching stability years apart rather than together. Check the maturity of the specific signal and language SDK you depend on; "OpenTelemetry is stable" is not a single fact.
- span name = the operation:
GET /orders/:id,orders.charge - attributes = the dimensions: tenant, region, result, retry count
- never encode values into span names — that is what attributes are for, and dynamic names destroy grouping
- context dies at the queue boundary — propagate trace context through message headers, not just HTTP; the mechanics are covered in the Messaging & Async Playbook
- traces exist but nobody queries them, so nobody notices they are broken
- 100% sampling on day one, then a bill, then 0% sampling
My preference on sampling: head-based to start — cheap and predictable. Tail-based when money allows — it keeps the slow and failed traces that head-based throws away.
- RED for services: rate, errors, duration — per service, per endpoint
- USE for resources: utilization, saturation, errors — for hosts, pools, queues; deeper runtime diagnostics live in the Node.js Runtime & Performance Playbook
- Histograms for latency: percentiles, not averages — p50 tells you the typical case, p99 tells you who is suffering, the average tells you neither
- Business metrics next to system metrics: signups, orders, and payment failures on the same dashboards as 5xx rates — a healthy service with zero orders is an incident
The same RED metrics gate canary rollouts; that loop is covered in the DevOps Delivery Playbook.
Labels are for bounded dimensions: status class, region, endpoint template.
- no user ids in labels
- no request ids, emails, or raw URLs in labels
- every unbounded label is a time-series explosion arriving on a delay
If you need per-user answers, that is a trace or a log query, not a metric.
Pick SLIs users can feel:
- availability — the fraction of requests that succeed;
- latency at a percentile — "p99 under 500 ms", not "average under 100 ms".
Then:
- An SLO is an agreement, not a decoration. Product and engineering both sign it; it prices reliability against feature speed.
- The error budget is spending money. Budget left — ship faster, take risks. Budget gone — reliability work wins the sprint.
- Alert on budget burn rate, not raw thresholds. A fast burn pages; a slow burn opens a ticket. Static thresholds page at 3am for problems that could wait until Tuesday.
- SLOs end the "100% uptime" fantasy. 100% is not a target; it is a refusal to make tradeoffs. The last nine costs more than the product earns.
- Page on symptoms, ticket on causes. Users seeing errors is a page. A disk filling over three weeks is a ticket.
- Every page is actionable and owned. If the response to a page is "acknowledge and go back to sleep", delete the alert.
- Alert fatigue is an outage generator. Every ignored page trains the on-call to ignore the one that matters.
- Runbooks linked from the alert itself. The person paged at 3am should be one click from "what this means and what to do", not searching a wiki.
- page: SLO burn rate, user-facing error spikes, hard dependency down
- ticket: certificate expiring in three weeks, single host degraded behind a healthy pool, slow disk growth
- neither: a metric wiggled
One dashboard per service, answering two questions: is it healthy, and who is it hurting.
- top row: the SLIs — request rate, error rate, latency percentiles
- below: the usual suspects — saturation, dependencies, queue depth
- every panel drills down: symptom to dashboard, dashboard to trace, trace to logs — all joined by the trace id
A wall of forty graphs is a museum, not a tool. If a panel has never changed a decision, remove it.
The single highest-leverage convention in this suite: one request_id, born at the edge, carried everywhere.
- generated (or accepted) at the edge, attached to the trace context;
- stamped on every log line the request produces, in every service;
- echoed in every error envelope, so a user's screenshot leads straight to the trace — the envelope shape is defined in the API Design Playbook and wired through services in the Backend Architecture Playbook;
- carried through queues, jobs, and events, so background work stays attributable — see the Messaging & Async Playbook.
A request that becomes three events and a retry is still one story. Correlation is what lets you read it.
- unstructured printf logging that only a human with grep can read;
error-level noise nobody reads, burying the one line that mattered;- averages hiding tail pain while p99 users quietly churn;
- an alert on every metric "to be safe";
- per-request
debuglogging left on in production; - metrics with unbounded labels melting the metrics backend;
- tracing you pay for but never query.
- OpenTelemetry documentation
- OpenTelemetry: context propagation
- W3C Trace Context
- Google SRE book: Service Level Objectives
- Google SRE book: Monitoring Distributed Systems
- Google SRE workbook: Alerting on SLOs
- Prometheus: histograms and summaries
MIT is a sensible default for a repository like this, but choose the license that fits how you want others to reuse the material.