Distributed Tracing: Following Requests Across Many Services

Hey everyone, welcome to the forty-seventh issue of The Main Thread.

Consider a scenario: checkout latency triples on a Tuesday afternoon, the gateway p99 jumps from 400ms to 1.8s. Auth is fine, cart is fine, pricing is fine, payments are fine. The user is waiting almost 2s and every component involved claims innocence.

This is normal, and it follows from arithmetic: percentiles don’t compose. A request touching 50 services collects latency from each hop, and slow edge requests are usually assembled from moments that look unremarkable inside any single service. Metrics tell us the system is slow; they aggregate away the identity of the request, so they cannot tell us where one slow request spent its time. Logs have the opposite problem: 50 services produce 50 disconnected narratives with no shared key and no causality we can trust across hosts.

Distributed tracing exists to answer exactly this question: for one specific request, reconstruct the full tree of work it caused across every service, with timing. Google published the design in the 2010 Dapper paper, Zipkin and Jaeger carried it into open source, and the ecosystem consolidated into OpenTelemetry in 2019. The ideas stabilized a decade ago; what still goes wrong is the plumbing: context that fails to propagate, spans that record the wrong things, sampling that discards the traces we needed, and bills that grow faster than traffic.

This issue covers the whole path: how trace context moves between services (W3C and B3, byte by byte), how to design spans worth reading, how to sample when we cannot keep everything, how the OpenTelemetry pieces fit in production, how to read the trace shapes that diagnose most incidents, and how to keep the bill sane.

A trace is the record of one request’s journey through the system. A span is one named, timed unit of work inside it: an HTTP handler, a database query, a call to a payment provider. Spans form a tree; each span knows its parent, and the root (the edge request) has none. That is the entire data model, unchanged since Dapper.

The slow checkout, rendered as a waterfall in every tracing UI, looks like this:

One picture ends the incident call. Auth, cart, and pricing are collectively responsible for under 500ms. The payment provider took 1.3s, and every service reporting a healthy p99 was telling the truth: the time went to an external dependency no per-service dashboard owns.

Each span creates a fixed set of fields:

A trace_id (16 bytes, shared by every span in the trace), a span_id (8 bytes), and the parent’s span_id. These three make the tree reconstructable from spans that arrive out of order, from different hosts, minutes apart.

Start and end timestamps, taken on the host doing the work.

POST /checkout, SELECT discounts. Low cardinality by convention: the route template, never the concrete URL with IDs in it.

Key-value metadata about this execution. (http.response.status_code = 402, cart.item_count = 17).

Timestamped point-in-time annotations inside the span, most importantly recorded exceptions.

OK or Error.

A span is a structured log line with a duration and a family tree: anything we would have logged during a request can live on a span instead. pre-joined to everything else that happened to that request.

One more field earns its keep: SpanKind. A cross-service call produces two spans, a CLIENT span in the caller and a SERVER span in the callee, and the gap between them is network time plus queueing, computable only because the kinds are marked. PRODUCER and CONSUMER play the same role across queues; INTERNAL marks everything else.

Tree reconstruction depends on one thing: when service A calls service B, B must learn A’s trace_id and span_id. Nothing about HTTP, gRPC, or Kafka carries the identifiers: every service must read them on the way in and write them on the way out, and this relay race is where tracing deployments actually fail. Drop the baton at one hop and every span downstream starts a fresh trace, orphaned from the request that caused it.

The traceparent header is the standard, a W3C recommendation since 2020 and OpenTelemetry’s default. One line, four dash-separated fields:

Each service extracts this header, parents its own span under parent-id and injects a new traceparent into every outbound call with its own current span as the new parent-id. The trace-id never changes; the parent-id changes at every hop.

The last byte (sampled flag) carries the sampling decision down the chain, so one decision at the edge governs the whole trace. This one byte makes head sampling cheap, and it is also head sampling’s built-in limitation; the sampling section returns to it. A second header tracestate, rides alongside as vendor-specific key-value pairs; we rarely touch it, we always forward it.

Before W3C standardization, B3 was the de facto format, from Zipkin (the name comes from BigBrotherBird, Zipkin’s original codename). The same information is spread across headers.

or packed into one: b3: {trace-id}-{span-id}-{sampled}. B3 is everywhere in older infrastructure: Istio, Linkerd, Spring Cloud Sleuth, anything Zipkin-era. A fleet where half the services speak traceparent and half speak B3 breaks traces at every boundary between the camps. OpenTelemetry’s composite propagators fix the migration: extract whichever format arrives, inject both on the way out, converge on W3C, then drop B3.

The header mechanics are trivial. The failures are structural, and they cluster in four places:

Context lives in a thread-local or a context variable. Hand work to a raw thread pool and the context stays behind on the submitting thread; spans created inside the pool silently start new traces. Instrumented executors exist for exactly this; the danger zone is custom concurrency code.

HTTP auto-instrumentation forwards headers for us. A Kafka producer does not, unless instrumentation injects traceparent into the message headers and the consumer extracts it. If we miss this, then every trace ends at the producer. The queue decouples our services, and left alone, decouples our traces too.

A proxy that strips unknown headers, a serverless function without the SDK, a service in a language nobody instrumented. One opaque hop cuts the tree. The telltale signature in the backend: a population of traces that all start at an internal service that should never be a root.

One consumer span processes 500 messages from 300 different traces. A parent-child edge is wrong here (which parent?). Span links exist for this: the consumer span links to all 300 producer contexts without claiming descent from any.

For a queue, the propagation we otherwise get for free has to be spelled out:

One more thing travels with the trace: baggage, a separate W3C spec for arbitrary key-value pairs propagated hop to hop (baggage: tenant.id=acme,deploy.canary=true). It lets a service deep in the stack know something only the edge knew (canary attribution, tenant-level debugging), and it is a footgun: every hop pays its size on the wire, and every hop can read it. Routing-grade facts go in; secrets and PII stay out.

Instrumentation quality decides whether traces answer questions or merely exist. The failure modes sit at both extremes: spans so sparse that waterfall shows shapes with no explanations, and a span per function call, drowning the signal and quadrupling the bill.

A good granularity rule is: a span for every network boundary, plus a span for each internal unit we would want as its own bar in the waterfall. Network hops are non-negotiable; latency and failure concentrate there. Internal spans have a concrete test: if this unit were slow, would knowing that change where we look next? Serialization of a 40MB response, a lock acquisition: yes. A utility function: no.

Everything else is choosing among the three places a fact can live:

Two decisions inside the above table have the most weight.

OpenTelemetry standardizes attribute names: http.request.method, http.response.status_code, url.path, db.system. Backends key their latency breakdowns, error rates, and service maps off these exact names; invent myapp.status instead and those features degrade to a generic list. Custom names are for domain facts (cart.item_count, payment.provider, tenant.id).

Metrics forbid user-ID labels because each value mints a new time series. A span attribute is a field on one record: user.id and order.id belong on spans, and they are exactly what makes tracing useful during an incident (“show me traces for the customer in this ticket”). Cardinality still hurts in the span name, which backends group by: name the span GET /users/{id}, put the ID in an attribute.

Instrumented by hand, a unit of work worth tracing looks like this:

Error handling has one subtlety: span status is about this operation's contract, and it does not automatically follow HTTP codes. A GET /users/23 returning 404 during an existence check did its job; marking it ERROR floods the error views with non-errors. Auto-instrumentation follows the conventions' defaults (5xx is an error on servers, 4xx and 5xx on clients); override deliberately where the semantics differ.

What stays off the span: request and response bodies, SQL with literal values inlined, authorization headers, anything a privacy review would flag. Traces land in a system with looser access control than the production database and persist for the retention window. The collector can scrub centrally; restraint at the source is still the policy.

Let’s see the arithmetic that dominates every deployment at scale. A mid-sized system: 2000 requests/sec at the edge, around 50 spans per request. That is 100K spans/second. 8.6 billion per day, and at a representative 500 bytes per encoded span, roughly 4.3 TB per day before replication or indexing. Dapper shipped with 1-in-1024 sampling in 2008 for exactly this reason. Sampling is unavoidable; the only real question is where the decision gets made.

The root service flips a coin when the trace starts, records the outcome in the traceparent sampled flag, and every downstream service honors it:

Head sampling is cheap (unsampled traces cost almost nothing at runtime) and operationally trivial. Its flaw is fundamental: the decision is made before anything has happened. At 1%, we keep 1% of the errors, 1% of the 8-second outliers, 1% of the traces from the customer in the support ticket. The traces we want are rare by definition, and a uniform coin flip is uniformly blind to them.

Every span ships to a collector, which buffers spans by trace ID, waits for the trace to finish, then applies policy with full knowledge of what happened:

The result is close to ideal. The costs are concrete:

1/ We pay to generate and transport every span, kept or discarded. The savings apply to storage and ingest only.
2/ All spans of a trace must reach the same collector instance, or policies judge fragments. This forces trace-ID-aware routing (the collector's loadbalancing exporter with routing_key: traceID) in front of the sampling tier.
3/ The buffer is state. 10 seconds of decision_wait at 100,000 spans per second is a million spans held in memory per second of window, and a trace outliving the window is judged incomplete.

The pragmatic answer at real size is layered: head sampling in the SDK at a rate generous enough to feed the tail tier, tail sampling at the collector for the keep/drop intelligence, rate limits as a circuit breaker so a traffic spike degrades sampling instead of collectors. And because the flag rides in traceparent, use ParentBased samplers everywhere so what survives is complete traces. Partial traces are worse than fewer traces.

OpenTelemetry won: OpenTracing and OpenCensus merged into it in 2019, every major vendor accepts its protocol (OTLP), and instrumenting against anything else in 2026 is a legacy decision. The architecture is more important than any individual API.

The API/SDK split is the load-bearing design decision. Libraries (the HTTP framework, the Postgres driver) instrument against the OpenTelemetry API, a facade of no-ops. Applications install the SDK, which makes the facade real and decides exporters, samplers, and resource attributes. This is why a library can ship instrumentation without forcing an observability stack on its users, and why auto-instrumentation works at all.

Auto-instrumentation covers the plumbing; manual covers the business. The Java agent and Python's opentelemetry-instrument wrapper patch the common clients and servers at load time: every inbound request gets a SERVER span, every outbound HTTP/DB/queue call gets a CLIENT span, propagation handled. That alone produces the waterfall skeleton; the manual layer (the compute_totals example) adds what the request means. Deploy it fleet-wide first; it is the highest-value single step.

Applications export to a Collector, never directly to the backend:

The reasons to insist on this shape:

1/ Vendor decisions become config. Switching backends, or running two during an evaluation, is an exporter stanza in one place instead of 50 repos.
2/ Policy is centralized. Tail sampling, PII scrubbing, and attribute trimming happen once, at the gateway.
3/ The app is insulated. Batching, compression, retry with backoff, and queue-on-outage live in the Collector. A vendor incident stops being a memory leak in our checkout path.

The last line of the diagram ties this to cost: the spanmetrics connector derives request/error/duration metrics from spans before sampling throws data away. Aggregates come from 100% of traffic; full traces are retained for 1%. Accurate percentiles and cheap storage stop being a trade-off.

Waterfalls repeat a small vocabulary of shapes, and knowing them converts a wall of bars into a diagnosis. There are three of them that cover a remarkable share of production incidents.

Each call starts where the previous one ended: the N+1 query problem promoted to the network. Nothing here is slow: the shape is slow, 12 hops at 100ms each. The fix is a batch endpoint or concurrent fan-out, and the trace even prices the fix: batched, this page costs ~110ms instead of 1240ms. Staircases hide inside ORMs, per-item enrichment loops, and sequential awaits nobody meant to serialize. On a dashboard this pathology is invisible; product-service p99 is a flawless 105 ms.

Time inside a span that no child accounts for is the gap. The service was doing something for 613 ms, and nobody instrumented it. Gaps are where tracing hands off to Brendan Gregg's territory, because a gap is almost always off-CPU time: the request was waiting, and waiting is invisible to instrumentation that only wraps calls.

The recurring culprits, in rough order of frequency, are: connection-pool acquisition, garbage-collection pauses, lock contention, synchronous DNS or TLS setup on a cold path, CPU starvation on a throttled host.

Each has a next move: a span around pool acquisition (the single highest-value manual span in most codebases), correlating the gap's timestamp with runtime GC metrics, or a profiler once request-scope data is exhausted. The gap does not name the culprit; it scopes the search to one service and 613 specific milliseconds.

So, we have three identical child spans, two failed. The client library retried, silently, with a 3-second timeout per attempt, and the caller experienced 9.3 seconds. Retries configured in a client library are invisible in code review and in metrics (the dependency's error rate counts attempts, so it can even look better while callers suffer).

In a trace they are three unmistakable bars, and they raise the right questions: why does the first attempt time out, is the retry budget sane against the caller's own deadline, is there backoff and jitter. Multiply this shape down a 5-deep call chain with 3 retries per hop and the bottom sees 3^5 = 243 requests for 1 at the top.

One caveat applies to all careful trace reading: timestamps come from different hosts, so clock skew is real. NTP-synced fleets sit within a few milliseconds, fine for waterfalls, useless for microsecond forensics. A child span apparently starting before its parent is skew. Within one host, timestamps share a clock and can be trusted.

Let’s return to the arithmetic: ~4.3 TB per day at 100% capture for a 2,000 rps system. At an illustrative $0.20 per ingested GB, that’s roughly $26,000 per month, scaling with traffic times instrumentation density, both of which only grow.

The levers that we can pull, in order:

1/ Sampling is the primary lever, and tail sampling makes it safe. The keep-errors, keep-slow, 1%-baseline policy cuts ingest by ~99% while retaining nearly every trace anyone will look at: the difference between $26,000 and a few hundred dollars a month, at the price of a collector tier costing an order of magnitude less than the ingest it saves.

2/ Derive metrics before sampling. The spanmetrics pattern, stated as economics: dashboards and alerts consume aggregates daily, and aggregates compress millions of spans into a handful of time series. Compute them from the full stream at the collector, then sample the traces.

3/ Trim at the collector. A 2 KB SQL text repeated on a million spans is 2 GB of the same string. Truncate long values, drop attributes no one queries, and drop whole span populations with no diagnostic value: health checks alone add millions of identical, worthless spans a day. Dropped at the agent, they never cross the network.

4/ Tier the retention. Trace value decays in hours: debugging happens against the last day, incident review against the last two weeks. Baseline retention of 7 days with 30 for error traces covers nearly every real access pattern at a fraction of flat 30-day retention. Aggregate trends are what the metrics from lever 2 are for.

5/ If self-hosting, choose object-storage-backed designs. Grafana Tempo bets that traces are fetched by ID far more often than searched, and stores compressed blocks at S3 prices. ClickHouse-backed stores make search cheap through columnar compression. Elasticsearch-era per-span indexing is the expensive ancestor; if a Jaeger deployment still writes to it, a 10x reduction is sitting there.

Metrics aggregate across requests. Logs fragment within them. A trace is the one artifact keyed by the request itself, which is why it answers the question the other two cannot: for this specific slow request, where did the time go. Everything in this issue serves that artifact's integrity end to end: propagation keeps the tree connected across 50 services and every queue between them, span design makes each node worth reading, sampling decides which trees survive, and the collector pipeline is where policy and economics live.

The failure mode to defend against is quiet degradation. Tracing rarely breaks loudly; it decays. A queue hop loses context and traces silently shorten. Head sampling at 1% quietly discards incident traces for months before anyone needs one and finds it missing. So treat the trace pipeline like the production system it is, and start this week: pull up 10 traces for the highest-traffic endpoint and read them against the three shapes. A staircase that should be a batch call, a 600 ms gap nobody instrumented, a retry hiding in a client library. One of them is in there, and it has been on every dashboard all along, averaged into invisibility.

If this clicked, forward it to whoever is currently grepping 50 services' logs for one request. And if you want more deep dives like this, subscribe to The Main Thread: practical distributed systems engineering, one essay per week.

Namaste!