Book Appointment

Head vs Tail Sampling: A Hospital IT Comparison

Healthcare IT

Hospital IT teams that run their own observability stack eventually face a budget conversation about traces. Every request through a patient-portal login, an invoice export or a records-access gateway can leave a trace, and keeping all of them gets expensive. This comparison looks at two ways to thin that stream, head sampling and tail sampling, as part of ObservabilityOps, the practice of running the telemetry system itself. It is a technical, administrative piece with no clinical content.

Why sampling is a platform decision

Sampling is not a developer preference that each service can settle alone. Where the decision sits, what it costs to run and which traces survive all depend on the platform around it. According to the Ops Atlas, sampling is one of the named concerns inside the ObservabilityOps scope the Atlas describes, next to cardinality and retention. The OpenTelemetry documentation, as the Atlas relays it, suggests starting to consider sampling once a system produces something like a thousand traces per second.

Below that rate, many departments are better off keeping everything and spending their attention elsewhere. Above it, the choice between the two approaches shapes both the bill and the debugging experience.

Head sampling in plain terms

With head sampling, the decision happens when a trace begins. The first service flips a weighted coin, and every downstream service honours the result. It is cheap, stateless and easy to reason about, and it needs no special infrastructure beyond configuration in the SDK or the first collector in the path.

The weakness is that the coin knows nothing about the future. A request that will fail two services later looks identical, at the moment of the flip, to one that will succeed. So a one-percent head rate keeps about one percent of the errors too, which can leave the incident you most want to examine unrecorded.

Tail sampling in plain terms

Tail sampling holds back until a trace has finished, and only then makes its choice, with the whole picture in hand. You can keep every trace that contained an error, every one slower than a threshold, and a small random slice of ordinary traffic. The Atlas describes it as stateful and hard to operate, and the reason is mechanical: all spans belonging to one trace must reach the same collector instance, which usually means routing by trace identifier and holding spans in memory until a decision timer expires.

  • Memory and buffering: spans wait in the collector while the decision window runs.
  • Routing: a load-balancing tier sends each trace to a single instance.
  • Failure behaviour: if an instance restarts, in-flight traces can be lost or decided on partial data.

Side-by-side: what you get and what you pay

Think of the two options along four dimensions rather than as good versus bad.

  • Error coverage. Head sampling gives proportional coverage only; tail sampling can guarantee that failures and slow requests are kept.
  • Operating effort. Head sampling is a configuration line. Tail sampling is a small distributed service that someone must size, monitor and upgrade.
  • Cost shape. Head sampling cuts cost before data leaves the application. Tail sampling still pays network and collector cost for everything, then saves on storage.
  • Derived numbers. Request rates and error ratios should be derived from the complete span stream ahead of the sampler, otherwise dashboards shrink in step with the traces they came from.

That last point is easy to miss. If rate metrics are built from sampled traces, they under-report in proportion to the sampling rate, and an executive dashboard may show demand that never matched reality.

Trading cost against detail

The Atlas includes a budget squeeze as a teaching case. Management wants the monitoring invoice cut by 30%. Traces make up 45% of it and logs another 40%, with roughly 2,000 traces arriving each second. Path one: keep only about 1% of traces through tail sampling, but never drop the failed or sluggish ones. Path two: leave sampling alone, hold traces for a shorter time and push verbose debug logs into archive storage. The Atlas presents neither as obviously right. It is a judgement call that hinges on how your people really investigate problems and how much running effort they can spare.

For a hospital’s administrative systems, a reasonable lens is audit and support. If your help desk regularly investigates one specific user complaint days later, longer retention for a thin slice of traces may serve better than aggressive discarding. If incidents are mostly short bursts of errors, keeping every failed trace matters more than keeping every successful one.

Context propagation still comes first

None of this works if traces break across hops. The default W3C trace context header carries the sampling flag, so a proxy that drops the header, or a mix of old and new header formats, makes each side start fresh traces and sample independently. Queues need span links rather than a straight parent relationship. A team that is weighing the two sampling styles should first confirm that a single request produces a connected trace end to end.

For wider background on running dependable platforms, see this write-up on delivery, reliability and cloud-native engineering, and the companion piece on cloud operations and managed delivery services.

A pragmatic way to choose

The Atlas’s decision heuristics lean toward starting simple. Use head sampling or no sampling while volume is modest. Move to tail sampling once you have one large producer whose cost dominates the bill and a platform team able to run stateful collectors. Roll it out for that single producer first, measure how many errors you retain, and expand only if the effort proves tolerable.

  1. Measure trace volume per service for a normal week.
  2. Identify the largest producer and confirm its traces connect across hops.
  3. Compute rate and error metrics before sampling.
  4. Pilot tail sampling there, with an always-keep rule for errors and slow requests.
  5. Review disk, memory and lost-trace behaviour after a month, then decide on a wider rollout.

Done in that order, the comparison stops being theoretical. You end up with a documented reason for each choice, and a clear point at which to revisit it when traffic or budgets change.