Sampling

Sampling keeps a fraction of your traces and discards the rest. It is the largest single lever on trace volume, and the one most likely to be misconfigured in a way that costs you the traces you needed.

Sampling applies to traces. Logs are reduced by log level and by filtering; metrics are aggregates and are not sampled — dropping metric samples changes the values rather than the volume of evidence. See Reducing what you store.

The rule that matters most

A sampling decision must be made once per trace and honoured by every service in it.If each service decides independently, you do not get fewer traces — you get broken traces: service A kept its span, service B dropped its own, and the trace you open has a hole where the failure was. This is worse than no sampling, because it is invisible until you need it.With head sampling, the decision travels in the traceparent header and downstream services must respect it — which parentbased_traceidratio, the default, does. Do not set a bare ratio sampler in some services and not others.

Head sampling

The decision is made when the trace starts, before anything is known about how it went.

OTEL_TRACES_SAMPLER=parentbased_traceidratio
OTEL_TRACES_SAMPLER_ARG=0.1          # keep 10%

Sampler	Behaviour
`always_on`	Keep everything. The default, and the right starting point.
`always_off`	Keep nothing.
`traceidratio`	Keep a fixed fraction, deciding locally.
`parentbased_always_on`	Follow the parent’s decision; start new traces on.
`parentbased_traceidratio`	Follow the parent’s decision; sample new traces at the ratio. Use this one.

Head sampling is cheap and unavoidably ignorant. It costs nothing — nothing is generated, buffered or transmitted for a dropped trace — but it decides before knowing whether the request errored or took nine seconds. At 10%, nine out of ten of your incidents are gone.That is the trade. Use it when volume is the binding constraint and you accept losing individual examples.

Ratio sampling is deterministic on the trace id, so the same trace is kept or dropped consistently everywhere — that is what makes the decision coherent across services.

Tail sampling

The decision is made after the trace completes, in a Collector, when its outcome is known. This is what most teams actually want.

processors:
  tail_sampling:
    decision_wait: 10s
    num_traces: 50000
    policies:
      - name: keep-errors
        type: status_code
        status_code: { status_codes: [ERROR] }
      - name: keep-slow
        type: latency
        latency: { threshold_ms: 1000 }
      - name: baseline
        type: probabilistic
        probabilistic: { sampling_percentage: 5 }

That keeps every erroring trace, every trace slower than a second, and 5% of everything else. Most teams find they lose nothing they use while cutting volume by an order of magnitude.

Tail sampling requires that every span of a trace reaches the same Collector instance. The processor buffers spans until the trace looks complete, so a trace split across two load-balanced Collectors is evaluated twice, with half the evidence each time.With more than one Collector you need a two-stage pipeline: agent Collectors forward to a gateway using the load-balancing exporter keyed on trace id, and tail sampling runs only on the gateway. Do not enable it on both stages.

Tail sampling also costs memory and adds decision_wait of latency before export — bounded by num_traces, which is a cap you should set deliberately rather than leave at a default.

Choosing

	Head sampling	Tail sampling
Where	In the application	In a Collector
Decides on	Trace id only	Errors, latency, attributes — anything
Keeps every error	No	Yes
Cost to run	None	Memory and a buffering delay
Multi-Collector setup	No constraint	Needs trace-id-aware routing
Reduces network from your app	Yes	No — everything is sent to the Collector

Start with no sampling

Until volume is a real problem, sample nothing. Sampling a system you do not yet understand hides the thing you were about to learn.

Then filter before you sample

Dropping health checks and metrics scrapes is free and loses nothing. It is frequently a larger reduction than sampling would have been. See Collector configuration.

Then tail sample

Keep all errors, keep the slow tail, take a small baseline of the rest.

Use head sampling only if the volume leaving your app is the problem

A network or CPU constraint at the application, rather than a storage one.

What sampling does to your other data

Metrics computed from sampled spans are wrong unless the pipeline accounts for it. If you sample 10% and then count spans, your request rate is a tenth of reality.Take rates and error ratios from metrics, which are not sampled, rather than by counting traces. Use traces to explain what the metric shows, not to measure it.

Log-to-trace links also break for dropped traces: the log line still carries a trace id, and clicking through finds nothing. That is expected and is not a bug — it is the cost of the trace you chose not to keep.

Verify

After changing sampling, confirm the shape rather than assuming it:

Trace volume should fall roughly as expected within a few minutes.
Error traces should still be present. Cause a failure and look for it. This is the check that catches a tail-sampling policy that silently is not matching.
Traces should still be complete. Open one and confirm it has spans from every service you expect, not a hole in the middle.

Collector configuration

Where tail sampling and filtering live.

Exporters

Batching, retries and queueing — what happens to a dropped export.

Data retention

The other half of controlling volume.

Overview

Platforms

OpenTelemetry

Other sources

The rule that matters most

Head sampling

Tail sampling

Choosing

What sampling does to your other data

Verify

Next

Collector configuration

Exporters

Data retention

​The rule that matters most

​Head sampling

​Tail sampling

​Choosing

​What sampling does to your other data

​Verify

​Next

Collector configuration

Exporters

Data retention

The rule that matters most

Head sampling

Tail sampling

Choosing

What sampling does to your other data

Verify

Next