> ## Documentation Index
> Fetch the complete documentation index at: https://docs.aiaxoniq.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Sampling
> Head and tail sampling for traces, which to choose, and the rule that stops sampling from producing broken traces instead of fewer ones.
Sampling keeps a fraction of your traces and discards the rest. It is the
largest single lever on trace volume, and the one most likely to be
misconfigured in a way that costs you the traces you needed.
**Sampling applies to traces.** Logs are reduced by log level and by
filtering; metrics are aggregates and are not sampled — dropping metric
samples changes the values rather than the volume of evidence. See
[Reducing what you store](/concepts/retention#reducing-what-you-store).
## The rule that matters most
**A sampling decision must be made once per trace and honoured by every
service in it.**
If each service decides independently, you do not get fewer traces — you get
*broken* traces: service A kept its span, service B dropped its own, and the
trace you open has a hole where the failure was. This is worse than no
sampling, because it is invisible until you need it.
With head sampling, the decision travels in the `traceparent` header and
downstream services must respect it — which `parentbased_traceidratio`, the
default, does. Do not set a bare ratio sampler in some services and not
others.
## Head sampling
The decision is made when the trace **starts**, before anything is known about
how it went.
```bash theme={null}
OTEL_TRACES_SAMPLER=parentbased_traceidratio
OTEL_TRACES_SAMPLER_ARG=0.1 # keep 10%
```
| Sampler | Behaviour |
| :------------------------- | :------------------------------------------------------------------------------ |
| `always_on` | Keep everything. The default, and the right starting point. |
| `always_off` | Keep nothing. |
| `traceidratio` | Keep a fixed fraction, deciding locally. |
| `parentbased_always_on` | Follow the parent's decision; start new traces on. |
| `parentbased_traceidratio` | Follow the parent's decision; sample new traces at the ratio. **Use this one.** |
**Head sampling is cheap and unavoidably ignorant.** It costs nothing —
nothing is generated, buffered or transmitted for a dropped trace — but it
decides before knowing whether the request errored or took nine seconds. At
10%, nine out of ten of your incidents are gone.
That is the trade. Use it when volume is the binding constraint and you
accept losing individual examples.
Ratio sampling is deterministic on the trace id, so the same trace is kept or
dropped consistently everywhere — that is what makes the decision coherent
across services.
## Tail sampling
The decision is made **after** the trace completes, in a Collector, when its
outcome is known. This is what most teams actually want.
```yaml theme={null}
processors:
tail_sampling:
decision_wait: 10s
num_traces: 50000
policies:
- name: keep-errors
type: status_code
status_code: { status_codes: [ERROR] }
- name: keep-slow
type: latency
latency: { threshold_ms: 1000 }
- name: baseline
type: probabilistic
probabilistic: { sampling_percentage: 5 }
```
That keeps **every** erroring trace, **every** trace slower than a second, and
5% of everything else. Most teams find they lose nothing they use while cutting
volume by an order of magnitude.
**Tail sampling requires that every span of a trace reaches the same
Collector instance.** The processor buffers spans until the trace looks
complete, so a trace split across two load-balanced Collectors is evaluated
twice, with half the evidence each time.
With more than one Collector you need a two-stage pipeline: agent Collectors
forward to a gateway using the load-balancing exporter keyed on trace id, and
tail sampling runs only on the gateway. Do not enable it on both stages.
Tail sampling also costs memory and adds `decision_wait` of latency before
export — bounded by `num_traces`, which is a cap you should set deliberately
rather than leave at a default.
## Choosing
| | Head sampling | Tail sampling |
| :---------------------------- | :----------------- | :--------------------------------------- |
| Where | In the application | In a Collector |
| Decides on | Trace id only | Errors, latency, attributes — anything |
| Keeps every error | No | **Yes** |
| Cost to run | None | Memory and a buffering delay |
| Multi-Collector setup | No constraint | Needs trace-id-aware routing |
| Reduces network from your app | **Yes** | No — everything is sent to the Collector |
Until volume is a real problem, sample nothing. Sampling a system you do
not yet understand hides the thing you were about to learn.
Dropping health checks and metrics scrapes is free and loses nothing. It is
frequently a larger reduction than sampling would have been. See
[Collector configuration](/send-data/otel/collector-config).
Keep all errors, keep the slow tail, take a small baseline of the rest.
A network or CPU constraint at the application, rather than a storage one.
## What sampling does to your other data
**Metrics computed from sampled spans are wrong unless the pipeline accounts
for it.** If you sample 10% and then count spans, your request rate is a
tenth of reality.
Take rates and error ratios from **metrics**, which are not sampled, rather
than by counting traces. Use traces to explain what the metric shows, not to
measure it.
Log-to-trace links also break for dropped traces: the log line still carries a
trace id, and clicking through finds nothing. That is expected and is not a
bug — it is the cost of the trace you chose not to keep.
## Verify
After changing sampling, confirm the shape rather than assuming it:
* **Trace volume** should fall roughly as expected within a few minutes.
* **Error traces should still be present.** Cause a failure and look for it.
This is the check that catches a tail-sampling policy that silently is not
matching.
* **Traces should still be complete.** Open one and confirm it has spans from
every service you expect, not a hole in the middle.
## Next
Where tail sampling and filtering live.
Batching, retries and queueing — what happens to a dropped export.
The other half of controlling volume.