> ## Documentation Index
> Fetch the complete documentation index at: https://docs.aiaxoniq.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Exploring traces

> Find a slow or failing request, read a trace waterfall, and diagnose the two reasons a trace arrives with a hole in the middle.

A **trace** is one request's journey across your services. A **span** is one
operation within it. Traces answer *where the time went* and *which hop
failed* — questions no metric and no log line can answer on their own.

<Info>
  **Before you start.** You need traces arriving — see
  [Instrument your application](/send-data/otel/zero-code) or
  [zero-code instrumentation](/zero-code/overview).
</Info>

## Finding the trace you want

You rarely browse traces. You arrive at one from something else:

| From           | How                                                                  |
| :------------- | :------------------------------------------------------------------- |
| A metric spike | Narrow the time range, then filter traces to that window and service |
| A log line     | Its `trace_id` links to the trace that produced it                   |
| An alert       | The alert's window and service scope you down to a handful           |
| A service      | Open the service and look at its slowest and failing traces          |

Filter by service, operation, duration and status. **Slowest first, errors
only** is the filter that answers most questions.

## Reading a waterfall

Each bar is a span; nesting is causality. Three things to read, in order:

<Steps>
  <Step title="Total duration versus the widest child">
    If the root is 900ms and its widest child is 880ms, the time is in that
    child — follow it down. If the root is 900ms and every child is small, the
    time is in **your** service, between the calls.
  </Step>

  <Step title="Sequential versus parallel">
    Bars that step down one after another are sequential calls. Ten 50ms calls
    in series is 500ms that could have been 50ms. This is the single most
    common finding in a first trace review.
  </Step>

  <Step title="The failing span, not the failing trace">
    A trace marked failed shows the error at the root and at the actual source.
    The deepest failing span is the cause; everything above it is propagation.
  </Step>
</Steps>

## Spans, attributes and events

A span carries a name, a duration, a status, and attributes describing the
operation — `http.method`, `http.route`, `http.status_code`, `db.system`. Events
are timestamped moments inside a span.

Span fields are queryable, and any name not in the promoted set falls through
to an attribute lookup — the same behaviour as
[log search](/guides/logs/search).

## Traces with a hole in them

<Warning>
  **A trace missing a service in the middle has one of two causes.**

  **Context propagation stopped.** A service received a request without a
  `traceparent` header, so it started a new trace. Common where a call goes
  through a queue, a thread pool, a detached background task, or an HTTP client
  the SDK does not instrument.

  **Sampling dropped it.** If services sample independently rather than
  honouring the parent's decision, each drops its own spans and you get partial
  traces rather than fewer traces. See
  [Sampling](/send-data/otel/sampling#the-rule-that-matters-most).

  The two are distinguishable: a propagation break makes the downstream work
  appear as a **separate root trace**; sampling makes it disappear entirely.
</Warning>

## Logs for a span

A log line written while a span was active carries its trace id and is
reachable from the trace.

<Warning>
  **This requires a trace-aware logging integration.** A `print()`, a bare file
  logger, or a logging framework without the OpenTelemetry bridge produces log
  lines that are correct, searchable, and permanently disconnected from every
  trace.

  Empty "logs for this span" on a service you know is logging means the logger,
  not the trace.
</Warning>

## Metrics derived from traces

Request rate, error rate and duration per service are computed from spans, so
instrumenting for traces gives you service-level metrics without extra work.

<Warning>
  **If you sample traces, do not count spans to get a rate.** At 10% sampling,
  counting traces reports a tenth of reality. Take rates and error ratios from
  metrics, which are not sampled, and use traces to *explain* what the metric
  shows.
</Warning>

## Retention

Traces are kept for **30 days**, like logs. Unlike metrics, there is no
long-lived rollup — a trace is either inside the window or gone. If a specific
trace matters for a post-incident review, capture it before it ages out.

## Next

<CardGroup cols={3}>
  <Card title="Service map" icon="diagram-project" href="/guides/services/service-map">
    The dependency graph these traces build.
  </Card>

  <Card title="Manual instrumentation" icon="code-branch" href="/send-data/otel/manual-instrumentation">
    Add spans where auto-instrumentation cannot see.
  </Card>

  <Card title="Searching logs" icon="magnifying-glass" href="/guides/logs/search">
    The other half of an investigation.
  </Card>
</CardGroup>
