> ## Documentation Index
> Fetch the complete documentation index at: https://docs.aiaxoniq.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Manual instrumentation

> Add your own spans, attributes and events on top of auto-instrumentation — what is worth instrumenting by hand, and the mistakes that make it useless.

Auto-instrumentation covers your framework boundaries: inbound requests,
outbound calls, database queries. Manual instrumentation is how you add what
only your code knows.

<Info>
  **Start with auto-instrumentation and add to it.** Manual spans are meant to
  sit *inside* the trace auto-instrumentation already produces, not to replace
  it. See [Instrument your application](/send-data/otel/zero-code).
</Info>

## What is worth a span

<Warning>
  **Do not instrument every function.** A span per function produces traces
  hundreds of spans deep that nobody reads, and it costs real money to store.

  A span is worth creating when the operation could plausibly be **slow**,
  could **fail on its own**, or is a **unit you would want timed separately** —
  a batch job stage, a cache lookup, a third-party call, an expensive
  computation.

  If you would not put it on a timeline while debugging, it does not need a
  span. Use an attribute or a log line instead.
</Warning>

Good candidates:

| Operation                                               | Why                                                       |
| :------------------------------------------------------ | :-------------------------------------------------------- |
| A third-party API call your library does not instrument | Frequently the actual latency, and invisible without this |
| A batch job's stages                                    | The only way to see which stage is slow                   |
| An expensive computation                                | Distinguishes "slow database" from "slow us"              |
| A cache lookup                                          | Hit and miss paths have completely different shapes       |
| A background task                                       | Otherwise it has no trace at all                          |

## Creating a span

Every SDK follows the same shape: get a tracer, start a span, do the work, end
the span. Language-specific syntax is in the OpenTelemetry documentation for
your SDK; the parts that decide whether it is *useful* are below.

<Steps>
  <Step title="Name the operation, not the instance">
    `charge-payment`, not `charge-payment-8817`. A name containing an
    identifier produces one operation per request, which makes aggregation —
    the whole point — impossible.
  </Step>

  <Step title="Attach the identifiers as attributes">
    `order.id`, `customer.tier`, `payment.provider`. This is where per-request
    values belong, and where they are cheap.
  </Step>

  <Step title="Record the outcome">
    Set the span's status to error when it fails, and record the exception.
    A span that failed but reports success is worse than no span — it is
    evidence pointing the wrong way.
  </Step>

  <Step title="End it, always">
    Use your language's scope or context manager so it ends on the error path
    too. A leaked span is a trace that never completes and never arrives.
  </Step>
</Steps>

## Attributes, events and status

| Use            | For                                                                                |
| :------------- | :--------------------------------------------------------------------------------- |
| **Attributes** | Facts about the whole operation: `order.id`, `retry.count`, `cache.hit`            |
| **Events**     | Something that happened at a moment inside it: `retry attempted`, `lock acquired`  |
| **Status**     | Whether the operation succeeded. Set `ERROR` on failure, and record the exception. |

<Note>
  **Prefix your own attribute names.** `acme.order.id`, not `order.id`, so
  nothing you invent collides with an OpenTelemetry semantic convention added
  later. Follow the conventions where one exists — see
  [Resource attributes](/send-data/otel/resource-attributes#conventions-worth-following).
</Note>

<Warning>
  **Never put a secret, a credential, a password, a token or personal data on a
  span.** Telemetry is append-only and retained for its full window, and there
  is no surgical delete. Anything you attach here, you have stored.

  If it must be captured, hash or redact it at the Collector before it leaves
  your network — see
  [Collector configuration](/send-data/otel/collector-config).
</Warning>

## Context propagation

A manual span joins the current trace automatically **if the context reaches
it**. That is where manual instrumentation usually goes wrong.

<Warning>
  **Context does not survive a thread pool, a queue or a detached task on its
  own.** Work handed to a background executor starts a *new* trace unless you
  explicitly carry the context across, and the symptom is a trace that stops
  exactly where the interesting work began.

  Every SDK provides a way to capture and reattach the current context. If your
  manual spans are appearing as separate root traces, this is why.
</Warning>

Across a network boundary, propagation is the `traceparent` header, and
auto-instrumentation handles it. If you make an HTTP call with a client the SDK
does not instrument, you must inject the header yourself or the trace ends
there.

## Connecting logs to traces

A log line written while a span is active carries that span's trace id, and
becomes clickable from the trace.

<Warning>
  **This only works through a logging integration that is trace-aware.** A
  `print()`, a bare file logger or a logging framework without the
  OpenTelemetry bridge produces correct, searchable log lines that are
  permanently disconnected from every trace.

  If "view logs for this span" is empty on a service you know is logging, the
  logger is the thing to fix — not the trace. See
  [Sending logs as well as traces](/send-data/otel/zero-code#sending-logs-as-well-as-traces).
</Warning>

## Custom metrics

Where a span answers "what happened in this request", a metric answers "how
often, across all of them". Counters, histograms and gauges are all available
through the SDK.

<Warning>
  **Metric labels must be low-cardinality.** A counter labelled with a user id
  or an order id creates one series per value, permanently. Per-request
  identifiers belong on spans; metrics take bounded dimensions like status,
  route pattern and region.

  This is the most expensive mistake available in instrumentation, and it is
  entirely silent — see
  [cardinality](/concepts/data-model#cardinality-is-the-cost-you-cannot-see).
</Warning>

## Verify

<Steps>
  <Step title="Trigger the code path">
    Then open **Traces** and find the trace.
  </Step>

  <Step title="Confirm your span is nested, not a root">
    A manual span appearing as its own root trace means the context did not
    reach it — see [Context propagation](#context-propagation).
  </Step>

  <Step title="Confirm the failure path">
    Force an error and check the span is marked as failed and carries the
    exception. This is the half people forget to test, and it is the half you
    need during an incident.
  </Step>
</Steps>

## Next

<CardGroup cols={3}>
  <Card title="Auto-instrumentation" icon="code" href="/send-data/otel/zero-code">
    The layer this sits on top of.
  </Card>

  <Card title="Resource attributes" icon="tags" href="/send-data/otel/resource-attributes">
    Naming, conventions and cardinality.
  </Card>

  <Card title="Explore traces" icon="share-nodes" href="/guides/traces/overview">
    Finding what you instrumented.
  </Card>
</CardGroup>
