Collector configuration

The Collector is one binary with a pipeline: receivers take telemetry in, processors transform it, exporters send it on. This page is the reference for building that pipeline. The platform pages (Docker, Linux, Kubernetes) give you a working one to start from.

The minimum that works

config.yaml

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318
  hostmetrics:
    collection_interval: 30s
    scrapers:
      cpu:
      memory:
      disk:
      filesystem:
      network:
      load:

processors:
  # Order matters and is not alphabetical. memory_limiter can only shed load it
  # sees before anything has buffered it, so it goes first; batch should group
  # records after every other processor has finished changing them, so it goes
  # last. A pipeline that batches first and limits afterwards will still run out
  # of memory under exactly the load the limiter was added for.
  memory_limiter:
    check_interval: 1s
    limit_percentage: 80
    spike_limit_percentage: 25
  resourcedetection:
    detectors: [env, system]
    system:
      hostname_sources: [os]
  batch:
    timeout: 5s
    send_batch_size: 1000

exporters:
  otlphttp/aiaxoniq:
    endpoint: ${env:OIQ_ENDPOINT}
    headers:
      X-License-Key: ${env:OIQ_LICENSE_KEY}
    compression: gzip

service:
  telemetry:
    metrics:
      readers:
        - pull:
            exporter:
              prometheus:
                host: 0.0.0.0
                port: 8888
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, resourcedetection, batch]
      exporters: [otlphttp/aiaxoniq]
    metrics:
      receivers: [otlp, hostmetrics]
      processors: [memory_limiter, resourcedetection, batch]
      exporters: [otlphttp/aiaxoniq]
    logs:
      receivers: [otlp]
      processors: [memory_limiter, resourcedetection, batch]
      exporters: [otlphttp/aiaxoniq]

Everything below is something you add to that.

The pipeline

service:
  pipelines:
    traces:
      receivers:  [otlp]
      processors: [memory_limiter, batch]
      exporters:  [otlp]

Processor order is execution order, and it matters.

memory_limiter must be first. Its job is to reject work before the process runs out of memory, which it cannot do from the middle of a pipeline.
batch should be last, or as close as possible. Batching before a filter means you built batches out of records you then threw away.
Anything that drops data — filtering, sampling — belongs before anything expensive.

A pipeline that lists the right processors in the wrong order is valid configuration and does the wrong thing silently.

A component defined but not listed in a pipeline does nothing. This is the most common Collector configuration mistake: the processor is configured correctly, the file is valid, and it is never executed because it was not added to service.pipelines.

Processors worth adding

memory_limiter — stop the Collector from being the outage

processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 80
    spike_limit_percentage: 25

Without it, a traffic spike can push the Collector into the kernel’s out-of-memory killer — and on a Kubernetes node, the thing that dies may not be the Collector. Always include it, first.

batch — fewer, larger requests

processors:
  batch:
    timeout: 5s
    send_batch_size: 8192
    send_batch_max_size: 10000

The receiver rate-limits requests, not records, so batching is what keeps you inside it. send_batch_max_size also stops a batch growing past the body size limit. See Exporters.

filter — drop what you will never read

Usually the single largest reduction available, and it costs you nothing.

processors:
  filter/health:
    error_mode: ignore
    traces:
      span:
        - 'attributes["http.route"] == "/health"'
        - 'attributes["http.route"] == "/metrics"'

Health checks and metrics scrapes are the highest-frequency, lowest-value traffic in most systems.

resource — set or normalise resource attributes

processors:
  resource:
    attributes:
      - key: deployment.environment
        value: production
        action: insert

insert only fills a missing value; upsert overwrites what the application sent. Prefer insert unless you mean to override applications. See Resource attributes.

redaction and transform — keep sensitive values out

Telemetry pipelines are append-only, and there is no surgical delete once data has arrived. Masking in flight is the only place this is cheap.

processors:
  attributes/scrub:
    actions:
      - key: http.request.header.authorization
        action: delete
      - key: user.email
        action: hash

This is the mechanism behind the warning on Data retention.

k8sattributes — Kubernetes metadata

processors:
  k8sattributes:
    extract:
      metadata:
        - k8s.namespace.name
        - k8s.pod.name
        - k8s.node.name
        - k8s.deployment.name

Needs read access to pods. The Kubernetes page has the working manifests.

tail_sampling — keep the traces that matter

Only on a gateway Collector, and only where every span of a trace reaches the same instance. See Sampling.

Receivers beyond OTLP

The Collector can collect as well as receive, which is where host and infrastructure telemetry comes from:

Receiver	Produces
`otlp`	Whatever your applications send. The one you always have.
`hostmetrics`	CPU, memory, disk, filesystem, network for the host
`docker_stats`	Per-container resource usage
`kubeletstats`	Per-pod and per-container usage on a node
`filelog`	Log files, tailed and parsed
`prometheus`	Scrapes Prometheus endpoints, including the Collector’s own

This is why a Collector is worth running even when your applications export directly. Nothing an SDK does produces host CPU, disk pressure or container restarts, and those are frequently the explanation for what the application telemetry shows.

Environment variables and secrets

exporters:
  otlp:
    endpoint: ${env:OIQ_ENDPOINT}
    headers:
      X-License-Key: ${env:OIQ_LICENSE_KEY}

Never commit a license key into a Collector configuration file. Use environment variable substitution, and supply the value from a secret store, a Kubernetes Secret, or a 0600 environment file.A configuration file with a key in it tends to end up in a repository, in a container image, and in a support ticket.

Validate before restarting

otelcol validate --config=config.yaml

A Collector with an invalid configuration does not start. It logs the error and exits — it does not fall back to a previous configuration. On a gateway that every service exports to, that is an outage of your telemetry rather than a warning, so validate first.

Watch the Collector itself

The Collector exposes its own metrics, and two of them answer most questions:

Metric	Meaning
`otelcol_exporter_sent_spans`	Data is leaving. If this is zero, nothing is being exported.
`otelcol_exporter_send_failed_spans`	Exports are being rejected — check the log for the status code.
`otelcol_processor_dropped_spans`	Something in the pipeline is discarding data, usually the memory limiter.

Scrape them with the prometheus receiver and send them to aiAxonIQ, so your telemetry pipeline is itself monitored. A silently failing Collector otherwise looks exactly like an application that stopped producing traffic.

Troubleshooting

The collector starts, then logs 'Exporting failed. Will retry'

Read the error field on that line — it names the cause exactly.

Exporting failed. Will retry the request after interval.
  {"kind": "exporter", "name": "otlphttp/aiaxoniq",
   "error": "failed to make an HTTP request: Post \"…/v1/metrics\":
             dial tcp: lookup … : no such host", "interval": "5.2s"}

no such host — the endpoint hostname does not resolve. Check $OIQ_ENDPOINT against the value on Get Started.
connection refused — the host resolves but nothing is listening on that port.
401 Unauthorized — the key is missing, malformed or revoked.
404 — the endpoint already ends in /v1/…. It must be the base URL; the collector appends the signal path itself.

The collector retries with backoff and does not drop data while it retries, so a transient failure here is not a loss.

Nothing at all in the logs after 'Everything is ready'

The collector is running and receiving nothing. That is an application-side problem, not a collector one — your services are not exporting to it.Check that your application’s OTEL_EXPORTER_OTLP_ENDPOINT points at the collector’s OTLP port (4318 for HTTP, 4317 for gRPC), not at aiAxonIQ.

401 with a key you know is correct

Two causes that are not about the key’s value:

A trailing newline. A key read from a file created by a shell heredoc, or a Kubernetes Secret made with --from-file, carries the newline as part of the value. Use --from-literal, or printf rather than echo.
A validation outage. If the receiver cannot reach the service that validates keys it fails closed and returns the same 401. A sudden 401 across every service at once, with a key you have not changed, is far more likely to be this. Check $OIQ_ENDPOINT/health first.

Config changes appear to do nothing

The collector reads its configuration only at startup. Restart it after any edit, and confirm the file you edited is the one mounted into the process — a bind mount pointing at a path that does not exist silently yields the image’s default config rather than an error.

Exporters

Batching, retries, queueing and multiple destinations.

Sampling

Head and tail sampling, and the rule that keeps traces whole.

Kubernetes

A DaemonSet and gateway pair, with working manifests.

Overview

Platforms

OpenTelemetry

Other sources

Collector configuration

The minimum that works

The pipeline

Processors worth adding

Receivers beyond OTLP

Environment variables and secrets

Validate before restarting

Watch the Collector itself

Troubleshooting

Next

Exporters

Sampling

Kubernetes

​The minimum that works

​The pipeline

​Processors worth adding

​Receivers beyond OTLP

​Environment variables and secrets

​Validate before restarting

​Watch the Collector itself

​Troubleshooting

​Next

Exporters

Sampling

Kubernetes

The minimum that works

The pipeline

Processors worth adding

Receivers beyond OTLP

Environment variables and secrets

Validate before restarting

Watch the Collector itself

Troubleshooting

Next