> ## Documentation Index
> Fetch the complete documentation index at: https://docs.aiaxoniq.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Send data from Kubernetes

> A DaemonSet collector for node and pod telemetry, where the license key goes, and the two collectors most clusters end up needing.

aiAxonIQ has no Kubernetes agent of its own. It accepts OTLP, so the thing you
install is the upstream OpenTelemetry Collector, pointed at your ingest
endpoint. Everything below is stock chart configuration.

<Info>
  **Before you start, you need two values.**

  1. **Your base endpoint** — shown on **Get Started** in the dashboard. These
     pages write it as `$OIQ_ENDPOINT`.
  2. **A license key** — created in **Settings → License Keys**, starting
     `oiq_`. Requires the Admin role. See
     [Create a license key](/get-started/license-keys). These pages write it as
     `$OIQ_LICENSE_KEY`.

  Export both before running anything below:

  ```bash theme={null}
  export OIQ_ENDPOINT="https://app.aiaxoniq.com/otlp"   # or your own
  export OIQ_LICENSE_KEY="oiq_..."
  ```
</Info>

You also need a cluster you can install a Helm chart into, with `kubectl` and
`helm` configured against it.

## The shape most clusters need

There are two jobs, and they want different deployment modes.

<CardGroup cols={2}>
  <Card title="Per-node telemetry" icon="server">
    Container logs, host metrics, and OTLP from workloads on that node. One
    collector per node: `mode: daemonset`.
  </Card>

  <Card title="Cluster-level telemetry" icon="sitemap">
    Object counts, deployment replica status, events. Exactly one per cluster:
    `mode: deployment` with `replicaCount: 1`.
  </Card>
</CardGroup>

<Warning>
  Running the cluster receiver as a DaemonSet is the usual first mistake. Every
  node reports the same cluster-wide numbers, and the result is not wrong so
  much as multiplied by your node count.
</Warning>

Start with the DaemonSet. Add the second collector when you want workload status
alongside the telemetry.

## Store the key as a Secret

```bash theme={null}
kubectl create namespace observability

kubectl create secret generic aiaxoniq \
  --namespace observability \
  --from-literal=license-key="$OIQ_LICENSE_KEY"
```

The key is an ingest credential for the whole workspace. It belongs in a Secret
referenced by the collector, never in a values file committed to a repository.

<Tip>
  Use `--from-literal`, not `--from-file`. A file created by a shell heredoc
  carries a trailing newline, and that newline becomes part of the key value —
  producing a `401` that looks nothing like a formatting problem.
</Tip>

## Install the node collector

```bash theme={null}
helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
helm repo update

helm install aiaxoniq-node open-telemetry/opentelemetry-collector \
  --namespace observability \
  --values node-values.yaml
```

```yaml node-values.yaml theme={null}
mode: daemonset
image:
  repository: otel/opentelemetry-collector-contrib

extraEnvs:
  - name: OIQ_LICENSE_KEY
    valueFrom:
      secretKeyRef:
        name: aiaxoniq
        key: license-key

presets:
  # Container logs from the node, with pod/namespace/container metadata attached.
  logsCollection:
    enabled: true
  kubernetesAttributes:
    enabled: true

config:
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: 0.0.0.0:4317
        http:
          endpoint: 0.0.0.0:4318
    hostmetrics:
      collection_interval: 30s
      scrapers:
        cpu: {}
        memory: {}
        disk: {}
        filesystem: {}
        network: {}

  processors:
    memory_limiter:
      check_interval: 1s
      limit_percentage: 80
      spike_limit_percentage: 25
    batch:
      timeout: 5s
      send_batch_size: 1000

  exporters:
    otlphttp/aiaxoniq:
      # The base URL shown on Get Started in the dashboard. A self-hosted
      # deployment behind the bundled nginx serves OTLP under /otlp; a hosted
      # one does not. Use the value the dashboard gives you.
      endpoint: ${env:OIQ_ENDPOINT}
      headers:
        X-License-Key: ${env:OIQ_LICENSE_KEY}
      compression: gzip

  service:
    pipelines:
      logs:
        receivers: [otlp]
        processors: [memory_limiter, k8sattributes, batch]
        exporters: [otlphttp/aiaxoniq]
      metrics:
        receivers: [otlp, hostmetrics]
        processors: [memory_limiter, k8sattributes, batch]
        exporters: [otlphttp/aiaxoniq]
      traces:
        receivers: [otlp]
        processors: [memory_limiter, k8sattributes, batch]
        exporters: [otlphttp/aiaxoniq]
```

<Note>
  **`memory_limiter` goes first, `batch` goes last.** The order of `processors`
  is the order data passes through them. `memory_limiter` can only shed load it
  sees before anything has buffered it, and `batch` should group records after
  every other processor has finished changing them. A pipeline that batches
  first and limits afterwards will still OOM under the load the limiter was
  added for.
</Note>

## Add the cluster collector

```bash theme={null}
helm install aiaxoniq-cluster open-telemetry/opentelemetry-collector \
  --namespace observability \
  --values cluster-values.yaml
```

```yaml cluster-values.yaml theme={null}
mode: deployment
replicaCount: 1
image:
  repository: otel/opentelemetry-collector-contrib

extraEnvs:
  - name: OIQ_LICENSE_KEY
    valueFrom:
      secretKeyRef:
        name: aiaxoniq
        key: license-key

presets:
  clusterMetrics:
    enabled: true
  kubernetesEvents:
    enabled: true

config:
  exporters:
    otlphttp/aiaxoniq:
      endpoint: ${env:OIQ_ENDPOINT}
      headers:
        X-License-Key: ${env:OIQ_LICENSE_KEY}
      compression: gzip
```

The `clusterMetrics` preset is what populates the Kubernetes pages in the
product — clusters, nodes, namespaces, pods and containers are all derived from
`k8s.*` metrics. Without it those pages stay empty however much application
telemetry is flowing, because they are describing the cluster rather than the
workloads.

## Point applications at the node collector

Inside the cluster, applications export to their own node's collector rather
than to aiAxonIQ directly. The DaemonSet service address is stable:

```yaml theme={null}
env:
  - name: HOST_IP
    valueFrom:
      fieldRef:
        fieldPath: status.hostIP
  - name: OTEL_EXPORTER_OTLP_ENDPOINT
    value: "http://$(HOST_IP):4318"
```

This keeps the license key in one place — the collector — instead of in every
workload's environment, and it means the batching and retry behaviour is
configured once.

## Verify

First confirm the pods are up — one per node, plus one cluster collector:

```bash theme={null}
kubectl -n observability get pods
```

**Expected output:**

```text theme={null}
NAME                                 READY   STATUS    RESTARTS   AGE
aiaxoniq-cluster-6b9f7c8d4-x2kqp     1/1     Running   0          62s
aiaxoniq-node-4tzrw                  1/1     Running   0          71s
aiaxoniq-node-9mp2s                  1/1     Running   0          71s
```

A DaemonSet pod count that does not match your node count means a node is
tainted or short of resources — `kubectl describe` on the DaemonSet says which.

Then read the collector's own metrics:

```bash theme={null}
kubectl -n observability port-forward svc/aiaxoniq-node 8888:8888 &
curl -s localhost:8888/metrics \
  | grep -E 'otelcol_(receiver_accepted|exporter_sent|exporter_send_failed)'
```

**Expected output:**

```text theme={null}
otelcol_receiver_accepted_log_records{receiver="filelog"} 2841
otelcol_exporter_sent_log_records{exporter="otlphttp/aiaxoniq"} 2841
otelcol_exporter_send_failed_log_records{exporter="otlphttp/aiaxoniq"} 0
```

`send_failed` at zero with `sent` climbing is a working install. Then open
**Kubernetes** in the product, set the range to the last 15 minutes, and confirm
your nodes and pods are listed.

## Troubleshooting

<AccordionGroup>
  <Accordion title="401 in the collector logs" icon="key">
    The key did not reach the pod, or reached it with a trailing newline.

    ```bash theme={null}
    kubectl -n observability get secret aiaxoniq -o jsonpath='{.data.license-key}' \
      | base64 -d | wc -c
    ```

    A key is 36 characters. **37 means a trailing newline** — recreate the Secret
    with `--from-literal`, which does not add one, rather than `--from-file`.

    Then confirm `extraEnvs` names the same Secret and key.
  </Accordion>

  <Accordion title="Kubernetes pages are empty while application telemetry flows" icon="dharmachakra">
    Those pages are built from `k8s.*` metrics, which come from the **cluster**
    collector, not from your services or the DaemonSet. Confirm the
    `clusterMetrics` preset is enabled and the cluster collector pod is running.

    This is the most common Kubernetes install outcome that looks like a bug and
    is not.
  </Accordion>

  <Accordion title="Every cluster metric appears multiplied by the node count" icon="copy">
    The cluster collector is running as a DaemonSet. It must be
    `mode: deployment` with `replicaCount: 1` — every node is reporting the same
    cluster-wide numbers.
  </Accordion>

  <Accordion title="A DaemonSet pod is CrashLoopBackOff" icon="rotate">
    Almost always the config. The collector refuses to start on an invalid
    configuration rather than running degraded.

    ```bash theme={null}
    kubectl -n observability logs -l app.kubernetes.io/name=opentelemetry-collector \
      --tail=50 --previous
    ```

    `--previous` is the important flag — without it you read the logs of the
    container that is about to fail rather than the one that already did.
  </Accordion>

  <Accordion title="Applications cannot reach the node collector" icon="plug">
    Confirm `HOST_IP` is being injected. The `fieldRef` on `status.hostIP` must
    be declared **before** the variable that references it in the same `env`
    list, because Kubernetes resolves `$(VAR)` in declaration order.
  </Accordion>
</AccordionGroup>

<AccordionGroup>
  <Accordion title="The collector starts, then logs 'Exporting failed. Will retry'" icon="arrows-rotate">
    Read the `error` field on that line — it names the cause exactly.

    ```text theme={null}
    Exporting failed. Will retry the request after interval.
      {"kind": "exporter", "name": "otlphttp/aiaxoniq",
       "error": "failed to make an HTTP request: Post \"…/v1/metrics\":
                 dial tcp: lookup … : no such host", "interval": "5.2s"}
    ```

    * `no such host` — the endpoint hostname does not resolve. Check
      `$OIQ_ENDPOINT` against the value on **Get Started**.
    * `connection refused` — the host resolves but nothing is listening on that
      port.
    * `401 Unauthorized` — the key is missing, malformed or revoked.
    * `404` — the endpoint already ends in `/v1/…`. It must be the **base** URL;
      the collector appends the signal path itself.

    The collector retries with backoff and does not drop data while it retries,
    so a transient failure here is not a loss.
  </Accordion>

  <Accordion title="Nothing at all in the logs after 'Everything is ready'" icon="ear-listen">
    The collector is running and receiving nothing. That is an application-side
    problem, not a collector one — your services are not exporting to it.

    Check that your application's `OTEL_EXPORTER_OTLP_ENDPOINT` points at the
    collector's OTLP port (`4318` for HTTP, `4317` for gRPC), not at aiAxonIQ.
  </Accordion>

  <Accordion title="401 with a key you know is correct" icon="key">
    Two causes that are not about the key's value:

    * **A trailing newline.** A key read from a file created by a shell heredoc,
      or a Kubernetes Secret made with `--from-file`, carries the newline as
      part of the value. Use `--from-literal`, or `printf` rather than `echo`.
    * **A validation outage.** If the receiver cannot reach the service that
      validates keys it fails closed and returns the same `401`. A sudden `401`
      across every service at once, with a key you have not changed, is far more
      likely to be this. Check `$OIQ_ENDPOINT/health` first.
  </Accordion>

  <Accordion title="Config changes appear to do nothing" icon="file-pen">
    The collector reads its configuration only at startup. Restart it after any
    edit, and confirm the file you edited is the one mounted into the process —
    a bind mount pointing at a path that does not exist silently yields the
    image's default config rather than an error.
  </Accordion>
</AccordionGroup>

## Next

<CardGroup cols={3}>
  <Card title="Verify your data" icon="circle-check" href="/get-started/verify-data">
    Confirm it arrived.
  </Card>

  <Card title="Instrument the applications" icon="code" href="/send-data/otel/zero-code">
    Zero-code setup per language.
  </Card>

  <Card title="Endpoints and errors" icon="list" href="/send-data/endpoints">
    Every status code and limit.
  </Card>
</CardGroup>