Skip to main content
For services on a Kubernetes cluster. This path has the most decisions that are painful to change later, so it is worth an hour of planning before the first helm install.
About an hour to first data, a day to something useful, a week or two to a team using it. Most of that is not installation.

​
The shape

Two tiers, and this is the standard arrangement rather than an advanced one:
1

A DaemonSet — one Collector per node

Collects node and pod telemetry, receives OTLP from workloads on that node, and attaches Kubernetes metadata. Applications export here.
2

A gateway — a Deployment of a few replicas

Everything the DaemonSets collect goes here. Holds the license key, and is the one place tail sampling, redaction and filtering are configured.
Why two tiers rather than one. Kubernetes metadata must be attached on the node that ran the pod, which only a DaemonSet can do. Tail sampling requires every span of a trace to reach the same instance, which only a gateway can guarantee. Neither tier can do the other’s job.It also means the license key lives on a few gateway pods rather than on every node.
Working manifests are on Kubernetes.

​
The four decisions to make first

1. Cluster name. Set one on every cluster reporting to the organization. Without it, two clusters are indistinguishable — the same namespace and service names from two places, silently merged. Correcting it later means every historical record carries the wrong value.
2. Environment separation. A deployment.environment attribute in one organization, or two organizations. The attribute is what most teams want; separate organizations are the answer only when the requirement is access control. See Organizations and multi-tenancy.
3. Service naming. From the pod spec, via OTEL_SERVICE_NAME — not inferred. A Deployment name is a reasonable default, a pod name is never one: a pod name in service.name produces a new service per replica, forever.
4. Sampling. Decide before the volume arrives, and enable tail sampling only on the gateway. Enabling it on the DaemonSets as well means each node decides on the fragment of a trace it happens to hold, producing broken traces rather than fewer traces. See Sampling.

​
Day one

1

Prove the path

Quickstart from inside the cluster — a pod with curl. This proves cluster egress, which is the constraint that most often surprises people.
2

Store the license key as a Secret

Never in a ConfigMap, never in a values file you commit. Only the gateway needs it.
3

Deploy the gateway, then the DaemonSet

In that order — a DaemonSet with nowhere to forward to just buffers and then drops.
4

Confirm Kubernetes metadata is attached

Open a record and check it carries pod, namespace and node. If not, the Kubernetes attributes processor lacks read access to pods — see Infrastructure and Kubernetes.
5

Point one workload at the DaemonSet

Then verify, before rolling it out to everything.

​
Week one

Every service one real request touches. Then open a trace and confirm it spans all of them — a hole means propagation broke or sampling is inconsistent. See Traces with a hole in them.
And include the memory limiter processor, first in every pipeline. Without it a traffic spike can push a Collector into the node’s out-of-memory killer — and on a shared node, what dies may not be the Collector.See Collector configuration.
Kubernetes probes every pod constantly. On a cluster this is frequently the largest single share of telemetry volume, and nobody has ever read one.
Scrape their own metrics and send them along. A silently failing gateway looks exactly like a quiet cluster.
The rule that catches a broken pipeline. See Alerting.

​
Coverage for what you cannot instrument

Every cluster has them: vendor images, a legacy service nobody owns, something in a language no one left uses. The eBPF agent observes them at the kernel with no code change, as a second DaemonSet.
It requires kernel capabilities on every node, so it needs a security conversation rather than only a technical one — and on EKS and AKS it needs a node sysctl set, or it silently collects nothing.It is early access and not integrated with the product. See zero-code on Kubernetes and the security model.
It does not work on Fargate, GKE Autopilot or virtual nodes. No node access means no DaemonSet. If you run there, the SDK is the only route.

​
Next

Kubernetes install

The working manifests.

Sampling

Gateway tail sampling, done correctly.

Enterprise

Many teams, single sign-on and a rollout plan.