The two that matter most
deployment.environment is the second. Without it, staging traffic and
production traffic are the same service, and every chart is the sum of the two.
Get Started in the dashboard shows the exact base endpoint for your
deployment next to a license key you create there, with copy buttons, and
then watches for your first records. Where these pages write
$OIQ_ENDPOINT, that page has the real value.The attributes aiAxonIQ builds on
Set these and the product works as designed. Omit them and specific features degrade in specific ways.
On Kubernetes, pod, namespace, node and workload attributes are added by the
Collector or the zero-code agent — you do not set those by hand. See
Kubernetes.
Naming services
Names are what everyone will read at 3am, so it is worth spending five minutes on a convention:1
Use the name the team uses
checkout-api, not svc-prod-ckt-01. If nobody says it out loud, it is
the wrong name.2
Keep it stable across environments
The same service in staging and production should carry the same
service.name and differ by deployment.environment. Naming them
checkout-api-staging and checkout-api-prod makes them two services
forever, and no comparison between them is possible.3
Do not encode the instance
A pod name or host in
service.name produces one service per replica.
Instances belong in host.name and the Kubernetes attributes.Resource attributes versus span attributes
Two levels, and the distinction decides both what you can query and what you pay.- Resource attributes describe the emitter and are attached to every record it produces. Set them once, at startup.
- Span and log attributes describe one event —
http.method,http.status_code,order.id. Set them per operation.
Put per-request values on the span, never on the resource. A resource
attribute is repeated on every record the process emits, so a request id
there is both useless and expensive. The rule is: does this value change
between two requests? If yes, it is a span attribute.
Cardinality: the cost you cannot see
For metrics, every distinct combination of label values is a separate stored series. This, not the number of samples, is what drives cost.Setting them
Environment variables — works with every SDK
Environment variables — works with every SDK
The portable way. Every OpenTelemetry SDK reads these without any code.Comma-separated
key=value pairs. OTEL_SERVICE_NAME takes precedence
over a service.name inside OTEL_RESOURCE_ATTRIBUTES.In the Collector — for telemetry that arrives without them
In the Collector — for telemetry that arrives without them
Useful when a source you do not control emits nothing useful, or when you
want to guarantee a value regardless of what an application sets.
upsert overwrites what arrived; insert only fills in a missing value.
Prefer insert unless you specifically intend to override applications.In Kubernetes — from the pod spec
In Kubernetes — from the pod spec
Set the variables from the downward API so they follow the workload rather
than being pasted per deployment.
Conventions worth following
OpenTelemetry publishes semantic conventions — agreed names for common attributes. Following them is what makes tooling work without configuration.Custom attributes are fine — prefix them with something of your own, like
acme.tenant_tier, so they never collide with a convention added later.
Attribute keys are case-sensitive throughout.Verify what actually arrived
Instrumentation intent and reality diverge. Confirm:1
Open Services
Your service should be there under the name you chose. An
unknown_service
entry means service.name did not reach us from something.2
Open a log record or a span and read its attributes
Resource attributes are shown alongside record attributes. This is the
fastest way to catch a typo in an attribute name.
3
Filter on one
Next
Sampling
Keep the traces that matter and drop the rest.
Collector configuration
Where to add, rename and redact attributes in flight.
Signals and the data model
How attributes connect logs, metrics and traces.