Skip to main content
aiAxonIQ stores four kinds of telemetry. They are not interchangeable, and the value of having all of them comes almost entirely from being able to move between them — which depends on two attributes you control.

​
The four signals

Logs

Discrete events with a body, a severity and a timestamp. Answers what happened, in the words your code chose.

Metrics

Numeric samples over time, with labels. Answers how much, how often, how slow — cheaply, over long windows.

Traces

A causally-linked tree of spans across services. Answers where the time went and which hop failed.

Synthetic results

Outcomes of checks run against your endpoints from outside. Answers is it up, from somewhere that is not inside your network.
Choosing between them for a given question is most of the skill:

​
The two attributes that join them

Correlation is not automatic. It works because records carry matching values, and the values come from your instrumentation.
1

service.name — the join across signals

The name your service reports itself as. It appears on logs, spans and metrics, and it is what the service inventory, the service map, and every “show me everything for this service” view are built on.
2

trace_id — the join across services

Propagated from caller to callee on every hop. It is what turns twelve unrelated spans into one trace, and what lets a log line link to the request that produced it.
A log line only links to a trace if the trace context was in scope when it was written. OpenTelemetry logging integrations do this automatically; a print() or a bare file logger does not, and its lines will be correct, searchable, and permanently disconnected from every trace.If “view logs for this span” is empty on a service you know is logging, the logger is the thing to look at — not the trace.
service.name is unset by default, and the default is unknown_service. Every service that does not set it collapses into a single entry that appears to be enormous and is not anything. Set it before you set anything else. See Resource attributes.

​
Resource attributes versus record attributes

Two levels, and the distinction decides both what you can filter on and what you pay for.
  • Resource attributes describe the emitter and are attached to every record it produces: service.name, service.version, deployment.environment, host.name, k8s.pod.name.
  • Record attributes describe one event: http.method, http.status_code, db.system, exception.type.
Both are searchable. In the log query syntax, the named fields — level, service, body, trace_id, host, version and their aliases — are promoted columns; anything else you write falls through to an attribute lookup. That is why http.status_code:>=500 works without anything being declared in advance.
An unrecognised field name is not an error, it is an attribute lookup that matches nothing. levle:error is a perfectly valid query for an attribute called levle. Re-read a zero-result query for a typo before concluding the data is missing. Attribute keys are case-sensitive.

​
Cardinality is the cost you cannot see

A metric’s cost is driven by the number of distinct label combinations, not by the number of samples.
The second is the same data with one label added, and it can be a thousand times more expensive. Every unique combination is a separate stored series, forever.
Never put a user id, request id, session id, full URL path or raw error message in a metric label. Put them on a log record or a span attribute, where per-event data belongs and costs what one event costs.The classic instance is an un-templated URL: /orders/8817 and /orders/8818 are two series. Report the route pattern — /orders/{id} — and use a span attribute for the actual id.

​
Derived data you did not send

Some of what you see is computed rather than received. It is worth knowing which, because it explains why some things appear a minute late and why some survive longer than the data behind them.

​
Next

Resource attributes

The attributes worth setting, and the ones that cost you.

Searching logs

The full query syntax and the complete field list.

Platform architecture

Which store answers which question.