Log ingest rules

Logs are usually the biggest and the least curated thing you send: health checks every few seconds, debug output nobody reads, request headers with credentials in them. Ingest rules decide what is kept, before anything is stored. Open Logs → Ingest rules. Everyone in the account can read the rules (they explain why a field is missing from a log); only an admin can change them.

​

Where the rules run

Rules run in the aiAxonIQ receiver, as each log arrives and before it is queued, stored, indexed for search or shown in live tail. Something a rule drops or removes is never written anywhere in aiAxonIQ, so it costs nothing to store and cannot leak from a later copy. A saved change applies to logs received more than 30 seconds after the save. Logs already stored are not changed — to shorten how long stored data is kept, use retention. The order is fixed:

1. Drop rules, checked on the log exactly as it arrived.
2. Field rules on the logs that are kept.
3. Detectors on the message and attribute values.
4. aiAxonIQ’s own credential redaction, which always runs and which your rules cannot turn off.

​

Drop logs

A drop rule is a list of conditions; a log that matches every condition of any one rule is not stored, searched or counted. Operators: equals, does not equal, contains, does not contain, starts with, ends with, exists, does not exist, and > ≥ < ≤ for numbers. Examples:

• Health checks: http.route equals /health
• Heartbeats: message contains heartbeat
• Debug noise from development: level equals debug and deployment.environment equals development
• A noisy service: service.name equals health-check

There are no regular expressions. A pattern that backtracks badly on one large log could slow ingest for everyone, and the operators above cover the common cases; this may change when a linear-time matcher is available.

​

Remove or sanitize fields

A field rule names a field and what happens to it: The field is a dotted path, matched ignoring case, in:

• log attributes, including nested ones (request.headers.authorization);
• resource attributes;
• a JSON message, by the same path ({"user":{"email":…}} is user.email);
• key=value pairs in a text message (password=…).

Start a path with *. to match the key at any depth: *.password matches password, user.password and request.body.password.

​

Fields that cannot be changed

Some fields are what the rest of the product is built on. A rule may read them in a drop condition, but cannot remove or rewrite them:

• service.name, service.version, deployment.environment, and the host.*, os.*, cloud.*, k8s.*, container.* and telemetry.sdk.* resource attributes — services, hosts and Kubernetes views read them;
• trace_id, span_id and their other spellings, and event.name — logs are linked to traces through them;
• the message, level and timestamp as whole fields;
• any field a log retention rule matches on — rewriting it would change how long those logs are kept. The retention page refuses the reverse, too.

The page lists them and refuses a rule that names one.

​

Detectors

Detectors find a kind of value wherever it appears in the message or an attribute value, whatever the field is called:

• Email addresses
• Card numbers — 13 to 19 digits that pass the Luhn check, so order numbers, ids and timestamps are left alone.

They are off by default. A detector that matches too much destroys exactly the data you debug with, so try it in the preview with your own logs first. Phone numbers are deliberately not offered: every pattern we tried also matched ids, ports and durations.

​

Preview before you save

The Preview panel runs a sample log through the rules you are editing — unsaved changes included — with the same code the receiver runs. It shows whether the log would be dropped, every field removed or sanitized and where, and the log as it would be stored. Paste a real log from your services to check a rule before it affects anything. A save that drops logs or removes fields asks you to type drop data: data a rule drops or removes is never stored and cannot be recovered.

​

What the rules saved

The top of the page shows the last seven days, as counted by the receiver: logs checked, logs dropped (and their share), fields removed, values sanitized, and the matches for each rule. The sizes are the dropped logs and removed fields as uncompressed JSON when they arrived. Storage is compressed, so the disk saved is smaller than that, and it is not measured — the page does not claim a disk saving it cannot show.

​

History

Every save is a new version: who saved it, when, and what it held. Restore brings back an older version as a new one, so the history is never rewritten. Every save is also recorded in the account’s audit log.

​

If the rules cannot be loaded

If the receiver cannot load your rules — for example during an outage — it keeps using the last rules it saw. If it has never seen them, it refuses the request with 503 (gRPC UNAVAILABLE) and a Retry-After, rather than storing logs without your rules. OpenTelemetry exporters retry these automatically, so logs are delayed, not lost and not stored unsanitized.

​

Not covered yet

• Browser logs from the RUM agent and AgentSight events.
• Traces and metrics.
• Logs that fail to parse are rejected before any rule could run on them.

​

Related