SLO vs SLI Instrumentation for High-Cardinality Event Streams

High-cardinality event streams make SLO instrumentation a choice about which evidence survives. Metric-based SLOs keep alerting simple, while event-based SLOs preserve the detail needed to explain breaches. ObsessionDB is an option for keeping that event data queryable at scale, but storage alone cannot settle the choice. Accurate, debuggable SLIs depend on preserving the right dimensions before aggregation destroys them.

Why the canonical SLI definition breaks at high cardinality

An SLI looks deceptively simple, namely good events divided by total events over a defined measurement window. The numerator says what counts as good, and the denominator says which events count at all. An SLO sets a target for that measure. The formula seems straightforward because it leaves the collection pipeline out of view. Yet the ratio is only as sound as the inputs that pipeline delivers.

With user ID, customer tier, feature flag, and request path as labels, each unique label combination becomes a distinct series. Tracking every combination is expensive, while collapsing them removes distinctions that may explain a breach. A service-wide result can therefore be accurate for the whole service without revealing a serious problem for one customer segment. The arithmetic still works, but the assumption that one compact ratio preserves the relevant service experience does not.

A later investigation can isolate a deployment variant or request class only if the relevant dimensions were retained at instrumentation time. Cardinality also puts pressure on more than disk space. A surge in unique labels can saturate indexing state or a series-ID allocation service, creating ingestion backpressure before storage fills. The pipeline must handle both the volume of measurements and the growing number of identities attached to them.

How metric-based SLOs trade dimensional richness for operational simplicity

Metric-based SLOs resolve much of that pressure by evaluating pre-aggregated counters, including through a “good minute or bad minute” classification. When cardinality is kept in check, each series costs little to store and alerting works on a small stream that keeps flowing. This approach fits naturally into Prometheus-lineage stacks. Tools that support metric-based and monitor-based SLI definitions also link reliability dashboards to the traces and incidents that go with them. For an established, stable definition of service health, those are substantial operational advantages.

The Google SRE Workbook gives a familiar starting point for multi-window burn-rate alerting. Its fast-burn parameters are 14.4× over one hour confirmed over five minutes, while medium burn is 6× over six hours confirmed over a short confirmation window. These parameters assume continuous, low-latency measurement. They are not a universal guarantee that any pipeline producing a ratio can support timely alerts.

The limitation is what the counter no longer contains. A counter grouped only by status code cannot later be sliced by customer tier or feature flag. If a changed threshold needs detail that was thrown away, changing a query cannot bring it back either. Re-instrumentation can make future measurements better, but it cannot fix the record of the past. Metric-based SLOs therefore offer simplicity when the retained dimensions answer the important questions.

How event-based SLOs preserve investigative surface and what that costs

Event-based SLOs keep the evidence by evaluating individual structured events. When an event has status code, duration, user information, and feature flags, SLI definitions can come straight from those fields. Thresholds can change after collection, and retained dimensions can support new definitions without another instrumentation cycle. This late binding lets an investigation move from a breach to the users, tenants, or request classes affected.

Seznam.cz DevOps moved from a Prometheus-only SLO implementation to processing individual events through slo-exporter at the system edge. That move reduced cardinality issues and removed complex SLO classification recording rules from Prometheus. It also enabled real-time root cause analysis broken down to application and endpoint.

Keeping that detail has a real cost. Storage and ingestion costs depend on event volume and cardinality as well as the number of series. During an incident, event volume can spike just as engineers need to concentrate on resolution. Longer retention adds another tradeoff, since cold object storage lowers retention cost but brings retrieval latency and per-GB retrieval charges when archived events are needed. Event-based SLOs preserve a broader investigative surface, but the team must pay to collect it, retain it, and make it usable.

The aggregation decision: where information is permanently destroyed

The decisive choice happens before alert configuration, when the pipeline emits a counter or preserves a structured event. Once a field has been discarded, no downstream query can reconstruct it from the aggregate. That makes aggregation an evidence policy as much as a storage policy. Alerting requires both detection of elevated errors and retained data that can explain whose requests failed and what those requests shared.

Metrics describe what happened and how much, such as an elevated error rate or high p99 latency. Structured events keep the detail about each request needed to explain why, while traces show where an error or delay happened within a distributed call. None replaces the others. The mistake is expecting a cheap aggregate to answer a question whose necessary dimensions exist only in the raw event.

Normalization makes the same tradeoff on a smaller scale. Replacing parameterized URLs with route patterns, IP addresses with subnets, or session IDs with time-bucketed groups reduces detail before storage. Those choices should be explicit for each dimension because each changes the questions a breach investigation can answer. ObsessionDB separates compute from durable storage and caches frequently accessed dimensions locally through its cache-mesh architecture, keeping high-cardinality slicing feasible without collapsing those dimensions first.

Burn-rate alerting assumptions that break when SLI freshness is not guaranteed

Preserving events does not by itself make their measurements timely. A burn-rate window shows how the service is behaving now only if the data that feeds it arrives and is checked promptly. Batch SQL queries against an event store introduce a measurement schedule, while queries routed through cold storage introduce retrieval delay. In either case, a mathematically correct window can describe past conditions. The alerting model inherits the freshness of its inputs.

A fast-burn alert meant to fire within minutes may arrive after the incident has moved on. A delayed ingestion spike can also produce an alert after ingestion has recovered, leaving responders to interpret a retrospective signal as a live one. The distinction matters because the alert may be correct about the recorded events while misleading about current conditions. Reliability status and measurement status are not the same thing.

A freshness SLO makes this dependency explicit through a percentile ingest-lag target over a rolling window. It adds a second commitment to underwrite the first, so its complexity belongs in the pipeline design, before the incident review. Always-hot caching, including the approach used by ObsessionDB, can remove cold-storage round-trips from frequently accessed event queries. That addresses the query-latency part of freshness, but it does not make ingestion lag disappear. Both collection and evaluation determine how current the SLI really is.

Instrumentation decisions that determine whether SLIs are both accurate and debuggable

Instrumentation begins with user-facing behavior. Successful payment authorizations divided by total attempts is a clear place to start because it states what outcome is being measured. If teams may need to look into that outcome by customer tier, feature flag, or request class, keep a structured event at the system edge. Measurement close to the user provides an authoritative source for SLO events without relying on classification scattered deep in the call stack. OpenTelemetry auto-instrumentation can cover common frameworks, with manual instrumentation reserved for custom business logic it cannot reach.

A route pattern can keep what a path tells teams without keeping every parameter value, while customer tier should keep all its detail. Standard resource attributes, including service name, namespace, and deployment environment, let metrics, traces, and events correlate without label mismatches. Dimensions needed only for investigation can live in a flexible event tail with no indexed labels. That costs less to index and still lets teams query the data.

Evaluation placement determines how that evidence becomes an alert. Collection-time classification can produce unified SLI metrics with controlled cardinality and keep classification logic out of the query layer. Query-time evaluation lets teams use late-bound definitions, but it needs low-latency access to the events they have kept. A hybrid emits both a pre-classified counter for burn-rate alerts and a structured event for investigation. It also needs the same classification across both pipelines.

Treat the fraction of expected events arriving within the measurement window as an SLI, and alert on sustained gaps. Publish the telemetry freshness SLO so dependent teams can see its parameters. A missing event is an ingestion failure. If backpressure drops failures from the denominator, the apparent improvement in service health may be a measurement defect at the moment it matters most.

Storage and query architecture for event-level SLI data at scale

Ingestion is sustained and append-heavy, while incident queries are ad hoc, high-dimensional, and sensitive to latency. Compliance review needs long retention but infrequent access. No single storage tier optimizes all three. The practical choice is how to keep active investigation fast without treating every retained event as equally urgent.

Typed columns are central to that choice. Bare untyped storage is orders of magnitude slower for SLI queries at scale. Frequently queried fields, including status code, duration bucket, and customer tier, should be extracted into typed columns at ingestion, with variable tails kept in flexible storage. This preserves adaptability without forcing routine SLI evaluation through the least structured part of the event. Retaining a dimension is useful only if the store can query it fast enough during a breach.

A hot-data cache addresses the next constraint by avoiding repeated retrieval from cold object storage. For this workload, ObsessionDB is a strong fit among event-store approaches because its cache mesh keeps hot data in memory and never routes queries through S3. Its p99 latency for ad hoc customer-segment investigations stays flat as event volumes grow into the hundreds of terabytes. Compute-hour pricing also avoids penalizing high-cardinality instrumentation or exploratory queries as separate dimensions of usage. These properties address the cost and latency of event-based investigation, rather than changing which SLI should be measured.

Retention remains a separate decision. Cold tiers can make long review windows affordable, but repeated retrieval during an investigation adds both cost and delay. Teams should set retention windows from actual compliance review and incident replay needs. The goal is accessible evidence retained for a defined period with a plan for using it.

How agentic SLO pipelines change the instrumentation calculus

Agent access adds an accountability and isolation requirement to that storage design. Every autonomous query needs a record of what was asked, by which agent, and at what time. Without that record, an unusual cost spike or access pattern becomes indistinguishable from general load. Investigation compute also needs isolation from production traffic so parallel probing does not degrade the service being measured. ObsessionDB supports isolated, read-only compute over shared data with full query logging for this access pattern.

Isolation alone does not make existing SLIs describe agent workloads correctly. Human-shaped instrumentation assumes bounded concurrency, predictable query plans, and deliberate session structure. Agents instead choose database calls based on tool descriptions and the task at hand, which leads to non-deterministic call trees and exploratory queries that run in parallel. Each additional agent multiplies possible decision paths. Classification designed around human queries can therefore undercount failures in query classes it has never represented.

A pipeline must count the work actually attempted while retaining enough detail to attribute failures to the workload that produced them. Otherwise, agent investigations can both escape the measurement and disturb the system behind it. The durable architectural choice remains the one made at instrumentation time.