Compute-Storage Separation Economics in Analytical Databases

Compute-storage separation gets talked about as a feature a platform either has or doesn't. That framing misses the point: cost and latency depend on what mediates between compute and storage, which varies wildly across architectures, and a query can return far faster on one system than on another with the same "separated" label on the box, with the difference having nothing to do with whether separation exists. It has to do with what sits in the gap.

Picture the moment an engineer gets paged because a dashboard query that normally takes a second is now taking twenty. Nothing in the architecture diagram changed. The data volume barely moved. What changed is contention somewhere in the path between compute and the bytes it needs, and depending on where that path leads, the fix is either a quick cache tweak or a rearchitecture.

Three positions anchor this spectrum. At one end sits raw object storage, queried directly, with nothing structured standing between the compute engine and the bytes. In the middle sit page-server meshes that keep local caches and serve requests without touching the object store every time. At the far end, you find NVMe-backed, in-memory cache layers built specifically to keep hot data close to compute. Each occupies a different point on the latency-cost curve, and none of them is free.

The most capable systems on this spectrum treat that mediation layer as structured infrastructure. An addressable layer maps logical files to physical storage locations, and it can be reconfigured without the compute side ever noticing. This was a deliberate engineering choice: the people who build these systems for a living have already concluded that the gap between compute and storage is the thing worth engineering, not an afterthought. ObsessionDB, a managed analytical database service whose cache-mesh architecture keeps hot data in distributed NVMe memory and never routes queries through S3, is one concrete example of that philosophy applied at terabyte-to-petabyte scale.

Someone might reasonably object here: every modern cloud warehouse already separates compute and storage, so isn't this distinction academic? It would be, if separation were implemented the same way everywhere, and it isn't. OvalEdge's 2026 comparison of six major platforms, Snowflake, BigQuery, Redshift, Databricks, Azure Synapse, and Microsoft Fabric, found that every one of them claims compute-storage separation, and their latency and pricing behavior still diverge sharply. The label is shared, but the implementation underneath it is not.

What direct S3 routing costs at query time

Object storage is cheap to hold data in. It is not necessarily cheap to query, and the two facts get confused constantly. The byte price on S3 is low. The cost that shows up on the bill comes from somewhere else entirely: the protocol overhead paid every time compute reaches out to fetch something.

S3 runs on HTTP/1.1, a protocol that introduces head-of-line blocking and carries more TCP overhead per request than newer alternatives. Google Cloud Storage, by contrast, benefits from HTTP/2's multiplexing and header compression. Connection setup, request queueing, and timeout handling eat into every single query, and none of that work shows up as a line item anyone budgets for in advance.

The expense comes not from failed requests but from idle compute. A query engine that's waiting on a connection to establish, or waiting in a queue behind other requests, is still running. The cloud provider bills for that time whether the compute is doing useful work or just waiting. That idle time is the real cost, and it gets billed regardless of outcome.

This tax scales with the number of requests, not the size of the dataset, so it doesn't shrink as data grows, and it compounds hard the moment queries run concurrently or an automated workload fires many short probes in parallel.

BigQuery's on-demand pricing model makes the related problem of scan amplification easy to see. OvalEdge's 2026 comparison confirms the on-demand rate runs $6.25 per terabyte scanned, and a single unoptimized query against a large table can rack up hundreds of dollars in a single run. Keeping that cost under control takes disciplined partitioning and clustering, and building that discipline into a team's query habits is itself ongoing engineering work.

Redshift Spectrum offers a related capability: it queries S3 directly without first loading data into the warehouse, which is genuinely useful for teams already inside the AWS ecosystem. Provisioned Redshift clusters bill whether queries are running or sitting idle, so reaching out to S3 only relocates the idle-compute cost. It just moves where the idle time happens.

A fair pushback here: Athena-on-S3 stacks are supposed to be radically cheaper than full warehouses, so doesn't that undercut the whole argument? Partly. They're cheaper on storage, and cheaper on light, infrequent workloads. If query frequency rises, the protocol tax and idle compute both come back, because the arithmetic that made the light workload cheap depends entirely on how rarely compute has to reach out and pay that tax.

How page-server and cache-mesh architectures interrupt the tax

Moving up the spectrum changes the math. A structured mediation layer, whether built as page servers with local caches or as an NVMe-backed mesh, breaks the direct dependency on object-storage protocol latency, and that changes the cost curve for any workload touching hot data repeatedly.

The design pattern behind this is concrete: a Microsoft patent, US 12,499,122 B2, covers the separation of logical and physical storage in a distributed database system. The mechanism it describes: a logical file can move to a different physical storage configuration while the compute layer's addressing stays untouched. Compute never has to know the move happened, because reconfiguration happens at the mediation layer instead of forcing a change on every client that touches the data.

An earlier patent in the same family lays out the mechanics further: page servers can use striped and strided layouts, spreading data across multiple servers so retrieval happens in parallel rather than serially through a single endpoint. A single S3 bucket, fundamentally, is one endpoint. A page-server mesh is many, working together.

The economic payoff follows directly from that design. Hot data served from a cache or from NVMe skips the protocol tax entirely and skips the idle-compute cost that comes with it. The query finds what it needs close by, so compute cycles go toward actual computation instead of waiting on a network round trip, and p99 latency stays flat even as data volume climbs.

This isn't a theoretical benefit. Dynatrace built its data lakehouse, called Grail, on compute-storage separation with exactly this kind of mediation layer in place. A vendor operating at production scale chose to build its own data backbone this way, a strong signal the gains show up in practice and not just in a whitepaper. Separately, Databend's architectural writing makes a related point: moving metadata into its own independent service lets multiple compute clusters work against the same underlying data without colliding.

One objection deserves a direct answer here: doesn't adding a cache layer just add operational complexity and a new failure point? It adds complexity, but a managed cache mesh shifts that operational burden away from the engineering team running queries day to day. And the failure modes of a well-designed cache layer tend to be more predictable and more bounded than the failure modes of S3 protocol variability under high query concurrency. Trading an unpredictable tax for a bounded, manageable one is most of the appeal.

How workload mix determines which mediation strategy wins

None of this makes cache meshes or page servers a universal answer. The right mediation strategy depends on what the workload actually looks like, not on which architecture happens to be the most sophisticated on paper.

Consider a team running SQL once a day against a modest dataset, touching different slices of data each time. A sophisticated cache layer does nothing useful for that team. Keeping data hot in memory costs money and engineering attention, and if queries never return to the same hot set twice, that investment just sits there unused.

There's a mirror version of this problem on the other side. Provisioning always-on compute so queries are instantly ready costs money too, and warehouses often stay provisioned continuously at a premium that genuinely batch-oriented workloads never earn back. Paying for readiness nobody needs is its own kind of waste, just as real as paying the S3 protocol tax.

So which workload actually stresses the mediation layer the hardest? Not the big overnight batch job. The pattern that matters most is high-concurrency, low-latency, hot-data access: customer-facing analytics dashboards, real-time risk engines, observability query streams, and increasingly, automated workloads that fire many short probes in parallel.

Two brief examples show what a well-designed intermediary layer can do for that pattern specifically. Apache Pinot's Star-Tree index pre-computes aggregations inside the mediation layer itself, so a query across billions of rows can finish in a fraction of the time a naive scan would take. The speed doesn't come from faster disks. It comes from restructuring what the intermediary layer already holds, so the query barely has to compute anything new. Apache Iceberg's hidden partitioning works on a related but different principle: Trino can automatically skip irrelevant data files using partition metadata, so a date-filtered query only touches the files it actually needs. Partition pruning inside the intermediary layer cuts S3 scan volume even without a cache sitting anywhere in the path.

Neither example is meant as the final word on either technology. Both are illustrations of one underlying principle: how the intermediary layer is designed shapes cost, independent of whatever raw storage sits underneath it. And the workload profile that rewards a cache mesh most, high-frequency, low-latency, hot-data access, happens to be the same profile that's growing the fastest across the industry: real-time systems and automated, agent-driven workloads.

How agentic query patterns strain S3-routed architectures

That last point deserves its own close look, because AI agents aren't just one more workload type added to the mix. They generate a query pattern that looks nothing like what human analysts produce, and that pattern runs straight into the protocol tax described earlier, repeatedly and in parallel.

Research on the Data Intelligence Agents (DIA) system lays out what this looks like in practice. Agents fire lightweight queries against schema and statistics just to confirm a join key actually exists. They sample column values to figure out formatting. They check cardinality assumptions directly against the data instead of guessing from column names the way a rushed analyst might. None of this is one query. It's a sequence of small probing queries, and agentic explorers typically keep no memory across sessions, so every new session starts the probing over from scratch.

Compare that to how retrieval-augmented generation worked as recently as 2023 and 2024, when a typical system made one retrieval call per conversational turn. Agentic RAG systems in 2026 make multiple dynamic retrieval calls per turn, and that alone multiplies total query volume well beyond what the earlier generation produced. TiDB reports that a large majority of new daily TiDB Cloud clusters are now created by AI agents rather than human users: agents are already the primary database user in a lot of production environments right now, not a future workload to plan for.

MotherDuck has pointed to serverless, scale-up columnar engines as a better fit for agent workloads specifically because agents need sub-second interactive latency. An agent doesn't wait around the way a human does. A human tolerates a few seconds of cold-start latency on a dashboard without much complaint. An agent firing dozens of probe queries per task does not have that patience built in, because the whole point of the agent is to move fast through a decision loop.

This is where the earlier protocol tax compounds in a way that's worse than anything a human workload produces. A human analyst fires one query and waits for the result. An agent fires many probes at once, and each one of those probes pays its own TCP setup cost and its own queueing delay. The idle-compute cost from the earlier section doesn't just repeat for agents, it multiplies, because the parallelism that makes agents fast also makes the protocol tax hit many times simultaneously instead of once.

That multiplication forces a related requirement into view: isolation. Agents need read-only compute over shared data so a malformed probe can't corrupt anything. They need full query logging so every probe is auditable after the fact. They also need sandboxed access, so a runaway agent can't touch production directly. None of that is a feature a raw S3 endpoint provides on its own. All of it points toward a mediation layer built with explicit access controls from the start, not bolted on after the fact.

Operational reliability under a well-designed mediation layer

The benefits of a well-designed mediation layer go further than query speed and bill size. They extend into what's recoverable, and how fast.

Open table formats built for this kind of separation show the clearest example. Apache Iceberg keeps snapshot history as metadata, so rolling back to an earlier state is a metadata operation, not a data-movement operation. Reading snapshot history runs in constant time. Rolling back runs in constant time too, and the only operation that carries real cost is VACUUM, which cleans up old snapshots later. Recovery becomes a quick metadata update instead of an expensive, slow data copy, which is a large part of why Iceberg and Delta have come to dominate modern lakehouse design.

The Microsoft patent mentioned earlier applies that same principle to reconfiguration rather than rollback. A logical database file can move between physical storage configurations while the intermediary layer handles the move, updating the compute node's endpoint mapping without ever exposing data to the compute layer mid-move. That's a reliability primitive direct S3 routing simply doesn't have available to it, because there's no addressable layer in between to absorb the change.

None of this happens by accident. It happens because someone designed the mediation layer to support it, the same way someone designed it to absorb the protocol tax or serve hot data locally. Compute-storage separation, treated seriously, turns out to be less about where bytes live and more about what stands between them and the compute that needs them. Before choosing a platform or a pattern, the question is not whether separation exists, but what, specifically, is doing the mediating, and whether that choice matches the shape of the workload that depends on it.