Query Result Caching Strategies for LLM Agent Workloads

Agents query databases nothing like humans do, and the caching systems built for human query patterns miss most of the reuse available in agentic workloads. That mismatch, not any flaw in conventional caching itself, is why a layered, purpose-built caching strategy matters for anyone running agents against a database at scale.

Why agents are a poor fit for human-pattern caches

Conventional query result caching rests on two assumptions. The same string or near-identical string shows up again with high syntactic fidelity. There is meaningful idle time between requests, giving a cache room to serve a hit before the next write invalidates it. Agentic workloads break both.

Human users already repeat exact queries at a low rate. Agents repeat them at an even lower rate still, because an agent formulates a query dynamically from the state of its reasoning rather than retyping a phrase a person already typed once before. Exact-match caching, built for the human pattern, therefore captures very little of the reuse actually available. The AgentSysBench study documents heavy cross-request redundancy in search queries and web fetches, redundancy that a strict string-match cache cannot see because the queries themselves rarely look alike even when they ask for overlapping information.

The rhythm compounds the problem. Agents fire queries in bursts with very short gaps between them, rather than the spaced-out, idle-time-rich pattern a human browsing session produces. A cache warm-up strategy tuned for human pacing has no time to do its job before the next burst arrives. And the bottleneck isn't where most teams assume it sits. AgentSysBench found non-LLM components, tool calls, database fetches, retrieval steps, dominating end-to-end latency in 5 of 10 agentic applications studied. The model itself is frequently not the slow part. The data access layer is.

Every uncached tool call also gets more expensive as a session runs. Tool schemas, system prompts, and the context overhead from each new observation accumulate in the context window turn after turn, crowding out productive compute. The same properties that make agents hard to cache, high query volume, low repetition, bursty timing, are the properties that make caching worth the engineering effort in the first place: there's more traffic to intercept, more latency on the table, and more cost compounding with every additional turn. That is the case for the layered approach that follows.

The three caching layers that together cover the agentic stack

No single cache sitting in front of the database solves this. Three layers, operating at different points in the stack, are needed together.

| Layer | Where it sits | What it caches | |---|---|---| | 1. KV / prompt caching | Inside the inference framework | Stable prefixes: system prompts, tool schemas | | 2. Semantic caching | Application layer, before the model or database | Past responses, matched by embedding similarity | | 3. Tool-result / agentic-plan caching | Between the agent and its tools or database | Outputs of external tool calls and database queries |

The serving infrastructure manages Layer 1, not application code. It reuses already-computed attention tensors for the parts of a prompt that don't change call to call. Layer 2 lives in the application layer and intercepts a query before it ever reaches the model or the database, matching it against prior requests even when the wording differs, which is what compensates directly for the low exact-repetition rate described above. Layer 3 is where database query result caching actually lives: it stores what a tool call or a database query returned, so the next equivalent call collapses into a cache lookup instead of a round trip.

These three target distinct bottlenecks, so running them together produces gains that stack. The Workload-Aware Caching for Multi-Agent Systems study found a 31.1% average latency reduction over the next-best finite-capacity baseline, and the same research shows workload-aware content caching working as a complement to plan-level caching and parallel execution, not a substitute for either. The next four sections take each layer in turn, starting with the one closest to the model.

KV and prompt caching: stabilizing the input side before the first database call

Prompt caching is the most automatic of the three layers, since the serving infrastructure largely handles it. It delivers real, consistent savings, but only when the cache boundary is drawn around content that is actually stable: system prompts and tool schemas. Dynamic tool results placed inside that same cached prefix defeat the purpose.

An evaluation by PricewaterhouseCoopers across three major LLM providers found that prompt caching cuts API costs by 41 to 80% and improves time-to-first-token, with one strategy standing out: caching the system prompt alone delivered the most consistent benefits across both cost and latency. Full-context caching, by contrast, can make things worse. When dynamic tool results get folded into the cached prefix, the system writes to cache for content that will never recur across sessions, and that write has a cost without a matching payoff.

The practical rule that follows is straightforward: keep dynamic tool results out of the prompt cache boundary and place them at the end of the context, caching only the stable, session-invariant prefix that precedes them. Done that way, this layer handles the input side of the stack cleanly. It does nothing for the output side, the actual database query results getting injected back into context turn after turn. That gap is what the next two layers exist to close.

Semantic caching: recovering reuse when queries never repeat exactly

Semantic caching recovers reuse that exact-match caching simply cannot see. Instead of requiring a new query to match a prior one character for character, it compares the new query's embedding against embeddings of past queries and returns a stored response once the cosine similarity between them crosses a configured threshold. Two queries worded completely differently but asking for the same underlying information can both hit.

How well this works depends heavily on the workload. Agent tool calls built on fixed schemas, FAQ-style bots, and retrieval-augmented generation over a stable corpus all produce materially higher hit rates: the space of distinct underlying questions is naturally limited even when the surface wording varies. Multi-turn chat and creative generation are at the other end: each output depends on a shifting conversation history or is meant to be novel by design, so query uniqueness is close to guaranteed and a semantic cache has little to find.

The similarity threshold is a real design decision with a cost on either side. A threshold set too loose makes the cache return answers to queries that only resemble each other on the surface, serving a wrong or stale result as if it were a hit. A threshold set too tight makes the cache reject genuine paraphrases, which falls back to the full model-and-database path far more often than necessary and gives up the latency and cost savings the layer exists to provide. Tuning that threshold against the actual distribution of queries a given agent produces is the work, not a one-time setting.

The distinction that actually decides what belongs in this layer is whether a query is stateless or stateful. A query whose correct answer depends only on its own inputs, not on anything earlier in the conversation, is a safe target for caching. A query whose correct answer depends on session context is not. Caching a stateful query doesn't just waste a cache slot. It produces an agent response that's wrong for the session it's returned into, which is a correctness failure, not a performance one.

Tool-result caching: where database query results live in the agentic stack

Tool-result caching is the highest-leverage intervention available on the database side of an agentic stack. For a stable, read-only result, it removes the call-wait-inject sequence entirely, turning a database round trip that costs tens to hundreds of milliseconds into a cache lookup that resolves in under a millisecond. AgentSysBench found that tool-result caching removes more than a third of redundant search calls in production agentic workloads and saves nearly a fifth of aggregate search latency, without requiring any change to agent logic or tool interfaces.

The same stateless-versus-stateful classification from the semantic caching layer governs what belongs here, applied tool by tool. Product catalogs, pricing tables, user profile metadata, historical analytics, and other read-only, slow-changing reference data are strong candidates, since they get read constantly and change rarely. Tools with side effects, writes or sends, real-time data such as live inventory counts or queue lengths, write-then-read sequences, authentication tokens, and anything whose staleness inside the TTL window could cause the agent to take a wrong action belong on the other side of that line and should stay uncached. Getting a tool's classification wrong doesn't just lower the hit rate. It produces incorrect agent behavior and pollutes the cache with results the agent should never have been given, so the decision needs to be made per tool.

Analytics databases serving agents get a structural advantage here. Where a security-conscious deployment restricts agents to read-only access, the only source of cache staleness becomes the underlying data changing, since there's no agent-initiated write to invalidate against. That constraint, covered in more detail in the next section, simplifies the invalidation logic considerably.

The implementation itself can stay simple. A thin proxy sits between the agent and its tools: on each tool call, it checks the cache; on a miss, it executes the tool and stores the result with a configured time-to-live; on a hit, it returns the cached result. That proxy pattern is also exactly where the next layer of the stack, the integration protocol connecting agents to their tools and data sources, naturally sits.

Where MCP moves caching logic in the stack

The Model Context Protocol, MCP, has become the dominant way agents connect to tools and data sources, but the protocol itself provides no caching. Query-result caching still has to be implemented somewhere specific: inside the MCP server's handler logic, or in a caching proxy placed in front of the database. MCP gives the stack a standard interface. It does not give the stack a cache.

What MCP does provide is a session model that REST APIs lack. MCP connections are stateful and session-aware, so the server can maintain context across the multiple steps of a workflow. That property is what makes it possible to track, within a session, which tool results are safe to reuse and which depend on something that happened earlier in that same session.

The security posture of a sound MCP deployment does real work for caching correctness, not just for safety. Least-privilege, read-only tool permissions on analytics databases mean agents cannot write, so the only trigger for cache invalidation is the underlying data changing. TTL-based expiry becomes sufficient for most stable analytics data, because there's no agent-initiated write to race against. Prompt injection turns this into a sharper concern than it might first appear: if an MCP server exposes a tool that can execute arbitrary queries, a crafted prompt could use that tool to pollute the cache with incorrect results that get served to later, unrelated requests. Restricting tools to validated, read-only actions functions as both a security control and a cache-correctness control at once.

For internal systems, company databases, proprietary data stores, the case for self-hosting the MCP server is strong. Internal schema has no reason to pass through a third-party proxy, and a self-hosted server is the only place where per-tool caching configuration can actually be tuned to match how fast internal data changes. Whatever the deployment model, full logging of every agent query through the MCP layer is a prerequisite for any of this to improve over time. Without a complete record of what was asked, when, and what came back, there's no basis for adjusting TTLs, similarity thresholds, or the stateless/stateful classification as the workload evolves, which is exactly the record that makes the next layer, eviction policy, possible to tune well.

Workload-aware eviction: using DAG structure to decide what stays in cache

Once the three caching layers are in place, the eviction policy governing them becomes its own source of gains, and conventional policies are a poor fit for multi-agent workloads. LRU and LFU both treat every cached entry the same way, judging value purely by access history. That assumption fails badly when the workload is a multi-agent DAG, a directed acyclic graph of task decompositions, where an entry's structural position and recomputation cost predict its value far better than how recently it was touched.

The failure case is concrete. Picture a node accessed recently that feeds exactly one downstream calculation and costs almost nothing to recompute if evicted. Now picture a node accessed much longer ago that feeds many downstream nodes and would cost far more to recompute if it had to be rebuilt. LRU evicts the second node first, because it only looks at recency. The right call is the opposite: keep the expensive, heavily-depended-on node, and let the cheap, lightly-depended-on one go.

Mohamed et al., from the University of Minnesota, Google, and IIT Guwahati, built an eviction policy around exactly this logic. It scores each cache entry on three signals at once: recomputation cost, the number of downstream nodes depending on it (outgoing edges in the DAG), and how frequently the agent invokes it. The entries that would be most expensive to recover from losing are the ones the policy fights hardest to keep.

Across three multi-agent benchmarks, this scoring approach reduced latency by up to 64.7% against an uncached baseline and produced a substantial average latency reduction over the next-best finite-capacity baseline, closing much of the gap to what an unbounded cache would achieve. The signals this depends on, recomputation cost, dependency count, invocation frequency, are already present in any multi-agent planning system that decomposes tasks into nodes with explicit dependency edges. Nothing extra needs to be instrumented to get them. The structure is sitting there in the execution graph already. Most teams simply aren't reading it.

Schema and context design choices that make all three layers work better

Every caching layer described above has a ceiling set well before any cache is consulted, by decisions made at the schema and context level. How much schema gets exposed to an agent, how incoming queries get classified before they're routed, and how narrowly tools get scoped all shape how high that ceiling actually sits.

A tool scoped tightly around a single, well-defined, read-only operation is far easier to classify correctly as stateless than a general-purpose query tool that can touch anything in a schema. The narrower the tool, the cleaner the cache-or-don't-cache decision becomes, and the less likely a classification error slips through and pollutes the cache with a result that should never have been stored. Schema design carries similar weight. A schema that separates stable reference data, catalogs, pricing, historical analytics, from frequently-changing operational data makes it possible to apply different caching rules to each without special-casing every table by hand.

Query routing decisions made upstream, before a request ever reaches the semantic or tool-result cache, determine how much of the traffic those layers even get a chance to act on. An agent architecture that forces every request through a single, undifferentiated query path gives the caching layers less to work with than one that separates stateless lookups from stateful, session-dependent operations at the point the query is first constructed. None of the three caching layers can fix a routing or schema design that blurs that distinction. The caching strategy and the schema it runs against are, in the end, the same design problem viewed from two different angles. That pairing also shapes the choice of database underneath: a database built for real-time analytics at terabyte-to-petabyte scale that keeps hot data in a distributed cache mesh rather than routing queries through S3 is designed so agentic read workloads land on always-warm data without per-query cost penalties.