Sandboxed Branching for Agent Experiments on Production Data

Agents need production data because that's where the real edge cases, real distributions, and real schema quirks live. But give an agent direct access to that production data, and the risk stops being hypothetical. In April 2026, a Cursor coding agent wiped the production database for PocketOS, a SaaS platform serving car rental businesses, membership clubs, and auto auctions, with a single Railway API call: a GraphQL volumeDelete mutation.

Why agents need production data

Stale data disqualifies most of the obvious workarounds. Data warehouses refresh on batch schedules, often an hour or longer between updates, and an agent making decisions about money, inventory, or a customer's order can't work from information that old. A financial agent reading yesterday's positions might generate trades that no longer make sense by the time they execute. A support agent pulling from a cached order table might tell a customer their package is on the way when it already arrived.

Synthetic or sampled data creates a different kind of failure. Stripping out the edge cases, the odd distributions, and the messy schema quirks also strips out the very things agents are supposed to surface. A cleaned-up copy of production data gives you cleaned-up, wrong answers.

The PocketOS incident isn't an isolated case. In 2025, a Replit agentic workflow deleted a production database after overriding an active code freeze and running destructive commands with no human approval in the loop, a direct result of development and production environments not being kept separate. Both incidents point to the same root cause: governance built for human users doesn't hold up once an agent is making the calls. Agents don't carry the semantic judgment a human engineer uses to tell "I have permission to touch this" apart from "I should touch this." They can hallucinate a query against a dataset they were never meant to reach, and once that happens, the error moves downstream fast. So the obvious next move is to patch this with read-only access, staging copies, or transaction wrapping. Each of those, it turns out, runs into its own wall.

Why transactions and staging copies cannot substitute for true isolation

Transactions and staging environments are the two most common fixes teams reach for, and each one fails in its own distinct way once agents start generating real workload.

Transactions can fake a limited kind of exploration. An agent can drop a savepoint at every step and roll back when it wants to try something else. BranchBench, a benchmark released by Columbia University in April 2026, tests this exact pattern as a baseline and finds it works, but only as a fast, lightweight option across a narrow slice of workflows. The limit appears fast in production: transactions don't support concurrent branches. An agent running Monte Carlo Tree Search needs to explore many paths at once, but a transaction-based approach forces it to go depth-first instead, one branch at a time, which kills the throughput the technique depends on. Worse, that isolated state disappears the moment the transaction ends. There's no branch left over to come back to, compare against another, or merge selectively, because its entire lifetime is tied to that one transaction.

Staging environments fail differently. A staging copy is a snapshot frozen at some point in time, not a live view of production, so the staleness problem that direct production access was supposed to avoid creeps right back in through another door. Keeping a staging environment useful means refreshing it constantly, which adds operational overhead without actually closing the gap.

Neither approach comes close to the scale agentic workloads actually produce. BranchBench found that a single agent task running Monte Carlo Tree Search can spin up thousands of branches, some living just milliseconds and others lasting minutes, and all of them hitting branch lifecycle latency, metadata scalability, garbage collection, and cross-branch reads at the same time. Read-only access sidesteps the mutation risk, sure, but it kills the entire point of the exercise: an agent that can't write hypothetical state can't test a hypothesis, and testing hypotheses is what the branch–mutate–evaluate–prune loop exists to do. What's actually needed is state that's isolated, writable, concurrent, and temporary, built on top of real data rather than a copy of it.

Sandboxed branching and zero-copy branching

Sandboxed branching answers that need directly. It gives an agent its own writable, short-lived slice of state that shares its underlying data with production through copy-on-write mechanics, rather than duplicating that data.

The natural shape of agentic work is a loop: branch, mutate, evaluate, prune. An agent picks a branch, forks a child branch off it, makes some changes, checks the results through queries or by comparing across branches, and then either keeps that branch or throws it away. A single high-level task might run this loop thousands of times before it's done.

Zero-copy branching is what makes that loop affordable. Creating a branch copies no data; it's just a metadata pointer aimed at a specific point in the parent's state. Only when the agent actually changes something does the system write a new version of that piece of data; until then, the branch and its parent share the same storage underneath. That's why a sandbox can be forked from production in under a second and explored freely, with cost scaling only to what actually changed, not to the size of the dataset it was forked from. Throwing away a branch that never touched anything costs close to nothing, just some metadata cleanup.

The sandbox boundary is what keeps this safe. Inside its branch, an agent gets full read-write access, and every query it runs is logged. None of that activity reaches production unless someone, a human or a governed process, makes an explicit decision to merge or promote it. Isolation extends past the data layer too: the process actually running the agent's code and queries is sandboxed on its own, so a bad query or a code-generation mistake can't reach host systems or another tenant's data.

How existing systems implement branching under agentic load

The architecture sounds clean on paper. In practice, BranchBench found a real tension running through every production-ready system it tested: tune a system for fast branch creation and reads get slower as branches pile up deeper; tune it for fast reads and branching gets too expensive to use the way agents need it used. No system in the benchmark managed to sit at both ends of that trade-off at once.

Neon, the serverless Postgres platform Databricks acquired in May 2025, separates compute from storage and handles copy-on-write at the storage layer by marking the appropriate point in the write-ahead log, letting a branch share unchanged pages with its parent; branches provision in under a second. On BranchBench, Neon finished the Failure Reproduction workload cleanly but only partially completed Software Development, Data Cleaning, and Monte Carlo Tree Search, because its hosted setup caps how many branches can stay live at once, and that cap turns out to be the real bottleneck for agentic work. There's also a cost to letting compute scale down: once a node goes idle and scales to zero, bringing it back takes anywhere from 500 milliseconds to a few seconds, a real tax on agents that jump between many branches quickly.

Dolt and DoltgreSQL take a different route, built on a content-addressed storage engine. On BranchBench, Dolt performs in line with Neon across most data operations, DDLs and range queries aside, though on the Monte Carlo Tree Search workload, Chronos runs 6.0 times faster than Doltgres.

Xata takes yet another approach, handling copy-on-write at the block level on top of an unmodified PostgreSQL engine, which buys simpler branching at the cost of coarser granularity compared to storage-layer designs. Filesystem snapshots and physical cloning get close to the same goal but carry heavier overhead and much coarser granularity, and BranchBench treats them as baselines to measure against, not real contenders.

No system in the benchmark completed the full suite at production scale. A database built specifically for agentic branching doesn't exist yet in a form ready for production.

What Chronos contributes to cross-store branching

Real agentic systems rarely live inside one database. They span a relational store, a vector index, and a filesystem all at once, and every one of the systems above only branches a single store at a time. That leaves the application stuck with the job of keeping all those stores lined up by hand, and a merge done badly can leave the system in a state where one store disagrees with another.

The problem gets concrete fast. Picture a coding agent investigating a bug: it updates source files, writes provenance records, and touches a vector index for semantic search, all inside one branch. So when that branch merges back, a vector index entry can't point to a file that doesn't exist yet on disk. It's a correctness failure, not a slowdown.

Chronos, a system out of MIT CSAIL and the University of Arizona published in September 2026 by Xinjing Zhou, Jason Mohoney, Samuel Madden, Michael Stonebraker, and Lei Cao, takes this on directly, adding branching across different kinds of data stores without changing any of their underlying engines. It rests on two ideas: a compact, interval-based way of tracking versions that makes branching and data-sharing cheap through simple query rewriting, and a bolt-on design that keeps branch management separate from each store's normal data path. Because of that separation, Chronos can make a merge visible across every store at the same instant, so no store ever shows a partial result of a merge.

Chronos works today across PostgreSQL, SQLite, DuckDB, Qdrant, and a DBMS-backed filesystem, which happens to be close to the exact mix of stores enterprise agentic platforms already run on. On performance, it beats the next-best versioning system by a wide margin across PostgreSQL's TPC-C and YCSB-C benchmarks, DuckDB's TPC-H benchmark, and Qdrant vector search, and it runs MCTS-style exploration substantially faster than existing approaches. Compared to running those stores unversioned at all, Chronos only adds a modest slowdown, a real cost but a tolerable one.

The bolt-on design matters beyond the numbers. Because Chronos doesn't require replacing PostgreSQL, Qdrant, or any other store a team already runs, the barrier to trying it drops considerably. That's not a small point when so much of the cost of adopting new infrastructure comes from migration, not from the technology itself.

Chronos isn't confined to a research paper. It already runs in production at CMS Computing Operations at CERN, serving as a storage layer for agent knowledge management, data sandboxing, and ingestion staging. That deployment matters because CMS Computing Operations is an organization that manages scientific data at real scale and relies on agent sandboxing and branching as working infrastructure.

The five agentic workload patterns that branching must support

BranchBench doesn't treat agentic work as one undifferentiated blob. It breaks it into five representative workloads that together map out the shape of speculative branching and push on different parts of a branchable database. Agentic software engineering, where agents propose and test code changes across branches running concurrently, is probably the pattern most teams run into first, and it leans hardest on fast branch creation and high throughput. Failure reproduction looks different: an agent forks from some point in the past, replays a sequence of events to recreate a bug, and checks the outcome. What matters here is forking quickly from arbitrary historical states rather than running many branches at once. Data curation sends agents through cleaning, labeling, or transforming data across branches and then comparing results across them, so the pressure falls on cross-branch analytics and comparison, not on raw mutation speed.

Monte Carlo Tree Search is the heaviest of the five. An agent explores a tree of possible actions by forking thousands of branches, running short sequences of mutations in each, checking the outcomes, and picking the best path forward, and it does all of this while simultaneously stressing branch lifecycle latency, metadata scalability, garbage collection, and cross-branch reads. The payoff for getting this right is large: BranchBench points to Terminal-Bench results showing task success climbing from a very low baseline up to 30.6% once MCTS is applied, and that jump comes from being able to fork and discard states efficiently, not from any improvement in the underlying model's reasoning. Simulation rounds out the list, with agents running parameterized scenarios across isolated branches and periodically aggregating results across them in the background, which puts the load on managing long-lived branches and keeping aggregation fast.

The practical takeaway for a team building one of these systems is to figure out which of these five patterns actually describes their workload before picking a branching mechanism. A system tuned for failure reproduction, built around fast historical forking with low concurrency, is a poor match for Monte Carlo Tree Search, which needs the opposite profile: high concurrency and branches that live for seconds, not minutes.

Auditability and governance requirements for agents operating on production-derived data

Isolation keeps an agent's exploration from touching production, but isolation by itself doesn't answer the question a compliance officer or an engineering lead will eventually ask: what did the agent actually do inside that branch? Without a full record of every query, every mutation, and every decision to promote or discard a branch, the sandbox turns into a blind spot rather than a safety mechanism. An agent that's free to explore thousands of hypothetical states needs that freedom paired with a log precise enough to reconstruct what it tried, what it changed, and why a particular branch was or wasn't merged back. Isolation answers the question of whether an agent can hurt production. Auditability answers the question of whether anyone can explain, after the fact, what the agent was doing and whether it should have been trusted to do it. Both questions matter, and neither one, on its own, is a substitute for the other.