RaceLab Inspector
RaceLab

Two agents checked the budget. Both said yes.

When AI agents share a budget, each one reads the balance, thinks, then spends. If two of them read before either writes, both see room and both approve. The money is gone twice and the database reports no error at all.

What actually happened

One account. A hard budget of $100. Twenty agents, each wanting to approve about $45. Here is a real run:

1

An agent reads the account. Nothing spent yet. Plenty of room, so it approves $45. This is correct.

$0
2

At the same moment two more agents read the account. They also see nothing spent, because nobody has finished writing yet. Both approve $45 too. Also correct, given what they saw.

$0
3

All three writes land. The account is now $35 over budget — and every agent reasoned correctly, so nothing anywhere reported a problem.

$135

Why doesn't the database just block this? The rule is "everything added together must stay under the limit." That is a statement about many rows at once, so it cannot be written as a check on any single row. The database is doing exactly what it was asked. Nobody asked it this question.

Now you try it

Everything above is a recording. This is not — it runs in your browser, right now, with the same rules the real system follows. Pick how the agents should behave when they collide, and watch what the account ends up at.

When two agents collide, the loser should…
$0 approved so far
Press Run it.
cap $80
budget $100
$0$200
read the balance thinking wrote collided — someone wrote first

A faithful model of the protocol, not a replay of the measurements — it runs the same read → think → write cycle, the same collision rule, and the same three responses to a collision. The measured results from 250 real runs are further down the page, and they are what any claim here rests on.

Then it gets harder

Part-way through the run, someone lowers the approval cap from $80 to $60. That is a policy change. It lives in the agent's notes — the text it looks up before deciding — and nowhere in the database.

So there are now two different rules that can break, and they break for completely different reasons:

Rule 1 · in the database Stay under $100

An agent that re-reads the balance can catch this, because the balance is a real number it can look up again.

Rule 2 · only in memory Stay under $60

Re-reading the balance tells you nothing about this. The cap changed in the notes, not in the data. You have to go back and re-read the notes.

The notes the agents actually read

These are the real rows in the memories table on CockroachDB. Before each decision an agent asks — and gets the closest matches back by meaning, not by keyword — CockroachDB's vector index over a VECTOR(1024) column, embedded with Amazon Bedrock Titan.

One of these arrives after the run starts. That is the policy change, and it is the only place the new cap exists.

Why this is the whole point. The $100 budget is a column — re-reading the balance recovers it. The cap in these notes is not a column anywhere, so no amount of re-reading the balance will tell you it moved. The only way to notice is to read the notes again.

Five ways to handle it

Every one runs the identical workload — same twenty agents, same seed, same arrival timing. They differ only in what happens when the database says "someone changed this underneath you." Pick one to see inside it.

Why are there two "no protection" rows? A is stock PostgreSQL and A-rc is CockroachDB at the same default isolation level. We report both because comparing across two different databases also compares two different network latencies — and a headline claim we built on exactly that comparison turned out to be confounded, so we withdrew it. A-rc is the honest control: same cluster, same latency, only the isolation level changes.

Watch the money

The line is the account total as the run proceeds. The dashed line is the approval cap — watch it drop from $80 to $60 partway through.

—

Show all 20 agents, one row each

Each row is one agent, from the moment it arrives to the moment it finishes. Every dot is one moment the agent stopped and thought. A single dot means it made one decision and stuck to it. Several dots mean it threw its answer away and worked it out again. Click any row to see what that agent was looking at.