A neutral verifier for DeFi-execution agents

An agent optimized against the quoted number learns to pick the venue that promises the most, not the one that delivers the most. To train or evaluate a DeFi-execution agent you need an independent measurement of the realized outcome, scored the same way every time. That measurement is a verifier, and it is the piece that is hard to build.

ClearTrace already runs one. This note describes the mechanism, shows a worked example on real data, and asks a question at the end.

The mechanism

The verifier has two parts: a step function that realizes a proposed action, and a rubric that scores it.

Step function: propose calldata, get the realized fill. Given a swap the agent proposes (the aggregator's own swap calldata), the engine simulates it against current chain state using Alchemy eth_simulateV1, sent from a token-rich holder, and reads the bought-token transfer to the taker. The output is the amount the trade would actually have produced, on live liquidity, not a modeled estimate. This is the environment step a DeFi-execution agent needs: propose a route, receive the realized fill.

Rubric: decompose the gap into attributable components. The realized output is compared to what was quoted, and the shortfall is decomposed:

output_gap_bps  = (quoted_out - realized_out) / quoted_out * 1e4  =  fee_bps + mev_bps + slippage_bps
all_in_cost_bps = output_gap_bps + gas_bps

Corrected 2026-08-21. The worked example below originally read "gas ≈ 0" and gave an all_in_cost_bps of 9.71. That zero was structural, not measured: until 2026-08-21 the sampler stored gas_bps = 0.0 on every row rather than leaving it unset, so all_in_cost_bps equalled output_gap_bps throughout. Rows now carry a gas_valuation column and both figures are null where gas could not be priced. See docs/METRICS.md.

One term deserves a caveat. mev_bps is a reserved slot for realized on-chain fills, where a sandwich is observable. A fork simulation has no adversary in it, so under fork-simulation mev_bps is zero by construction and every row in the sampler carries 0.0. What this engine actually resolves is the venue's explicit fee against organic slippage. Sandwich exposure is measured separately, on-chain, and is not folded into this number.

Every score is anchored to quoted_out, the promise the agent was optimizing toward. The one cross-venue-comparable number compares like-for-like against a neutral Uniswap on-chain baseline inside a 300-second window. That is the natural reward signal: did the agent's route beat the neutral baseline, measured the same way for everyone?

A trust gate on the reward. Not every measurement is stable enough to reward on. A cell (a source / pair / trade-size bucket) is treated as rated only with at least 30 realized samples and at least a 7-day span; thinner cells are shown but never ranked. This is the eval-quality discipline an RL reward or a benchmark needs so a burst of correlated samples cannot masquerade as signal.

A worked example

One realized sample from the public dataset, venue anonymized because the point is the mechanism, not a ranking:

The agent's route was not overtly taxed by an explicit fee or a sandwich; it lost about a tenth of a percent to organic price movement between quote and fill. A different route on the same pair and size in the same window might have lost less, or more, and the verifier would say so in the same units. That comparability is the product.

What it produces at scale

The public sampler currently holds 63,912 quote realizations (56,750 successfully realized), Ethereum only, across 11 aggregators, 13 pairs (majors plus several long-tail tokens), and 4 size cohorts ($1k / $10k / $100k / $1m), over ~74 days (2026-06-23 to 2026-09-04). That resolves into 325 source/pair/cohort cells, of which 298 already carry at least 30 realized samples over a 7-day span.

Across those cells the median realized gap ranges from roughly 0 bps (the fill matches the promise) to about 50 bps (the fill lands half a percent below the promise). A verifier that can separate a 0-bps route from a 50-bps route, like-for-like and without trusting either venue's self-report, is the object an eval or an RL reward function is built around.

Reliability is measured separately, on-chain rather than in simulation: genuine user-flow revert sits at 0.1–3.2% across venues and chains (a machine-readable user-only rate, routers only, since an unfilled batch-auction or intent order never becomes a transaction), after filtering out solver and keeper bots whose failed transactions otherwise inflate the raw rate by up to ~57x. An agent that routes into a frequently-reverting path is paying gas for trades that never land, and that is a distinct axis of "did it execute well" from the quote gap.

Three ways this plugs into agent work

1. Offline benchmark. Score a DeFi-execution agent (or a model prompted to route trades) against a frozen, dated slice of realized outcomes, reported in all_in_cost_bps vs the neutral baseline. A held-out, methodology-locked eval set for "can this agent execute a swap well."
2. Online reward. Use the step function as an RL environment and the matched-gap-vs-baseline number as the reward. The environment is non-stationary by nature (liquidity, MEV, and routing shift constantly), so it resists the "task saturates once the model passes it" decay that static replicas suffer.
3. Neutral referee. Compare two agents, two models, or two routing policies in identical units, with published methodology and a public, dated record of every figure that has been withdrawn or requalified (see the Edition 2 errata). The independence is the point: the referee does not have a route to sell.

Limits

• Coverage. Ethereum only today; ~74 days of continuous history. Deep, not yet broad.
• Simulatability. Fork-simulation cannot realize intent / RFQ venues that do not expose swap-builder calldata (e.g. CoW), so those are marked unrealized rather than scored.
• The reward is not the agent's action space yet. The engine realizes an aggregator's own calldata. Wiring an agent-chosen action space and a task generator on top is real work, not a repackage. This note describes the verifier, not a finished gym.
• Two revert signals are different. The sampler's unrealized rows are a coverage artifact, not user pain; the on-chain user-only rate is the real reliability number. They are never conflated.

The question

This is a listening post, not a sale. If you are building agents that touch on-chain execution, at a lab, an agent-infrastructure company, or a protocol shipping its own agent, I want to know what a verifier like this would have to be for you to actually use it. More chains? A specific action space? A particular reward shape? Offline eval versus online environment? Exclusivity on a slice?

The engine, the published methodology, and the dated-correction record already exist and run continuously. What is unproven is whether anyone building crypto agents needs a neutral verifier badly enough to pull it into their loop. That is the only thing worth finding out before building further.

Contact: andrew@cleartracedata.com · a two-line reply on what is missing is more useful than a booked meeting.