If you are building an agent that trades on-chain, how do you score whether it executed well? The obvious signal is the quote the agent got. That signal is also the wrong one. A quote is a promise; the realized fill is a different number, and the gap between the two is where fees, MEV, and slippage live.
An agent optimized against the quoted number learns to pick the venue that promises the most, not the one that delivers the most. To train or evaluate a DeFi-execution agent honestly you need an independent measurement of the realized outcome, scored the same way every time. That measurement is a verifier, and it is the piece that is hard to build.
ClearTrace already runs one. This note describes the mechanism, shows a worked example on real data, and asks a question at the end.
The verifier has two parts: a step function that realizes a proposed action, and a rubric that scores it.
Step function — propose calldata, get the realized fill. Given a swap the agent
proposes (the aggregator's own swap calldata), the engine simulates it against current chain state
using Alchemy eth_simulateV1, sent from a token-rich holder, and reads the
bought-token transfer to the taker. The output is the amount the trade would actually have
produced, on live liquidity, not a modeled estimate. This is the environment step a DeFi-execution
agent needs: propose a route, receive the realized fill.
Rubric — decompose the gap into attributable components. The realized output is compared to what was quoted, and the shortfall is decomposed:
output_gap_bps = (quoted_out - realized_out) / quoted_out * 1e4 = fee_bps + mev_bps + slippage_bps
all_in_cost_bps = output_gap_bps + gas_bpsEvery score is anchored to quoted_out, the promise the agent was optimizing toward.
The one cross-venue-comparable number compares like-for-like against a neutral Uniswap on-chain
baseline inside a 300-second window. That is the natural reward signal: did the agent's route
beat the neutral baseline, measured the same way for everyone?
A trust gate on the reward. Not every measurement is stable enough to reward on. A cell (a source / pair / trade-size bucket) is treated as rated only with at least 30 realized samples and at least a 7-day span; thinner cells are shown but never ranked. This is the eval-quality discipline an RL reward or a benchmark needs so a burst of correlated samples cannot masquerade as signal.
One realized sample from the public dataset, venue anonymized because the point is the mechanism, not a ranking:
output_gap_bps = 9.71 bps.all_in_cost_bps = 9.71.
The agent's route was not overtly taxed by an explicit fee or a sandwich; it lost about a tenth of a percent to organic price movement between quote and fill. A different route on the same pair and size in the same window might have lost less, or more, and the verifier would say so in the same units. That comparability is the product.
The public sampler currently holds 22,832 quote realizations (20,774 successfully realized), Ethereum only, across 11 aggregators, 11 pairs (majors plus several long-tail tokens), and 4 size cohorts ($1k / $10k / $100k / $1m), over ~27 days (2026-06-23 to 2026-07-20). That resolves into 304 source/pair/cohort cells, of which 278 already carry at least 30 realized samples over a 7-day span.
Across those cells the median realized gap ranges from roughly 0 bps (the fill matches the promise) to about 50 bps (the fill lands half a percent below the promise). A verifier that can separate a 0-bps route from a 50-bps route, like-for-like and without trusting either venue's self-report, is the object an eval or an RL reward function is built around.
Reliability is measured separately, on-chain rather than in simulation: genuine user-flow revert sits at 0.2–2.7% across venues and chains (a machine-readable user-only rate), after filtering out solver and keeper bots whose failed transactions otherwise inflate the raw rate by up to ~20x. An agent that routes into a frequently-reverting path is paying gas for trades that never land, and that is a distinct axis of "did it execute well" from the quote gap.
all_in_cost_bps
vs the neutral baseline. A held-out, methodology-locked eval set for "can this agent execute a
swap well."This is a listening post, not a sale. If you are building agents that touch on-chain execution, at a lab, an agent-infrastructure company, or a protocol shipping its own agent, I want to know what a verifier like this would have to be for you to actually use it. More chains? A specific action space? A particular reward shape? Offline eval versus online environment? Exclusivity on a slice?
The engine, the methodology, and the honesty discipline already exist and run continuously. What is unproven is whether anyone building crypto agents needs a neutral verifier badly enough to pull it into their loop. That is the only thing worth finding out before building further.
Contact: andrew@cleartracedata.com · a two-line reply on what is missing is more useful than a booked meeting.
seed/quote_samples.csv
(22,832 rows as of 2026-07-20). Fork-simulation via Alchemy eth_simulateV1; the gap is
decomposed into fee, MEV, and slippage against the quoted output; a cell is rated only at ≥30 realized
samples and a ≥7-day span. On-chain reliability uses a user-only revert rate that filters solver and
keeper bots; retired solver-inflated revert headlines are deliberately excluded.