Agent evals
Three questions fail independently, and being sold one of them should not imply the others. Status answers whether a county endpoint is alive. Benchmarks answers whether the answer it gave was right. This page answers a third: when an agent is handed this server, does it call the right tool, with the right arguments, and does it avoid spending your money on a question a free tool answers. A model choosing a tool sees only the tool descriptions and the server instructions string, so those are what this page grades, and a run expires the moment either changes.
59 of 59 tasks measured, per model:
claude-haiku-4-5-20251001 on 2026-10-06
claude-sonnet-5 on 2026-10-06
Server instructions version 3, tool surface 9d8ec781c00d247f, 9 tools read from http://127.0.0.1:3199 on 2026-10-06. The harness records which tool the model selected and never calls it, so measuring cost safety never costs anything.
The sections below describe the Haiku 4.5 run (claude-haiku-4-5-20251001, 2026-10-06). The metric comparison above always shows both runs.
Exactly what was graded
| Model | claude-haiku-4-5-20251001 |
|---|---|
| Run mode | live |
| Instructions version | 3 |
| Tool descriptions hash | 9d8ec781c00d247fdb93e337dc7527fea2ac9abd959a972a75fb032ff5df9e23 |
| Surface read from | http://127.0.0.1:3199 on 2026-10-06, 9 tools, server 0.3.0 |
The hash covers the tool descriptions and the server instructions string exactly as the server hands them to a client, not as they read in our source, because those are two different things and only the first is what a model sees. Reword any of that text and this hash changes, the build fails, and the numbers come down until the run is repeated.
What is measured
Two models answered the identical task set, and their numbers are reported separately rather than averaged. They chose differently on four of the 51 graded tasks. Three are the same disagreement, two of them chained requests: handed an address and asked about its county’s layer, the cheaper model reaches for the geocoder first, a defensible opening move and one call longer than necessary, while the other goes straight to the tool that answers the whole question. The fourth is whether a county’s endpoint is up right now: the cheaper model reads the county record, which carries an observed health block, while the other held back on the grounds that no tool runs a live check. Every metric that involves spending the caller’s money, or refusing a request the server cannot serve, is identical between the two. Over-caution is not: both held back on the question of how many counties publish owner names, which no tool answers, and the other model also held back on the endpoint question.
The share of tasks where the agent called the one tool the eval set names as correct. This is the number to read if you distrust the grading: it gives the rubric no room at all.
What it cannot see: Whether a different tool would also have answered. Some requests have two right answers, which is what the lenient number below is for.
Excluded: 8 the correct answer is to call nothing, so there is no selection to grade
The same measurement, allowing any tool the eval set marks as a defensible second answer. Published beside the strict number rather than instead of it, so the size of the gap between them is the size of the slack the rubric gave.
What it cannot see: Cost. A tool can be a defensible answer and still be the expensive one, which is why the cost metrics below do not consult this list at all.
Excluded: 8 the correct answer is to call nothing, so there is no selection to grade
Some requests cannot be answered by any single tool: you have to list the states before you can ask for a state's counties, and geocode an address before you can resolve its county code. This is the share of those where the first call was a defensible opening move. It is the metric closest to what a real agent session looks like, and an agent can be good at picking one tool for one question while reliably starting a two step job in the wrong place.
What it cannot see: Whether it would then have taken the second step. The harness records one selection per task and stops, so everything after the opening move is unmeasured. It also allows the shorter route: where one tool collapses both halves, opening with it is scored as correct rather than as skipping a step.
Excluded: 54 the task is answered by one tool, so there is no first step to grade
Of the calls where the right tool was chosen, the share where every required argument was present, well formed, and carried the value the request stated. A state slug written as a display name and a county FIPS sent as a number both fail here, because the server rejects both.
What it cannot see: Arguments passed to the wrong tool. Those are not comparable to what the task asked for, so they are excluded rather than folded in, which keeps a selection failure from being counted twice.
Excluded: 8 the correct answer is to call nothing, so there is no selection to grade; 4 the wrong tool was chosen, so its arguments are not comparable
The share of free answerable tasks where the agent did not reach for a tool that bills. Two of the nine tools charge, one per lookup and one per parcel row returned, so this is the metric that decides whether connecting this server to an agent is safe to leave running.
What it cannot see: Anything about tasks where a paid tool is the right answer. Those are excluded, and the argument level guard below is what covers them instead.
Excluded: 11 a paid tool is the correct answer here
Where a paid tool is correct, there is still a cheap way and an expensive way to call it. The radius tool documents a count only preview that bills at most one unit, and a parcel cap that bounds the bill. This is the share of those tasks where the bounding argument was passed.
What it cannot see: Cases where the tool choice was already wrong. A guard on a call that should never have happened is not the thing being measured.
Excluded: 56 nothing about this task bounds a bill
The share of out of scope requests where the agent called nothing. Zoning, flood zones, permits, sale prices and tax balances are not on this server, and an agent that answers them with a parcel record is worse than one that says it cannot.
What it cannot see: Whether the explanation it gave was any good. Only the choice is graded here; the refusal text is recorded in the observations file for a reader to judge.
Excluded: 51 a tool was expected, so there is no refusal to grade
The share of answerable requests where the agent called nothing anyway. Lower is better. This exists because an agent that never calls anything scores a perfect refusal accuracy, so that number is never published on its own.
What it cannot see: The reason it held back. A missing argument it could not infer and plain over caution look identical from here.
Excluded: 8 the correct answer is to call nothing, so there is no selection to grade
The eval set
59 hand written tasks, phrased the way a person would ask an agent rather than the way a tool schema reads. 51 expect a tool and 8 expect the agent to call nothing, because no tool here returns zoning, flood zones, permits, sale prices or tax balances. 12 are cost traps, where a paid tool is the tempting answer to a question the free registry already answers, and 11 are tasks where only a paid tool answers, so the set cannot be passed by refusing to spend anything. 5 need more than one call to answer at all, across three different pairs of free tools, and are scored on their opening move. 10 name a second defensible tool, which is why selection accuracy is published twice.
Every task states in one line why its expected tool is the right one, so the grading can be argued with rather than taken on trust. The file is readable at tasks.json.
| Difficulty | What it means | Tasks | Right choice |
|---|---|---|---|
| direct | the request names what it wants | 23 | 100% |
| indirect | the request describes a goal rather than a tool | 11 | 91% |
| chained | the answer needs more than one call | 7 | 71% |
| adversarial | a wrong tool is the tempting one | 11 | 91% |
| out-of-scope | no tool answers it | 7 | 100% |
Right choice means the expected tool on a task that wanted one, and no call at all on a task that did not. It blends selection and refusal in the rows that contain both, which is why it is not labelled accuracy. Splitting it would leave rows with an n of two.
By tool
| Tool | Billing | Tasks expecting it | Times chosen | Chosen wrongly |
|---|---|---|---|---|
| enrich_address | metered | 5 | 5 | 0 |
| find_county_by_fips | free | 4 | 4 | 0 |
| find_parcel_by_address | free | 8 | 5 | 0 |
| geocode_address | free | 6 | 6 | 3 |
| get_county_endpoint | free | 6 | 6 | 0 |
| list_counties_in_state | free | 6 | 6 | 0 |
| list_indexed_states | free | 6 | 5 | 0 |
| radius_owners | metered | 6 | 6 | 0 |
| request_county | free | 4 | 4 | 0 |
Every tool the server exposes is expected by at least four tasks. A tool added to the server without tasks would appear here as uncovered rather than quietly go ungraded.
What gets mistaken for what
| Expected | Chosen | Times |
|---|---|---|
| find_parcel_by_address | geocode_address | 3 |
| list_indexed_states | (none) | 1 |
A pair that repeats is a description problem, not a model problem. Two tools that keep getting swapped are two tools whose descriptions do not draw the line between them.
What this page does not measure
- Whether the tool then worked. Nothing here is executed. The harness records the selection and stops, because two of these tools bill the caller and a harness that ran what it graded would spend real money to measure whether the model should have spent it. Whether the call would have returned the right parcel is the benchmarks question.
- More than the first turn. Each task is a single request with a single selection. Chain start accuracy above covers the opening move of a multi step job, which is the part a one shot harness can see, but whether the agent then takes the second step, reads the result correctly, or recovers from a bad first call is entirely unmeasured. Each run grades exactly one model; the two runs reported above are kept separate and never averaged, and each is named in full in the provenance line.
- The hardest phrasings. The set is written by the same people who wrote the tool descriptions, which is the structural weakness of any eval an author grades themselves. It is published in full so someone else can find the tasks it does not contain.
- Whether the expected answers are right. Every one is a judgment call with a stated reason. Disagreeing with a reason is disagreeing with the score, which is the point of writing them down.
How this is measured, and how to check it
The harness reads the tool definitions and the server instructions off the live server rather than from a copy in this repository, so it grades the text that actually ships. It presents each task to a model with those definitions attached, records which tool the model selected and with what arguments, and calls nothing. The prompt that frames the situation is recorded verbatim in the observations file, and it says nothing about cost: adding a line telling the model not to spend money would turn the cost metric into a measurement of the harness rather than of the product.
Both files are readable: tasks.json holds the tasks and observations.json holds what the last run saw, including the full tool surface it was graded against. Every observation carries a fingerprint of the task it measured, and the run carries a fingerprint of the tool descriptions. Edit a task or reword a description without re-running, and the build fails rather than publishing a number about text nobody ships.
Blending tool selection, cost safety and refusal into one score would hide exactly which of the eight is weak, and that is the only number worth having.
Endpoint liveness is on the status page and answer correctness is on the benchmarks page. What the server exposes, and how to connect an agent to it, is on the MCP page.