How we measure — and how we label

We borrowed this from drug regulation, because the problem is the same one: a number that is easy to measure gets used to stand in for the thing you actually care about, and sometimes it moves the right way while the real thing moves the wrong way. The convention has three steps. A candidate measure is still being evaluated — useful for steering our own work, not for deciding anything. A reasonably likely measure is one the field accepts as tracking the thing that matters — good enough to decide with, as long as its known weaknesses are printed next to it. A validated measure has evidence that moving it actually moves the buyer's real-world result, and accounts for that whole effect.

Where we stand, stated plainly: nothing we own has reached the third step, for any measure. The check behind the outcomes on this site is at the first step — it is our own check, it tops out at a perfect score too easily, and it has never been validated against a customer's realised result. That is why every outcome on this site is a preview, and why the next thing we ship is a measurement the rest of the field already accepts rather than a better version of our own.

"Cleared m of n declared goals." m counts only the goals whose measured result clears the bar declared in advance, on the stated set. n counts every declared goal, measured or not. Goals we have not measured are reported as not yet measured — never guessed, never blank, and never counted as cleared. There are no letter grades and no weighting: a goal you told us matters 10 out of 10 and a goal that matters 7 out of 10 both count once, and the importance is shown beside each one so you can weight them yourself.

The set is always stated inside the grade, because "3 of 3 on a development set" and "3 of 3 on a locked-away set" are different claims and must never look identical.

A preview card carries real measurements from a real evaluation run. What it does not carry is a run against a locked-away set.

The distinction is worth the extra word. A development set is a set of requests the assistant has not seen, drawn from the pool we are allowed to work with — we can measure against it repeatedly, and we do. A locked-away set is used for nothing at all until the day it decides something: no training, no choosing between versions, no designing the test itself, and no looking at individual answers. It is opened once and then retired, so the same questions cannot be reused to flatter a later result.

For the banking router that set exists — 400 requests, drawn by a fixed rule from the requests no part of our work has touched, fingerprinted, and unopened. We have deliberately not opened it, for a reason we would rather publish than hide: the check we would score it with has not earned the right to decide anything yet, and spending a single-use set to obtain a number we are not allowed to publish as a decision would waste the most valuable asset we have. One honest disclosure about that set: it was included once in an all-requests-at-once average in July, before any of this, which produced no per-request record and changed nothing about the assistant. So it is unused rather than never-touched, and this sentence exists so nobody can say we implied otherwise.

Because a scorecard that only contains passes is an advertisement, and you already know how to read one of those.

Three things follow from that, and all three cost us something. A declared goal that fails stays on the front page, in the same size type as the ones that passed, sorted by how much it matters rather than by how it turned out. A number we published and later found unsound comes off the page and the reason goes on the corrections page with the date — we supersede, we do not delete. And a result we cannot support gets no replacement number at all, because filling the hole with a weaker claim is the same mistake wearing different clothes.

The most useful thing we can tell you about this assistant is what it does badly, since that is the part you have to design around. That is the permission finding, and it is one click from the front page.

These answer one question: how does this assistant do on a test the whole field already uses, on requests that are published, against a fixed answer key. The point of them is that you do not have to take our word for anything — you can run them yourself.

Nothing is published here yet. The accepted public test for picking the right action is BFCL, and version 3 of its structural check is the one that applies to us. We have the software, the settings are registered, and we have run zero jobs against it. When it lands, three things appear with it: the version, since version 3 and version 4 are not comparable and the larger vendors' cards quote version 4; the result broken out by category with a count for each, because a single average hides where a tool-picker actually fails; and two weaknesses of the test itself, which we print whether or not they flatter us — an audit of tool-calling benchmarks found roughly 18.5% of disagreements between the answer key and a careful re-reading, and the same kind of checker has been measured swinging up to 10 points between two ordinary settings.

Separately, we re-run a public knowledge test — MMLU-Pro — against our own copy of a model whose score its maker has already published. This checks our measuring equipment, not the assistant: if we cannot reproduce someone else's published number on their own model, no number of ours means anything. Because the published figure is a single total with no per-question data, this comparison can only be made against a fixed margin, not question by question, and that limits what it can tell us. The comparison and its limits live in the technical report.

These answer a different question: does this assistant do the job you declared, on requests shaped like yours. That is the thing you are buying, and it is also the thing no public test measures, because your job is not on anyone's leaderboard.

The cost of that is real and we state it before the results, not after: an outcome result is not independently reproducible, because we do not release the requests. They are generated from our own customer models, they stay on our own network, and a locked-away set is single-use by design. What we put in place of "take our word for it" is everything else — the fingerprint of the exact request file, the job identifier, the settings the assistant actually ran with, the answers fixed in advance, and a write-once record that a result's fingerprint must match before it is allowed into a report. On top of that, an auditor under agreement can watch a re-run on our network and take the report away with them. We label that witnessed, not verified, because the person watching is not independent of us and pretending otherwise would be the same species of claim this page exists to avoid.

One more limit, stated because it is the one buyers most often assume away: these results describe the stated set of requests. They are not a prediction about the share of your traffic the assistant will handle correctly, and there is no accepted method for turning the first into the second. Every absolute number on this site is read as "on this stated set, with this many requests".