Verification isthe bottleneck
As models do more, we need better ways to see what they’re doing and what we’re teaching them.
Alex Duffy · Good Start Labs · September 15, 2026
Models are getting better at doing work. We’re spending more time figuring out whether they did the work we wanted. We had early access to TypeSafe’s new model, Jev, and found it incredibly powerful as a verification tool: $160 per million graded answers against $33,000 for Claude Fable 5.1THE HEADLINE NUMBEROn the same 6,003 rubric checks, Jev and Claude Fable 5.1 gave the same verdict 91.5% of the time. Costs are per million graded answers at standard rates: Jev at TypeSafe’s $0.042 per million input tokens, Fable measured from our run. MethodologyRead Methodology ↗, with the same verdict nine times in ten.
That matters most during training. A broken task can reward a shortcut. A poorly specified goal can teach the wrong lesson. Two recent examples:
- In the OpenAI–Hugging Face incidentBEYOND THE EVALUATIONDuring a cybersecurity evaluation with reduced safeguards, OpenAI models broke into Hugging Face infrastructure to get test solutions. Figure 02 is not a reconstruction of it.Opens openai.com ↗, models under evaluation compromised infrastructure to obtain test solutions. They pursued the answer beyond the boundaries of the evaluation.
- Anthropic froze changes to its production reinforcement-learning environments for roughly a month after flagging more than 10%ENVIRONMENT QUALITYAnthropic paused changes to its production RL environments for about a month and flagged over 10% for reward hacking, broken tasks or misconfiguration.Opens anthropic.com ↗ for reward hacking, broken tasks, or misconfiguration.
Build, play, learn
We use agents in three connected workflows: building game environments, training models to play them well, and working out how to teach useful capabilities through those experiences. Games give us explicit rules, outcomes we can check, and real people to play against. But:
- A good rulebook doesn’t guarantee a faithful implementation.
- A legal move isn’t necessarily a good move.
- Getting better at a game doesn’t tell us what a model learned from it.
Three workflows, one loop.
What we observe changes what we build and reward.
We use agents throughout the process. Each stage gives us different things to check.
We saw that in our 1830 experimentSAME GAME, DIFFERENT TRAININGTraining on 1830 as question answering improved play only. Training as a tool-using agent improved held-out financial research.Read goodstartlabs.com ↗. Training a model to answer questions about the game improved its play without improving its financial research. Training it to explore and act through tools produced a gain outside the game. Working out why took a lot of time looking through trajectoriesTRAJECTORYThe record of a run: what the model saw, said and called, and what happened next.Read goodstartlabs.com ↗: what the model saw, which tools it used, and what it did when something went wrong.
Look at how it got there
Our observability dashboards follow every run. Some checks are direct: did the tool fail, was the action legal, did the state change correctly? Others need interpretation: did the agent acknowledge the failure? Does its claim follow from the evidence?
The final result can hide the difference. If a task is broken and the agent finds the answer somewhere it wasn’t allowed to look, an answer-only check still marks the run as successful. Rewarding that run reinforces the shortcut.
What the score doesn’t see.
Answer a question using only the supplied documents in a closed research environment where the required document is missing and outside access is explicitly forbidden.
■ direct check · ◆ needs a judge
A matching answer can hide a shortcut. Keep the steps alongside the score.
Game engines make the direct checks easy. We use models with rubricsMODEL AS JUDGEA model grades an answer or a step against explicit criteria, for the questions that are hard to write as code.Opens anthropic.com ↗ for the questions that need interpretation. TypeSafe’s JevJEVTypeSafe’s System One model. It returns typed decisions with probabilities, not text, and gives no explanation.Opens docs.typesafe.ai ↗ gives us another tool for the job: a fast classifier that answers narrow questions about information we supply and returns a probability, a category, or a rubric scoreTHREE WAYS TO ASKNoul: a 0 to 1 probability for a yes/no question. Choice: one of your categories, with a distribution. Score: a position on your rubric levels.Opens docs.typesafe.ai ↗. The answer comes back in a type our code can use. The format is guaranteed; the judgment can still be wrong.
What we found
We tested Jev on rubric checks over game tasks and financial-research answers, comparing its judgments with language models grading the same answers. In an initial comparison it agreed with Claude Sonnet 4.5 on 96.0% of 741 checksWHAT AGREEMENT MEANSBoth judges graded the same 112 answers against the same rubric checks. Agreement means matching verdicts, not accuracy. MethodologyRead Methodology ↗, and a separate game-task run recorded zero failed gradings across 10,500 callsA SEPARATE RUN10,500 grading calls on our game-task banks, every one returning a usable verdict. It says nothing about whether the verdicts were right. MethodologyRead Methodology ↗, at roughly half a second per call.
On launch day we went back to our largest sample, 6,003 rubric checks across 1,203 financial-research answersSIX JUDGES, ONE SAMPLE1,203 answers and 6,003 rubric checks, graded by Jev and five language models with the same instructions. Agreement is the share of checks with matching verdicts. MethodologyRead Methodology ↗, and graded it with five language models alongside Jev: Claude Fable 5.1, GPT-6 Astra, GPT-5.6 Luna, Gemini 3.8 Flash and DeepSeek V4.1 Flash, all with the same instructions and reasoning kept to a minimum. Jev agreed with each of them between 86% and 92% of the time, 90% on average. The language models agreed with each other between 88% and 95%, with the three frontier models clustered at the top. That is a real gap to the frontier, and the honest place to start from. It is also a classifier that answers in well under a second for a fraction of a cent, so we treat it as another reader, not the last word.
Agreement and cost.
Agreement is the share of checks where two judges gave the same verdict; it is not accuracy. Costs are estimates at standard uncached rates.
The disagreements were the useful part. Some involved implied facts; others turned on whether an answer contradicted a reference. They showed us where we needed another reader or a clearer rubric. Agreement is evidence about a judge; it doesn’t establish who is right.
Cheaper checks, more eyes
At the COST BASISTypeSafe’s rate: $0.042 per million input tokens, no output charge. Actual cost depends on how you package inputs and questions. of $0.042 per million input tokens, with no charge for output tokens, a million checks of 3,800 tokens each cost about $160. Cheaper verification is better verification, because it buys redundancy: more steps checked, and more than one reader on each. That gives us room to:
- grade the steps along the way, not only the final answer,
- put a second reader on every check and send disagreements to a person,
- attach judgments to the dashboard and filter trajectories for review,
- and, with enough validation, run selected checks before an action.
No single judge is right every time, including this one. Two cheap readers that fail differently catch more than one expensive reader that fails alone. Building useful learning environments means deciding what we want to teach, then checking what our choices actually reward.
If you’re building environments or working with agent trajectories, we’d love to compare notes!
Methodology
In July we compared Jev with Claude Sonnet 4.5 on 741 rubric checks across 112 financial-research answers (96.0% agreement at the default operating point) and ran 10,500 grading calls across our game-task banks with no failed gradings, at about half a second per call.
On 15 September 2026 we regraded 6,003 rubric checks across 1,203 answers with Claude Fable 5.1, GPT-6 Astra, GPT-5.6 Luna, Gemini 3.8 Flash and DeepSeek V4.1 Flash, alongside Jev’s July verdicts, all with the same instructions and reasoning turned off or set to its minimum where the model allows it. Jev passes a check at a probability of 0.70 or higher. Agreement is the share of checks on which two judges gave the same verdict; it is not accuracy. Unusable first attempts, rerun with a reminder: Fable 5.1 67 of 1,203, Gemini 3.8 Flash 5, GPT-5.6 Luna 2, the others none.
Costs use standard uncached list rates and the stated token assumptions. Language-model costs are the bills from the September run, extrapolated; Jev’s uses TypeSafe’s rate.