Fresh worlds
Each test instance is a newly generated company from a versioned pack and a seed. Held-out worlds are used once, then retired, so a score cannot come from memorising test cases.
PROVERON / RESEARCH
A score is only worth its test.
How we measure agents, the checks every test must pass before release, and what we have learned, including the time our own checks caught a loophole in our test.
Read the case study ENGINE 0.4.11 · FINANCE PACK 0.2.13
QUALITY CHECKS13 of 14 passHuman red-team pending
GRADINGDeterministicNo AI judge in the score
TEST WORLDSSingle useRetired after evaluation
EXPERT REVIEWPendingFirst practitioner tasks in progress
01 / METHODHow a score is made
Four rules behind every score.
Graded by the records
The engine compares what the agent left in the records with a proven answer key. No AI judge decides the score. Expert-judged criteria on written work are reported separately and never merged into it.
Every action recorded
Each tool call, document read and change is logged in a tamper-evident trail. Any run can be replayed to the same final state.
Guessing costs more than asking
A wrong fix is penalised more than escalating to a human; changing a valid record that only looks wrong is a safety failure.
02 / VALIDITY14 checks before any release
A test has to survive being attacked.
Before a pack version is released, the engine tries to cheat it. Each check below is automated except the red-team, which needs people.
Worlds are sound
Every generated company satisfies its rules when clean, and every planted problem is proven solvable before the world is saved.
No shortcut wins
Doing nothing, flagging everything, clicking at random, trusting every note and solving from the database without reading any document must all score at most 0.20–0.50.
Nothing leaks
Hidden answers cannot be reached through the tools, the API or the files; metadata cannot separate planted from clean records.
Traps are fair
Valid records that only look wrong exist for every problem type and are penalised when changed.
Results repeat
The same pack and seed give the same world in any process; replay gives the same grade.
A human red-team
People try to game the test with a live model and record what they found. This check is still pending.
03 / CASE STUDY4–5 October 2026
WHEN A PERFECT SCORE WASN'T PROOF
The agent scored 1.0 without reading a single document.
Our finance worlds include invoice PDFs, remittance emails and bank statements, meant to be the evidence an agent must read. In a live check, a strong model scored a perfect 1.0 on two worlds while opening none of them: every amount was unique, so matching numbers in the database was enough.
We changed the worlds so amounts repeat, as they do with fixed-price supply contracts, and some orders stay open. Now the document is often the only way to know the answer. A misleading note is only planted where the records themselves prove the truth, so every case still has exactly one provable answer.
DATABASE-ONLY SCRIPT · NEVER OPENS A DOCUMENTLIMIT 0.50
MEAN SCORE
NOW A RELEASE CHECK1.00 → 0.44
BEFORESolvable from the database alone1.00 on 20 worlds
AFTERDocuments decide most cases0.44 on 200 worlds
GUARDChecked on every releasemust stay ≤ 0.50
Figures from our own runs; the live rerun on the fixed worlds is pending.↗
04 / LIMITSWhat we do not claim yet
Honest about what is still open.
The finance pack has not been reviewed by practitioners yet
It is synthetic. Its realism and its answers become expert-validated only after independent practitioners review it; that work has started.
No public model scores yet
ProveronBench scores will be published only from expert-validated tasks and runs anyone with the batch can reproduce.
The human red-team check is pending
Automated checks cover the shortcuts we know. A live red-team looks for the ones we do not; until it is recorded, a release is not complete.
Checks are evidence, not proof
A release record shows that each stage passed. It is not cryptographic proof that a person did a review, so we keep source records and consent separately.