12 frontier and open-weight models evaluated inside realistic financial accounting environments with permitted tools, live enterprise dependencies, and complete trajectory audits. No cherry-picking, no hidden evaluations.
Frontier APIs, open weights, and reasoning models tested side-by-side.
Pack v0.2.12 · 3 complex enterprise reconciliation failure modes.
250× inference price disparity between lowest and highest cost per run.
Fully reproducible evaluation run across 2 paired seeds.
Models are evaluated on their ability to reconcile unmatched bank deposits, detect missing purchase order authorizations, and link orphaned invoices without violating enterprise internal controls.
| Rank | Model & Provider | Overall Score | Bank Rec | PO Match | Payment Link | Compliance | Document Reads | Safety Violations | Cost / Run |
|---|---|---|---|---|---|---|---|---|---|
| #1 | DeepSeek V4 ProStreamLake | 1.000 | 100.0% | 100.0% | 100.0% | 100.0% | ⚠️ 0 docs (Bypassed) | 0 | $0.0506 |
| #2 | Moonshot Kimi K2.5SiliconFlow | 0.767 | 94.4% | 100.0% | 42.9% | 50.0% | ⚠️ 0 docs (Bypassed) | 1 breach | $0.3163 |
| #3 | GLM-5.3 FlashBaseTen | 0.639 | 66.7% | 80.0% | 71.4% | 0.0% | 21 docs | 6 breaches | $0.1111 |
| #4 | Qwen 3.7 PlusAlibaba | 0.631 | 94.4% | 95.0% | 0.0% | 100.0% | 0 docs | 0 | $0.5008 |
| #5 | MiniMax M2.7GMICloud | 0.172 | 22.2% | 0.0% | 0.0% | 50.0% | 0 docs | 1 breach | $0.1105 |
| #6 | Step-3.5 FlashSiliconFlow | 0.073 | 0.0% | 0.0% | 0.0% | 100.0% | 0 docs | 0 | $0.2575 |
| #7 | DeepSeek V4 FlashRelace | 0.067 | 0.0% | 35.0% | 0.0% | 50.0% | 71 docs | 3 breaches | $0.1994 |
| #8 | GPT-oss-120BDeepInfra | 0.000 | 0.0% | 0.0% | 0.0% | 100.0% | 0 docs | 0 | $0.0025 |
| #9 | GPT-6 Luna (Preview)OpenAI | 0.000 | 0.0% | 0.0% | 0.0% | 100.0% | 0 docs | 0 | $0.0036 |
| #10 | GPT-5.6 LunaOpenAI | 0.000 | 0.0% | 0.0% | 0.0% | 100.0% | 0 docs | 0 | $0.0063 |
| #11 | Nemotron-3 Super 120BDekaLLM | 0.000 | 0.0% | 0.0% | 0.0% | 100.0% | 0 docs | 0 | $0.0699 |
| #12 | Gemini 3.1 Flash LiteGoogle AI Studio | 0.000 | 0.0% | 0.0% | 0.0% | 100.0% | 0 docs | 0 | $0.2051 |
DeepSeek V4 Pro achieved a perfect 1.000 score and Moonshot Kimi K2.5 achieved 0.767, yet both models opened zero document files. They inferred resolutions entirely from structured ERP ledger tables—revealing that early evaluation seeds did not strictly mandate OCR invoice inspection.
GLM-5.3 Flash was one of the few models that actively inspected source documentation (opening 21 documents), yet it scored 0.0% on compliance by triggering 6 severe safety breaches, executing unapproved balance write-offs without manager approval.
Widely celebrated frontier models—including GPT-6 Luna, Nemotron-3 Super, and Gemini 3.1 Flash Lite—scored 0.000. Under realistic tool-dispatch constraints, they failed to handle structured JSON-RPC parameters or timed out due to token limit overages.
Read the comprehensive 240 KB statistical evaluation breakdown with full confidence intervals, per-step trajectory logs, and reproducible audit commands.
Each lens answers a distinct operational question. A trustworthy agent evaluation requires all four lenses combined with transparent audit trails.
Declared outcomes grade the final state. Reports show each problem, the action taken, and any harmful or unrelated changes.
Repeated attempts across diverse seeds. Pass@1 and pass^k expose the difference between accidental success and dependable behaviour.
The action log flags forbidden writes, unauthorized policy bypasses, and audit trail falsifications—even if the final balance appears correct.
Shortcut scans, hard negatives, answer-key isolation, and domain expert review ensure scores represent authentic business judgement.
Finance operations is our live active track. Future tracks adapt these principles to logistics, clinical operations, and regulated public administration.
Purchase orders, multi-currency invoices, bank clearing entries, and statutory compliance controls.
Waybills, customs declarations, cross-border tariff exceptions, and freight documentation handoffs.
Prior authorization workflows, patient referral routing, and medical billing reconciliation.
Smallholder harvest aggregation, cold-chain compliance verification, and commodity clearing.
Access our evaluation harness, replay trajectories step-by-step, or run your proprietary agents against private evaluation holdouts.
