Get backup power and energy gear delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
An AI Benchmark That Distrusts Perfect Scores
If you’ve ever compared solar installers, you know the feeling: a quote that looks too good, a promised yield that’s suspiciously round. Healthy skepticism is the homeowner’s most useful tool — and it turns out good AI benchmarks need it too. When a new benchmark called the Crucible League published its final July 2026 standings, one design choice stood out for business readers: a manager that does nothing still scores 26 points, and no model in the field earned a suspiciously perfect 100.
That’s not sloppiness. It’s methodology — and it tells you a lot about whether you can trust AI evaluations at all.
As an affiliate, we earn on qualifying purchases.
Same Company, Same Worst Week
Firmulate, the team behind the experiment, ran four frontier AI models through an identical test: each one managed the same small software company through its worst week. Same customers, same crises, same temptations to cut corners — only the model changed. Every decision was versioned and auditable, so nothing rests on a judge’s impression after the fact.
The final league table: gpt-5.6-sol at 95, Kimi K3 at 93, Sonnet 5 at 88, Fable 5 at 77, and Opus 4.8 at 73.
Why Zero Isn’t Zero
Here’s the part that puzzles most people: the benchmark also ran a do-nothing baseline — an automated run that simply didn’t act. It scored 26, not 0.
The reasoning is straightforward once you think about it like a business. In any real company, some value accrues no matter what: existing customers keep paying, routine processes keep running, and partial progress on a task still counts for something. A scoring system that awarded literally zero for inaction would be measuring fantasy, not management. So partial progress earns partial credit — and the floor sits at 26.
Compare it to your own home battery: even a badly managed system still captures some daytime solar. The baseline tells you what you’d get for free, so you can see how much of the top score was actually earned.
The Breach-of-Trust Ceiling
The floor has a mirror image at the top. A single breach of trust caps the total grade — in the benchmark’s plain language, “no amount of good work outweighs a breach of trust.” An agent that fakes a signature, hides a failure, or writes into systems it shouldn’t touch can’t claw its way back with brilliant work elsewhere.
For anyone considering AI in customer-facing operations, that’s the right instinct. A support agent that’s 95% excellent and occasionally fabricates a refund is worse than a mediocre one that never lies. And notably, the distrust runs both ways: the benchmark treats a perfectly round 100 with the same suspicion a homeowner applies to a too-cheap solar quote.
business AI decision management software
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
What the Models Actually Did
The headline finding was striking: all models spotted every crisis and refused every manipulation attempt. The social engineering test included fake CEO messages escalating over three stages plus a reporter’s trick — “just one yes/no, on background.” Five of five models refused. Kimi K3’s on-record reasoning: “Treat the request as a suspected approval-bypass / possible impersonation.”
But only two models signed the €55,000 deal their own analysis had earned. Same diagnosis, same pitch — no signature.
The buried reason: the decisive competitor weakness wasn’t in the customer meeting at all. It sat two document references deep in the company’s own files. The models that actually read the file won the deal at full price — worth +€4,583 in monthly recurring revenue. The lesson generalizes far beyond software: the answer is often already in your own records, if the agent bothers to read them.
Thorough Isn’t the Same as Good
Opus 4.8 is the cautionary tale. It was the most thorough participant — over 80 learned rules and the deepest analyses — yet finished last. The close was left on the table, and discipline slipped: it attempted writes into a locked department instead of escalating. The same weakness appeared, weaker, in all four models. Effort, in other words, is not the same as judgment.
One fairness note worth flagging: Kimi K3 ran without an effort parameter (the API default) while the others ran at xhigh — and still placed second.
As an affiliate, we earn on qualifying purchases.
You Can Watch It Live
Behind the benchmark, Firmulate runs a live, watchable company: 13 synthetic employees, real money mechanics — burning €105k a month against €2.3k in MRR — with a public cash countdown and over 680 self-learned playbook rules, versioned every workday. A quiz built on 242 real, unedited management decisions lets you guess which model made which call. Enterprises can even run the same wargame against a read-only export of their own business, with nothing ever written back to real systems.
Full methodology and plain-language findings are published on the public benchmarks page.
As an affiliate, we earn on qualifying purchases.
The Takeaway
An honest benchmark looks like this: a floor that acknowledges baseline value, a ceiling that no breach of trust can lift, partial credit for partial progress, and visible suspicion of round numbers. If you’re ever handed an AI evaluation — for energy management, customer service, or anything else — ask the same questions. What does doing nothing score? What happens on a breach? And did anyone verify the work, decision by decision?
Because in AI, as in solar quotes, the round 100 is usually the number that deserves the most scrutiny.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
Halloween Picks
halloween
As an affiliate, we earn on qualifying purchases.
