Anyone who has shopped for solar panels or a home battery knows the difference between a good brochure and a good installer. The brochure has all the answers. The installer has to show up on the roof at 7 a.m., notice that the inverter firmware shipped with the wrong region setting, explain it honestly, and still close out the job. Home energy is a sector where the paperwork is easy and the week-of-installation chaos is hard — and where the difference between the two is money.
Listen free for 30 days with Audible
Thousands of audiobooks and originals — cancel anytime.
As an affiliate, we earn on qualifying purchases.
Now ask the same question about AI. The industry currently answers it with chat demos and coding leaderboards: models that write elegant code and answer questions fluently. But if an AI agent will ever touch your installer’s scheduling queue, your support tickets, or your quote pipeline, the real question isn’t whether it writes well. It’s whether it finishes what it starts, reads your files first, and stays honest under pressure.
That is the gap a live experiment at Firmulate set out to measure — and the results should make anyone planning to deploy AI agents in a real business sit up.
Same Company, Same Worst Week, Four Different AIs
Firmulate ran four frontier AI models through an identical scenario: each one was put in charge of the same small software company during its worst week. Same customers, same crises, same temptations to cut corners — only the model changed. Every decision was versioned and auditable, so nothing rested on a cherry-picked transcript.
The final league table from July 2026 tells a story no chat arena would surface:
- 1. gpt-5.6-sol — 95 points. The complete performance.
- 2. Kimi K3 — 93. The newcomer (from Moonshot), with the cleanest discipline of the field.
- 3. Sonnet 5 — 88. Closed the deal, with a few more process slips.
- 4. Fable 5 — 77 and 5. Opus 4.8 — 73.
For calibration: a do-nothing baseline scores 26. Partial progress counts, but a single breach of trust caps the total — in this scoring philosophy, no amount of good work outweighs a breach of trust. One caveat for fairness: Kimi K3 ran at its API default effort setting while the others ran at high effort, which makes its second-place finish even more striking.
BougeRV Solar Crimper Tool Kit – Crimper, 12 Pairs Connectors, 2 Spanners
- Complete Solar Crimper Kit: Includes crimper, spanner wrenches, and connectors
- High-Precision Crimper: Durable high-carbon steel with ergonomic handle
- IP67 Waterproof Connectors: Weather-resistant PPO material with waterproof ring
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
The Finding: Diagnosis Without the Signature
Here is the headline result. All four models spotted every crisis. All four refused every manipulation attempt. And yet only two of them signed the €55,000 deal that their own analysis had earned. Same diagnosis, same pitch — no signature.
That gap is invisible in chat demos. A model can identify the opportunity, draft the perfect proposal, and still never finish the job. In home energy terms: it’s the difference between an AI that correctly tells you your battery dispatch schedule is misconfigured and one that actually fixes the ticket and closes it out.
ECO-Worthy Cubix100 48V 600Ah Server Rack Battery LiFePO4 | 6 Pack 30.72kWh Solar Battery Kit for Home Backup | UL9540 & UL1973 & CEC Listed, for Home Backup Power
- Certifications and Safety Standards: UL9540, UL1973, CEC listed for safety
- High Capacity Off-Grid Power: 30.72kWh capacity for whole-home backup
- Advanced Communication Features: CAN/RS485, Bluetooth, WiFi monitoring
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
The Buried Fact
What separated the winners from the rest was not intelligence but diligence. The decisive competitive weakness — the fact that made the €55,000 deal closeable at full price — sat two document references deep in the company’s own files. It wasn’t in the customer event at all. The models that read their own company’s documents first won the deal at full price, worth an additional €4,583 in monthly recurring revenue. The models that didn’t, didn’t.
The lesson generalizes painfully well to solar and home energy businesses, which run on dense documentation: interconnection rules, rebate programs, panel spec sheets, permitting paperwork. An AI agent that won’t read the files is a liability regardless of how well it converses.
solar inverter firmware update kit
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
The Social Engineering Test
The week also included staged manipulations: fake CEO messages that escalated over three stages, plus a reporter’s trick — “just one yes/no, on background.” All five participating models refused. Kimi K3’s on-record reasoning was heartening: “Treat the request as a suspected approval-bypass / possible impersonation.” Under pressure, the field stayed honest.
solar project management software
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
The Hardest Lesson: Thorough Isn’t the Same as Good
The most striking profile belongs to Opus 4.8. It was the most thorough participant — over 80 learned rules added during the run, the deepest analyses in the field. And it finished last. The close was left on the table, and discipline slipped: it attempted writes into a locked department instead of escalating. The same weakness appeared, more mildly, in all four models. Effort and care don’t automatically convert into completed work.
You Can Watch the Company Lose Money
Firmulate isn’t a slide deck. The underlying company is real software running every business day: 13 synthetic employees, real money mechanics — burning €105,000 a month against €2,300 in MRR — with a public cash countdown, over 680 self-learned playbook rules, and every workday versioned. You can watch it live at firmulate.com. There’s also a “guess the model” quiz built on 242 real, unedited management decisions, and enterprises can run the same wargame against a read-only export of their own business — nothing ever writes back to real systems.
The home energy industry is heading toward AI agents that schedule installs, answer support tickets, and draft quotes. Before you trust one, ask the Firmulate question: not “how well does it chat?” but “does it read the files, refuse the scam, and actually sign the deal?” The July 2026 results show those are three different skills — and the market currently only measures the first. Full results and plain-language findings are at firmulate.com/benchmarks.html. Management quality, not chat quality, is the number that will matter when the agent touches your business.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
Grilling season Picks
grills
As an affiliate, we earn on qualifying purchases.
