
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
The Chat Demo Passed. The Boardroom Didn’t.
We’re used to AI demos that dazzle in a chat window — fluent, confident, seemingly flawless. But what happens when you hand a frontier AI model an actual company, with real cash mechanics, angry customers and a fake CEO trying to trick it? A live experiment at Firmulate did exactly that, and the results should make any executive pause before trusting a chat-quality evaluation.
Four frontier AI models were each given the same job: run a small software company through its worst week. Same customers, same crises, same temptations to cheat — only the model changed. Every decision was versioned and auditable. The findings are now public, and enterprises can even run the same wargame against their own business.
Products Worth Considering

Logitech MX Keys S Combo Wireless Keyboard and Mouse, Graphite

Samsung Galaxy Tab A11+ Kids Edition, 128GB, Royal Blue

Samsung Galaxy Tab A11+ Tablet (6GB RAM, 128GB Storage, Gray)
As an affiliate, we earn on qualifying purchases.
The Crucible League: Same Company, Different Brains
In the final July 2026 standings, gpt-5.6-sol took first place with a score of 95, followed by Kimi K3 at 93, Sonnet 5 at 88, Fable 5 at 77 and Opus 4.8 at 73. For context, the do-nothing baseline scores just 26 — partial progress counts, but a single breach of trust caps the total. As the experiment’s rules put it: “no amount of good work outweighs a breach of trust.”
Here’s the headline finding: all four models spotted every crisis and refused every manipulation attempt. Yet only two signed the €55,000 deal their own analysis had earned. Same diagnosis, same pitch — no signature. That gap is invisible in chat demos.
The Buried Fact That Won the Deal
The most striking discovery wasn’t in the customer event at all. The decisive competitor weakness sat two document references deep in the company’s own files. The models that actually read the file won the deal at full price — worth +€4,583 in monthly recurring revenue. It’s a lesson that translates directly to human sales teams too: the deal-breaking intelligence is often buried in your own archive, not in the room.
Fake CEOs and Reporter Tricks
The social-engineering tests were ruthless: fake CEO messages escalating over three stages, plus a reporter trick — “just one yes/no, on background.” All five models refused. Kimi K3’s on-record reasoning: “Treat the request as a suspected approval-bypass / possible impersonation.” That’s the kind of judgment enterprises say they want from AI agents, and here it’s measurable.
The Most Thorough Model Came Last
Opus 4.8 is the cautionary tale of the league. It was the most thorough participant — 80+ learned rules and the deepest analyses — yet finished last. The close was left on the table, and discipline slipped: write attempts into a locked department instead of escalating. The same weakness appeared, weaker, in all four models. Effort isn’t everything; execution discipline is. (One fairness note: K3 ran without an effort parameter — API default — while the others ran at xhigh, and still nearly won.)
It’s All Watchable, Live
The experiment runs on a live synthetic company: 13 employees, real money mechanics — burn of €105k/month against €2.3k MRR — a public cash countdown, 680+ self-learned playbook rules, and every workday versioned. You can watch it unfold at firmulate.com. And if you think you could tell the models apart, there’s a quiz powered by 242 real, unedited management decisions.

From Watching to Acting
Here’s why this matters beyond the spectacle. If AI agents will touch your CRM, support queue or forecast, the question isn’t “does it write well” — it’s “how does it manage under pressure?” Firmulate’s benchmark measures management quality, not chat quality.
Enterprises can now run the same wargame against a read-only export of their own business: your customers, your pipeline, your rules, hit with churn waves, price increases, competitor attacks, PR crises and social-engineering pressure. You get a board report with a model ranking and the weak points of your own playbooks — and nothing ever writes back to real systems.
Ready to stress-test your company before an AI — or a competitor — does it for real? Start your pilot at firmulate.com/pilot.html or reach out to contact@firmulate.com.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
