AI Models Show Hidden Strengths and Weaknesses in Business Crisis Tests

In the rapidly evolving world of AI, what truly matters isn’t just how convincingly a model can chat—it’s whether it can get the job done when real pressure hits. Recent experiments reveal a stark truth: some AI systems not only spot crises but also seal the deal, while others falter despite impressive chat demos.

Testing AI in the Real World: More Than Just Chat

Imagine running four different AI models through the same simulated week of a small software company facing multiple crises—ranging from customer issues to internal manipulations. The goal wasn’t just to see which AI could hold a convincing conversation but to assess their actual management capabilities under pressure. This experiment was conducted by Firmulate, a company dedicated to measuring AI management performance in real-world scenarios.

The Experiment Setup

Each model was tasked with navigating the same set of crises, making decisions, and ultimately closing a critical €55,000 deal. All decisions were versioned and auditable, ensuring that each step was transparent and comparable. The models, from the latest GPT-5.6 to a custom enterprise system called Kimi K3, faced identical temptations—like social engineering attacks and fake CEO messages—yet only some rose to the occasion.

The Results: Crisis Detection and Deal Closure

Remarkably, all four AI models detected every crisis presented, refusing manipulation attempts at every turn. That’s a clear sign: today’s models can recognize trouble when it’s staring them in the face. But here’s where things get revealing: only two models managed to close the deal that their own analysis had earned them. The others left the opportunity on the table, despite having diagnosed the issues correctly.

For instance, the top-scoring system, GPT-5.6, identified the buried fact hidden within company files—information crucial to sealing the deal—and successfully closed at full price. In contrast, the poorest performer, Fable 5, failed to execute the deal, even though it knew what was necessary. That’s a stark illustration: chat demos and superficial assessments don’t reveal an AI’s true management capabilities.

Discipline Under Pressure and the Hidden Weaknesses

Digging deeper, the experiment uncovered a significant weakness. The AI models that relied heavily on reading detailed internal documents were more successful at closing deals, while those that failed to read or escalate proper processes left opportunities unfulfilled. For example, Opus 4.8, the most thorough participant with over 80 learned rules, was last in closing, illustrating that thoroughness alone doesn’t guarantee execution.

Beyond Chat: Measuring the Invisible

This experiment demonstrates that the real strength of AI in business isn’t just in how well it can mimic human conversation but in its ability to follow through, read critical information, and resist manipulation when it matters most. As AI starts touching your CRM, support queues, or forecasting tools, the question isn’t whether it can talk convincingly—it’s whether it can deliver measurable business outcomes under real pressure.

Why This Matters for Business Leaders

Current AI benchmarks often focus on chat quality or superficial scores. But as this test shows, the true test of management AI is its discipline and execution capabilities. Companies considering AI integration should look beyond chat demos and ask: Can my AI finish what it starts? Will it read the essential files before acting? And, crucially, under pressure—will it stay honest and deliver results?

More Than a Test: A Benchmark for the Future

Firmulate’s ongoing league table ranks models by their real-world management scores, with GPT-5.6 leading at 95, closely followed by Kimi K3 at 93. Despite high scores, only the top two models managed to close the deal, highlighting that scoring high doesn’t necessarily mean successful execution. The key takeaway: measurable management strength—such as closing deals and resisting manipulation—is invisible in chat demos but vital for real-world success.

See It Live

For business leaders and developers wanting to test their own AI systems, Firmulate offers live experiments and pilots. These wargames simulate real crises, decisions, and manipulations, providing a clear picture of an AI’s management capabilities before deployment. Visit firmulate.com to watch these experiments unfold and to learn how your AI can be truly tested in the wild.

Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html

Powered by Thorsten Meyer AI