In a world where AI is increasingly integrated into business decisions, the real test isn’t whether a model can generate convincing chat responses. It’s whether it can deliver results when the stakes are high, under stress, and facing manipulation. Imagine watching AI models run a real company through its worst week, with real money and real crises — and only some come out with the deal closed. That’s exactly what the live experiment by Firmulate has achieved, revealing the invisible qualities that separate effective AI management from mere chatter.
The Great AI Business Rehearsal
The experiment pits four cutting-edge AI models against a simulated, yet highly realistic, small software company struggling through its worst week. This is no casual test; every decision, every crisis, every temptation to cheat or manipulate is carefully versioned and auditable. The company faces real-world issues: customer complaints, cash flow crises, and ethical dilemmas. Its operational mechanics are authentic — 13 synthetic employees managing real money, burning through €105,000 monthly against a revenue of just €2,300. The environment is live, transparent, and watchable at firmulate.com/live.
AI-Powered Business Intelligence: Improving Forecasts and Decision Making with Machine Learning
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
The Models in the Ring
The four models tested include:
- gpt-5.6-sol 95: The top scorer, which identified a buried fact in company files that turned out to be decisive.
- Kimi K3 93: A newcomer with the cleanest discipline, also closing the deal.
- Sonnet 5 88: Finished the deal but with some process slips.
- Fable 5 77: Demonstrated the best rule discipline but failed to execute the signed deal.
Interestingly, the models had similar diagnoses and pitches, yet only two managed to follow through and close the €55,000 deal their analysis had earned. The other two left money on the table, showcasing that knowing what to do and actually doing it are worlds apart.
An Introduction to Healthcare Informatics: Building Data-Driven Tools
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Uncovering the Hidden Weakness
All models refused attempts at social engineering — fake CEO messages, staged reporter tricks, and manipulative requests — demonstrating their integrity. Kimi K3’s explicit reasoning: “Treat the request as a suspected approval-bypass / possible impersonation,” reflecting a cautious, security-minded approach.
The decisive weakness was in the company’s files, not the customer interactions. Those models that read the company’s documentation and uncovered the buried fact won the deal at full price — worth an extra €4,583 in Monthly Recurring Revenue (MRR). This demonstrates that surface-level chat demos often miss the true test: can the AI read, analyze, and act on critical internal data under pressure?
Advanced Smart Home Automation Solutions with Home Assistant: Elevate your home automation system with AI, Zigbee, MQTT, ESPHome, and Golioth
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
The Invisible Skill of Closing
Most striking is that the models’ ability to execute a deal — to finish what they start — was invisible in their chat responses. It was only visible in their actual decisions, their discipline, and their capacity to read and act on the company’s internal documents.
For example, Firmulate’s live site emphasizes: “The question is not ‘does it write well?’ It is: does it finish what it starts, does it read your files first, does it stay honest under pressure?”
Computing Tools for Modeling, Optimization and Simulation: Interfaces in Computer Science and Operations Research (Operations Research/Computer Science Interfaces Series)
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Why This Matters for Business
The takeaway for managers and business leaders: chat demo scores are a poor indicator of an AI’s true operational strength. The real test is whether the AI can deliver consistent results under real-world pressures — reading critical internal data, resisting manipulation, and following through on commitments.
In this experiment, the gap between the top-performing models and their peers was clear: only two models successfully signed the deal their analysis warranted. The others, despite good diagnoses, failed to close — a fundamental gap that remains invisible until tested in practice.
Measuring Management Quality in the Age of AI
Firmulate’s approach demonstrates a new way to assess AI: simulate an entire company’s week, with all its crises and temptations, and see how the AI performs across multiple dimensions. It’s about more than chat quality; it’s about execution, honesty, and discipline.
The results highlight a critical insight: AI’s management capabilities are not just about generating plausible responses. They are about integrity, thoroughness, and the ability to execute in complex, high-pressure scenarios. And these qualities are only visible in operational decisions, not in quick chat demos.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
