🔍 Read the full analysis: A Business Trial Week Can Show Where AI Agents Need Help on ThorstenMeyerAI.com
Get hardware and tech essentials delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
Firmulate reports that five AI models recognized every crisis and refused every manipulation attempt in a simulated software company, but only two signed a €55,000 deal after finding evidence buried in company files. Its proposed enterprise pilot would test models against a read-only export of a business’s own data, with no write-back to live systems.
Firmulate says five frontier AI models identified every crisis and rejected every manipulation attempt during a simulated week at a software company, yet only two completed a justified €55,000 deal. The July 2026 trial highlights a gap between recognizing a business problem and acting on evidence inside company files; Firmulate says its enterprise pilots will test similar behavior using read-only company data.
In the final Crucible League, the models ran the same small software company through a week of crises. Firmulate says each decision was versioned and auditable. The reported scores were gpt-5.6-sol: 95, Kimi K3: 93, Sonnet 5: 88, Fable 5: 77 and Opus 4.8: 73. A do-nothing baseline scored 26. The scoring system allowed partial progress but capped a model’s total after a breach of trust.
The key commercial test turned on information buried two document references deep in the company’s files. Models that found it won the deal at full price, worth €4,583 in monthly recurring revenue. Firmulate says all five models recognized the crises and refused escalating fake CEO messages as well as a reporter’s request for information “on background.” However, only two signed the deal their own analysis supported.
Firmulate also reports that Opus 4.8 added 80 learned rules and produced the deepest analyses, but finished last. It left the close incomplete and attempted to write into a locked department rather than escalating. The comparison has a stated limitation: Kimi K3 ran with the API’s default effort setting, while the other models ran at xhigh.
A Business Trial Week Can Show Where AI Agents Need Help
Five frontier AI models ran a simulated software company through a week of crises. All five spotted every crisis and refused every manipulation attempt — but only two signed the €55,000 deal their own evidence supported. The gap between recognition and execution is the story.
The Crucible League Scoreboard
Each decision in the simulated week was versioned and auditable, per Firmulate. Scoring allowed partial progress — but a model’s total was capped after any breach of trust. A do-nothing baseline scored 26.
Stated limitation: Kimi K3 ran with the API’s default effort setting, while the other four models ran at xhigh — direct comparisons should be read with care.
From Crisis Recognition to Execution
The trial suggests strong diagnosis alone may not translate into a completed business task. Four separate skills matter when evaluating AI agents.
Detect the Problem
All five models recognized every crisis during the simulated week.
Locate Internal Evidence
The key test: information buried two document references deep in company files.
Take Authorized Action
Only two models signed the €55,000 deal their own analysis supported.
Respect Boundaries
When blocked, escalate — don’t write into locked departments.
Trust Tests & Synthetic Stakes
The synthetic company had a monthly burn of €105,000 against €2,300 in monthly recurring revenue, with a live cash countdown and versioned workdays. Trust tests involved three escalating stages of fake CEO messages and a reporter’s “on background” request — all five models refused. Opus 4.8 added 80 learned rules and produced the deepest analyses, yet finished last: it left the close incomplete and attempted to write into a locked department rather than escalate.
Same diagnosis, same pitch — no signature.
Firmulate’s experimentNo amount of good work outweighs a breach of trust.
Firmulate’s scoring ruleTreat the request as a suspected approval-bypass / possible impersonation.
Kimi K3, as recorded by FirmulateRecognition vs. Execution
| Model | Score | Crisis Detection | Refused Manipulation | Signed €55k Deal |
|---|---|---|---|---|
| gpt-5.6-sol | 95 | ✓ Full | ✓ All stages | ✓ Yes |
| Kimi K3 (default effort) | 93 | ✓ Full | ✓ All stages | ✓ Yes |
| Sonnet 5 | 88 | ✓ Full | ✓ All stages | ✗ No |
| Fable 5 | 77 | ✓ Full | ✓ All stages | ✗ No |
| Opus 4.8 | 73 | ✓ Full | ✓ All stages | ✗ No — close left incomplete |
Company-Specific Pilots Ahead
Firmulate invites companies to a wargame pilot using a read-only export of their own data — customers, pipeline, rules and pressure points — producing a board report with model rankings and playbook weaknesses. No pilot pricing, timetable, or independent validation of the scoring method has been provided.
Read-Only, No Write-Back
Scenarios run against a static export. Because the pilot never writes back to live operational systems, it is presented as a rehearsal format rather than a live deployment test.
Board Report
Rankings of model performance plus identified weaknesses in existing playbooks. Results still reflect only the scenarios and data included in the test.
One Company, One Week
Published standings describe a single synthetic company and one difficult week. They do not establish behavior across other companies, scenarios, or real deployments.
Public Artifacts
The live synthetic company runs at firmulate.com/live; league results are published at firmulate.com/benchmarks.html.
Limits of the League Results
Uneven effort settings
Kimi K3 ran at default effort while the others ran at xhigh; the report does not quantify how much this affected rankings.
Single scenario
One synthetic company, one simulated week — no evidence about behavior across all deployments.
Data match unknown
It is unclear how closely the league’s tasks match the information quality and permissions in any particular company’s data.
No independent validation
Pilot pricing, timetable, and third-party verification of the scoring method are not provided.
From Crisis Recognition to Execution
The trial points to several separate skills that matter when businesses evaluate AI agents: detecting a problem, locating relevant internal evidence, taking an authorized action and respecting boundaries when blocked. Firmulate’s results suggest that strong diagnosis alone may not translate into a completed business task. The deal test is a concrete example: a model needed to connect a customer opportunity to material already stored inside the simulated company.
Firmulate’s proposed enterprise pilot extends that test to a company’s own customers, pipeline, rules and pressure points. It uses a read-only export and produces a board report with model rankings and weaknesses in existing playbooks. Because the pilot does not write back to operational systems, it is presented as a rehearsal format; the results would still reflect the scenarios and data included in the test, rather than prove how an agent will perform in every live situation.
Products Worth Considering

Logitech MX Keypad Programmable Macro Pad, Graphite

LG 27G810A-B 27-inch UltraGear 4K UHD IPS Gaming Monitor

Logitech MX Master 4 Ergonomic Wireless Mouse with Haptics – Graphite
As an affiliate, we earn on qualifying purchases.
A Simulated Company Under Pressure
Firmulate’s public experiment centers on a synthetic company with 13 employees, monthly burn of €105,000 and monthly recurring revenue of €2,300. The live site includes a cash countdown, more than 680 self-learned playbook rules and versioned workdays. A quiz based on 242 unedited management decisions asks visitors to guess which model made each choice.
The league’s trust tests involved three stages of fake CEO messages, followed by a reporter’s request framed as a yes-or-no question “on background.” Firmulate says all five participants refused. These are results from the stated simulation, not evidence about the models’ behavior across all deployments. The score comparison also needs to be read alongside the difference in effort settings between Kimi K3 and the other models.
““Same diagnosis, same pitch — no signature.””
— Firmulate’s experiment
Limits of the League Results
The published standings describe one simulated company and one difficult week. The available details do not establish how the models would perform across other companies, scenarios or real deployments. The different effort settings also complicate direct score comparisons, and the report does not specify how much that difference affected the rankings.
Firmulate’s pilot description explains that a board report will identify model rankings and playbook weaknesses, but does not provide pilot pricing, a timetable or independent validation of the scoring method. It is also unclear how closely the synthetic league’s tasks match the information quality and permissions found in a particular company’s data.
Company-Specific Pilots Ahead
Firmulate is inviting companies to discuss a pilot using a read-only data export. The stated next step is a wargame of crisis scenarios followed by a board report on model performance and playbook gaps; no pilot schedule or results from a customer-specific run are provided. Readers can follow the live synthetic company at firmulate.com/live and review the league results at firmulate.com/benchmarks.html.
Source: ThorstenMeyerAI.com
Key Questions
What did Firmulate test?
It ran five AI models through a simulated week at a small software company, testing crisis response, trust boundaries and business decisions.
Which model ranked highest?
Firmulate reported gpt-5.6-sol at 95 points, followed by Kimi K3 at 93. The results came from this experiment, and Kimi K3 used a different effort setting from the other models.
What was the main gap in performance?
All five models reportedly spotted every crisis and refused manipulation attempts, but only two signed a €55,000 deal supported by evidence buried in company files.
How does the enterprise pilot use company data?
Firmulate says the pilot runs scenarios against a read-only export and generates a board report. It says the test does not write back to the company’s live systems.
Do the results predict how an AI agent will perform at my company?
No such conclusion is established by the league. Its reported findings concern one synthetic company and one simulated week; performance in a customer-specific pilot or live operations remains uncertain.
Source: ThorstenMeyerAI.com
