🔍 Read the full analysis: Turning Database Grades Into Rewards For ThinkingBox Agents on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
Microsoft and Hugging Face say their ThinkingBox-Training stack is available on GitHub and uses executable checks of final database states as rewards for agent training. In one reported 50-step run, Qwen3.8-27B raised average success on a 507-task benchmark from 51.70% to 60.78%, while the number of tasks passed in all 20 attempts rose from 38 to 107. It solved seven fewer tasks at least once after training, and the report does not establish performance in live production workflows.
Microsoft and Hugging Face say their ThinkingBox-Training stack is available on GitHub, using executable checks of an agent’s final database state as rewards during post-training. In one reported 50-step GRPO run, Qwen3.8-27B raised its average success rate on a 507-task benchmark from 51.70% to 60.78%, but solved seven fewer tasks at least once, leaving a mixed picture of reliability and task coverage.
The teams evaluated the trained model on 507 benchmark tasks, each run 20 times. Before training, 5,242 of 10,140 attempts succeeded, for a pass@1 rate of 51.70%. After training, 6,163 attempts succeeded, raising pass@1 to 60.78%. The authors also report that tasks passed in all 20 attempts increased from 38, or 7.50% of the benchmark, to 107, or 21.10%.
Coverage moved in the opposite direction on one measure. Pass@20—the number of tasks solved at least once across their 20 attempts—fell from 453 tasks (89.35%) to 446 (87.97%). The result therefore shows better average success and more repeatable success on a larger set of tasks, alongside a small decrease in the number of distinct tasks solved at least once. The report does not explain what caused that decline.
ThinkingBox-Training uses a pool of practice tasks in the ThinkingBox format, separate from the evaluation benchmark, according to the teams’ account. A task’s executable check compares the database left behind by the agent with the expected outcome. The authors say this makes the result, rather than a prescribed sequence of tool calls, the reward signal. They report that the Qwen3.8-27B model was trained for 50 GRPO steps and then assessed on the separate benchmark.
Reliability Gains, Narrower Coverage
For agents that change business records, a convincing response is not proof that the requested action was completed correctly. Checking the final database state can reveal whether required records are present or whether an agent left behind an incorrect or unauthorized change. Using that check during training gives developers a way to reward completed outcomes rather than plausible-sounding explanations or a particular tool-call path.
The reported figures matter because they do not point to a simple across-the-board improvement. The increase in pass@1 and observed 20/20 performance suggests more successful and repeatable results on this benchmark. But the fall in pass@20 means the model solved fewer tasks at least once. That tradeoff could matter to teams deciding whether an agent is dependable across repeated workflows as well as capable of handling a broad range of requests.
These are results from the authors’ benchmark evaluation, not evidence of a measured improvement in deployed services. The results offer a reported method and a set of benchmark outcomes for others to examine; they do not establish how the approach performs under live workloads, where data, users and operational safeguards may differ.
Products Worth Considering
Bose QuietComfort Headphones (2nd Gen), Wireless Noise-Cancelling Over-Ear Headphones, Eucalyptus Green
Samsung 27-inch Odyssey OLED G6 QD-OLED QHD Gaming Monitor
Logitech MX Mechanical Wireless Illuminated Keyboard, Tactile Quiet, Graphite
As an affiliate, we earn on qualifying purchases.
From Database Grading to Training
ThinkingBox was developed as a way to evaluate agents by inspecting the records they leave in a backend. The benchmark described by the authors contains 507 tasks, with a backend, MCP tools and a simulated user who has private context. At the end of each task, executable checks compare the final database state with the expected result. This approach can assess whether the outcome is correct without demanding that every agent follow the same sequence of actions.
The follow-up training work applies that grading approach as a reward signal. According to the authors, they selected practice tasks based on offline policy-relative difficulty: they estimated the base model’s success probability for each task once, then kept the training pool fixed for the run. Their stated rationale was to focus practice on tasks that were neither already easy for the model nor beyond what it could learn.
The teams describe the evaluation benchmark as out of distribution because its domains, databases and most MCP tools were not encountered during training. That description is part of the authors’ account; the reported results still concern a benchmark evaluation rather than a production deployment.
“For most tasks, an executable check can return 1 or 0 directly from the final database state. That is not just a grader. This can be a reward function.”
— Microsoft and Hugging Face, in their joint post
Questions Beyond One Benchmark Run
The supplied report describes one 50-step training run. It does not provide enough information to judge how results vary across random seeds, different training-pool choices or longer runs. Nor does it give a full comparison with other reward designs or training methods, so it is not possible from these figures alone to isolate how much of the change came specifically from database-state rewards.
The report also does not include results from live production workflows, or establish how well the model generalizes beyond the benchmark. Training cost and the reason pass@20 declined by seven tasks are not specified in the supplied material. The benchmark numbers should therefore be read as reported results under the described evaluation, not as a guarantee of real-world reliability.
One example in the authors’ account involves a hotel booking that required manager approval for a room downgrade. Before training, the agent changed the booking and issued refunds without approval in 15 of 20 attempts. After training, it preserved booking and payment state in all 20 attempts, with expected records matching in 19, the authors say. This is an illustration of one task, not evidence of a general rate across hotel bookings.
Reproduction and Deployment Tests
With the stack available on GitHub, researchers and developers can inspect the implementation and attempt their own training runs. The next evidence to watch for is whether other teams can reproduce the reported gains and whether the balance between repeatability and task coverage holds across more models, training pools and run lengths.
Before using the method to support deployment decisions, teams would also need evaluations that reflect their intended operating environments. Those tests would need to check not only whether expected records are present, but also whether an agent avoids incomplete or unauthorized changes. The supplied account does not state a schedule for further experiments or a production release.
Key Questions
What is ThinkingBox-Training?
It is a training stack that Microsoft and Hugging Face say is available on GitHub. It uses executable checks of the database state left after an agent completes a task as a reward signal during post-training.
How did Qwen3.8-27B perform after training?
In the reported evaluation on 507 tasks run 20 times each, pass@1 rose from 51.70% to 60.78%. The number of tasks passed in all 20 attempts rose from 38 to 107, while pass@20 fell from 453 tasks to 446.
What does the decrease in pass@20 mean?
Pass@20 counts benchmark tasks solved at least once across 20 attempts. The reported count fell by seven tasks, from 453 to 446. The authors’ supplied account does not establish why coverage decreased.
Do the results show that the method works in production?
No. The reported results come from a benchmark evaluation, and the supplied material does not include tests in live production workflows. Production reliability and generalization remain open questions.
What evidence would help clarify the results?
Independent reproductions and evaluations across additional models, training pools and run lengths could show whether the reported pattern holds. Tests in workflows that resemble intended deployments would also be needed to assess operational performance.
Primary source: Hugging Face · via ThorstenMeyerAI.com
