Beyond the score.
The leaderboard shows who comes out ahead. These charts help explain why: whether the ranking holds up, how agents dig into problems, when they ask the client, what goes wrong, and which methods they reach for. Each chart redraws a figure from the paper; open “View chart values” under it for the exact numbers.
Does the ranking hold up under stricter scoring?
DAQ is an average, so strong results on some tasks can make up for failures on others. DAQα adds a setting, α: the higher it is, the harder low-scoring tasks pull the score down.
Stricter scoring widens the gap to the experts
Each line follows one model as α rises from 1 to 3. Every model’s score falls, the order never changes, and the experts stay at 1 throughout.
Hover or drag across the chart to read every model at one α. Select a model above to highlight it.
The dashed line marks the experts at 1. At α = 1, each value is the model’s DAQ from the leaderboard divided by 100.
View chart values
| α | Claude Opus 5 | DeepSeek-V4.1-Flash | Doubao-Seed-2.1-Pro | Gemini 3.8 Flash | GLM-5.3-Flash | GPT-5.6 Sol | Grok 4.6 | Hy3 | Hy4 preview | Muse Spark 1.3 | Qwen3.8-Max |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 1.0 | 0.774 | 0.693 | 0.648 | 0.829 | 0.632 | 0.850 | 0.865 | 0.438 | 0.669 | 0.729 | 0.780 |
| 1.1 | 0.754 | 0.667 | 0.620 | 0.813 | 0.603 | 0.835 | 0.853 | 0.403 | 0.641 | 0.706 | 0.760 |
| 1.2 | 0.735 | 0.643 | 0.594 | 0.796 | 0.574 | 0.820 | 0.841 | 0.371 | 0.615 | 0.684 | 0.742 |
| 1.3 | 0.716 | 0.619 | 0.568 | 0.780 | 0.548 | 0.806 | 0.829 | 0.342 | 0.590 | 0.663 | 0.724 |
| 1.4 | 0.698 | 0.597 | 0.544 | 0.765 | 0.522 | 0.792 | 0.818 | 0.316 | 0.565 | 0.643 | 0.706 |
| 1.5 | 0.680 | 0.575 | 0.521 | 0.750 | 0.498 | 0.778 | 0.807 | 0.291 | 0.542 | 0.623 | 0.689 |
| 1.6 | 0.663 | 0.555 | 0.499 | 0.735 | 0.475 | 0.765 | 0.796 | 0.269 | 0.520 | 0.604 | 0.672 |
| 1.7 | 0.646 | 0.535 | 0.477 | 0.721 | 0.453 | 0.751 | 0.785 | 0.248 | 0.499 | 0.586 | 0.656 |
| 1.8 | 0.629 | 0.515 | 0.457 | 0.707 | 0.432 | 0.739 | 0.775 | 0.229 | 0.479 | 0.568 | 0.640 |
| 1.9 | 0.613 | 0.497 | 0.438 | 0.693 | 0.412 | 0.726 | 0.764 | 0.211 | 0.460 | 0.551 | 0.625 |
| 2.0 | 0.598 | 0.479 | 0.419 | 0.680 | 0.393 | 0.714 | 0.754 | 0.195 | 0.441 | 0.535 | 0.610 |
| 2.1 | 0.583 | 0.462 | 0.402 | 0.666 | 0.375 | 0.701 | 0.744 | 0.180 | 0.423 | 0.519 | 0.595 |
| 2.2 | 0.568 | 0.445 | 0.385 | 0.654 | 0.358 | 0.689 | 0.735 | 0.166 | 0.406 | 0.503 | 0.581 |
| 2.3 | 0.553 | 0.429 | 0.368 | 0.641 | 0.341 | 0.678 | 0.725 | 0.153 | 0.390 | 0.488 | 0.567 |
| 2.4 | 0.539 | 0.414 | 0.353 | 0.629 | 0.325 | 0.666 | 0.716 | 0.141 | 0.374 | 0.474 | 0.554 |
| 2.5 | 0.526 | 0.399 | 0.338 | 0.617 | 0.311 | 0.655 | 0.707 | 0.130 | 0.359 | 0.459 | 0.541 |
| 2.6 | 0.512 | 0.385 | 0.324 | 0.605 | 0.296 | 0.644 | 0.698 | 0.120 | 0.345 | 0.446 | 0.528 |
| 2.7 | 0.499 | 0.371 | 0.310 | 0.593 | 0.283 | 0.633 | 0.689 | 0.111 | 0.331 | 0.432 | 0.515 |
| 2.8 | 0.487 | 0.358 | 0.297 | 0.582 | 0.270 | 0.622 | 0.680 | 0.103 | 0.317 | 0.420 | 0.503 |
| 2.9 | 0.474 | 0.345 | 0.284 | 0.570 | 0.257 | 0.612 | 0.671 | 0.095 | 0.305 | 0.407 | 0.491 |
| 3.0 | 0.462 | 0.333 | 0.273 | 0.560 | 0.245 | 0.602 | 0.663 | 0.087 | 0.292 | 0.395 | 0.479 |
Do agents get to the bottom of problems?
When a run fails or a result looks wrong, a good analyst guesses the cause and explains it, runs a test, reads the result, and then changes the model. The paper’s experts traced how far agents got each time.
How far agents get before they stop
Each bar splits one model’s cases by the last step it reached, and adds up to 100%. Every task counts equally.
Hover over a segment to see its percentage. With a keyboard, Tab to a segment and use the arrow keys. On touch screens, tap a segment to show or hide its value.
Where the numbers come from. Whole percentages marked “printed” are copied from labels in the paper’s figure; the rest are measured from the figure and are approximate.
View chart values
| Model | No hypothesis | Hypothesis only | Explained, not tested | Tested, results not read | Results read, nothing changed | Went all the way |
|---|---|---|---|---|---|---|
| Claude Opus 5 | 19.0% (printed) | 11.5% | 11.4% | 0.0% | 17.0% (printed) | 41.0% (printed) |
| DeepSeek-V4.1-Flash | 16.0% (printed) | 14.0% (printed) | 18.0% (printed) | 0.0% | 16.0% (printed) | 35.0% (printed) |
| Doubao-Seed-2.1-Pro | 19.0% (printed) | 16.0% (printed) | 14.0% (printed) | 1.8% | 6.2% | 43.0% (printed) |
| Gemini 3.8 Flash | 26.0% (printed) | 10.1% | 14.0% (printed) | 1.1% | 6.3% | 42.0% (printed) |
| GLM-5.3-Flash | 12.9% | 18.0% (printed) | 23.0% (printed) | 0.3% | 11.3% | 35.0% (printed) |
| GPT-5.6 Sol | 41.0% (printed) | 16.0% (printed) | 10.8% | 3.8% | 1.9% | 26.0% (printed) |
| Grok 4.6 | 14.0% (printed) | 19.0% (printed) | 9.3% | 2.0% | 11.2% | 45.0% (printed) |
| Hy3 | 12.2% | 18.0% (printed) | 19.0% (printed) | 0.0% | 12.2% | 39.0% (printed) |
| Hy4 preview | 6.8% | 18.0% (printed) | 22.0% (printed) | 0.5% | 12.0% | 41.0% (printed) |
| Muse Spark 1.3 | 29.0% (printed) | 15.0% (printed) | 11.9% | 0.0% | 20.0% (printed) | 24.0% (printed) |
| Qwen3.8-Max | 16.0% (printed) | 14.0% (printed) | 21.0% (printed) | 0.9% | 21.0% (printed) | 27.0% (printed) |
Careful digging does not guarantee a good result
Across: how well each step was done, as rated by experts, counting only the cases a model followed all the way through. Up: the model’s overall score (DAQ).
The rating covers only cases followed all the way through; DAQ counts plans that break a rule as zero. Qwen3.8-Max has the best rating but finishes only 27% of its cases, while GPT-5.6 Sol scores high with a lower rating.
View chart values
| Model | Expert rating (%) | DAQ (%) |
|---|---|---|
| Claude Opus 5 | 59.7 | 77.44 |
| DeepSeek-V4.1-Flash | 61.2 | 69.28 |
| Doubao-Seed-2.1-Pro | 62.6 | 64.82 |
| Gemini 3.8 Flash | 58.9 | 82.94 |
| GLM-5.3-Flash | 61.0 | 63.21 |
| GPT-5.6 Sol | 45.8 | 85.02 |
| Grok 4.6 | 57.3 | 86.55 |
| Hy3 | 52.7 | 43.77 |
| Hy4 preview | 57.0 | 66.85 |
| Muse Spark 1.3 | 57.6 | 72.86 |
| Qwen3.8-Max | 67.2 | 77.97 |
When do agents ask the client?
Agents can ask ORSim, the simulated client, at any stage. In practice they ask most of their questions before they start writing solver code.
When agents ask, and whether it helps
Bubble size: the share of a model’s questions asked at that stage. Color: how often those questions ended up fixing a real problem.
How to read this. A question counts as fixing a real problem only if it was about a concrete issue, the agent acted on the answer, and the result confirmed the fix. Routine questions and unclear outcomes still count in the total. Later stages have few questions, so their percentages rest on small numbers. Values are approximate readings of the paper’s figure.
View chart values
| Model | Before writing solver code | Before the first solver result | Before a first solution | Before a complete solution | Before passing its own checks | Before submitting |
|---|---|---|---|---|---|---|
| Claude Opus 5 | 94.8% / 32.6% | 1.1% / 66.7% | 0.3% / 0.0% | 2.2% / 30.0% | 0.9% / 0.0% | 0.7% / 0.0% |
| DeepSeek-V4.1-Flash | 90.8% / 32.3% | 0% / – | 0.9% / 0.0% | 6.2% / 56.2% | 1.7% / 0.0% | 0.4% / 0.0% |
| Doubao-Seed-2.1-Pro | 92.7% / 19.4% | 0% / – | 2.6% / 71.4% | 3.8% / 50.0% | 0.8% / 66.7% | 0% / – |
| Gemini 3.8 Flash | 91.4% / 15.4% | 0.1% / 100.0% | 0.8% / 20.0% | 7.3% / 8.9% | 0.4% / 100.0% | 0% / – |
| GLM-5.3-Flash | 93.7% / 37.0% | 0% / – | 0.8% / 50.0% | 3.7% / 2.8% | 1.8% / 0.0% | 0% / – |
| GPT-5.6 Sol | 86.7% / 16.7% | 0.2% / 0.0% | 0.8% / 42.9% | 6.6% / 22.9% | 2.2% / 100.0% | 3.5% / 19.0% |
| Grok 4.6 | 98.3% / 13.4% | 0% / – | 0.9% / 0.0% | 0.4% / 50.0% | 0.5% / 50.0% | 0% / – |
| Hy3 | 94.4% / 34.6% | 0% / – | 0% / – | 1.1% / 100.0% | 3.7% / 100.0% | 0.7% / 0.0% |
| Hy4 preview | 87.1% / 34.5% | 0.3% / 0.0% | 2.8% / 16.7% | 3.4% / 57.1% | 4.8% / 71.4% | 1.7% / 50.0% |
| Muse Spark 1.3 | 93.4% / 10.6% | 0% / – | 0.6% / 0.0% | 4.5% / 21.7% | 0.6% / 50.0% | 1.0% / 0.0% |
| Qwen3.8-Max | 95.3% / 29.3% | 0.5% / 100.0% | 0% / – | 2.8% / 60.0% | 0.5% / 0.0% | 1.0% / 50.0% |
What goes wrong?
Experts went through each agent’s work and sorted every mistake by where it started. For 9 of the 11 models, the most common mistakes are in how the problem was set up, not in the code.
How often plans break a rule, and why
Bar length: the share of tasks where the model’s plan broke a rule (100% minus its feasibility). Colors: the mix of mistakes experts found in its work.
Hover over a segment to see that kind of mistake’s share and the model’s failure rate. With a keyboard, Tab to a segment and use the arrow keys. On touch screens, tap to show or hide the values.
Colors show each model’s mix of mistakes, stretched to the length of its bar. One run can contain more than one mistake.
View chart values
| Model | Plans that break a rule | Misread the problem | Wrong data or parameters | Decisions set up wrong | Rules missed or wrong | Changes over time mishandled | Uncertainty mishandled | Wrong goal or priorities | Code or output errors |
|---|---|---|---|---|---|---|---|---|---|
| Claude Opus 5 | 22.6% | 4.9% | 14.0% (printed) | 0.0% | 42.7% (printed) | 5.6% | 6.3% | 12.6% (printed) | 14.0% (printed) |
| DeepSeek-V4.1-Flash | 30.3% | 16.7% (printed) | 2.8% | 0.0% | 14.8% (printed) | 4.6% | 15.7% (printed) | 28.7% (printed) | 16.7% (printed) |
| Doubao-Seed-2.1-Pro | 34.8% | 20.8% (printed) | 8.3% | 0.0% | 27.1% (printed) | 25.0% (printed) | 5.2% | 4.2% | 9.4% |
| Gemini 3.8 Flash | 17.4% | 3.8% | 23.8% (printed) | 0.0% | 5.0% | 6.2% | 25.0% (printed) | 26.2% (printed) | 10.0% |
| GLM-5.3-Flash | 37.4% | 19.1% (printed) | 3.5% | 0.0% | 20.6% (printed) | 25.5% (printed) | 7.1% | 6.4% | 17.7% (printed) |
| GPT-5.6 Sol | 15.5% | 4.3% | 5.8% | 0.0% | 5.8% | 20.3% (printed) | 7.2% | 23.2% (printed) | 33.3% (printed) |
| Grok 4.6 | 11.6% | 0.0% | 10.3% | 0.0% | 0.0% | 10.3% | 34.5% (printed) | 34.5% (printed) | 10.3% |
| Hy3 | 54.8% | 10.1% | 3.6% | 5.8% | 10.8% | 11.5% | 12.2% (printed) | 12.9% (printed) | 33.1% (printed) |
| Hy4 preview | 33.5% | 2.2% | 23.5% (printed) | 0.0% | 2.9% | 28.7% (printed) | 3.7% | 30.1% (printed) | 8.8% |
| Muse Spark 1.3 | 25.8% | 3.8% | 6.4% | 0.0% | 5.1% | 6.4% | 34.6% (printed) | 35.9% (printed) | 7.7% |
| Qwen3.8-Max | 21.3% | 5.1% | 2.6% | 0.0% | 26.5% (printed) | 3.4% | 8.5% | 28.2% (printed) | 25.6% (printed) |
Which methods do agents use?
Experts labeled the methods behind each final solution and compared them with the experts’ own solutions. Agents lean more on simple rules and on listing out candidates; experts more often split the problem up and narrow the search.
How often each method is used
11 models + Expert · 31 tasks
| Model | General solver | Specialized algorithm | Greedy or rule-based | Local search | Metaheuristics | Enumeration | Tree search |
|---|---|---|---|---|---|---|---|
| Claude Opus 5 | 80.6 | 6.5 | 25.8 | 3.2 | 6.5 | 12.9 | 0.0 |
| DeepSeek-V4.1-Flash | 74.2 | 0.0 | 25.8 | 3.2 | 9.7 | 12.9 | 0.0 |
| Doubao-Seed-2.1-Pro | 87.1 | 3.2 | 25.8 | 6.5 | 3.2 | 12.9 | 0.0 |
| Gemini 3.8 Flash | 83.9 | 0.0 | 25.8 | 0.0 | 6.5 | 6.5 | 0.0 |
| GLM-5.3-Flash | 71.0 | 6.5 | 29.0 | 12.9 | 12.9 | 22.6 | 3.2 |
| GPT-5.6 Sol | 67.7 | 9.7 | 38.7 | 9.7 | 9.7 | 9.7 | 0.0 |
| Grok 4.6 | 74.2 | 9.7 | 38.7 | 22.6 | 9.7 | 22.6 | 0.0 |
| Hy3 | 77.4 | 3.2 | 25.8 | 9.7 | 6.5 | 3.2 | 0.0 |
| Hy4 preview | 83.9 | 6.5 | 29.0 | 6.5 | 6.5 | 16.1 | 0.0 |
| Muse Spark 1.3 | 71.0 | 0.0 | 32.3 | 29.0 | 6.5 | 3.2 | 0.0 |
| Qwen3.8-Max | 87.1 | 6.5 | 29.0 | 9.7 | 12.9 | 12.9 | 0.0 |
| Expert | 93.5 | 19.4 | 19.4 | 9.7 | 6.5 | 3.2 | 16.1 |
Each cell is the share of the 31 tasks where the final solution uses that method.
What they optimize, and how they use the problem’s structure
Same model order · 31 tasks
| (a) What gets optimized | (b) Using the problem’s structure | ||||||
|---|---|---|---|---|---|---|---|
| Model | The stated business goal | A self-made stand-in score | Simplify the model | Split into subproblems | Special-structure algorithm | Narrow the search | |
| Claude Opus 5 | 67.7 | 32.3 | 12.9 | 19.4 | 6.5 | 29.0 | |
| DeepSeek-V4.1-Flash | 77.4 | 22.6 | 12.9 | 16.1 | 0.0 | 25.8 | |
| Doubao-Seed-2.1-Pro | 80.6 | 19.4 | 9.7 | 19.4 | 3.2 | 32.3 | |
| Gemini 3.8 Flash | 64.5 | 35.5 | 12.9 | 25.8 | 0.0 | 29.0 | |
| GLM-5.3-Flash | 80.6 | 19.4 | 12.9 | 25.8 | 6.5 | 41.9 | |
| GPT-5.6 Sol | 67.7 | 32.3 | 19.4 | 32.3 | 9.7 | 22.6 | |
| Grok 4.6 | 71.0 | 29.0 | 9.7 | 35.5 | 9.7 | 25.8 | |
| Hy3 | 51.6 | 48.4 | 12.9 | 19.4 | 3.2 | 9.7 | |
| Hy4 preview | 83.9 | 16.1 | 19.4 | 19.4 | 6.5 | 38.7 | |
| Muse Spark 1.3 | 80.6 | 19.4 | 9.7 | 19.4 | 0.0 | 9.7 | |
| Qwen3.8-Max | 83.9 | 16.1 | 12.9 | 22.6 | 6.5 | 35.5 | |
| Expert | 61.3 | 38.7 | 6.5 | 41.9 | 6.5 | 45.2 | |
Each cell is a share of the 31 tasks. Both panels use the same 0–100% color scale.