OR-License
Leaderboard

A plan has to work before it can be good.

11 models, each run 5 times on all 31 tasks. Even the top model scores below the expert solutions that clients accepted.

Top overall score (DAQ)86.55%Grok 4.6 · ± 12.97
Highest feasibility88.39%Grok 4.6 · ± 10.28
Quality of its feasible plans97.92%Grok 4.6 · ± 14.67
Experts100%Solutions the clients accepted
Overall score (DAQ)
Run in Claude Code · 31 tasks · 5 runs per task
Experts · 100%
  1. Grok 4.6high
    86.55%
  2. GPT-5.6 Solxhigh
    85.02%
  3. Gemini 3.8 Flashhigh
    82.94%
  4. Qwen3.8-Maxxhigh
    77.97%
  5. Claude Opus 5xhigh
    77.44%
  6. Muse Spark 1.3high
    72.86%
  7. DeepSeek-V4.1-Flashhigh
    69.28%
  8. Hy4 previewhigh
    66.85%
  9. Doubao-Seed-2.1-Prohigh
    64.82%
  10. GLM-5.3-Flashmax
    63.21%
  11. Hy3high
    43.77%
A plan counts only if it passes every hidden check; plans that fail score 0. Scores under stricter weighting

Full results Paper, Table 1

CSV
# Model Reasoning effort
1 Grok 4.6 high 88.39± 10.28 86.55± 12.97
2 GPT-5.6 Sol xhigh 84.52± 17.02 85.02± 17.78
3 Gemini 3.8 Flash high 82.58± 17.49 82.94± 19.06
4 Qwen3.8-Max xhigh 78.71± 23.11 77.97± 23.71
5 Claude Opus 5 xhigh 77.42± 4.98 77.44± 7.26
6 Muse Spark 1.3 high 74.19± 12.04 72.86± 12.34
7 DeepSeek-V4.1-Flash high 69.68± 14.60 69.28± 15.93
8 Hy4 preview high 66.45± 9.95 66.85± 10.38
9 Doubao-Seed-2.1-Pro high 65.16± 17.81 64.82± 18.47
10 GLM-5.3-Flash max 62.58± 25.35 63.21± 25.68
11 Hy3 high 45.16± 19.58 43.77± 19.50

Each model ran 5 times on each of the 31 tasks; ± shows how much results on the same task varied from run to run (standard deviation). A task’s plan is feasible only if it passes every test case. The experts score 100%, and a model can score above 100% on a task if its plan beats the expert’s.

FeasibilityDoes the plan follow every rule?

The share of tasks where the submitted plan passes all of the hidden checks, such as capacity, timing and demand rules.

DAQThe overall score

Each task is scored 0 if the plan breaks a rule. Otherwise its result is compared with the expert’s, where 1 means just as good. The overall score is the average over all 31 tasks.

DAQ = 1/N · Σi 1[feasiblei] · qi

qi compares the result on task i with the expert’s.

DAQαA stricter version

An average lets good tasks make up for bad ones. DAQα weighs low-scoring tasks more as α rises from 1 to 3. At α = 1 it equals DAQ; the experts stay at 1 throughout.

The coding tool matters too: in a separate run with Codex instead of Claude Code (paper, Appendix D.7), GPT-5.6 Sol scored 93.59% ± 7.31. Those runs are not part of this ranking.