Maths graders vs the enumerator, by subtask
| subtask | model | n | r | MAE | bias | exact | within1 | human_floor_agree | cost_per_child |
|---|---|---|---|---|---|---|---|---|---|
| bleffcomp | openai/gpt-5-mini | 12 | 0.35 | 1.25 | -1.25 | 42 | 58 | 87.8 | 0.014 |
| bleffcomp | anthropic/claude-sonnet-5 | 13 | 0.45 | 1.46 | -1.46 | 38 | 46 | 87.8 | 0.108 |
| bleffcomp | google/gemini-2.5-pro | 116 | 0.44 | 1.72 | -0.58 | 33 | 60 | 87.8 | 0.113 |
| bleffcomp | google/gemini-3-flash-preview | 25 | 0.08 | 2.32 | -0.16 | 20 | 52 | 87.8 | 0.008 |
| idnummag | google/gemini-3-flash-preview | 37 | 0.4 | 2.54 | -2.0 | 16 | 43 | 80.9 | 0.008 |
| idnummag | google/gemini-2.5-pro | 153 | 0.27 | 3.09 | -2.59 | 11 | 36 | 80.9 | 0.113 |
| idnummag | openai/gpt-5-mini | 36 | 0.22 | 3.28 | -3.22 | 14 | 33 | 80.9 | 0.014 |
| idnummag | anthropic/claude-sonnet-5 | 30 | 0.11 | 3.33 | -2.8 | 17 | 30 | 80.9 | 0.108 |
| numrep | google/gemini-2.5-pro | 155 | 0.61 | 2.09 | -1.97 | 14 | 39 | 86.8 | 0.113 |
| numrep | google/gemini-3-flash-preview | 38 | 0.42 | 2.18 | -1.87 | 11 | 39 | 86.8 | 0.008 |
| numrep | anthropic/claude-sonnet-5 | 33 | 0.38 | 2.52 | -2.33 | 6 | 36 | 86.8 | 0.108 |
| numrep | openai/gpt-5-mini | 36 | 0.5 | 2.67 | -2.67 | 8 | 28 | 86.8 | 0.014 |
| patterns | google/gemini-3-flash-preview | 38 | 0.47 | 2.68 | -2.53 | 21 | 37 | 86.1 | 0.008 |
| patterns | anthropic/claude-sonnet-5 | 33 | 0.15 | 2.76 | -2.45 | 27 | 39 | 86.1 | 0.108 |
| patterns | google/gemini-2.5-pro | 161 | 0.4 | 2.78 | -2.64 | 28 | 39 | 86.1 | 0.113 |
| patterns | openai/gpt-5-mini | 36 | 0.25 | 2.86 | -2.69 | 22 | 42 | 86.1 | 0.014 |
| wrdpblm | google/gemini-2.5-pro | 155 | 0.7 | 1.46 | -1.29 | 45 | 63 | 87.0 | 0.113 |
| wrdpblm | anthropic/claude-sonnet-5 | 31 | 0.48 | 1.52 | -1.0 | 52 | 74 | 87.0 | 0.108 |
| wrdpblm | google/gemini-3-flash-preview | 35 | 0.46 | 1.66 | -0.97 | 46 | 69 | 87.0 | 0.008 |
| wrdpblm | openai/gpt-5-mini | 35 | 0.44 | 1.74 | -1.11 | 46 | 63 | 87.0 | 0.014 |