Comprehension graders vs the enumerator
| model | n | r | MAE | bias | exact | within1 | questions_found | cost |
|---|---|---|---|---|---|---|---|---|
| google/gemini-3-flash-preview | 53 | 0.73 | 1.04 | -0.7 | 40 | 79 | 5.7 | 0.0012 |
| anthropic/claude-sonnet-5 | 52 | 0.75 | 1.13 | -1.06 | 37 | 73 | 5.6 | 0.0121 |
| openai/gpt-5-mini | 53 | 0.74 | 1.17 | -1.06 | 34 | 75 | 5.7 | 0.003 |
| deepseek/deepseek-v3.2 | 53 | 0.66 | 1.3 | -0.89 | 28 | 66 | 5.7 | 0.0002 |
| google/gemini-2.5-flash-lite | 53 | 0.22 | 1.57 | 0.66 | 34 | 64 | 6.0 | 0.0002 |