The evaluation

Every model we tested, one table per subtask, the human floor and the baseline beside every number, and the stack we recommend

The evaluation: every scorer we tried, beside the humans

Data: 187 enumerator recordings from the May 2026 Rawalpindi EGRA/EGMA study that join to a child's tablet scores and a reviewer's re-listen; 8 kHz classroom audio. Scripts: harness/01…15 (numbered for reproduction; every model reply is cached). Model spend for the whole evaluation: ≈ $70 (LLMs through OpenRouter $57.77, whisper-1 $3.47, Soniox ≈ $8.50; SpeechAce on the existing plan; Azure unavailable).

1. How a score is judged

2. The human floor

Subtask Reviewer agreed with enumerator Disagreed Could not tell
letterid_urd 89% 8% 3%
letterid_eng 90% 5% 5%
idwrd_urd 51% 30% 17%
idwrd_eng 56% 25% 18%
pw_eng 52% 21% 24%
orf_urd 54% 25% 20%
orf_eng 62% 18% 20%
listcomp_urd 81% 10% 9%
rdcomp_urd 73% 11% 14%
idnummag 81% 4% 14%
numrep 87% 6% 7%
computation 88% 6% 6%
word_problems 87% 6% 6%
patterns 86% 5% 9%
blfluency_l1 90% 4% 6%

On the timed word lists and stories two trained humans agree only about half the time on this audio; on letters and maths about nine times in ten. That is the bar.

3. Reading windows: counts and per-word marks

3a. The baseline on every recording (speech-to-text + alignment)

Subtask n wrong-window cases on the rest: r MAE bias within ±5 human floor
English word list (60 s) 137 28% 0.93 5.9 -4.7 60% 56%
Urdu word list (60 s) 157 12% 0.97 5.9 -5.5 57% 51%
English letters (60 s) 170 62% 0.95 11.7 -5.5 16% 90%
Urdu letters (60 s) 154 84% 0.71 11.4 -11.1 28% 89%
English story (60 s) 109 20% 0.97 5.6 -2.0 64% 62%
Urdu story (60 s) 137 24% 0.98 10.4 -9.6 35% 54%
English pseudowords (60 s) 132 24% 0.24 2.2 -0.3 91% 52%

"Wrong-window cases" are recordings where the segmenter cut the wrong part of the 20-minute session (|error| > 20 words); the product design removes them (one voice note per block). Letters fail for the machine as they nearly fail for the reviewer's ear: letter names came back in three scripts.

3b. Every scorer on the same sixty-second windows, with the printed text supplied

Audio scorers heard the clip; transcript scorers read the STT transcript of it; every scorer saw the exact printed text and returned a verdict per word. Baseline = alignment on the very same windows. Sorted by MAE within each subtask.

Urdu story (60 s) — human floor 54%

Scorer input n count r MAE bias within ±5 word precision word recall $/window
Baseline: STT + alignment transcript 129 0.87 15.7 -15.0 25% 0.0000
Gemini 3.1 Pro audio 28 0.97 2.6 -2.0 89% 0.59 0.81 0.0551
Gemini 3.8 Flash audio 129 0.90 5.5 -1.2 74% 0.48 0.52 0.0097
Gemini 3.5 Flash-Lite audio 34 0.84 7.0 -0.1 62% 0.39 0.68 0.0037
Claude Sonnet 5 text 24 0.95 8.5 -8.1 46% 0.19 0.78 0.0418
GPT-5 mini text 31 0.85 12.8 -12.4 26% 0.18 0.89 0.0081
GPT audio audio 15 0.64 13.1 -9.2 27% 0.13 0.81 0.0302
Gemini 2.5 Flash audio 34 0.49 13.6 +10.1 62% 0.57 0.25 0.0047
MiMo 2.5 audio 33 0.51 13.8 +7.7 58% 0.18 0.15 0.0014
Gemini 3 Flash text 34 0.87 14.9 -14.8 29% 0.16 0.91 0.0029
Nemotron omni (free) audio 15 nan 17.8 +17.8 53% 0.00 0.00 0.0000
Voxtral small audio 11 nan 20.0 +20.0 45% 0.00 0.00 0.0065
DeepSeek 3.2 text 34 0.80 21.5 -21.5 24% 0.13 0.97 0.0007
GPT audio mini audio 34 0.15 25.9 -11.1 29% 0.10 0.72 0.0036

Urdu word list (60 s) — human floor 51%

Scorer input n count r MAE bias within ±5 word precision word recall $/window
Baseline: STT + alignment transcript 149 0.86 9.0 -8.7 49% 0.0000
Gemini 3.1 Pro audio 33 0.99 1.7 +0.1 97% 0.75 0.79 0.0566
GPT audio audio 12 0.93 3.3 +1.3 92% 0.32 0.44 0.0294
Gemini 3.8 Flash audio 149 0.93 3.8 +0.2 86% 0.68 0.59 0.0098
Claude Sonnet 5 text 27 0.93 4.2 -3.7 89% 0.46 0.83 0.0379
Gemini 3 Flash text 35 0.90 6.7 -6.1 63% 0.31 0.86 0.0025
Gemini 3.5 Flash-Lite audio 35 0.76 8.0 -0.9 69% 0.26 0.52 0.0032
GPT-5 mini text 34 0.81 8.9 -8.4 65% 0.23 0.85 0.0077
DeepSeek 3.2 text 35 0.86 11.5 -11.1 40% 0.20 0.92 0.0008
GPT audio mini audio 35 0.60 11.9 -2.1 57% 0.19 0.64 0.0030
MiMo 2.5 audio 34 0.60 11.9 +7.3 62% 0.25 0.31 0.0011
Gemini 2.5 Flash audio 35 0.56 12.8 +10.9 57% 0.30 0.17 0.0043
Nemotron omni (free) audio 15 0.52 15.3 +15.3 47% 0.00 0.00 0.0000
Voxtral small audio 10 nan 22.8 +22.8 40% 0.00 0.00 0.0065

English story (60 s) — human floor 62%

Scorer input n count r MAE bias within ±5 word precision word recall $/window
Baseline: STT + alignment transcript 104 0.80 11.2 -8.2 52% 0.0000
GPT audio audio 11 0.94 4.2 -4.0 73% 0.18 0.33 0.0293
Gemini 3.5 Flash-Lite audio 25 0.95 5.1 -2.4 72% 0.25 0.38 0.0033
Gemini 3.8 Flash audio 104 0.89 5.8 -0.2 74% 0.31 0.38 0.0092
Gemini 3.1 Pro audio 24 0.94 6.1 -1.5 71% 0.19 0.31 0.0573
Gemini 2.5 Flash audio 25 0.86 7.7 +5.2 68% 0.22 0.13 0.0046
MiMo 2.5 audio 23 0.82 9.1 +4.2 52% 0.29 0.28 0.0011
Claude Sonnet 5 text 20 0.86 9.2 -8.4 55% 0.20 0.42 0.0330
Gemini 3 Flash text 25 0.86 11.3 -10.8 44% 0.18 0.49 0.0027
DeepSeek 3.2 text 23 0.84 11.6 -10.7 43% 0.19 0.48 0.0016
GPT-5 mini text 24 0.47 20.7 -19.0 42% 0.16 0.77 0.0082
Nemotron omni (free) audio 12 nan 21.5 +21.3 42% 0.00 0.00 0.0000
Voxtral small audio 10 nan 22.2 +22.0 30% 0.00 0.00 0.0065
GPT audio mini audio 24 -0.05 31.7 -17.0 29% 0.12 0.77 0.0033

Baseline per-word precision / recall on all recordings (same rule, words both sides reached):

4. Comprehension answers (Urdu, six questions), graded from the transcript

Human floor for this subtask: 73%. The grader sees the passage and the diarised question window.

Grader n r MAE (questions) bias exact within 1 $/child
Gemini 3 Flash 53 0.73 1.04 -0.70 40% 79% 0.0012
Claude Sonnet 5 52 0.75 1.13 -1.06 37% 73% 0.0121
GPT-5 mini 53 0.74 1.17 -1.06 34% 75% 0.0030
DeepSeek 3.2 53 0.66 1.30 -0.89 28% 66% 0.0002
Gemini 2.5 Flash-Lite 53 0.22 1.57 +0.66 34% 64% 0.0002

5. Finding the windows (segmenter labeller on STT timestamps)

The labeller reads the diarised, timestamped transcript and names first/last turn per subtask; the clock is Soniox's. Compared with the enumerator's tablet taps, which precede the child's first word by 5–15 s.

Labeller files subtasks matched start within 5 s within 15 s mean abs error $/file
Gemini 2.5 Flash 11 137 44% 72% 21.8 s 0.0039
Gemini 2.5 Pro 181 2352 43% 70% 25.6 s 0.0958
Gemini 3 Flash 23 301 42% 68% 19.1 s 0.0064
Claude Haiku 4.5 23 282 43% 67% 25.5 s 0.0145
GPT-5 mini 22 243 40% 64% 29.7 s 0.0146
Gemini 2.5 Flash-Lite 23 281 28% 49% 34.3 s 0.0014

Audio-native models were also tried as the clock: Gemini 2.5 Pro drifted 20–75 s and Gemini 2.5 Flash returned a transcript dump. The clock must come from speech-to-text; a text labeller then only has to name turns. In the product the coach's cue phrase replaces most of this.

6. Maths from the spoken transcript

The grader sees the diarised maths-block transcript with no item bank (the study export has none). Items whose question is spoken are gradable; items whose stimulus is on paper are not from audio alone, and the product will have the bank.

Subtask Grader n r MAE (items) bias exact within 1 human floor $/child
wrdpblm Gemini 2.5 Pro 155 0.70 1.46 -1.29 45% 63% 87% 0.113
wrdpblm Claude Sonnet 5 31 0.48 1.52 -1.00 52% 74% 87% 0.108
wrdpblm Gemini 3 Flash 35 0.46 1.66 -0.97 46% 69% 87% 0.008
wrdpblm GPT-5 mini 35 0.44 1.74 -1.11 46% 63% 87% 0.014
bleffcomp GPT-5 mini 12 0.35 1.25 -1.25 42% 58% 88% 0.014
bleffcomp Claude Sonnet 5 13 0.45 1.46 -1.46 38% 46% 88% 0.108
bleffcomp Gemini 2.5 Pro 116 0.44 1.72 -0.58 33% 60% 88% 0.113
bleffcomp Gemini 3 Flash 25 0.08 2.32 -0.16 20% 52% 88% 0.008
numrep Gemini 2.5 Pro 155 0.61 2.09 -1.97 14% 39% 87% 0.113
numrep Gemini 3 Flash 38 0.42 2.18 -1.87 11% 39% 87% 0.008
numrep Claude Sonnet 5 33 0.38 2.52 -2.33 6% 36% 87% 0.108
numrep GPT-5 mini 36 0.50 2.67 -2.67 8% 28% 87% 0.014
idnummag Gemini 3 Flash 37 0.40 2.54 -2.00 16% 43% 81% 0.008
idnummag Gemini 2.5 Pro 153 0.27 3.09 -2.59 11% 36% 81% 0.113
idnummag GPT-5 mini 36 0.22 3.28 -3.22 14% 33% 81% 0.014
idnummag Claude Sonnet 5 30 0.11 3.33 -2.80 17% 30% 81% 0.108
patterns Gemini 3 Flash 38 0.47 2.68 -2.53 21% 37% 86% 0.008
patterns Claude Sonnet 5 33 0.15 2.76 -2.45 27% 39% 86% 0.108
patterns Gemini 2.5 Pro 161 0.40 2.78 -2.64 28% 39% 86% 0.113
patterns GPT-5 mini 36 0.25 2.86 -2.69 22% 42% 86% 0.014

7. English pronunciation services: SpeechAce vs Azure

SpeechAce was run on the first 29 s of every English window (its scripted endpoint caps audio at 30 s on our plan; the reference was cut to the words the child reached). Compared on words both sides reached, quality score < 40 = wrong:

Subtask n count r MAE (words, 29-s window) per-word precision per-word recall F1 Gemini 3.8 Flash on the full 60 s, same children
English word list 137 0.89 6.8 0.40 0.46 0.43 count MAE 5.9, per-word F1 0.34
English story 109 0.93 6.7 0.45 0.38 0.41 count MAE 5.8, per-word F1 0.34

So on English, SpeechAce's per-word marks are the best we measured (the audio LLMs are stronger on Urdu, where SpeechAce cannot run), its counts are on par, and it is the only scorer that also names the mispronounced sound. Costs: $0.008 per 15 s of audio (about 3 cents per 60-s window in two calls) plus a plan tier; en-US/en-GB dialect models only; every number here is on 8 kHz audio and should be re-measured on 16 kHz voice notes.

Azure Pronunciation Assessment was not evaluated: both documented keys return 401 and the same key is on the production service, so English reading has been on the silent fallback path (P1 bug filed). Given the SpeechAce result there is no reason to regenerate it: SpeechAce covers the same job with child-speech evidence, nonword phone-list scoring and a working key.

8. What it adds up to: the stack we recommend

Job Use Because Fallback
Cutting the audio into subtask windows The coach's cue phrase per block + speech-to-text timestamps (Soniox for Urdu, whisper-1 forced to English) STT clocks were exact to the second; every audio LLM's clock drifted A cheap text labeller (Gemini 3 Flash tier) on the diarised transcript when a coach forgets the cue
Counting words correct in 60 s, Urdu and English Gemini 3.8 Flash on the clip with the printed text (Gemini 3.1 Pro where a second opinion is worth 6× the cost) Best MAE and rank agreement of everything tried, on par with or better than the alignment baseline while also giving per-word marks; ~1¢ per window STT + alignment (the round-1 baseline) if the audio model is unavailable
Per-word chips for the coach Gemini 3.1 Pro / 3.8 Flash verdicts, shown only where the model and the alignment agree Their precision is the best measured but still well under 1, so chips are suggestions, not marks Coach marks the word
Letters and initial sounds Coach marks; keep the audio for a later open forced-alignment / GOP layer Letter names transcribe in three scripts; the window is lost most of the time —
Comprehension answers A text grader on the diarised question window (Gemini 3 Flash tier is enough) Within one answer of the enumerator for ~75–80% of children at ~$0.001 Claude Sonnet 5 / GPT-5 mini give the same result at higher cost
Spoken maths Transcript + the item bank + a number-word grammar; text grader only for word problems Word problems r ≈ 0.7 without the bank; number ID needs the bank Coach marks
Written maths (photo) Gemini 3.1 Pro vision with the answer key Best on handwriting benchmarks; not yet tested on our sheets Coach marks
English per-word marks and phoneme detail SpeechAce scripted scoring, two 30-s calls per window, plus its phone-list endpoint for nonwords Best per-word F1 on English of anything tried (0.41–0.43), counts on par with Gemini, names the mispronounced sound; retire Azure rather than regenerate its dead key Gemini 3.8 Flash counts with coach-decided chips
Coach protocol audit Gemini-class audio model over the whole block, once, in the background Caught a slip the human reviewer missed —

Models tried and set aside: GPT audio mini and Nemotron omni (counts wrong by 15–30 words); MiMo 2.5, Voxtral small and Gemini 2.5 Flash (lenient, +8–10 words); every transcript-only LLM (no better than alignment, and harsher); GPT audio (good on word lists, poor on the Urdu story, 4× the cost of Gemini 3.8 Flash).