07 — Model deep dive: what fits better for scoring 6–11-year-old Pakistani children (Urdu + English reading, spoken maths), Sep 2026
Written: 2026-09-15 · Builds on:02_model_landscape.md (vendor/API tables — not repeated here). Method: primary sources fetched today (arXiv, ISCA archive, ACL Anthology, Hugging Face model cards and the HF model API, vendor docs, GitHub). Every claim carries a URL; anything I could not confirm against a primary page is marked unverified. Cost arithmetic ("≈") is mine, based on 206 recordings × 20 min = 68.7 h of audio.
Input constraints assumed throughout: WhatsApp voice notes (16 kHz Opus) and SurveyCTO audio audits (8 kHz AAC, ~12 kbps); ~5-min child battery inside a ~20-min enumerator recording; a coach corrects the pre-score (HITL), so the target is high-recall flagging with calibrated confidence, not autonomous scoring.
0. What changed since the first report (the six findings that move the plan)
#
Finding
Why it matters
1
An open, MIT-licensed, EGRA-specific child ASR model now exists — AI-for-Education/sw-tz-child-egra-fastconformer-ctc-110m (Kiswahili, Tanzania, FastConformer-CTC, val WER 13.0 / test WER 16.8, letters 3–18 % WER, syllables 19–23 %, "as little as 50 hours of child-speech data" made the difference) (model card). Built by Zevo Tech / Fab Inc / Laterite / Stellenbosch for the Digitisation-of-EGRA project.
It is the recipe we should copy for Urdu (and reuse directly for TZ): a small CTC model fine-tuned on real child EGRA audio, per-subtask WER reported. It also gives us a public yardstick: ~13–17 % WER on child EGRA speech is the realistic bar.
2
Meta Omnilingual ASR (Nov 2025) is Apache-2.0 and lists urd_Arab, urd_Latn, pnb_Arab, hin_Deva, swh_Latn (lang_ids.py, repo, HF card). CTC sizes 300 M / 1 B / 3 B / 7 B; 7B-LLM needs ~17 GiB VRAM; inputs must be < 40 s.
Replaces MMS-1b-all (CC-BY-NC, a production blocker flagged in §B of the first report) as the permissive Urdu CTC backbone for forced alignment and CTC-GOP. Per-language Urdu CER is in the repo's CSV but I could not read it today — unverified.
3
Two 2026 Urdu Whisper fine-tunes beat every number in the first report: emads/whisper-large-v3-urdu-v2 (Aug 2026, Apache-2.0): FLEURS-ur 16.65 WER, Common Voice 15.98, Roman-Urdu CER 14.7, code-switch benchmark 35.4 WER, three output registers (Urdu script / Roman / mixed) via control tokens, trained only on CC-0/CC-BY data (Common Voice, IndicVoices, Kathbath, PRUS, MUCS Hinglish) (card); awaaz-se-alfaaz/whisper-turbo-v3-urdu (LoRA on large-v3-turbo, LaTeLL 2026, Apache-2.0): FLEURS 24.07→16.86, CSaLT 27.51→20.77, CV-23 37.23→25.78, real-world YouTube 25.33→24.20 (→21.75 with Gemma3-12B post-edit) (card).
The "WER ~26 on CV" baseline from the first report is stale; open Urdu ASR is now ~16–17 WER on read adult speech, on a par with Scribe's 14.4 FLEURS claim, and self-hostable. The 35 % code-switch figure is the number to worry about for maths items.
4
Common Voice 26.0 (June 2026) Urdu = 95.5 h total / 88.5 h validated / 1,127 speakers / 40,385 clips; 10.2 % of clips from "teens", 0 % under-13 (cv-corpus-26.0 JSON). Hindi 436 h, Swahili 157 h in the same file.
Confirms there is no public Urdu child speech at all; our 206 recordings are the only Urdu child-EGRA audio anyone has.
5
Gemini timestamp drift is a documented, unanswered bug, not our artefact. Google forum thread (opened 7 Mar 2026, still open 13 Aug 2026): Gemini 3 Flash compresses an 11:49 file into 0:00–9:14 (clock ~22 % fast, −157 s at the end); 3.1 Pro drifts −16 s non-linearly; only the Flash-Lite variants stay < 9 s; workarounds = 5-min chunks, two-pass (Lite for time, Pro for text), disable thinking (thread). The 2026 temporal-grounding literature says the same of every LALM (TAG-Bench best mIoU 31.2, recall@IoU≥0.7 only 21.5 %) (TAG-Bench, TEMPO).
No audio LLM returns trustworthy word timestamps natively in Sep 2026. Timestamps must come from a CTC/attention aligner; the LLM only ever sees ≤5-min segments.
6
The hard case is exactly ours, and it is much harder than the MyST numbers suggest. Dutch grade-3 readers recorded "in diverse, noisier environments" (DART) gave 70.4 % WER for the best fine-tuned Whisper-medium vs 5.5 % on clean JASMIN; the authors' fix is utterance-level prompt matching — where ASR output equals the read prompt exactly, precision is 98.3–99.3 %, covering 18–42 % of utterances (Lathouwers et al. 2026).
Design the HITL around this: auto-accept only exact matches, route everything else to the coach with the aligner's evidence. Expect the Urdu struggling-reader tail to look like DART, not MyST.
1. Children's speech ASR — state of the art, 2025–2026
Whisper-small (244 M) fine-tuned wins 6/8 splits; OGI-Script 1.8 %, OGI-Spon 11.6 %; OWSM-1B 12.8 % MyST; performance plateaus or degrades beyond ~1 B params; multi-corpus training lifts OGI-Spon 59.3→24.2 %; WER by age band 4–7 ≫ 8–10 ≫ 11–15
WavLM-based recogniser best; children's mispronunciations "persist longer than previously reported", esp. certain manners of articulation — i.e. many "errors" are developmental, not reading errors
Reading of this for us. (a) Small models fine-tuned on in-domain child audio beat big zero-shot models; whisper-small/medium or a 110 M FastConformer is the right size, not large-v3. (b) Age 6–8 will run ~2× the WER of age 9–11 whatever we do — report by grade. (c) Cross-corpus transfer is poor, so a MyST-trained English model will not save us; we need our own audio in the training mix. (d) The 8 kHz SurveyCTO channel is a first-order problem: run the eval separately on WhatsApp-16 kHz vs SurveyCTO-8 kHz.
1.2 Public child-speech fine-tunes on Hugging Face (Sep 2026)
From the HF model API (search=child, ASR pipeline): the useful ones are the Kiswahili EGRA FastConformer (above), AlexXu811/child-adult-joint-asr-diarization (Whisper, Jan 2026 — joint child/adult ASR + diarization, relevant for enumerator/child separation), dysata/Wav2Vec2-Ru-Child, tartuNLP/xls-r-300m-et-children, kawchar85/ASR_Bangla_Child_Voice (the only South-Asian child ASR checkpoint on the hub; no card metrics), Korean/Estonian/Dutch/Icelandic child Whisper fine-tunes (HF API listing). No Urdu or Hindi child ASR checkpoint exists on the hub.bookbot/wav2vec2-adult-child-cls is a ready adult/child classifier useful for segmenting enumerator vs child turns (card).
1.3 South-Asian child-speech work
ASER (Pratham) dataset: 5,301 children 6–14, 81,658 clips, Hindi/Marathi/English, expert level labels not transcripts — still the only large Indian child reading corpus; not public as audio (arXiv 1912.04381).
HiACC (2025): first Hinglish adult + child code-switched corpus — 1,858 child segments, 5.24 h total, Zenodo (Data in Brief, PMC). Tiny, but it is the nearest public sample of South-Asian children code-switching — use for a sanity probe on maths items.
Google Read Along / Bolo: on-device, Hindi + Urdu + English, product claims only ("64 % improved in 3 months"); no technical report or API (Wikipedia, Slashdot 2019).
AI4Bharat / Bhashini / EkStep / Karya: no child-speech evaluation of IndicConformer or IndicWhisper found in any search; IndicVoices is adult (23.7 k h collected, 11.2 k h transcribed as of Dec 2025) (IndicVoices). "Aksharajaani" returned nothing — unverified that it exists as a speech product.
Malay/Tamil (Singapore, 2025): 654 P1–P2 children recorded in classrooms; fine-tuned ASR → objective metrics → GPT classifier beat GPT-4o-audio direct (arXiv 2505.19671) — the closest published design to what we are building, in a Tamil-script language.
No Pakistani child corpus: searches over CLE/FAST/NUST found only adult UrduSER, UrduSpeech (156 h, adult) and a preschool narrative-skills dataset (audio not for ASR) (UrduSpeech, UrduSER).
1.4 What we would fine-tune, on what, for how much
Target
Base model
Data
Compute
Notes
Urdu child EGRA ASR
emads/whisper-large-v3-urdu-v2 (best open Urdu WER, Apache) orwhisper-small from scratch on Urdu (ESPnet finding: small wins on child data) oromniASR_CTC_300M/1B (urd_Arab) FastConformer-style
Child turns from our 206 recordings (≈ 30–40 h after enumerator removal — my estimate), with verbatim transcripts derived by forced-aligning the item text and applying the coach corrections (correct→item word, error→what the coach heard)
Pashto precedent: whisper-large-v3-turbo full FT on 113 h ran ~5.5 k steps on one A40; LoRA on whisper-base failed (54.6 vs 21.2 WER) — do full FT (arXiv 2604.06507). On Modal an A10G is ~$1.10/h, L4 ~$0.80/h (Modal pricing, comparison) → ≈ $10–30 per training run.
The Kiswahili EGRA card says ~50 h of real child audio was the lever; we have ~two-thirds of that. Hold out by school and by grade.
English child ORF (PK accent)
whisper-small/medium (Kid-Whisper recipe) seeded from MyST checkpoint, then our English segments
Same recordings, English block
Same
Do not expect MyST transfer (CSLU 55 % WER above).
Urdu phone recogniser
anyspeech/zipa-large-crctc-500k (MIT)
Not initially — use zero-shot; ZIPA saw 10 h 19 min of Urdu in IPAPack++ (ZIPA §3)
Inference only
Fine-tune later on coach-labelled letter-sound rows if PFER on Urdu letters is poor.
FLEURS 16.65, CV 15.98, Roman CER 14.7, code-switch 35.4
Apache-2.0
via aligner
Emits Urdu script, Roman, or mixed with English islands by control token — the only model that treats code-switch output as a first-class register (card)
Evaluation gotchas specific to Urdu. (1) Script/romanisation inflation: SN-WER transliterates reference and hypothesis to one canonical script before scoring and removes up to 12 % of "model gap" on FLEURS and 67 % of artificial romanisation-induced WER — adopt it, because children and enumerators mix scripts and the ASR may emit Roman islands (arXiv 2606.02548). (2) Diacritics: Urdu text omits short vowels; normalise by stripping U+064B–U+0652 on both sides before alignment (standard practice in the Urdu survey, arXiv 2411.14493). (3) Numerals: Urdu digits are U+06F0–U+06F9 (۰–۹), not Arabic U+0660–U+0669 (٠–٩); the glyphs for 4/5/6/7 differ by locale, and the Urdu forms "are not encoded separately from the Persian variants" (Wikipedia). Map both ranges to ASCII before scoring maths. (4) Code-switching: 35 % WER on the emads code-switch benchmark and seamless-class models that translate instead of transliterate (WER-We-Stand) mean English numbers inside Urdu answers are the single biggest transcript risk — keyterm/hotword biasing (Deepgram, Soniox) and the emads mixed-register token are the two mitigations to test.
3. Phone recognition, forced alignment and GOP for Urdu
pip install onnxruntime soundfile librosa lhotse torch; python inference/inference.py audio.wav --model-path checkpoints/zipa-cr-small-300k/exp/model.onnx --model-type ctc; also zipa_ctc_inference.py Python API with batch padding; 16 kHz in
IPA string over a 127-symbol simplified inventory
PFER seen-lang avg 2.71 (large, noisy-student) / 3.22 (small); unseen 3.20 / 3.24; smooths sociophonetic variation toward dictionary forms; vowels hardest; length marks and glottal stops most deleted — i.e. it under-detects exactly the vowel errors children make
Practical recipe (Urdu letter sounds, phoneme awareness, nonwords) and hardware.
1. Segment the enumerator recording into child turns with bookbot/wav2vec2-adult-child-cls + Silero VAD (CPU).
2. For each 10-letter row: (a) ZIPA-large-CR-CTC → IPA hypothesis; (b) Omnilingual omniASR_CTC_1B (urd_Arab) → CTC posteriors → alignment-free CTC-GOP against the expected letter-sound sequence with substitution clusters (Parikh 2025). Score = per-item GOP + IPA edit distance; flag when the two disagree.
3. Reference phones: espeak-ng ur via phonemizer for letter names; hand-written table for letter sounds (short vowels are unrecoverable from text — the table must list allowed variants, e.g. /ə/ vs /a/).
4. Hardware: ZIPA-large (300 M) and omniASR-CTC-1B are single-forward CTC models; on a Modal L4 ($0.80/h) a 5-min clip should take well under a minute — RTF not published for either; benchmark before committing. The whole 206-recording set ≈ 69 h audio ≈ 3–7 GPU-hours ≈ $3–6 of L4 time per model pass (my estimate).
5. Keep MMS_FA only for offline research (CC-BY-NC).
4. Prosody, fluency and disfluency — what is reusable on children
System / paper
What it measures
Result
Reuse
Frontiers/SMU prosody scoring (2024)
9 openSMILE ComParE features (F0, RMS energy, ZCR, spectral bands 250–650 Hz / 1–4 kHz, centroid, sharpness) → NAEP 4-pt; 5,841 recordings, 1,811 grade-2–4 children, 30 passages; no ASR alignment
cross-domain 62.5 % micro-acc / QWK 0.38; 76 % with all features in-domain
Feature set is cheap and language-agnostic — use as a coarse prosody prior only (paper)
Harmsen et al., Interspeech 2025
15 ASR-derived phrasing/smoothness/pacing measures; 244 recordings, 131 Dutch children 6–13
12/15 correlate strongly with human-transcript measures
Directly portable: pause count/duration, repetition rate, mean words per phrase, WCPM — needs word timestamps, i.e. the aligner (ISCA)
omission/insertion/mispronunciation + "Expression" (monotone, long pauses, missing breaks) — it is Azure PA underneath
independent study found false positives on acceptable pronunciations
Confirms Azure PA's prosody error types are the same ones we get from the API (Lodz study, MS support)
What to build: reading rate (WCPM) and pause statistics from aligner timestamps (Harmsen's 15 measures), repetition/false-start counts from a verbatim transcript (CrisperWhisper for English; whisper-timestamped [*] on the Urdu fine-tune), and a NAEP-style prosody label trained on coach labels with the 9-feature openSMILE set as input. Do not ship an LLM-judged prosody score.
5. English pronunciation for South-Asian-accented children — vendors re-checked today
Vendor
Dialect / accent options
Max audio per call
Price (fetched)
Bias / child evidence
Azure PA
Locales include en-IN (plus en-US/GB/AU…); hi-IN and ta-IN exist, ur-* does not
30 s single-shot with miscues; continuous mode beyond
Real-time STT $1.00/h + $0.30/h PA add-on (batch: add-on free, $0.18/h) — from an MS Q&A answer and a secondary pricing analysis; the public page shows "$-" placeholders (Q&A, secondary, pricing page)
No published bias study by L1; MS says only ">0.5 Pearson" and to loosen thresholds for children; BYJU'S (India) is a cited child customer (GA post). Whether en-IN changes phoneme scores vs en-US is untested in any paper I found — we must A/B it ourselves.
SpeechAce
en-us, en-gb only (+ fr-fr/fr-ca/es-es/es-mx); no en-IN / Pakistani option despite the search snippet suggesting otherwise (supported languages)
45 s for Score Text passages (Score Text); Score Phone List (ARPABET g|ao|ch|ah, en-us/en-gb) scores non-words and returns per-phone quality_score, sound_most_like, extent ms (Score Phone List)
plans $40/$80/$125 per month; overage $0.008 per 15 s (first report)
Vendor kids study (350 children, F1 0.86); accent policy: "mere presence of accent is not a criterion for penalising" (SpeechAce); one EFL classroom study (CALL-EJ)
Language Confidence
"standard US or UK accents" as reference only (docs)
not stated
Pronunciation $0.008 / Scripted $0.010 / Unscripted $0.0125 per 15 s (Startup $100/mo), 10 rps (pricing)
none
ELSA API
English; metered "per time or characters"; REST recommended only for ≤ ~30 s, websocket beyond (General API info)
~30 s REST
on request
none
SpeechSuper
has an Indian-English variant (first report)
—
$0.004–0.008/request
none
Bottom line: the only vendor with an Indian-English locale is Azure (en-IN). No vendor has a Pakistani-English option and no published bias analysis for South-Asian children exists — our 206-recording comparison of Azure en-IN vs en-US vs SpeechAce en-us on the same clips would be a first. For nonwords, SpeechAce Score Phone List is the only API that scores an arbitrary phone string.
Answer to "is there a model that returns trustworthy word timestamps natively?" Among audio LLMs, no. The only native word timestamps come from ASR models (Gemini 3.5 Transcribe — no Urdu; Voxtral/Qwen3-ASR — no Urdu; whisper-1 API; Soniox/Deepgram/Scribe/AssemblyAI-U2 — Urdu yes) or from CTC/DTW alignment on a self-hosted model (CrisperWhisper 30 ms; Omnilingual/IndicMFA). Our 20–75 s drift on a 17-min file with Gemini 2.5 Pro is consistent with the public bug on 3.x. Rule for the harness: any LLM audio call gets a ≤ 5-min segment plus reference text, and its time references are ignored in favour of aligner timestamps.
7. Maths from speech — number normalisation and digit accuracy
Need
What exists
Gap
Urdu ITN (spoken → digits)
None. NeMo-text-processing and EkStep indic-punct ship WFST ITN for 11 Indic languages — Hindi, Tamil, Telugu, Kannada, Gujarati, Marathi, Odia, Bengali, Assamese, Malayalam, Punjabi — no Urdu (indic-punct, paper, NeMo)
Write our own: Urdu cardinals 0–100 are irregular lexical items (ایک…سو), plus ہزار/لاکھ/کروڑ; ~150-entry table + a 3-rule combiner covers EGMA (numbers ≤ 999). Accept Hindi-Urdu homophones and Roman spellings.
Urdu num2words (digits → words, for building the table and TTS prompts)
num2words has no ur; indic-num2words "planning to add" Urdu (repo); num2fawords (Persian) is the closest (PyPI)
emads mixed-register token; Deepgram/Soniox keyterms; Whisper initial_prompt with the 8 target numbers
Test all three on the maths block; the 35 % code-switch WER is the risk
Child digit ASR evidence
Old but clean: speaker adaptation on a single digit cut child WER 9.6 → 4.2 %, seven digits → 3.5 % (ResearchGate review); Kaldi-NL scored a digits-in-noise test at 5 % WER (ScienceDirect 2025); HiACC gives 1,858 Hinglish child segments for a code-switch probe (HiACC)
No Urdu child digit study. Constrained decoding (grammar of allowed answers) is the cheap win: number-identification and addition items have a finite answer set.
Recipe: ASR (Urdu fine-tune with mixed-register token, or Soniox) → strip diacritics, map ۰–۹/٠–٩→0–9 → Urdu+English number-word grammar → take the last numeric token in the child turn (children count aloud) → compare to key. Keep the raw transcript for the coach.
8. Vision for written maths and photographed EGMA answer pages
Evidence
Numbers
Source
17-model handwritten-forms benchmark (Apr 2026) — Claude Haiku 4.5/Sonnet 4.6/Opus 4.6, GPT-4.1/5-Mini/5-Nano/5.2/5.2-Pro/5.4, Gemini 2.5 Flash/Pro, 3.1 Flash-Lite/Pro, Qwen2-VL-2B/2.5-VL-3B, Grok 4.1, Dots.ocr; 49 real health forms with illegible and out-of-box writing
Gemini 3.1 Pro best overall (85 % accuracy, F1 0.91 discrete fields, WER 0.50 free text); Claude Sonnet 4.6 best on formatted numeric/date fields (77 % median); GPT-5.4 lowest hallucination (6 %) and 100 % on noisy dates in one section; Gemini 3.1 ~8 % hallucination
Recipe for a photographed EGMA answer page under classroom light: (1) photograph with the answer key's layout known; (2) ask the VLM for per-cell {expected, read, confidence, crossed_out?} with the key in the prompt (FN-first framing from the exam-grading paper); (3) run two models (Gemini 3.1 Pro + Claude Sonnet 4.6 — the two field-level leaders) and route disagreements to the coach; (4) normalise digits from both Unicode ranges; (5) log lighting/blur so we can report accuracy by photo quality. Children's digits and Urdu numerals are unbenchmarked — the 50-page probe from the first report remains the gate.
9. Concrete shortlist for the eval harness (206 recordings, per-item human ground truth)
Cost basis: 206 × 20 min = 68.7 h. Where a stack only touches one block (e.g. English ≈ 5 min per recording ≈ 17 h), the block-scoped cost is given. GPU costs assume Modal L4 $0.80/h and my RTF guesses (marked ≈). Metrics: per-item correct/incorrect F1 vs coach (primary), WCPM MAE, letter-sound accuracy, timestamp error vs 30 hand-anchored words, exact-match auto-accept precision/coverage (DART method), all reported separately for WhatsApp-16 kHz vs SurveyCTO-8 kHz and by grade.
Block A — English decoding + ORF (letters, nonwords, words, passage)
Rank
Stack
Endpoint / model id
Cost (English ≈ 17 h)
Judged on
1
Azure PA en-IN scripted (≤ 30 s rows) + continuous for the passage, our diff for miscues
Speech SDK PronunciationAssessmentConfig, locale en-IN; run en-US as the A/B arm
≈ $1.30/h → $22 (real-time) or $0.18/h batch → $3
word F1, phoneme NBest usefulness (en-US only), en-IN vs en-US delta
2
SpeechAce Score Text (45-s chunks) + Score Phone List for nonwords
Overall ranking of what to run first (by expected information per dollar): (1) Block B #1 + #2 and Block A #1 — they decide the core architecture; (2) Block C #1 + #2 — the Urdu-only gap; (3) Block A #3 — tells us whether our own fine-tuning is worth it; (4) Blocks D/E on top of those transcripts (near-free); (5) Block F; (6) Block G last, as a control. Total cash for the whole harness ≈ $150–300 plus two vendor monthly plans, dominated by SpeechAce/LC subscriptions and the GPT-Realtime-2 arm; every open-weight arm is single-digit dollars.
Open verification items (new since the first report)
Read the Omnilingual per-language CSV for urd_Arab CER (repo file not readable via fetch today).
Extract the CHiPSAL 2025 Urdu WER table and the Vistaar Urdu rows (PDFs did not parse).
Confirm the IndicMFA licence and the Ultravox v0.7 language list from the raw files.
Benchmark RTF for ZIPA-large, omniASR-CTC-1B and the two Urdu Whisper fine-tunes on an L4 with 8 kHz-upsampled audio.
Obtain the Imagine Worldwide / RTI Tanzania SA-EGRA/SA-EGMA 2025 pilot report (403 today) — it is the only field validation of tablet-administered, speech-scored EGRA/EGMA in Kiswahili and may already contain child-ASR reliability numbers (PDF).
Ask Fab Inc / Zevo Tech whether the Kiswahili EGRA FastConformer recipe (data schema, per-subtask manifests) can be reused for Urdu.