Every pronunciation, ASR, phone-recognition, audio-LLM and vision option checked, with locale lists and prices
Written: 2026-09-15 · Scope: which speech / phoneme / audio-LLM / vision models can pre-score a coach-recorded, ~5-minute child battery (ages 6–10, Pakistani government schools) for a human-in-the-loop coach. Method: vendor docs, model cards and papers fetched 2026-09-15; every claim carries a URL. Anything not confirmed against a primary page is marked unverified. Companion docs: ../BASELINE.md (enumerator timings, 8 kHz SurveyCTO audio), ../probes/.
Headline: English has three mature scripted pronunciation stacks (Azure PA, SpeechAce, Language Confidence) with phoneme scores and kids evidence. Urdu has zero commercial phoneme-scoring APIs (Azure PA supports hi-IN and ta-IN but not ur-; SpeechAce/ELSA/SpeechSuper/Language Confidence/iFlytek are English-or-8-language only). Urdu therefore rides on (a) commercial ASR with word timestamps (Soniox, Deepgram, Scribe, AssemblyAI Universal-2, Speechmatics, Whisper) + edit-distance alignment — our existing FluencyService pattern — and (b) open-weight phone recognisers / forced aligners that do cover Urdu (ZIPA, MMS_FA, IndicMFA, mms-1b-all) for letter-sound and phoneme-awareness items. Audio-native LLMs are useful as verbalisers* of externally computed evidence, not as standalone per-word scorers: the 2026 literature documents prior-over-evidence diagnosis, ASR that silently corrects children's errors, and leniency on low-quality speech.
| Vendor | Languages (Urdu / Hindi / Kiswahili / Arabic?) | Phoneme-level output | Word timestamps | Scripted / unscripted | Max audio per call | Child-speech evidence | Price | Notes |
|---|---|---|---|---|---|---|---|---|
| Microsoft Azure Speech — Pronunciation Assessment | 33 locales. ur-PK / ur-IN: NO. sw-KE / sw-TZ: NO. hi-IN, ta-IN, ar-EG, ar-SA, en-IN, en-US, en-GB: yes (locale include file) | Phoneme AccuracyScore in all locales; phoneme names, IPA alphabet, syllable groups, spoken-phoneme NBestPhonemes ("sounded most like") are en-US only (SAPI names also zh-CN) (how-to) |
Yes — Offset/Duration in 100-ns units at word, syllable and phoneme level (same page) |
Both. Scripted = ReferenceText; unscripted uses a different STT model (MS recommends STT-then-scripted for accuracy). Scripted ErrorType ∈ {None, Omission, Insertion, Mispronunciation, UnexpectedBreak, MissingBreak, Monotone} |
30 s per single-shot call; >30 s must use continuous mode, where EnableMiscue is not supported — you must diff recognised words vs reference yourself. Streaming mode is unbounded (Foundry demo allows 60 min) |
No published child numbers; the responsible-AI page reports only ">0.5 Pearson with human judges" and advises a looser mispronunciation threshold for children (characteristics & limitations). Independent: on speechocean762 (half children) Azure PA PCC = 0.70 accuracy / 0.715 fluency / 0.842 prosody / 0.782 total (arXiv 2509.02915 Table 4) | Billed as STT Standard; a MS Q&A answer quotes $1.32/hr real-time ($0.66/hr short-audio endpoint), prorated per second (MS Q&A). Prosody is an add-on above baseline (tool page §Pricing). Public pricing page shows "$-" placeholders (pricing) | Prosody + content scores en-US only; content assessment (grammar/vocab/topic) via SDK retired July 2025 (release notes via search). Regions: "each language is available in all speech-to-text regions" (language-support). Rejects mixed-lingual and multi-speaker audio; wants ≥16 kHz, close mic. |
| SpeechAce | en-US, en-GB, fr-FR, fr-CA, es-ES, es-MX only — no Urdu/Hindi/Swahili/Arabic (api-plans, llms.txt) | Yes: phoneme quality_score + "sound most like" substitutions; syllable scores; letters/phonics via Score Phone List (endpoint list in llms.txt; field-level page 404'd on fetch — field names unverified today) |
Yes — word/syllable/phoneme extents (unverified today, page 404) | Both: Score Text (scripted), Score Speech / Score Task (unscripted), Score Phone List (phonics) | Tier-capped: Basic 30 s, Pro 45 s, Premium 2 min (api-plans); we independently verified the 30-s cap on the scripted endpoint | Strongest vendor evidence: 5,000+ items from 350+ child speakers in homework-noise conditions, 86% accuracy / F1 0.86, general (not kids-trained) 9th-gen model, "Kindergarten and up", used for phonics, sight words, ORF (kids page, early-literacy solution) | Basic $40/mo (5,000 × 15-s requests), Pro $80/mo (10,000), Premium $125/mo; overage $0.008 per 15-s audio (Basic/Pro), $0.0125 (Premium) (api-plans) | Returns WCPM, pauses, IELTS/PTE bands ("Hello Fluency"). A 60-s ORF passage exceeds Basic/Pro caps → Premium or chunking. |
| ELSA API | English only (positioned as "English Language Speech Assistant"; accent coverage 100+ L1s) (elsaspeak.com/elsa-api, API docs overview) | Sentence/word/phoneme scores, word stress, mispronunciation hints (vendor copy); IPA/substitution format unverified (docs page fetched had no field spec) | unverified | Both (scripted "with transcript", unscripted adds grammar/vocab) (docs) | unverified; websocket streaming available (General-api-info) | Consumer-app efficacy studies only (e.g. POPS Kids integration); no child accuracy numbers found | "Pay per time or number of characters"; rates on request (General-api-info) | Regionally-distributed k8s streaming servers (SLaTE 2023 paper: Anguera et al.) |
| SpeechSuper | English, Mandarin, German, French, Russian, Korean, Japanese, Spanish (+ Indian-English accent variant). No Urdu/Hindi/Swahili/Arabic (GitHub samples, Indian English post) | Phoneme/syllable/word/sentence scores (vendor claim) | unverified | Both (sent.eval.promax, speak.eval.pro, word/paragraph coreTypes) |
unverified | None found | PAYG from $20/mo minimum; $0.004 (word) – $0.008 (paragraph) per request (pricing via search) | 16 kHz/16-bit mono recommended |
| Language Confidence | English only (US/UK accent reference) (docs overview) | Yes — phoneme, word, transcript "nativeness", Oxford IPA (phonetic system) | unverified | Scripted, Unscripted, and a scripted-only Pronunciation API | unverified | Age of Learning cited as a young-learner customer; no numbers (site) | Per 15 s: Pronunciation $0.008 (Startup $100/mo) → $0.006 (Scaleup $500/mo); Scripted $0.01→$0.008; Unscripted $0.0125→$0.01 (api-pricing) | Rate limits 10–50 rps |
| Pronounce (getpronounce.com) | English (US/UK) consumer app; no developer API found (pricing page) | n/a | n/a | n/a | n/a | none | consumer plans | Not a candidate |
| Speechmatics | STT only — no pronunciation-assessment product (speechmatics.com); see §C for its Urdu STT | — | yes (STT) | — | — | — | — | |
| iFlytek ISE | English and Mandarin only (ISE product page, API doc) | Syllable/phone scores (EN), initial/final/tone (ZH) | streaming | scripted | unverified | none found | unverified | Urdu appears only in iFlytek translation products |
| Google Cloud | No pronunciation-assessment API. Read Along (formerly Bolo) does on-device reading feedback in 10 languages including Urdu and Hindi but exposes no developer API (Read Along — Wikipedia, Android dev blog) | — | — | — | — | Read Along pilot: 64% of children improved in 3 months (product claim, VentureBeat) | — | Cloud STT covers ur-PK (§C) |
| Amazon | No pronunciation product; Amazon Transcribe does not list Urdu at all (full table: hi-IN, sw-KE/sw-TZ, ar-SA present; no ur-) (supported languages) |
— | yes (STT) | — | — | — | — | Not a candidate for Urdu |
| 2025–26 entrants | Arabic phoneme scoring is the active research frontier (IQRA 2026 Interspeech challenge, 19 teams; Harf-Speech clinical Arabic phoneme framework) (IQRA 2026, Harf-Speech). No Urdu commercial entrant found in any search. |
FluencyService already does). SpeechAce is the only vendor with published child-speech accuracy and explicit phonics/nonword support (custom pronunciations), at ~2× Azure's price and with a 45-s Pro cap.| Model / tool | Urdu inventory? | What it outputs | Accuracy evidence | Compute | Licence / source |
|---|---|---|---|---|---|
| ZIPA (ZIPA-CR / ZIPA-T, 64 M and 300 M) — ACL 2025 | Yes — urd and hin are in the 88-language IPAPack++ training set (arXiv 2505.23170 HTML) |
IPA phone string (CTC or transducer) | Avg PFER ≈ 2.7 on seen languages, 3.2 unseen (Zipformer beats 300 M XLS-R baselines with 64 M params); limitations on sociophonetic variation | Zipformer is designed for efficiency; trained on A40/A100; CPU inference plausible for 64 M model but not benchmarked | Permissive, github.com/lingjzhu/zipa |
| PhoneticXEUS — Interspeech 2026 | "70+ languages"; Urdu unverified (trained on IPAPack++ 17k h which includes urd) (GitHub, arXiv 2603.29042) | IPA | 17.7% PFER multilingual, 10.6% accented English | CPU or GPU per README | open |
| facebook/wav2vec2-xlsr-53-espeak-cv-ft | Trained on CommonVoice-13 phone labels for ~60 languages; Urdu not listed in the documented training set, but the model does zero-shot transfer via articulatory features (HF card, Xu et al. 2021) | espeak-style phone string | none for Urdu | 300 M; runs on CPU at a few × real-time (unverified) | Apache-2.0 |
| kgnlp/allophant | 34 CV languages; zero-shot to any PHOIBLE inventory (so an Urdu inventory can be supplied) (HF, paper) | phones + articulatory attributes | +17% phone accuracy on unseen indigenous languages | XLS-R-300m base | open |
Meta MMS mms-1b-all |
Yes — adapters urd-script_arabic, urd-script_devanagari, urd-script_latin (HF card) |
characters (ASR), CTC emissions usable for GOP | Urdu WER 39.7 (ARL read) / 34.6 (CSaLT) zero-shot, 28.4 / 26.9 fine-tuned; conversational 46.4 → 42.5 — worse than whisper-large (WER We Stand, Table 3) | 1 B params; GPU preferred | CC-BY-NC 4.0 (non-commercial — a blocker for production) |
torchaudio MMS_FA forced aligner |
Language-agnostic: 23k h / 1,100+ languages; text must be uroman-romanised to a 29-token Latin set (a–z, ', -, *) (torchaudio tutorial) | word- and character-level timestamps + per-frame CTC scores | none for Urdu published | CPU fallback in code; DSAlign-class CPU aligners run ~0.5 RTF (People's Speech paper) — MMS_FA on CPU unverified | MIT (torchaudio); model weights CC-BY-NC per MMS (verify before prod) |
| Montreal Forced Aligner + IndicMFA | Yes — AI4Bharat IndicMFA ships acoustic models for all 22 scheduled languages incl. Urdu (101.79 h); uses grapheme-to-grapheme instead of G2P (IndicMFA, releases page) | phone/word TextGrids | none published; MFA remains the reference aligner in the 2026 survey (MFA 2026 arXiv) | CPU (Kaldi GMM-HMM) — fine on a Railway/EC2 CPU box | MIT (MFA); IndicMFA licence unverified |
| WhisperX | Default align models for {en,fr,de,es,it,ja,zh,nl,uk,pt}; no Urdu aligner shipped — you must plug a HF wav2vec2 Urdu CTC model (e.g. kingabzpro/wav2vec2-large-xls-r-300m-Urdu) (WhisperX README) |
word timestamps | none for Urdu | GPU recommended | BSD |
| Qwen3-ForcedAligner-0.6B | No Urdu (11 langs: zh, en, yue, fr, de, it, ja, ko, pt, ru, es) (Qwen3-ASR GitHub) | timestamps ≤5 min | claimed better than E2E aligners | GPU | Apache-2.0 |
| Urdu G2P | espeak-ng lists ur (Indic family, ph_urdu table based on hi_base) (languages.md, phsource); phonemizer wraps it (PyPI); CLE Lahore's PronouncUR lexicon generator (arXiv 1801.00409); DL Urdu G2P (urdu-lexicon-generator) |
IPA | espeak Urdu quality unverified; short-vowel diacritics are usually absent in Urdu text, so any G2P must guess vowels — a known error source | CPU | GPL (espeak-ng) |
| GOP recipes | Kaldi egs/gop_speechocean762 (GMM/TDNN, English) (Kaldi); alignment-free CTC-GOP on wav2vec2/XLS-R (GOP-SA / GOP-SF, SOTA phoneme-level on speechocean762) (arXiv 2507.16838); GOP-CTC topped the NOCASA 2025 children's-L2 challenge (arXiv 2509.03256); segmentation-free GOP without time alignment (arXiv 2603.25150) |
per-phone log-posterior ratio | English-only evaluations | any CTC model's emissions — i.e. works on ZIPA / MMS / a fine-tuned Urdu wav2vec2 | open |
Child-speech accuracy for these on Urdu: none published. The closest data points are (i) the ASER dataset — 5,301 Indian children aged 6–14, 81,330 clips in Hindi/Marathi/English, ASR-based reading-level classifier 86% accurate for English (arXiv 1912.04381); (ii) EGRA-Xhosa (letters + words, wav2vec2/HuBERT/Whisper; performance dominated by training-data balance) (arXiv 2505.17371); (iii) Bambara children's reading ASR — WER 0.42→0.22 after adaptation, under-10s remain the main error source (arXiv 2606.31508); (iv) BabAR phoneme recognition → developmental measures for young children (arXiv 2603.05213). No Urdu children's speech corpus exists publicly: CLE's corpora are adult (ELRA S0403), UrduSpeech (156 h, 1,000+ speakers, 2026) is adult (arXiv 2605.17846). Our 207 SurveyCTO recordings + 187 coach reviews are therefore a genuinely scarce asset — but at 8 kHz/12 kbps AAC they are below every model's 16 kHz expectation (see BASELINE.md).
CPU vs GPU: MFA/IndicMFA and 64 M ZIPA are CPU-feasible; MMS-1B, XLS-R-300m phone models and WhisperX are practical on CPU only at low throughput (a 60-s clip in tens of seconds — unverified; benchmark on the Railway worker before committing). A single T4/L4 on Modal or an EC2 g5 covers the fleet.
| Provider / model | Urdu? | Word timestamps | Streaming | Price | Max length | Children's-voice evidence |
|---|---|---|---|---|---|---|
| Soniox stt-async-v5 / stt-rt-v5 (in use) | Yes, "native-speaker accuracy in Urdu and 60+ languages"; vendor claims 6.3% WER vs Google 13.4% on real-world Urdu (compare page); benchmark methodology not published on the Urdu page (speech-to-text/urdu) | Yes, included | Yes | $0.10/hr async, $0.12/hr real-time, diarization + language-ID + timestamps included (pricing) | 5 h/request (models) | none published |
| Deepgram Nova-3 | Yes — dedicated monolingual ur model, batch + streaming + keyterm prompting; WER chart referenced but numbers not in text (Urdu launch post, models overview). Urdu is not in the 10-language code-switching multilingual mode |
Yes | Yes | $0.0043/min batch, $0.0048/min streaming (promo); multilingual $0.0052 (pricing) | — | none |
| AssemblyAI | Urdu on Universal-2 (99 langs) only; Universal-3 Pro (6 langs) and 3.5 Pro (18 langs) do not support Urdu — a Sep-2026 routing bug briefly sent ur to 3.5 Pro (supported languages, changelog, Urdu page) |
Yes | Urdu added to Universal-Streaming | $0.15/hr Universal-2, $0.21/hr 3.5 Pro (pricing) | — | none |
| ElevenLabs Scribe v2 / v2 Realtime | Yes (urd); Scribe v1 FLEURS Urdu WER 14.4% ("Good" tier, 10–25%) (Urdu page); v2 claims 99 languages (Scribe v2 blog) |
Word + character level | v2 Realtime, 150 ms | $0.22/hr Scribe, $0.39/hr realtime on API pricing (pricing/api); Urdu page still shows $0.40/hr — inconsistent | — | none |
| Speechmatics Ursa 2 | Yes (ur), batch + real-time; language-ID not available for Urdu (languages, Ursa 2 post) | Yes | Yes | free 8 h/mo; per-hour rate unverified (pricing) | — | none |
| OpenAI | whisper-1 tokenizer lists ur, hi, sw (tokenizer.py); gpt-transcribe (Jul 2026) accepts ISO-639-1 language hints but Urdu not explicitly tabulated — unverified for gpt-transcribe; Azure Voice Live lists Urdu for gpt-4o-transcribe (Azure list) |
Only whisper-1 returns word timestamps (timestamp_granularities); gpt-transcribe / gpt-4o-transcribe do not (STT guide) |
gpt-live-transcribe | gpt-transcribe $0.0045/min, gpt-4o-transcribe $0.006, gpt-4o-mini-transcribe $0.003, whisper-1 $0.006, gpt-live-transcribe $0.017 (pricing) | 25 MB file | Whisper on children: MyST WER 13–14% zero-shot → ~9% fine-tuned (Kid-Whisper, Graave et al.); Ghana ORF (regional-accent children) Whisper-v2 WER 13.5, ORF score r = 0.96 with experts (arXiv 2310.17606); Dutch primary miscue detection Whisper-v2 WER 9.8%, miscue F1 0.52 (arXiv 2406.07060) |
| Google Gemini 3.5 Transcribe | Urdu NOT in the 85+ list (hi-IN, sw-KE, ar-EG are) (model page) | Yes, but "degrades transcription accuracy", incompatible with custom vocabulary, caps request at 30 min | -live, 10 min/session |
$0.003/min audio in + $0.002/min text out ≈ $0.005/min (pricing) | 1 h | none |
| Google Cloud STT v2 (Chirp 3 / Chirp 2) | ur-PK on chirp_3 and chirp_2, eu region only; no long/short model for Urdu, so word-level timestamps for Urdu are unverified (languages) |
see left | — | — | — | none |
| Azure STT (fast/batch/real-time) | ur-IN listed with fast transcription; ur-PK not found in the STT include file fetched — verify in portal (stt.md); sw-KE yes, sw-TZ no fast-transcription |
Yes | Yes | $1/hr-class Standard; fast $0.66/hr (MS Q&A) | — | none |
| Amazon Transcribe | No Urdu (table) | — | — | — | — | — |
| NVIDIA Parakeet-TDT 0.6b v3 / Canary-1b v2 | No — 25 European languages only (parakeet card, canary card) | yes | yes | open | — | Parakeet-TDT competitive with Whisper-large on MyST (15.8 vs 14.8 WER) (ChildVox) |
| Mistral Voxtral Transcribe 2 | No Urdu (13 langs incl. Hindi, Arabic) (docs) | yes | yes | — | 3 h | none |
| Kyutai STT | English/French only (kyutai.org/stt) | — | — | open | — | — |
| Qwen3-ASR 0.6B/1.7B | No Urdu (30 langs incl. Hindi, Arabic, Persian); community issue open for Urdu fine-tuning (GitHub, issue #87) | via ForcedAligner (no Urdu) | — | Apache-2.0 | — | — |
| Qwen3-Omni (audio LLM) | Yes — ur is one of 19 speech-understanding languages (tech report) |
no native word timestamps | — | open | 40 min | none |
| Meta MMS-1b-all | Yes (see §B) | via CTC alignment | — | CC-BY-NC | — | — |
| Whisper large-v3 / turbo, Urdu | zero-shot whisper-large WER 26.3 (ARL read) / 24.4 (CSaLT) / 18.3 (conversational); fine-tuned 23.8 / 22.4 / 17.9; seamless-large best on read speech (24.0 → 17.1) (WER We Stand, COLING 2025, Table 3); community fine-tunes: large-v3 WER 26.0 / CER 9.4, large-v3-turbo WER 26.2 / CER 8.8 on CV-17 Urdu (HF, turbo) | via WhisperX/MMS_FA | — | open | 30-s windows | none for Urdu |
| Pakistani / Urdu-specific | CLE Lahore (KICS-UET) offers Urdu STT web services and a district-names corpus (CLE, KICS); Kaldi Urdu recipes (PRUS, qoul) (LUMS Kaldi); Common Voice Urdu ~81 h, FLEURS 12 h (survey); MagicHub 436-h scripted corpus (commercial) (MagicHub) | — | — | — | — | no children's corpus |
Take-aways. (1) Word-timestamped Urdu ASR is cheap and plentiful (Soniox, Deepgram, Scribe, AssemblyAI-U2, Speechmatics, whisper-1) — $0.10–0.36/hr. (2) The big 2026 flagships have dropped Urdu from their newest tiers (AssemblyAI 3.x, Gemini 3.5 Transcribe, Qwen3-ASR, Voxtral, Parakeet/Canary) — pin to Soniox/Deepgram/U-2 and keep a Whisper/seamless fallback. (3) No provider publishes Urdu child-speech numbers; on English children, zero-shot Whisper-class WER is 10–15% and can be ~9% with 10–50 h of child fine-tuning (Kid-Whisper), but fine-tunes generalise poorly across child datasets (MyST → CMU Kids WER rises) (CHSER). (4) Dysfluent/hesitant reading is where strong ASRs hurt most: they auto-correct the child's errors ("stronger ASR models tend to correct pronunciation errors during transcription", smaller models detect more) (arXiv 2601.14744) — a direct threat to correct/incorrect-per-word scoring, and an argument for phone-level (§B) rather than word-level evidence on the decoding subtasks.
| Model | Native audio in? | Urdu audio? | Per-word timestamps? | Cost per 5 min child audio (est.) | Verdict for per-word scoring |
|---|---|---|---|---|---|
| OpenAI gpt-realtime / gpt-4o-audio | Yes | unverified (Azure Voice Live lists Urdu for gpt-4o-transcribe; realtime language list not fetched) | No | Audio in $32/M tokens (pricing); at OpenAI's ~600 audio tokens/min (unverified rate) ≈ $0.10 | Can be prompted with reference text; no published ORF per-word eval; MDD F1 46.3 on L2-Arctic-plus (P 52.7 / R 41.3) — below an instruction-tuned Whisper+Llama at 62.8 (arXiv 2601.14744) |
| GPT-5.x (text/vision) | Audio via transcription pipelines (gpt-transcribe) rather than native chat audio per OpenAI docs; third-party claims of native audio are unverified (GPT-5.5 multimodal, Framia — secondary) | — | — | — | Use as the judge/verbaliser over structured evidence |
| Gemini 3.x Pro / Flash (audio understanding) | Yes — 32 tokens/s, 9.5 h/prompt, MM:SS timestamp references (audio docs) | Not tabulated for general audio understanding; Urdu absent from 3.5 Transcribe's list (§C) — treat as unverified | Only coarse MM:SS via prompting; precise words only via 3.5 Transcribe | 3.1 Pro $2/M in, $12/M out (pricing) → 5 min ≈ 9.6k audio tokens ≈ $0.02 + output ≈ $0.04–0.05; 2.5 Flash audio $1/M → ≈ $0.01 | Gemini 2.5 Flash / 3.5 Flash scored "substantially worse" than specialised models across five public child-speech datasets (ChildVox); Gemini 1.5 Flash MDD training-free study exists (ResearchGate) |
| Claude (Fable 5.1 / Opus 4.7) | No audio input in the API — text, images, PDFs only (ParseJet 2026 guide — secondary; Anthropic docs list no audio content block) | — | — | — | Vision-only role (§F) and text judge |
| Qwen3-Omni / Qwen3.5-Omni | Yes, 19 speech languages incl. Urdu, 40-min inputs (report, 3.5 report) | Yes | No native | self-host GPU | Qwen2-Audio zero-shot on speechocean762: agrees ±2 with humans on good speech but over-predicts scores for poor speech and lacks precision in error detection (SLaTE 2025) |
| Voxtral (Mistral) | Yes | No Urdu | word timestamps in Transcribe 2 | — | not for Urdu |
Published evals and failure modes that matter for us
| Study | Population | System | Result |
|---|---|---|---|
| FLORA / Bolaños (2013) | US grades 1–4, 738 one-minute readings | Sonic ASR | WCPM within 3–4 words of human scorers (FLORA, Bolaños) |
| Nese & Kamata (2020) | US grades 2–4 incl. SWD/EL | ORF ASR | word-level agreement lower for students with disabilities; passage-level WCPM differences not exacerbated (ERIC ED607983) |
| Ghana ORF (2023) | Ghanaian primary, regional accents, classroom audio | Whisper-v2 zero-shot | WER 13.5 (vs 12.8 adults); ORF score r = 0.96 with experts (arXiv 2310.17606); a 2026 journal version reports WER 10.3 / r = 0.98 (ScienceDirect) |
| Dutch reading diagnostics (2023–24) | Dutch primary | Kaldi / Whisper / wav2vec2 | word-level MCC 0.63 best system (arXiv 2306.03444); isolated words much harder (κ/MCC 0.20–0.41) than sentences/stories (ASR tutor, arXiv 2306.04190); miscue detection Whisper P 0.52 / F1 0.52, wav2vec2 R 0.83 (arXiv 2406.07060) |
| Speech-enabled reading fluency validation (2025–26) | children, word decoding + passage | ASR instrument | moderate human agreement: MCC 0.43 (word decoding) / 0.55 (passage) (IJAIED — paywalled, numbers via search snippet, unverified) |
| Kid-Whisper / MyST | US elementary conversational | Whisper fine-tuned | 13.9 → 9.1 (small), 13.2 → 8.6 (medium); tiny.en 15.9 (arXiv 2309.07927, WOCCI 2025); MyST→CMU Kids transfer degrades (CHSER) |
| ChildVox (2026) | birth → school age, 17 datasets | Whisper-L-v3 14.8 WER on MyST; Parakeet 15.8; Qwen2-Audio 13.3; Gemini Flash worse | specialised > general LALMs (arXiv 2605.29257) |
| Arabic Little STT (2025) | Arabic children | Whisper-L-v3 | WER 66% — 4.1× adult Arabic (arXiv 2510.23319) — a warning for non-English scripts |
| ASER India (2019) | 5,301 children 6–14, Hindi/Marathi/English, field phones | ASR classifier | 86% reading-level accuracy (English) (arXiv 1912.04381) |
| Amira (US) | K–6 | proprietary | "consistent accuracy across subgroups", TPRI-licensed assessment; no public word-level numbers (Amira research, ORF blog) |
| Google Read Along | 10 langs incl. Urdu, Hindi | on-device ASR | product claims only (35–85% ORF gain for <45 WCPM readers, The National); no technical report found |
| Bambara children (2026) | 60 children | Fast-Conformer | WER 0.42→0.22; under-10s dominate errors (arXiv 2606.31508) |
| Xhosa EGRA (2025) | letters + words | wav2vec2/HuBERT/Whisper | data balance drives accuracy (arXiv 2505.17371) |
| Malay/Tamil fluency (2025) | children | fine-tuned ASR + GPT | beats GPT-4o-audio direct (arXiv 2505.19671) |
| Gates EGRA-AI (Kiswahili) | UCT, wav2vec-based EGRA | — | project page unreachable (HTTP 530) — unverified (Grand Challenges) |
Realistic targets: for passage ORF (English, regional accent, zero-shot Whisper-class), expect WER 10–15% and WCPM correlation ≥0.95 with humans — good enough for coach pre-fill. For isolated items (letters, nonwords, single words) word-level agreement drops to κ 0.2–0.4 unless the recogniser is constrained to the item set (phone-level or grammar-constrained decoding). Urdu / Hindi children: no published word-level agreement anywhere — we will have to produce it from our own 187 coach-reviewed EGRA recordings.
| Model | Handwriting evidence | Digits / forms | Urdu script |
|---|---|---|---|
| GPT-5 / 5.4 | IAM handwriting CER ≈ 1.22% (best) (CodeSOTA — secondary); lowest hallucination rate (6%) and best noisy-date extraction on a 17-model handwritten medical-forms benchmark (arXiv 2604.16504) | ~85% field accuracy / F1 ≈ 90% on discrete fields in that benchmark | Gemini-2.5-Pro beat GPT-4o on Urdu Nastaliq newspaper OCR (WER 0.133); 500-sample GPT-4o fine-tune cut WER 6.1% (LREC 2026 "From Press to Pixels") |
| Gemini 3 / 3.1 Pro | IAM CER ≈ 1.44%; 100/100 on AIMultiple cursive set (AIMultiple); best CER on METATR handwritten (28.5%) (arXiv 2605.26712); best free-text WER 0.50 on handwritten forms (arXiv 2604.16504) | Handwritten-KIE 94.5 (secondary) | best zero-shot Urdu OCR in the LREC study |
| Claude Opus 4.7 / Fable 5.x | IAM CER ≈ 1.31% (secondary); Claude Sonnet 4.6 best on dates and numerical fields in the forms benchmark (arXiv 2604.16504); Opus 4.5 second on METATR handwritten (31.2%) | strong on numeric cells | untested on Nastaliq |
| Exam-grading VLM (2026) | 98.4% on 3,141 handwritten answer positions across 61 exams; false-negative (correct marked wrong) 0.58% with reference-solution prompts; handles crossed-out and out-of-cell answers (arXiv 2606.11477) | directly analogous to a coach-photographed EGMA sheet | — |
| Specialised Urdu digit CNNs | 98–99% on isolated Urdu handwritten numerals (MDPI 2023, AlexNet transfer) | — | Urdu HTR low-resource CRR 92–96% (arXiv 2512.01348) |
Use: photograph the paper EGMA sheet (or the child's written answers) and send to Gemini 3.1 Pro or Claude with the answer key in the prompt, asking for per-item {expected, read, correct?}; keep the FN-first framing from the exam-grading paper. Children's digit handwriting specifically is unbenchmarked for VLMs — run a 50-sheet probe. No published VLM benchmark on children's Urdu handwriting exists.
Cost basis: a 5-minute battery split ~2 min English speech, ~2 min Urdu speech, ~1 min math speech (see BASELINE.md). Prices from the tables above; "≈" = my arithmetic.
| Subtask | Language | Stack 1 (recommended) | Stack 2 (alternative) | Est. cost / child | Biggest risk |
|---|---|---|---|---|---|
| Letter names & sounds | English | Azure PA en-IN/en-US scripted, one call per 10-letter row (word-level accuracy + phoneme scores; NBest sounded-like en-US) | SpeechAce Score Phone List (phonics) | Azure ≈ $0.02–0.04 (1–2 min @ $1.32/hr); SpeechAce ≈ $0.06 (8 × 15 s) | Isolated tokens are the weakest ASR case (κ 0.2–0.4 in Dutch data); "b" vs "d" vs "p" confusions need phoneme NBest, which is en-US only |
| Letter names & sounds | Urdu | ZIPA-CR (urd) phone string per 10-letter row → match against expected phone(s) with a confusion-tolerant matcher (self-host) | MMS_FA / IndicMFA forced-align expected letter names to the row audio, score per-letter CTC confidence (GOP-SA) | compute only (≈ $0.001 on GPU) | Zero published Urdu child accuracy; short vowel ambiguity in G2P; needs our own 187-review calibration set |
| Phoneme (initial-sound) awareness | English | Azure PA en-US with the target phone as reference + NBestPhonemes | wav2vec2-xlsr-53-espeak-cv-ft first-phone match | ≈ $0.01 | Children often say letter name not sound; must accept both via rule |
| Phoneme awareness | Urdu | Phone recogniser (ZIPA / allophant with Urdu PHOIBLE inventory) → first-phone equality | Qwen3-Omni (Urdu audio) prompted "which sound did the child say?" — eval-only | compute only | No Urdu MDD benchmark; LLM prior-over-evidence (§D) |
| Nonword decoding | English | SpeechAce Score Text with custom pronunciations (explicit nonword G2P) | Azure PA scripted (relies on internal G2P for nonwords — quality unverified) | SpeechAce ≈ $0.02 (2–3 × 15 s) | Strong ASR "corrects" nonwords to real words (§C/D) — phone-level evidence is safer |
| Familiar word reading (50 words, 60 s) | English | Azure PA continuous mode + our FluencyService diff for omissions/insertions (miscue flag unavailable >30 s) |
SpeechAce Pro (45-s cap → split into two rows) | ≈ $0.02 | 30-s single-shot cap; timed-out items must be excluded by timestamp |
| Familiar word reading | Urdu | Soniox stt-async-v5 (word timestamps) → edit-distance alignment vs list (current FluencyService pattern) |
Deepgram Nova-3 ur / AssemblyAI Universal-2, same alignment; MMS_FA for tighter timestamps |
≈ $0.002 | ASR normalises child errors to real words → false "correct"; no child WER data |
| Oral passage fluency (WCPM + per-word) | English | Azure PA continuous + timestamps → WCPM at 60 s; per-word from alignment | Whisper-large-v3 (or gpt-transcribe) + WhisperX/MMS_FA alignment; literature says r ≥ 0.96 WCPM at WER 10–15 | ≈ $0.02–0.03 | Miscue detection F1 only ~0.5 in Dutch child data → coach must review flagged words, not accept silently |
| Oral passage fluency | Urdu | Soniox/Deepgram Urdu + MMS_FA forced alignment of the reference → per-word CTC confidence + edit-distance (hybrid: ASR for insertions, aligner for omissions/mispronunciation) | Whisper-large-v3-urdu fine-tune (WER ~26 CV) self-hosted + same alignment | ≈ $0.002–0.01 | 8 kHz field audio; Urdu WER 20–40% on adults means high false-flag rate → HITL burden; needs our own calibration |
| Comprehension answers | Both | ASR transcript (Soniox for Urdu, Azure/gpt-transcribe for English) → text LLM judge (GPT-5 / Claude) with rubric | Gemini 3.1 Pro audio + rubric (Urdu audio support unverified) | ≈ $0.01 | Judging inferential answers in child Urdu; keep LLM as rubric-follower with quoted transcript |
| Number identification (spoken) | Urdu/English/mixed | Soniox (alphanumerics + code-switch) → numeral normalisation (Urdu number words → digits) | Deepgram ur with keyterm prompting of the 8 numbers |
≈ $0.001 | Urdu–English code-switching on numbers ("twenty-eight" vs "اٹھائیس"); ASR digit normalisation errors |
| Addition/subtraction (spoken answers) | mixed | same as above + expected-answer match | — | ≈ $0.001 | children count aloud before answering → take last number token |
| Word problems (spoken) | mixed | ASR → LLM judge with answer key | — | ≈ $0.005 | as comprehension |
| Written EGMA / photographed sheet | — | Gemini 3.1 Pro vision with answer key, FN-first prompt | Claude (numeric fields) / GPT-5.4 (lowest hallucination) | ≈ $0.01–0.02/page | Children's digits + Urdu numerals (۰–۹) unbenchmarked; photo quality |
Where Urdu has NO commercial phoneme-scoring option and the fallback stack: letter sounds, phoneme awareness, nonword-style decoding, and per-word mispronunciation on Urdu ORF. Fallback = (ASR words + timestamps from Soniox/Deepgram) ⊕ (forced alignment of the reference via MMS_FA or IndicMFA) ⊕ (phone posteriors from ZIPA / fine-tuned Urdu wav2vec2 for a CTC-GOP score per expected phone), fused by edit-distance alignment as FluencyService does today, with a text LLM only writing the coach-facing explanation. Licensing gate: MMS weights are CC-BY-NC; prefer ZIPA (permissive) or IndicMFA/MFA (MIT) for anything in production.
Battery-level cost: commercial-only English + Urdu ASR stack ≈ $0.05–0.10 per child; adding a Gemini/GPT audio pass for comprehension ≈ +$0.05; self-hosted Urdu phone models add GPU-seconds only. All well under the $0.43 coaching-session benchmark.
ur-PK vs ur-IN in the live portal (include file showed only ur-IN).quality_score, sound_most_like) and the Score Phone List nonword path — docs pages 404'd today.