Constructs recoverable from a read-aloud beyond WCPM, with evidence and phase
Type: research memo | Status: draft v1 | Date: 2026-09-15 | Author: Rumi research (Claude) | Scope: EGRA/EGMA-style, coach-administered, AI-scored battery; Urdu + English reading, spoken maths, optional photo of written work; children 6–11, Pakistani government schools; audio arrives as WhatsApp voice notes (Ogg/Opus, 16 kHz mono, ~16–24 kbps).
The operator's question: "Given that we want granularity, is there more stuff that we can infer from a recording? One can infer WCPM. What else could be inferred … we are inventing a new era of standardized assessments that are AI driven. What more could we glean?"
One-paragraph answer. A timed oral read is a time series, not a count. Once the audio is force-aligned to the passage text, every word carries an onset, an offset, a correctness label and a confidence, and every gap between words carries a silence duration. From that word-level table alone you can recover (a) per-word latency distributions, (b) a miscue record (substitution / omission / insertion / repetition / self-correction, with the content of each substitution), (c) pause-based prosody, (d) a per-grapheme error profile, (e) rate curves across the minute, and (f) a model-based fluency score with a per-child standard error — all of which the paper form discards. The acoustics add pitch contour, intensity and voicing for expression scoring, and speech-rate/pause markers for affect. The same recording also scores the assessor: whether they prompted, how long they waited, whether the timing was honest, how noisy the room was. The evidence base is strong for the word-level table (NAEP's machine WCPM correlates .96 with human; DIBELS/CORE psychometrics; Dutch/English miscue-detection F1 up to ~.73), moderate for prosody (automated NAEP-scale scoring reaches ~62–76% exact agreement, humans themselves only 58%), thin-but-real for maths strategy inference from latency (well-founded in Siegler/Thevenot chronometry, but no automated field product), and speculative for affect/anxiety. Everything is computable on 16 kHz WhatsApp audio except fine fricative/aspiration phonetics, which the ~16–24 kbps Opus encode degrades.
Each construct gets: how it is computed (from ASR, forced alignment, or raw acoustics), evidence (reliability/prediction, with URLs), and 16 kHz WhatsApp feasibility. Claims I could not verify against a primary source are tagged unverified. Vendor marketing numbers are tagged vendor claim. Section F ranks everything into v1 / v2 / research.
A design fact that shapes everything below: WhatsApp voice notes carry no item-level timestamps. The only clock inside the recording is the coach's own voice ("Begin", "next", the maths item read aloud). So the first processing step is always coach/child diarization + coach-utterance detection, and the coach's speech becomes the item-onset marker. That is what makes response latency (A1, B1) and protocol adherence (Section C) recoverable at all. WhatsApp encodes voice notes as Ogg/Opus at 16 kHz mono, roughly 16 kbps on capture and re-transcoded to ~24 kbps server-side (WhatsApp audio format; openclaw PR #144507 on WhatsApp's 16 kHz Opus expectation). Wideband Opus keeps everything up to ~8 kHz, so pitch (children's F0 is 200–400 Hz), intensity, timing and vowel formants survive; what is lost is fine detail above 8 kHz and some spectral resolution at low bitrate — relevant only to the phonetic constructs (A4, A5).
Computation. Transcribe with a child-adapted ASR (Whisper large-v2/v3 or wav2vec2/HuBERT fine-tuned on children), prompt the recogniser with the reference passage, force-align hypothesis ↔ reference (SCTK / Levenshtein alignment) to label each reference word as correct / substituted / omitted, and each extra hypothesis word as insertion. Keep per-word timestamps and per-word confidence.
Evidence. - NAEP's 2018 ORF study did exactly this at national scale: the machine-vs-human correlation for words-correct counts was .96 (per passage .93–.99), while two humans agreed at .99/.97 on word lists (NCES 2021-025, Methods + Appendix A). The engine accepted dialect and second-language pronunciations "as long as the speaking pattern remains consistent" and was validated as unbiased against ELs and Black students (Balogh et al. 2012). - Amira reports human-item-level agreement of 96% and WCPM correlation of 98–99% with human raters, similar across gender/race/MLL subgroups — vendor claim, no independent technical report located (Amira ISIP page; NCII progress-monitoring entry). - Child ASR is worse than adult ASR everywhere: Whisper reaches ~13% WER on children's English, ~9% after child fine-tuning (Adaptation of Whisper to child speech; Learning Agency summary); on Dutch primary readers Whisper large-v2 gave 9.8% WER child vs 6.7% adult (Gao et al., Interspeech 2024). - The key trap: a language-model-decoded ASR (Whisper) corrects the child's misreadings into the right word. In the Dutch study Whisper's "error ratio" was 0.58 (it under-detects reading errors), and recognition accuracy on incorrectly read words was only .53–.60 and on part-word attempts .23–.29, versus .90–.93 on correctly read words (Gao et al. 2024). The fix — prompting Whisper with the reference text seeded with synthetic mistakes, plus an LLM rescoring step and a hallucination filter (hypotheses >20% longer or >5% shorter than the text are discarded) — cut WER 9.4%→5.1% and lifted reading-mistake F1 from 0.39 to 0.73 (Gao et al., Interspeech 2025). A Kaldi system with passage-specific LMs matched or beat Whisper for error detection in the same lab (Automatic assessment of oral reading accuracy, 2023, word-level MCC = .63). - Urdu is the weak link. Off-the-shelf Whisper-Small gives ~34% WER on adult Urdu (Tiny 67%, Base 54%) (Lightweight Whisper for Urdu, 2025; N-shot Whisper on Pashto/Punjabi/Urdu, CHiPSAL 2025). On Arabic children, Whisper large-v3 hit 66% WER, 4.1× its adult Arabic rate (Arabic Little STT, 2025). There is no published child-Urdu read-speech corpus that I could find (unverified that none exists privately). Reference-prompted decoding and a closed passage vocabulary mitigate this a lot — the task is verification against a known text, not open transcription — but Urdu miscue precision must be measured, not assumed.
16 kHz feasibility. Yes; every cited system runs at 16 kHz. Low bitrate mainly hurts confidence calibration on fricatives.
Computation. From alignment: inter-word gap before each word, word duration, and the child's onset latency after the coach says "Begin". Summaries: median gap, 90th-percentile gap, count of gaps > 1 s and > 3 s (the EGRA prompt threshold), and gaps conditioned on word properties (length, frequency, decodability, position after punctuation).
Evidence. On the CORE dataset (2,094 Grade 2–4 US students, ~50–100-word passages), a random forest using only relative silence times before and after specific word chunks plus grade explained 58.4% of WCPM variance out of sample (52.7% without grade) (Kara et al., PTAM 2023). Pause statistics, syllable rate and intensity dynamics — with no ASR at all — discriminated decoding-deficit categories "close to" an ASR-based system (Predicting lexical skills from oral reading with acoustic measures, 2021). Children with weaker comprehension make more inappropriate pauses and more pauses before commas (Álvarez-Cañizo et al. 2015).
Feasibility. Trivial at 16 kHz; timing precision (~10–20 ms frames) is far below the 100 ms resolution that matters.
Computation. Three signals: (i) intra-word silences and multiple onsets for a single reference word (the alignment produces several short hypothesis fragments mapped to one word — Gao et al. label these "part-of-word attempts"); (ii) word duration normalised by syllable count (a 3-syllable word taking >1.5 s is effortful); (iii) phone-level decoding of the fragment to detect letter-sound-by-letter-sound production. Report a per-child "effortful decoding rate" = words with ≥2 attempts or intra-word pause ÷ words attempted.
Evidence. Hesitation is already an ORF construct ("hesitations for more than 3 seconds are marked", Acadience ORF guidance); the EGRA protocol prompts after 3 s (EGRA Toolkit V2, 2016). ASR recognises part-word attempts poorly (.23–.29 accuracy, Gao et al. 2024) — but detecting that an attempt happened (a fragment + a silence + a restart) is much easier than transcribing it. Phone-level CTC models trained on young readers reach PER ~28% and "better detect reading mistakes" because CTC forces monotonic attention (Gelin et al. 2021). No paper reports reliability for an "effortful decoding rate" as a score — unverified as a psychometric quantity; treat as v2 with a human-coded validation set.
Feasibility. Fragment/restart detection: yes. Letter-by-letter phone identification in Urdu: research-grade.
Computation. Omission/insertion/substitution fall directly out of alignment. Repetition = hypothesis word(s) re-matching the immediately preceding reference span. Self-correction = an error followed within N words by the correct token for the same reference position (EGRA already scores this as correct; EGRA Toolkit V2, Annex instructions "Count self-corrections as correct"). Self-correction latency = time from the erroneous token offset to the corrected token onset. Substitution sub-type = compare the spoken substitute to the target: graphophonic (shares onset/rime/letters), semantic (synonym/same class — use an LLM judge), visual (similar shape in Nastaliq — needs an Urdu grapheme-confusion table, to be built), nonword.
Evidence. - Miscue categories are the running-record tradition (Clay) and the Dutch team formalised them for ASR: Wav2Vec2-Large recall .83, Whisper precision .52 / F1 .52 overall, but substitution precision as low as .17 for Wav2Vec2 and insertion recall .17 for Whisper (Gao et al. 2024); with prompting, F1 .73 (Gao et al. 2025). Microsoft Reading Progress auto-detects errors and lets teachers re-label them by type (mispronunciation, omission, insertion, repetition, self-correction), and states auto-detect "is currently best for English" and "may not recognize accents and dialects well" (Reading Progress product guide). SoapBox's fluency API returns substitutions, omissions, repetitions at phoneme level (Kelly et al., Interspeech 2020; SoapBox technology page). - Self-corrections are predictive. In 3,184 running records from 140 struggling first-graders, self-corrections significantly and positively predicted reading progress (D'Agostino, Kelly & Rodgers 2019); a generalisability study shows accuracy and self-correction scores need several passages to be dependable (2021). Errors on content words hurt comprehension more than errors on function words; frequency and relevance of the misread word matter (Reading Errors of Importance, 2025). Error rate in Grades 2–3 has a non-linear relation to comprehension (Abbott et al. 2013). - Graphemic–phonemic classification of substitutions has a classic template: Labov's GAREC coded inner-city children's reading errors by grapheme, phoneme and dialect rule (GAREC).
Feasibility. Counts: yes. Substitution content (needed for sub-typing): depends on the ASR not auto-correcting; needs the prompted-decoding trick. In Urdu, expect lower precision until a child corpus exists.
Computation. From alignment: pause at punctuation vs mid-phrase ("pause intrusions"), phrase length between pauses, sentence-final pitch fall (F0 declination), pitch range/variability, intensity contour, word stress on multisyllables. Feature extractors: Praat, openSMILE (eGeMAPS) or wav2vec2 embeddings; target: NAEP 4-level scale (word-by-word → phrase groups → mostly phrases → sentence-level with expression) (NAEP scale and rubric, NCES 2021-025 Fig. 2).
Evidence. - Human ceiling is low: two NAEP scorers agreed exactly 58%, adjacent 39% (NCES 2021-025); a five-rater G-study on 177 Grade 1–3 children examined the NAEP and MDFS rubrics (Reading Psychology 2018). - Automated: Bolaños et al. (FLORA) used 5 lexical + 15 prosodic features and reached ~76% exact agreement in-domain (Bolaños et al. 2013; FLORA); a 2024 CNN on 5,841 CORE recordings from 1,811 Grade 2–4 students, using openSMILE features, got 62.5% accuracy / QWK .38 when both passage and student were unseen (Frontiers in Education 2024). A wav2vec2 model on 1,447 recordings of Indian 10–14-year-old ESL readers predicted the NAEP-style comprehensibility rating at Pearson .827 vs human (inter-rater .76), beating a 39-feature hand-crafted system at .794 — and it did so without the reference text (Vaidya, Sahoo & Rao, IIT Bombay 2024). Rubric-guided speech-LLMs are the 2026 frontier (arXiv 2603.16889). - Why it matters: children whose pause count fell from Grade 1 to 2 and who had an adult-like intonation contour early comprehended better in Grade 3 (Miller & Schwanenflugel 2008); pause intrusions, child–adult pitch match and sentence-final pitch change predicted comprehension in Taiwanese children (Acoustic Features of Oral Reading Prosody, 2022). In NAEP 2018, fourth-graders below Basic-Low averaged below expression level 3 — "they often paused in the middle of a phrase" (NCES 2021-025). Microsoft ships exactly the pause-based subset: monotone reading, long pauses, not pausing at full stops/commas, inflection on ?/!, stress on multisyllables (Microsoft Education blog, Mar 2022).
Feasibility. Pause-based prosody (the most predictive subset) is pure timing — yes. F0 contour survives Opus wideband — yes. Note: Urdu and Punjabi are prosodically different from English (Punjabi/Pothwari are tonal, Urdu–Punjabi–Pothwari comparison); the NAEP rubric transfers, the acoustic thresholds do not. Unverified: any Urdu reading-prosody norms.
Computation. For every substitution/nonword where the spoken form is recovered, align target grapheme string ↔ produced phone string, and tally which GPCs fail (e.g., English short vowels, digraphs th/sh/ch, silent-e; Urdu: the four z letters, aspirated vs unaspirated stops, retroflex/dental, nasalisation, short-vowel diacritics that are usually absent in print). Aggregate per child over Urdu + English passages and — critically — over the class (Section E2).
Evidence. Amira advertises "error types to reveal decoding vs language gaps" and a class-level Skills Status Report with red/yellow/green per skill (Amira 5 ways; Amira Theoretical Framework) — the closest shipped analogue, no independent validation found. Phone-level recognition for young readers is feasible (PER 28.1%, Gelin et al.) but degrades sharply in noise: PER 14.6% at SNR > 25 dB vs 40.6% at SNR < 10 dB (Self-supervised phoneme recognition for reading, 2025). The Dutch study shows ASR confusions on children cluster on specific phones (/v/→/f/, /z/→/s/) — i.e., the recogniser's own errors overlap with the child's likely errors (Gao et al. 2024). No study I found validates an automated GPC error map against a phonics diagnostic — unverified as a psychometric; it is a plausible v2 teaching output rather than a v1 score.
Feasibility. English: v2 with noise gating. Urdu: research; aspiration and sibilant contrasts are exactly what 16–24 kbps Opus blurs (my judgement, unverified).
Computation. Tag each passage word a priori as high-frequency sight word / regular decodable / irregular. Report accuracy and latency by class. Consistency = does the child fail the same GPC or the same word every time it recurs (within the passage, across Urdu/English, across sessions).
Evidence. DIBELS 8 separates Word Reading Fluency (mixed list) from ORF and nonsense-word fluency precisely to isolate decoding (DIBELS 8 Technical Manual); EGRA carries a non-word subtask for the same reason and Sindh's 2024 EGRA reports it separately (Sindh EGRA 2024). Deriving the split inside the passage instead of from a separate subtask is a design shortcut; its reliability is unverified and will be low with only ~60 passage words (few irregulars per passage).
Computation. Regressive repetitions of a phrase (not a single word), pauses > 1 s immediately before low-frequency words, and self-corrections on meaning-changing errors versus meaning-preserving ones.
Evidence. Repetition and self-correction are standard running-record codes; self-correction predicts progress (D'Agostino 2019). NAEP explicitly did not penalise expression for meaning-preserving substitutions (NCES 2021-025), which is the same distinction. Automated "monitoring" scores — unverified; treat as research.
Computation. Cumulative words-correct against time; fit slope for 0–20 s, 20–40 s, 40–60 s; report start latency and rate change.
Evidence. I found no published study on within-minute rate change in children — unverified construct. The nearest evidence is on duration: reading-duration effects on ORF reliability in middle school (Effect of Reading Duration, 2014) and Piper & Zuilkowski's 4,385-student Kenya study showing timed vs untimed EGRA did not change performance (Language Testing 2016). Cheap to compute, unknown meaning — log it, do not report it to teachers in v1.
Computation. Transcribe the child's answers; LLM-judge each against the rubric (correct / partial / wrong / no response); tag language of each answer (Urdu / English / Punjabi / mixed) and whether the child answered in a language other than the passage's.
Evidence. Open-ended child answers are scorable by LLMs at human-comparable flagging accuracy in far lower-resource settings than ours: Gemini-Flash scored MAIN narrative comprehension for 4–5-year-old Afrikaans/isiXhosa children with 89% accuracy (96% recall) at flagging who needs support, on ASR with 28–34% character error rate, with only five minutes of child speech used to train the ASR (Louw et al., SLaTE 2025). Pakistani classroom speech is heavily code-mixed (Urdu–Punjabi–Pothwari). Sindh EGRA 2024 found 46% zero scores on comprehension questions vs 28% on CWPM (Sindh EGRA 2024) — the language-of-answer tag is what would tell you how much of that is comprehension and how much is production language. Unverified: any validated code-switch measure for reading assessment.
Computation. Transcribe retell; LLM scores idea units against the passage's proposition list and a story-grammar checklist (setting, characters, problem, resolution); also words-in-retell (DIBELS-style).
Evidence. Meta-analysis of 23 studies (N = 4,705): retell correlates r = .46 with other comprehension measures, r = .58 when three prompts are given; scoring reliabilities .57–1.00 (Is Retell a Valid Measure?, 2021). Fluency-based (word-count) retell correlated poorly with comprehension (Bellinger & DiPerna 2011); Acadience keeps Retell as a check on children for whom ORF over-predicts comprehension (Acadience ORF & Retell). LLM scoring of narrative recall correlates r ≈ .65 with humans (LLM narrative recall scoring, 2025). Verdict: worth 30 s only if scored for content, never for word count.
Computation. Pool the child's spontaneous utterances (answers, retell, maths explanations); compute MLU in words, VocD/MTLD, and an LLM grammaticality judgement.
Evidence. Clinical language-sample analysis (SALT/CLAN) gets reliable MLU from 25–50 utterances (Using computer programs for LSA, LSHSS 2019); VocD is the strongest diversity measure (Lexical diversity in children's speech, 2022); ASR errors inflate/deflate complexity indices (Impact of ASR on complexity measures, 2021). Our battery yields perhaps 10–20 utterances per child — below the reliability floor; research only.
Computation. Item onset = end of the coach's spoken item (diarized); response onset = first child voicing; latency = the difference. Add: audible sub-vocal counting (a run of short, rhythmic, low-intensity syllables — detectable as a syllable-rate pattern even when unintelligible), and the slope of latency against operand size within the child (the problem-size effect).
Evidence. Siegler's strategy-choice work established that children mix retrieval, counting-on/min, counting-all and decomposition, and that averaging RTs over strategies is misleading (SCADS, Shrager & Siegler 1998; Assessing children's strategy choices, ERIC). Chronometry distinguishes them: a linear increase of RT with operand magnitude signals a sequential counting procedure — present even in adults' "retrieved" small sums (Thevenot et al. 2016; rejoinder 2020; problem-size effect, 2013). Children with dyscalculia show longer latencies, more finger counting and more procedural errors, and fail to shift from counting to retrieval (Geary line of work summarised in an ERP study, 2021; counting strategies in a computer environment, 2022). EGMA already asks the assessor to record strategy — in RTI's samples 52.3% of Level-1 addition items were solved with fingers, 1.6% with tick marks (EGMA Toolkit 2014, Table 4) — so the construct is field-standard; only the measurement is new. No automated field product does this from audio that I could find (unverified); it is a v2 build with a human-coded strategy-observation set for validation. Caveat: finger counting is silent; a long, silent latency is ambiguous between finger counting and not knowing — the coach's EGMA strategy tick remains the ground truth, and the audio latency becomes its cross-check.
Computation. Compare spoken answer to key: off-by-one/off-by-two (counting slip), answer = a−b when a+b was asked (operation confusion), answer with correct digits in wrong place or a digit dropped (place-value), answer equals an operand (no computation), no response. Classify with rules first, LLM only for verbal explanations.
Evidence. EGMA's addition/subtraction subtests have α = .81/.79 (Level 1) but Word Problems only .44 and Missing Number .58, and Rasch fit was poor for several items (EGMA Toolkit 2014, Table 1) — so per-item error types will be noisy with 5–10 items; the value is diagnostic, not psychometric. Rule-based typing is deterministic once the answer is recognised; the risk is ASR on Urdu number words (see B3).
Computation. Number-identification items: tag each numeral by structure (single digit, teen, decade, two-digit non-teen, three-digit with/without internal zero); recover the spoken form; classify inversion (24↔42), fragmentation ("thirty-six five" for 365), hundred-in-wrong-position, lexical (wrong word for a memorised item).
Evidence. Transcoding error taxonomies are well studied: inversion errors are frequent in inverted-number-word languages and cluster on teens in non-inverted ones; young children fragment three-digit numbers (Sixty-four or four-and-sixty?, 2014; Arabic units/decades order, 2021). Urdu/Hindustani number words are effectively unique lexical items for 1–99 (each must be memorised; teens are irregular) (Urdu numbers guide — a language-learning source, not a research citation); this predicts a lexical retrieval failure pattern rather than an English-style syntactic one, and makes the "which decade band fails" tally a genuinely Urdu-specific output. Automated number-word recognition is a small closed vocabulary — good ASR territory even in Urdu.
Computation. Within-child variance of latency after removing operand-size; proportion of items with audible counting; switch-rate between apparent strategies.
Evidence. Adaptive strategy choice (retrieval for easy, counting for hard) is the healthy pattern in Siegler's model (SCADS); "consistency" is therefore not a virtue in itself — the interpretable score is whether the child retrieves the small facts. Research, not v1.
Computation. Vision-LLM prompt over the photo: detect tally marks, decompositions, column algorithm with carries, crossed-out attempts; transcribe final answers; flag digit reversals.
Evidence. Smartphone-photo homework grading exists (BAGS, arXiv 1906.03767; EDM 2023 image-response auto-scoring, EDM 2023), but "even cutting-edge multimodal LLMs perform poorly on STEM handwritten solution recognition" (EDU-CIRCUIT-HW, 2026); children's handwriting needs its own models (children's Arabic handwriting ensemble, 2025). Digit reversal is developmentally normal to age ~7 (digit 3 reversed in 45.6% of cases, 4 in 16.2%; rapid decline after 6) (Fischer 2011; Fischer 2013) — report it as an observation, never a deficit. Strategy-trace detection from photos: unverified anywhere; v2 for "final answer + presence of tally marks", research for algorithm-error typing.
The recording captures the coach as well as the child. Five checks, all from diarized coach speech + timing:
| Check | Computation | Evidence that it matters |
|---|---|---|
| Protocol adherence (instructions given, practice item done, story read once/twice as scripted, comprehension questions asked verbatim) | Align coach utterances to the script with an LLM; count missing / extra / paraphrased steps | Trained, paid examiners on a Texas fluency study made uncorrectable administration errors (coaching students, wrong prompts, asking the child to repeat) in 8% of records and correctable scoring errors in 91% of the rest (Reed & Sturges 2013; assessment-fidelity review). |
| Prompting / correcting the child | Coach speech overlapping or inside the child's timed window; coach says the target word; coach repeats an item | EGRA allows one prompt after a 3-s hesitation and nothing else (EGRA Toolkit V2); EGMA scripts a single "do you know another way?" nudge and a 5-s allowance for inefficient strategies (EGMA Toolkit 2014). Anything beyond that is construct-irrelevant variance. |
| Timing honesty | Elapsed audio between "Begin" and "Stop" vs the nominal 60 s; whether the child kept reading after 60 s and those words were counted | The toolkit relies on a stopwatch and "time remaining" fields; Tangerine tablets replace human timing (Tangerine). With audio, the timer is the recording itself — the coach's stopwatch becomes redundant. |
| Room noise | SNR estimate on the child's segments; count of overlapping third-party voices | Phoneme error rate rises from 14.6% (SNR > 25 dB) to 40.6% (SNR < 10 dB) (2025); classroom-noise-robust ASR is an active field (CPT-boosted wav2vec2 for classrooms). A noise flag tells you which scores to trust and which schools need a quieter corner. |
| Child affect / anxiety cues | Long silences, falling speech rate, low pitch, minimal pitch variation, whisper-level intensity | A 3-minute speech task identified 3–7-year-olds with internalising disorders at 80% accuracy (54% sensitivity, 93% specificity) (McGinnis et al. 2019); math-anxious children respond more slowly and err more (The Role of Response Time in Math Anxiety, 2023). But: no evidence that these cues predict score validity in an assessment; the constructs confound with low skill. Log as a research signal only; never surface per child in v1. |
Two further points. First, RTI's own quality gate is inter-rater reliability against a gold-standard assessment, with ≥90% agreement typically required before enumerators go to the field (EGRA Toolkit V2, Annex K; Sindh 2024 reported IRR of 88–97% across subtasks (Sindh EGRA 2024); Liberia required ≥90% (World Bank Liberia EGRA evaluation)). With audio, IRR becomes continuous: every recording can be re-scored by machine, so coach drift is measurable per coach per week instead of once at training. Second, DIBELS 8 estimates inter-rater ICCs precisely because "test administrators must decide whether student pronunciation … is correct" (DIBELS 8 Technical Manual, Inter-rater Reliability); the coach's own live tally, compared with the machine tally, yields a per-coach leniency/severity estimate — the "coach quality" signal the operator asked for.
| Measure | Concurrent alternate-form | Delayed alt-form / test-retest | SEM | Source |
|---|---|---|---|---|
| DIBELS 8 ORF (words correct) | median .93 (Gr 1 .96) | delayed .88; test-retest .91 | 5–8 wcpm Gr 1; 9–11 Gr 2; ~10–12 Gr 3+ (Table 2.34: Gr 2 8.5–10.6; Gr 3 10.8–12.3) | DIBELS 8 Tech Manual |
| DIBELS 8 ORF accuracy (%) | median .83 | delayed .73; test-retest .75 | 5–10 pts Gr 1, 4–7 Gr 2, 1–2 by Gr 8 | same |
| DIBELS 8 Maze | .72 | .77 / .82 | 3–4 | same |
| CBM-ORF generally | — | AIMSweb test-retest .94 | 5–20, "realistic" 8–10 wcpm; 10–15 per occasion in progress monitoring | Christ & Silberglitt 2007; Barth et al. 2012 |
| EGMA subtests (Cronbach α) | Number ID .94, Add L1 .81, Sub L1 .79, Missing Number .58, Word Problems .44 | — | — | EGMA Toolkit 2014 |
Two implications. (i) A single 60-s passage carries a ±10 wcpm standard error for a Grade 2–3 child; a 95% CI of ±20 wcpm is wider than a term's expected growth, which is why Christ et al. conclude you need ~14 weeks of weekly probes for a reliable slope (Christ et al. 2013). (ii) Accuracy is less reliable than rate and the five-item EGMA subtests are unreliable — item-level reporting from them is diagnostic colour, not measurement.
Passages written to the same readability formula produced mean wcpm from 67.9 to 93.9 in the same Grade-2 children (Francis et al. 2008; DIBELS Next form effects). Comparability to Hasbrouck–Tindal norms (Technical Report 1702) or to Pakistan's adopted Urdu benchmark (up to 90 cwpm; ACR-Asia benchmarking) therefore requires equated passages, not "grade-level" passages. Sindh's 2024 baseline (Urdu-medium Grade 3: 19.25 cwpm mean, 34.81 excluding zero-scorers; 28% zero) is the local anchor (Sindh EGRA 2024); PRP reported 30→57 cwpm in Sindh/Urdu 2014→2017 (Creative Associates). If we want to compare to any of these, one of our passages must be (or be equated to) theirs.
CORE (IES-funded, Nese & Kamata) replaces the raw count with a latent model fitted to word-level data: a binomial accuracy component (per-word correct/incorrect with a person accuracy θ and passage difficulty) and a lognormal speed component (per-passage reading time with a person speed τ), estimated jointly by Bayesian MCMC, from which a model-based WCPM with a conditional standard error is derived. On 58 fourth-graders × 4 short passages the mean CSEM was 8.00 (range 3.42–12.93) vs DIBELS Grade-4 SEM 9.63–12.86; model-based and observed WCPM correlated .99; and simulation showed the number of passages matters more than sample size — N = 50 children suffices to calibrate if each reads enough passages (Nese & Kamata 2020; Nese 2022 growth comparison; equating extension Kara et al. 2024 — abstract only, details unverified; project list at CORE publications). This is precisely the machinery a WhatsApp-audio pipeline feeds naturally: every recording yields the per-word table the model needs.
True CAT needs a large calibrated item bank; ORF passages are testlets, and testlet models outperform unidimensional ones when several items share a passage (Item selection for CAT with passages, 2019). The realistic design for us is two-stage branching (a short screener decides which of 2–3 passage difficulties the coach opens) rather than item-level CAT. For single-word/lexical-decision tasks, CAT is proven: ROAR-CAT reached reliability .90 with 75 items vs 125 in random order, correlated .89 (Gr 1) with FAST CBMreading, and its item parameters were invariant across SES/EL/LD subgroups (r = .82–.93) (ROAR-CAT, 2025). MAP Reading Fluency runs a 20-minute adaptive, speech-scored ORF (NCII entry). Where response data are thin, item parameters can be predicted from item features (AutoIRT: AutoML + BERT features → explanatory IRT, then Fisher-information bandit selection) (Sharpnack et al. 2024) — relevant for calibrating new Urdu passages from text features before we have thousands of reads.
Report (a) the level, (b) the model-based score with its CSEM as a band, (c) item-level diagnostics explicitly labelled as low-reliability (EGMA α .44–.58 subtests; single-passage GPC tallies). Amira's shipped pattern — a class Skills Status grid, per-student skill mastery, generated grouping and lesson recommendations (Amira 5 ways; Amira Instruct) — and Read Along's per-word feedback (Google Read Along India) are the "assessment for learning" reference points; neither publishes the reliability of its skill-level cells, so treat them as UX precedent, not psychometric precedent.
Ranking criterion: value to a teacher/coach × strength of evidence ÷ cost. U = Urdu read, E = English read, M = spoken maths, P = photo.
| # | Construct | Block | How computed | Reliability / prediction evidence | Cost & complexity | Phase |
|---|---|---|---|---|---|---|
| 1 | Words correct, rate, WCPM, accuracy (machine) | U E | Prompted child-ASR + forced alignment | Machine–human r = .96 (NAEP); DIBELS ORF alt-form .93, SEM ±5–12 wcpm | Low (E), medium (U: no child corpus; must measure Urdu miscue precision) | v1 |
| 2 | Coach protocol adherence + prompting + timing honesty | all | Diarize coach; LLM-align to script; count prompts inside timed window | Examiner error 8% uncorrectable / 91% correctable (Reed & Sturges); EGRA IRR gate ≥90% | Low–medium | v1 |
| 3 | Room noise / recording-quality flag | all | SNR on child segments | PER 14.6%→40.6% from low to high noise | Low | v1 |
| 4 | Pause-based prosody (pauses at punctuation vs intrusions, phrase length, start latency) | U E | Alignment timing only | Relative silence times explain 58% of WCPM variance; pause intrusions predict comprehension; Miller & Schwanenflugel longitudinal | Low | v1 (report as descriptive, not a level) |
| 5 | Miscue counts: omission / insertion / substitution / repetition / self-correction (+ SC latency) | U E | Alignment rules | F1 .39→.73 with prompted Whisper (Dutch); SC predicts progress (D'Agostino) | Medium; Urdu precision unknown | v1 counts (E), v2 (U) |
| 6 | Per-word latency distribution; effortful-decoding rate (restarts, intra-word pauses) | U E | Alignment + fragment detection | Construct is standard (3-s hesitation rule); no published score reliability | Medium | v2 |
| 7 | Model-based WCPM with CSEM; passage calibration & equating | U E | Binomial-lognormal latent model on word-level table | CSEM 8.0 vs DIBELS 9.6–12.9; needs ≥3–4 short passages/child; N≈50 calibrates | Medium (statistics), low (data) | v2 (design the passage set for it in v1) |
| 8 | Comprehension answers: LLM scoring + language-of-answer + code-switch tag | U E | Transcribe answers; LLM rubric judge | LLM flagging 89% accuracy on 4–5-yr-old low-resource narratives; retell r = .46–.58 | Low–medium | v1 scoring, v2 language tags |
| 9 | Maths response latency; problem-size slope; audible counting | M | Coach-utterance offset → child onset; syllable-rhythm detector | Strong cognitive theory (Siegler, Thevenot, Geary); EGMA strategy tick is field-standard; no automated precedent | Medium | v2 (validate against coach's EGMA strategy tick) |
| 10 | Maths error typing (near-miss / operation / place-value / no-computation) | M | Rules on recognised answer | EGMA α .79–.81 (add/sub L1), .44–.58 (word problems, missing number) | Low | v1 as diagnostic colour, never as a score |
| 11 | Number-naming error by structure (teens, decades, 3-digit, Urdu lexical) | M | Closed-vocabulary ASR; structural tags | Transcoding taxonomy well established; Urdu-specific pattern is a hypothesis | Low | v2 |
| 12 | NAEP-scale expression (acoustic: F0 contour, intensity, stress) | U E | openSMILE/wav2vec2 → 4-level classifier | Automated 62–76% exact vs human–human 58%; wav2vec2 r = .83 on Indian ESL readers | Medium–high; needs Urdu-rated training set | v2 (E), research (U) |
| 13 | Phonics gap map (GPC error profile) per child and per class | U E | Phone-level decoding of substitutions vs target graphemes | Amira ships it (no independent validation); phone ASR PER ~28%, worse in noise; Opus blurs fricatives/aspiration | High | v2 (E, class-level only), research (U) |
| 14 | Photo: final answers + presence of tally marks / decomposition; digit reversal (non-judgemental) | P | Vision-LLM | Handwriting grading exists; MLLMs weak on STEM handwriting; reversals normal to age 7 | Medium | v2 answers, research strategy traces |
| 15 | Retell (30 s) idea units / story grammar | U E | Transcribe + LLM against proposition list | Retell r = .46 with comprehension (.58 with 3 prompts); LLM–human r ≈ .65 | Low–medium; costs 30 s per child | v2 only if a slot exists |
| 16 | Sight-word vs decodable split from passage words | U E | A-priori word tags | Few irregulars per 60-word passage; reliability unverified | Low | research (use a word list subtask instead) |
| 17 | Within-minute rate curve (warm-up/fatigue) | U E | Cumulative words vs time | No literature found | Trivial | research (log only) |
| 18 | Child affect / anxiety cues | all | Speech rate, pitch, long silences | 80% accuracy for internalising disorders in 3–7-yr-olds; zero evidence for score validity; confounds with skill | Low compute, high ethical cost | research (never per-child in product) |
| 19 | Expressive language (MLU, lexical diversity) | all | LSA on pooled utterances | Needs 25–50 utterances; we have ~10–20 | Low | research |
| 20 | Dialect-aware accepted-pronunciation lexicon; subgroup fairness report | U E | Cluster consistent variants per target word; per-subgroup WER and machine–human gaps | NAEP/Balogh precedent; PNAS 35% vs 19% WER warning; Bailey recommendations | Medium, ongoing | v1 fairness report; v2 lexicon |
NAEP 2018 ORF study — https://nces.ed.gov/nationsreportcard/subject/studies/orf/2021025_orf_study.pdf · Balogh et al. 2012 — https://doi.org/10.1177/0013164411412590 · DIBELS 8 Technical Manual — https://dibels.uoregon.edu/sites/default/files/DIBELS8-TechnicalManual_04152020.pdf · EGRA Toolkit V2 — https://ierc-publicfiles.s3.amazonaws.com/public/resources/EGRA%20Toolkit%20V2%202016.pdf · EGMA Toolkit 2014 — https://ierc-publicfiles.s3.amazonaws.com/public/resources/EGMA%20Toolkit_March2014.pdf · Sindh EGRA Baseline 2024 — https://rsusindh.gov.pk/contents/SELECT/EGRA_Baseline_Assessment_Report_Sindh.pdf · Nese & Kamata 2020 — https://pmc.ncbi.nlm.nih.gov/articles/PMC7425326/ · Kara et al. PTAM 2023 — https://www.psychologie-aktuell.com/fileadmin/Redaktion/Journale/ptam_2023-1/PTAM__1-2023_2_kor.pdf · Gao et al. 2024 — https://arxiv.org/pdf/2406.07060 · Gao et al. 2025 — https://www.isca-archive.org/interspeech_2025/gao25c_interspeech.pdf · Frontiers 2024 prosody — https://www.frontiersin.org/journals/education/articles/10.3389/feduc.2024.1440760/full · Vaidya, Sahoo & Rao 2024 — https://arxiv.org/pdf/2405.19426 · Bolaños et al. 2013 — https://www.sciencedirect.com/science/article/abs/pii/S0167639312000933 · Miller & Schwanenflugel 2008 — https://ila.onlinelibrary.wiley.com/doi/10.1598/RRQ.43.4.2 · Retell meta-analysis — https://pmc.ncbi.nlm.nih.gov/articles/PMC7806198/ · D'Agostino et al. 2019 — https://www.tandfonline.com/doi/abs/10.1080/02702711.2019.1629518 · Christ & Silberglitt 2007 — https://www.researchgate.net/publication/232440479 · Christ et al. 2013 — https://pubmed.ncbi.nlm.nih.gov/23375171/ · Francis et al. 2008 — https://link.springer.com/chapter/10.1007/978-1-4939-2803-3_9 · Reed & Sturges 2013 — https://sciencedirect.com/science/article/pii/S0022440514000491 · Thevenot et al. 2016 — https://pubmed.ncbi.nlm.nih.gov/26491834/ · Shrager & Siegler 1998 — https://journals.sagepub.com/doi/10.1111/1467-9280.00076 · Transcoding (PMC3990049) — https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3990049/ · Digit reversal — https://pubmed.ncbi.nlm.nih.gov/21377665/ · McGinnis et al. 2019 — https://pubmed.ncbi.nlm.nih.gov/31034426/ · Koenecke et al. 2020 — https://www.pnas.org/doi/10.1073/pnas.1915768117 · ROAR-CAT — https://pmc.ncbi.nlm.nih.gov/articles/PMC11732908/ · AutoIRT — https://arxiv.org/abs/2410.21033 · Louw et al. 2025 — https://arxiv.org/html/2507.13205 · Urdu Whisper — https://arxiv.org/pdf/2508.09865 · Arabic child ASR — https://arxiv.org/pdf/2510.23319 · Phoneme recognition in noise — https://arxiv.org/pdf/2503.04710 · Microsoft Reading Progress guide — https://learn.microsoft.com/en-us/training/educator-center/product-guides/reading-progress/ · Amira framework — https://explore.amiralearning.com/hubfs/Customer_Resources/AmiraTheoreticalFramework.pdf · WhatsApp Opus — https://chattopdf.app/blog/whatsapp-audio-format