Probe 1: an audio LLM on a whole session

Gemini 2.5 Pro and Flash segment and score one 17-minute enumerator recording

Probe 1 — can an audio LLM parse a whole enumerator session? (2026-09-15)

Input: AA_eb34954f…_enumerator.m4a — Madiha_Khan administering the full battery to a Grade-3, age-10 child at school 37330351 on 7 May 2026. 17 min 01 s, 8 kHz AAC (upsampled to 16 kHz MP3 for the call). Coach review of this recording: compliance 100%, reviewer kainat. Model: google/gemini-2.5-pro via OpenRouter, one call, whole file as input_audio. 101 s wall-clock, $0.144 (25.5k audio tokens in, 11.1k out incl. 4.8k reasoning). Script: probe1_segment.py; raw output: probe1_google_gemini-2.5-pro.md.

What it got right (checked against the child's SurveyCTO row 37330351-3-1-3)

Thing Enumerator (tablet) Gemini from audio alone
Subtask boundaries 12:00:35 listcomp → 12:01:47 letters → … → 12:09:05 math → 12:15:12 end 11 segments with instruction cues quoted in Urdu; letter-ID segment lands within ~15 s of the tablet timestamp once the recording offset is accounted for
Urdu letter identification 74 attempted, 70 correct, 4 incorrect 73 items transcribed, 68 correct, 5 incorrect, each with a note (e.g. "child said the letter name dal instead of the sound", "ر for ڑ")
Urdu familiar words child completed the list early (no 60-s cut) 46 words transcribed in Urdu script, 4 marked incorrect
Urdu ORF 59/59 correct recognised the passage ("Bilal … river Jhelum"), noted no explicit stop
English listening comp 0/6 "child cannot answer (نہیں آتا ہے)"
Math 7/8 num-id, 6/8 compare, 4/8 computation, 5/8 word problems, 6/8 patterns 34 items with question + answer; marked 2 word problems wrong (enumerator marked 3)
Protocol coach reviewer: 100% compliant 7 deviations with timestamps — including "English story read once instead of twice", which the human reviewer did not flag

What it tells us

  1. Segmentation of a continuous coach recording is solved-enough by a frontier audio LLM. We do not need the coach to press "next subtask"; one recording per child (or per block) can be cut by the model using the administration script's own cue phrases. This alone removes most of the tablet-tapping that costs enumerators 3–4 minutes per child.
  2. Letter-level and word-level scoring in Urdu is in the right range on 8 kHz noisy audio (68 vs 70 correct), but it is a single clip and a generalist model, not a phoneme scorer. It cannot tell ر from ڑ or ت from ٹ reliably at 8 kHz — exactly the confusions a per-item HITL tick-box must let the coach settle. A 16 kHz WhatsApp voice note should help; a dedicated forced-alignment/GOP layer for Urdu is still needed for item-level confidence.
  3. The model is a usable protocol auditor. It caught an unflagged deviation. That is the "coaches assessing enumerators" layer, automated — worth keeping as a by-product even in the coach-administered design (it will audit the coach).
  4. Cost/latency shape: ~$0.15 and ~100 s for 17 minutes of audio in one call. A 5-minute coach battery at the same rate is ~$0.05 and ~30 s — well inside a WhatsApp turn. Splitting per block (Urdu / English / Math) lets scoring run while the coach moves to the next block.
  5. Noise is the enemy, not the model. Fan at 12:44, teacher announcing lunch at 15:01, soft child voice. The product has to control the recording environment (headset / close mic guidance, a one-tap "re-record this item") more than it has to pick a cleverer model.

Not yet measured (next probes)