Investigation index

What was asked, what is in the folder, headline findings, open decisions

EGRA Student Assessment — coach-run, AI-prescored, HITL (Sep 2026)

Bead: bd-gqoz1 · Started: 2026-09-15 · Status: plan v2 delivered (harness over 187 recordings + batteries + granularity + maths + first-principles battery), awaiting operator decisions D1–D6 · Plan page: PLAN.html → public site https://hyasin270.github.io/egra-five-minutes/ (repo hyasin270/egra-five-minutes, rebuilt by build_site.py from this folder) and the Claude artifact https://claude.ai/artifact/PBN7cNc2hdJWtEecaawG6C

The ask (operator, 15 Sep 2026)

Replace the enumerator-run EGRA/EGMA battery (English, Urdu, maths) with a coach-administered version: the coach records the child, models pre-score, the coach corrects on a pre-filled form (same HITL pattern as NIETE /observe), both AI and coach values plus the audio are stored. Target 5 min per child, 5 children in 30 min. Pops up after an observation visit. Deliverable: a plan with flows, model per stage, and a roadmap to production.

What is in this folder

Path	What
PLAN.html	The plan artifact: target vs baseline, evidence, the 5-minute battery, coach journey, scoring pipeline, model per stage, correction form, data model, evaluation, W1–W6 roadmap, decisions D1–D5
BASELINE.md	Corpus description + enumerator timing baseline (median 21 min/child; Urdu 5.7, English 6.9, Maths 6.2 min) + the coach-review ground truth
audio/	207 unique recordings (8 kHz AAC), EGRA_EGMA client N=2233.xlsx (SurveyCTO export), Data.xlsx (1,462 coach reviews), join indexes _audio_index.csv, _child_index.csv, _coach_reviews.csv
probes/	Probe 1 (Gemini 2.5 Pro / Flash on a whole session) and Probe 2 (Soniox on the Urdu block) with findings
research/01..07	EGRA/EGMA structure; model landscape; current main-bot reading + ASER tool; NIETE observe HITL; what a recording yields; batteries walkthrough; model deep dive
harness/	The eval harness (scripts 01–09b, HARNESS_RESULTS.md, cached intermediates)
tips/	TIP diagnostic PDFs, page renders, TIP_NOTES.md
meetings/	Extended Mind exports of the adjacent meetings plus the 15 Sep alignment call (recovered: 2026-09-15_alignment-call.md + full transcript)
*.py	Reproducible download / baseline / duration scripts

16 Sep: the evaluation as one record

• harness/HARNESS_RESULTS.md is now the single canonical evaluation (generated by harness/15_evaluation_doc.py from every cached result; no round ledger): method, human floor, one table per task with every scorer tried (9 audio models, 4 transcript models, 5 comprehension graders, 4 segmenter labellers, 3 maths graders, SpeechAce, the alignment baseline), and §8 the recommended stack. Headline: Gemini 3.1 Pro / 3.8 Flash listening to the 60-s clip with the printed text land within 5 words of the enumerator for 74–97% of children; per-word flags F1 50–66 (Urdu); comprehension graded within 1 for 79% by a cheap text model.
• harness/examples/ — 12 worked-example pages (14_worked_examples.py), reviewed by a subagent (examples/REVIEW.md); child clips in a domain-shared Drive folder.
• Per-word metric corrected: the export writes 0 for both "correct" and "never reached"; comparisons now restricted to words both sides reached (06, 10).
• Site rebuilt and pushed; home page's evaluation and stack sections rewritten to the combined findings. PROJ-067 comment posted with charts. Round-2 OpenRouter spend $20.88.

v2 additions (15 Sep, evening)

• harness/ — full eval harness over 187 recordings with the human floor (HARNESS_RESULTS.md): counts r .93–.98 after window failures; per-word precision .12–.25; letters coach-decided; maths gradable only when the question is spoken (item bank fixes it in product); Azure keys dead (P1 bead); Soniox writes accented child English in Urdu/Devanagari unless the language is forced; audio-LLM clocks drift.
• meetings/2026-09-15_alignment-call.md — the call WAS in the DB (restricted row, hidden from the RO role; recording job stuck at processing); transcript recovered via the Attendee API + Soniox.
• tips/TIP_NOTES.md — the folder is the LEAPS/CERP TIP written diagnostic.
• research/05 (what a recording yields), 06 (batteries walkthrough), 07 (model deep dive).

Headline findings

1. Baseline 21 min/child → target 5 min is a 3.6–4.2× compression; it has to come from removing item-by-item tapping, stop rules, and a deliberate subset (60-s reads kept for benchmark comparability).
2. The field team rejected the WhatsApp ASER tool on 11 Sep (one prompt per item, 10–15 min/child, no offline, accent worries). Manual/override modes shipped since; the per-item shape and online-only did not.
3. Probes: a Pro-class audio LLM segments and labels a whole session and audits protocol (caught a slip the reviewer missed) but its timestamps drift by ~1 min; Soniox timestamps the 60-s window exactly and diarises child vs enumerator on 8 kHz audio. Rule: STT cuts, LLM labels.
4. No commercial Urdu phoneme API exists (Azure PA: 33 locales, no ur). Urdu = STT + forced aligner + per-phone GOP, calibrated on the 187 coach-reviewed clips. English = Azure / SpeechAce.
5. Zero new tables for v1: aser_sessions + reading_assessments + numeracy_assessments already hold the AI-vs-coach pair, who-scored, audio, transcript, word timestamps and per-word errors.
6. NIETE observe is two recordings (classroom + debrief), not three; best hook is the classroom-recording-received point; observation completion must stay independent.

Open decisions (operator)

D1 after-observe vs standalone-first · D2 which five minutes · D3 Urdu instrument choices (names vs sounds, aerab, passages) · D4 offline in v1 · D5 unshelve SpeechAce for English. Full context in PLAN.html §Decisions.

Next action

Operator answers D1–D5 → W1: eval harness over the 206 recordings + instrument decisions with Sabeena/Sameer.

2026-09-16 · Checking screen redone as a real WhatsApp Flow

Operator: the phone mockup was not how Flows work. Rebuilt from the NIETE observe FICO Flow pattern and Meta component limits (no audio inside a Flow; ChipsSelector max 20; one Footer per screen). Draft Flow JSON: harness/flow/egra-review-flow.json; renders: harness/mockups/build_flow_mockups.py → chat_cta.png, flow_urdu.png, flow_maths.png. PLAN.html §form replaced; site pushed (136cc58); PROJ-068 section swapped.

2026-09-27 · Review of the staging EGRA/EGMA test build

Sabeena's 8 commits on main-bot staging reviewed against the plan (bd-ak1go). Published: https://hyasin270.github.io/egra-five-minutes/staging-review/ . Sources: review-staging-build/ (src snapshot, findings, images, builder). Security follow-up: bd-e7pg4 (RLS off on egra_egma_ict_test_attempts, needs a go).