EGRA Test Build Review

On 28 September Sabeena pushed two more commits, now deployed on staging. They fix three of the five "fix first" items and the two worst coach-experience problems:

• Public access to the results tables is revoked. Row-level security is still off, so the grants are the only lock.
• Pictures now reach the coach. Links are signed before sending, and a failed send falls back to text. On 29 September five picture sends worked, with no errors.
• The voice-note columns now exist. This came one day late. On 28 September a coach's listening marks were lost to exactly the failure the review predicted.
• The listening-comprehension Flow is gone. The coach now reads the passage from a picture and marks the answers, and the voice note is optional and never blocks.
• Both stories now offer an optional voice note, which is the plan's direction.
• A drawn child must be confirmed. Backing out needs a reason (Absent or Other), and the reason is saved with the child's ID.

Still open: the letter formula, region-wide class access, the unsaved child ID on results, punctuation counted as words, numbered child cards, /cancel at typed prompts, the English-only coach text and missing answer key, and the untraced redraw through /cancel. The product-level gap from the plan is unchanged. One test session on 29 September took about 19 minutes for the Urdu block alone, which supports the 22-minute estimate. Full detail: the 30 September recheck.

Which bot. It is the main Rumi bot, not the NIETE bot. All ten commits are on the staging branch only, behind a switch that is on for staging and limited to Islamabad test accounts. Nothing has reached production.

What it is. A careful, well-documented first build that puts almost the whole May 2026 enumerator test into WhatsApp chat. The coach taps Correct or Incorrect for every item, one message at a time. No AI is used anywhere, and a database rule stops an AI score from ever being stored.

How it compares with the plan. It builds a different product. The plan is a five-minute test: the coach records one voice note per block while the child reads from a printed card, Rumi pre-fills a checking form, and the coach corrects it. This build takes about 22 minutes per child, which is today's enumerator time, and the one-message-per-item pattern is the one the field team rejected on 11 September.

Does it work today. Better than on 27 September: pictures arrive and saves with audio work. One session has completed the Urdu block. No session has finished all 14 subtests. Of the 6 saved results, none is linked to the child's record.

What to keep. The item bank, the code-drawn stimulus pictures, the random draw from the real roster, the switch-and-region gate, and the chat-loop fix are all good foundations for the plan's version.

The estimate walks the code message by message for one child: 113 bot messages and 96 coach actions (83 taps, 11 typed answers, 2 voice notes). The time with the child comes from the May enumerator medians for the same 14 subtests (13.8 minutes). WhatsApp round trips, typing wrong-item lists and re-asking questions for the voice note add the rest.

The code is organised sensibly: small single-purpose services, a gate, a state machine in one handler, and switches that keep it inert. The problems are in the edges: what happens when a send or save fails, what the tests fake, and a few formulas.

• Failed saves are silent. A failed picture send now falls back to text. A failed save is still logged and ignored: the coach is told everything worked, and the marks cannot be recovered. This is how the 28 September listening marks were lost.
• The tests cannot see the real failures. WhatsApp sends and database saves are faked to always succeed, and picture links are only checked for their shape. The voice-routing test searches the source text, and the Flow-response test covers a function nothing in production calls. All 132 tests pass while the build cannot complete a child.
• Scoring does not follow the instrument. Beyond the letter formula: the English story counts full stops and commas as words (+13% for a child who finishes); there is no 60-second cap; the Discontinue button marks the first ten items wrong on every grid, including the stories, which have no such rule; comprehension has no "not asked" option and is asked even when the child read nothing.
• Answers can land on the wrong question. Correct/Incorrect buttons do not say which item they belong to, so a double tap or a tap on an older message is recorded against the current item.
• Coaches can get stuck. /cancel becomes the child's name at the name prompt, "not a number" at the typed prompts, and the skip reason at the reason prompt. Other commands are swallowed for up to an hour. (The voice note is now optional, which removed the worst trap.)
• Children's audio and names can leak into general chat. A voice note sent outside the exact voice-note step goes through the normal pipeline: transcription, the conversations table, and the chat model.
• A shared prompt was changed for every region. The chat-loop fix edits the general-chat prompt all regions use. It is a good fix for Pakistan, but it points Palestine teachers to a /reading test that is disabled there, and Tanzania teachers to the English test.
• Data hygiene. One real tester phone number is committed in the test file. The roster query pulls every Islamabad child's name just to count them.

• Everything the coach reads is in English, hard-coded, including the Flow. Islamabad coaches work in Urdu, and our checking-form draft and the observation form are Urdu.
• Two instructions would invalidate the reading subtests. "You will read out each item to the child" and "or read it out per the item type" would have the coach read letters and words aloud for the child.
• No answer key is shown. Every item has one in the item bank, but no code sends it, so each coach decides what counts as right.
• Fixed 28 Sep The listening steps were in the wrong order: the child answered inside the Flow before any recording. The Flow is gone; the coach reads the passage from a picture and the voice note is optional.
• No sense of progress. The coach sees "Item 3/8" but never "subtest 6 of 14" or how much of the session is left, and gets no "child 2 of 5".
• The end result is a text list of raw per-subtest scores, only if all 14 subtests are finished. There is no level, no benchmark, nothing for the teacher and no class view.

The pictures are well made as images: clean, correct Unicode, correct Nastaliq joining and right-to-left order. The problem is what they are designed as. They are the enumerator's score sheet, shown to the child on the coach's phone.

• Every item carries a printed number. A real EGRA child card has none; numbers belong on the assessor's sheet.
• The stories are cut into numbered word boxes, so the child is not reading connected text, and the result is not an oral reading fluency measure. Punctuation gets its own box.
• The print is small on a phone. English story words are about 2 mm tall with the phone turned sideways, and 1.1 mm upright. On a printed A4 card each of the ten columns gets about 27 mm; on the phone it gets about 6 mm.
• One phone cannot be the child's card and the coach's score sheet. The coach has to hold the phone up, keep time with no clock, remember which of up to 100 items were wrong, then take the phone back to tap Stop and type the numbers. The plan's printed card avoids all of this.

What the call decided. The impact team's algorithm names the children from the class register; the coach never chooses. Each pick comes with named alternates for absent children (the Usser study took odd roll numbers and replaced an absent child with the next even number). Five children per visit.

What was built.

Four reviewers worked in parallel on a frozen copy of the 42 changed files at staging head 554c6a8d on 27 September; the two later commits were rechecked directly on 30 September. Every headline claim on this page was then checked again against the code, the live database (read-only, counts only), the staging logs or Meta.