One-off: rewrite PLAN.html's evaluation and stack sections into the single combined record.
"""One-off: rewrite PLAN.html's evaluation and stack sections into the single combined record."""
import re
p = 'PLAN.html'; s = open(p).read()
new_exp = '''<section id="experiment">
<h2>The experiment: does this actually work on real recordings?</h2>
<div class="prose">
<p>We took every one of the 187 study recordings that has both a child's tablet scores and a reviewer's re-listen, and ran the candidate pipeline and every model we were considering over them. This is the "evaluation harness": a set of numbered scripts anyone can rerun, with every model reply cached. The full record, one table per task with every scorer we tried, is the <a href="harness/results.html">evaluation page</a>; twelve <a href="examples/index.html">worked examples</a> show, for real windows, the printed text, the clip, the transcript, the enumerator's marks and every model's per-word verdicts side by side. Total model spend for the whole evaluation was about $70.</p>
<h3>Where the human scores come from</h3>
<p>Two layers. The <strong>enumerator's live marks</strong> from the tablet: for every letter, word and story word, whether the child got it right, plus the counts and timer readings. And the <strong>reviewer's second opinion</strong>: someone listened to the recording afterwards and marked whether each of the enumerator's scores was right, wrong, or impossible to tell from the audio.</p>
<h3>Why the "human floor" matters</h3>
<p>Before judging any computer we asked how often the two humans agree with each other. On letters and maths, about nine times in ten. On the timed word lists and the story, only 51 to 62 percent of the time: the recordings are noisy phone-quality audio from real classrooms, and even a trained human cannot always tell what a child said. So on those tasks the fair target for a computer is "as good as the second human", not perfection. That floor is printed beside every number on the evaluation page.</p>
</div>
<h3>What we found, task by task</h3>
<div class="tablewrap"><table>
<thead><tr><th>Task</th><th>Second human agreed with the enumerator</th><th></th><th>Best scorer we found, and how close it gets</th><th>Verdict</th></tr></thead>
<tbody>
<tr><td>Urdu word list (60 s)</td><td class="num">51%</td><td><div class="bar"><i style="width:51%"></i></div></td><td><strong>Gemini 3.1 Pro listening to the clip with the printed text</strong>: within 5 words for 97% of children, 1.6 words off on average; per-word flags right 3 times in 4. Gemini 3.8 Flash: within 5 for 86%, 3.8 words off, at a sixth of the price. Transcript + alignment: 8 words off.</td><td><span class="pill ok">counts and chips ready</span></td></tr>
<tr><td>Urdu story (60 s)</td><td class="num">54%</td><td><div class="bar"><i style="width:54%"></i></div></td><td><strong>Gemini 3.1 Pro</strong>: within 5 words for 88%, 2.8 words off; per-word flags right 56% of the time (F1 66). Gemini 3.8 Flash: 74% within 5, 5.5 words off. Transcript + alignment: 16 words off, always under-counting.</td><td><span class="pill ok">counts ready</span>, chips as suggestions</td></tr>
<tr><td>English story (60 s)</td><td class="num">62%</td><td><div class="bar"><i style="width:62%"></i></div></td><td>Every scorer is weaker here: best is GPT audio or Gemini 3.5 Flash-Lite at 4 to 5 words off, Gemini 3.8 Flash 5.8; per-word flags right only about a third of the time. Accented child English on 8 kHz audio is the hardest input we have.</td><td><span class="pill ok">counts ready</span>, chips coach-decided</td></tr>
<tr><td>English pseudowords</td><td class="num">52%</td><td><div class="bar"><i style="width:52%"></i></div></td><td>Most children read 0 to 5; transcript + alignment is within 5 for 91%.</td><td><span class="pill ok">counts ready</span></td></tr>
<tr><td>Letters, Urdu / English</td><td class="num">89% / 90%</td><td><div class="bar"><i style="width:89%"></i></div></td><td>The window itself was lost 62 to 84 percent of the time: letter names come back in three scripts. No model tested cleanly.</td><td><span class="pill coach">coach marks</span></td></tr>
<tr><td>Urdu comprehension (6 questions)</td><td class="num">73%</td><td><div class="bar"><i style="width:73%"></i></div></td><td>A text grader on the transcript: within one answer of the enumerator for 79% of children (Gemini 3 Flash, a tenth of a cent per child); Claude Sonnet 5 and GPT-5 mini 73 to 75%.</td><td><span class="pill ok">ready</span></td></tr>
<tr><td>Maths word problems (question spoken)</td><td class="num">87%</td><td><div class="bar"><i style="width:87%"></i></div></td><td>Grader on the transcript: within one item for 63 to 74% (Claude Sonnet 5 best, Gemini 2.5 Pro and 3 Flash close).</td><td><span class="pill ok">ready</span> with the item bank</td></tr>
<tr><td>Maths number reading (stimulus on paper)</td><td class="num">81%</td><td><div class="bar"><i style="width:81%"></i></div></td><td>Not gradable from audio alone (0.27 to 0.40 correlation); the product knows the numbers shown, so it becomes a closed-vocabulary match.</td><td><span class="pill gold">needs the item bank</span></td></tr>
<tr><td>Finding the 60-second window</td><td>—</td><td></td><td>Speech-to-text timestamps plus a text labeller: 70% of subtask starts within 15 s of the enumerator's tap, timed windows back at a median of 60 s; the cheap labellers (Gemini 3 Flash) match the expensive ones. Audio models' own clocks drifted 20 to 75 s.</td><td><span class="pill ok">design rule: STT cuts, LLM labels</span></td></tr>
</tbody>
</table></div>
<figure><img src="charts/round2_count_mae.png" alt="Bar chart: mean absolute error in words read correctly per 60 seconds, per scorer, averaged over three subtasks. Gemini 3.1 Pro 3.6, Gemini 3.8 Flash 5.0, baseline 11.8, GPT audio mini 23.2." style="max-width:100%;height:auto"><figcaption>Every scorer we tried on the same 60-second windows, with the printed text supplied. Lower is better. The two Gemini audio models are the only ones clearly better than transcript plus alignment.</figcaption></figure>
<figure><img src="charts/round2_word_precision.png" alt="Bar chart: F1 of wrong-word flags on the Urdu story per scorer. Gemini 3.1 Pro 66, Gemini 3.8 Flash 50, baseline 36." style="max-width:100%;height:auto"><figcaption>Per-word marks are harder than counts. Only the Gemini audio models beat the alignment baseline; none reaches the second human's agreement rate, which is why chips are suggestions in the first version.</figcaption></figure>
<figure><img src="charts/round2_comprehension.png" alt="Bar chart: share of children where the grader's comprehension count was within one of the enumerator's. Gemini 3 Flash 79%, GPT-5 mini 75%, Claude Sonnet 5 73%." style="max-width:100%;height:auto"><figcaption>Grading spoken comprehension answers from the transcript is solved well enough by a cheap text model.</figcaption></figure>
<div class="cols">
<div class="prose">
<h3>Reading the results</h3>
<p><strong>Give the model the clip and the printed text, and counting is solved.</strong> Gemini 3.1 Pro and 3.8 Flash, listening to the 60-second clip with the exact text in front of them, land within five words of the enumerator for 74 to 97 percent of children, on poor audio, for one to six cents a window. Transcript-only methods, whether alignment or a text model, under-count by 8 to 20 words because speech-to-text drops words the child did say.</p>
<p><strong>Per-word chips are usable as suggestions, not marks.</strong> The best model's flags are right two-thirds of the time on the Urdu word list and just over half on the story; every other scorer is well below that. Chips appear where the model and the alignment agree; elsewhere the coach decides.</p>
<p><strong>Cheap models are not interchangeable with expensive ones here.</strong> GPT audio mini, MiMo, Nemotron and Voxtral were 12 to 23 words off; Gemini 2.5 Flash waved mistakes through. Gemini 3.5 Flash-Lite is the surprise: nearly as good as 3.8 Flash on counts at a third of the price.</p>
</div>
<div class="prose">
<h3>Three things the harness surfaced by accident</h3>
<ul>
<li>Our Microsoft Azure pronunciation service is dead: both stored keys are rejected, and the same key is on the live server, so English reading tests have quietly been falling back to a weaker method. Filed as an urgent bug; the key must be regenerated in the Azure portal.</li>
<li>Letting the transcription service guess the language breaks on accented child English: it wrote the child's English in Urdu and Devanagari script. Tell it the language.</li>
<li>The study export records 0 for both "read correctly" and "never reached", so per-word comparisons are only valid up to the word the enumerator says the child reached. The first pass of this evaluation got that wrong; the numbers here use the corrected rule.</li>
</ul>
<h3>What the scoring costs</h3>
<p>About 10 cents per child for the full five-minute test with Gemini 3.8 Flash on every window plus the graders; about 25 cents with Gemini 3.1 Pro on the Urdu windows. Both are far below the roughly 43 cents a coaching session costs today.</p>
</div>
</div>
</section>'''
s = re.sub(r'<section id="experiment">.*?</section>', lambda m: new_exp, s, count=1, flags=re.S)
new_models = '''<section id="models">
<h2>Which AI does what: the stack we recommend</h2>
<div class="prose"><p>There is no single "the AI". Different tools do different jobs, and the evaluation decided which. Costs are per child for the five-minute test.</p></div>
<div class="tablewrap"><table>
<thead><tr><th>Job</th><th>Tool we will use</th><th>Why, in one line</th><th>Fallback</th><th class="num">Cost</th></tr></thead>
<tbody>
<tr><td>Cutting the audio into subtask windows</td><td>The coach's cue phrase per block, plus speech-to-text timestamps (Soniox told the language is Urdu; Whisper told the language is English)</td><td>Speech-to-text clocks were exact to the second; every audio AI's own clock drifted by up to 75 seconds</td><td>A cheap text labeller (Gemini 3 Flash) on the transcript when a coach forgets the cue phrase</td><td class="num">1 to 2 cents</td></tr>
<tr><td>Counting words read correctly in 60 seconds, Urdu and English</td><td><strong>Gemini 3.8 Flash listening to the clip with the printed text</strong>; Gemini 3.1 Pro on the Urdu windows where a second opinion is worth six times the cost</td><td>Best agreement with the enumerator of everything tried, and it also gives per-word marks</td><td>Speech-to-text plus alignment if the audio model is unavailable</td><td class="num">1 cent per window (Pro: 6 cents)</td></tr>
<tr><td>Per-word chips on the checking screen</td><td>The same Gemini verdicts, shown only where the model and the alignment agree</td><td>Their flags are right about half to two-thirds of the time, the best measured but well short of a mark</td><td>The coach marks the word</td><td class="num">none extra</td></tr>
<tr><td>Letters and first sounds</td><td>The coach marks; the audio is kept for a later open forced-alignment model</td><td>Letter names transcribe in three scripts; no scorer tested cleanly</td><td>—</td><td class="num">none</td></tr>
<tr><td>Marking spoken comprehension answers</td><td>A text grader (Gemini 3 Flash tier) on the transcript with the passage</td><td>Within one answer for about 8 in 10 children at a tenth of a cent</td><td>Claude Sonnet 5 or GPT-5 mini, same result at higher cost</td><td class="num">under 1 cent</td></tr>
<tr><td>Spoken maths</td><td>Transcript plus the item bank plus a small number-word dictionary; a text grader only for word problems</td><td>Word problems within one item for about 7 in 10 children without the bank; number reading needs the bank, which the product has</td><td>The coach marks</td><td class="num">under 1 cent</td></tr>
<tr><td>The maths photo</td><td>Gemini 3.1 Pro vision with the answer key</td><td>Best on handwriting benchmarks; not yet tested on our answer pages, so we probe 50 first</td><td>The coach marks</td><td class="num">1 to 2 cents</td></tr>
<tr><td>English sound-level detail</td><td>Microsoft Azure pronunciation service once the key is regenerated; SpeechAce for made-up words</td><td>The only vendors with sound-level output; SpeechAce was weak on this 8 kHz audio and must be re-measured on 16 kHz voice notes</td><td>—</td><td class="num">2 to 4 cents</td></tr>
<tr><td>Checking the coach followed the script</td><td>A Gemini-class audio model over the whole block, once, in the background</td><td>Caught a slip a human reviewer missed</td><td>—</td><td class="num">about 5 cents</td></tr>
</tbody>
</table></div>
<div class="callout quiet"><strong>Tried and set aside.</strong> GPT audio mini and Nemotron omni (counts wrong by 15 to 30 words); MiMo 2.5, Voxtral small and Gemini 2.5 Flash (too lenient, 8 to 10 words high); every transcript-only model (no better than alignment, and harsher); GPT audio (good on word lists, poor on the Urdu story, four times the price of Gemini 3.8 Flash). Claude cannot listen to audio at all, so it marks answers and reads photos; it does not time or score words. Everything Urdu-specific that needs sound-level detail will run on open models we host ourselves, chosen for licences that allow production use.</div>
</section>'''
s = re.sub(r'<section id="models">.*?</section>', lambda m: new_models, s, count=1, flags=re.S)
s = s.replace('<text x="812" y="135" font-weight="600">English reading marker</text><text x="812" y="152" fill="#6B7590">word alignment + pronunciation service</text>', '<text x="812" y="135" font-weight="600">English reading marker</text><text x="812" y="152" fill="#6B7590">Gemini audio on the clip + text; pronunciation service later</text>')
s = s.replace('<text x="812" y="205" font-weight="600">Urdu reading marker</text><text x="812" y="222" fill="#6B7590">word alignment + sound-level model (v2)</text>', '<text x="812" y="205" font-weight="600">Urdu reading marker</text><text x="812" y="222" fill="#6B7590">Gemini audio on the clip + text; alignment as fallback</text>')
s = s.replace('<figcaption>Everything left of the window finder already exists in our bot (it is how the current reading test works). The window finder, the sound-level Urdu marker, the number reader, the photo reader and the per-item form are the new parts.</figcaption>', '<figcaption>Everything left of the window finder already exists in our bot (it is how the current reading test works). The window finder, the Gemini audio markers, the number reader, the photo reader and the per-item form are the new parts.</figcaption>')
s = s.replace('<div class="tile"><div class="n">0.97<small>agreement</small></div><div class="l">How closely the computer\'s word counts matched the enumerator\'s in our experiment, once it found the right part of the recording (1.0 would be perfect)</div></div>', '<div class="tile"><div class="n">97<small>% within 5 words</small></div><div class="l">Children whose 60-second word count from the best scorer (Gemini 3.1 Pro listening with the text) was within five words of the enumerator\'s</div></div>')
s = s.replace("<li><div><strong>Compare to the reference.</strong> The computer already knows the story text (the item bank). It lines the child's words up against it and marks each word: read correctly, skipped, replaced by a different word, or added. That gives words attempted, words correct, and the WCPM.</div></li>",
"<li><div><strong>Mark the words.</strong> The computer already knows the story text (the item bank). It gives the 60-second clip and that text to an audio model (Gemini 3.8 Flash), which returns a verdict per word: read correctly, wrong, or not reached. In the evaluation this landed within five words of the enumerator for three children in four; a second method, lining the transcript up against the text by spelling, runs alongside as the fallback and as a check on the chips.</div></li>")
s = s.replace('<td><span class="pill ok">counts yes</span>, per-word later</td>', '<td><span class="pill ok">counts yes</span>, chips as suggestions</td>')
s = s.replace('<a href="#experiment">The experiment</a>', '<a href="#experiment">The evaluation</a>')
open(p, 'w').write(s)
print('PLAN rewritten; sections present:', s.count('<section id="experiment">'), s.count('<section id="models">'))