Weekly Learning Signal, ICT
You wanted to know whether the team is right that we can only talk about learning levels once a quarter, or whether a weekly or fortnightly read is possible. This page walks through the reasoning one step at a time, with the numbers, so the conclusion is yours and not something you have to take on trust.
The short version
The team did the right maths for the wrong question. Their calculation is the one you do before an impact study, where you need to prove one group learned more than another. You asked a monitoring question: is ICT as a whole moving, and in which direction. That is easier, because every week of data you add makes the previous weeks more useful, not less.
So: a fortnightly-refreshed ICT trend is real and honest, and it becomes readable about ten to twelve weeks after launch. A weekly number exists but is only for operations. Zone trends are quarterly. Clusters and schools, never.
The catch is not children or schools. It is the person holding the stopwatch. Coaches scoring their own schools drift by up to 0.35 SD in a week, twenty times the weekly learning signal. Unless every child's audio is banked and a slice is re-scored blind, the fortnight is a mirage and the team's "half a year" is correct.
01First, what "learning went up" even means
Everything below is measured in standard deviations, written SD. One SD is roughly the gap between an average child and a child at the 84th percentile of their class. Reading fluency in Pakistani Grade 2 to 5 classes spreads about 23 correct words a minute per SD, so 0.1 SD is a little over 2 words a minute.
How fast do children actually learn? Not fast. Pooled international norms put Grade 3 to 5 growth at 0.36 to 0.40 SD a year, and low-income-country fluency data at about 1.3 words a minute a month. Spread over 35 teaching weeks that is 0.01 to 0.02 SD a week, or a quarter to a third of a word a minute. That is the signal we are hunting. Hold that number in mind; everything else is about whether the noise is smaller than it.
02Noise: the bathroom-scale problem
A bathroom scale drifts by a few hundred grams between readings. If you weigh yourself twice a day and react to each change, you will "gain" and "lose" a kilo every day and learn nothing. If you plot thirty days and fit a line, the true trend appears, even though no single pair of readings could show it.
Our scale is the coach's sample. With 55 coaches making about 46 school visits a day and testing 5 children each, we get roughly 1,150 tests a week from about 230 schools. Two facts about that sample:
- Children cluster in schools. Two children from the same school tell you less than two from different schools, because schools differ a lot. The measure of this is the intra-class correlation, ICC, around 0.2 for early-grade reading. It means the useful sample is capped at roughly schools reached divided by ICC, about 1,955 "effective" children for all of ICT no matter how many children per class you test. Testing the whole class instead of five adds under 2× per school. This is why "test everyone" is the wrong lever: it buys almost nothing, and it costs the coach's whole day.
- What one week's sample can say. Taking clustering into account, the ICT-wide average from one week has a 95% band of about ±0.08 SD. That is a decent snapshot of where ICT stands. It is hopeless for comparing this week with last week, because the difference of two such bands is about ±0.11 SD and the true weekly change is 0.01 to 0.02. To see one week of learning as a week-on-week difference you would need 115,000 to 273,000 tests a week. The team was completely right about this. Nobody will ever post "up 2% on last week" honestly.
03The trick the call missed: pooling weeks into a slope
The team's frame was "compare two moments". The monitoring frame is "fit a line through all the moments so far". Those behave very differently as data accumulates.
Say each weekly point has an error of 0.04 SD. Comparing week 2 to week 1: error about 0.06 SD, six times the weekly signal. Useless. Now fit a straight line through 8 weekly points. The error of the slope is 0.04 divided by the spread of the weeks, which works out to about 0.004 SD per week. Your 0.01 SD-a-week signal is now 2.5 times its error, and by the usual standard (signal at least 2.8 times its error) it is readable at 10 weeks. The slope error falls with weeks to the power 1.5: doubling the weeks cuts the error by almost three.
This is what the first chart shows. With sampling noise only, a business-as-usual trend of 0.01 SD a week is detectable after 10 weeks of pooled samples; a faster 0.02 SD a week after 6. Those two figures assume the team's design, five new children every visit. The 3-returning-plus-2-new design gives 9 and 6 weeks: one week faster at most, because pairs only start arriving at week 6 and the slope has already averaged away most of the noise the panel removes. And unlike a quarterly snapshot, the slope is refreshed every fortnight from then on and keeps sharpening.
Two things the team said about design turn out to be minor for the trend. Repeating the same children ("the panel") helps a lot when you compare two moments a quarter apart (that is the 3.6 versus 5.4 point gap in the August plan), but adds only about 10% to a pooled slope. So the call's decision to test five new children every visit in the first quarter costs the trend almost nothing. Its one real cost is that we never learn how well a child's score in September predicts their score in November, the number every future panel design depends on. One returning child in five slots from week 5 fixes that for free.
04Why the instrument does matter, and how much
The team said standard deviations are standardised, so the instrument is irrelevant. That is true of the arithmetic and false of the measurement. Two reasons.
- A ruler with five marks. ASER is a ladder with five rungs: nothing, letters, words, paragraph, story. A child who goes from reading 20 words a minute to 40 stays on the "words" rung and registers zero change. And 60% of Grade 5 children in ICT are already on the top rung, where the ladder cannot see any growth at all. A timed count of correct words a minute (the EGRA approach) is a ruler with millimetre marks: it sees movement inside a rung and above the ceiling.
- Reliability shrinks the effect. A measure that is only 70% reliable shows a true 0.10 SD change as about 0.08. The timed read is 0.83 to 0.98 reliable.
How big is the difference in practice? Smaller than I expected before the study. If growth were spread evenly across rungs, the ladder costs only 4 to 27% more weeks. The real damage is at the ceiling and inside rungs, where it costs everything. So: use the timed measure, derive the ASER rung from the same read for comparability, and stop arguing about it. The battery fits five minutes a child: a 60-second Urdu read on a server-chosen passage, three comprehension questions, a 60-second maths items-per-minute task, and a floor branch to letter sounds for a child who cannot read five words.
05The real enemy: the person holding the stopwatch
Everything above assumes the score is about the child. It is also about the assessor. The one field study that randomly assigned assessors to pupils (Kenya, 2023) found that who tested the child explained 12% of the score variance, and that assessors got more generous by 0.35 SD over their first six working days. Our assessors are the coaches, testing their own schools, with no one watching.
Now compare sizes. Weekly learning: 0.01 to 0.02 SD. Assessor drift: up to 0.35 SD. The drift is twenty times louder than the thing we want to hear. Worse, drift is sticky: a coach who was a little generous this week is usually a little generous next week too, so on a chart it looks exactly like real improvement. A fleet of untrained coaches ramping up produces a fake gain of 0.04 SD a week for two months, three times the true growth, and false alarms 13 to 45% of weeks.
Imagine the first 30 weeks after launch. Children learn at the normal pace, and then in week 20 something works: a phonics push lifts every child by 0.10 SD, about two words a minute. That is seven weeks of normal growth arriving at once, and it is exactly the kind of thing you want the RDF to notice. The grey line is what is really happening. The dots are what the weekly numbers show, with their bands.
Left: the coaches' scoring is held steady, because audio is banked, a slice is re-scored blind, and each coach's habit is modelled out. The dots track the grey line and the alarm fires the week after the jump. Right: the same children, the same jump, but a small, sticky drift in how generously coaches score (0.05 SD, the "mitigated" case) is left in. Nothing else changed. The dots wander above and below the truth for weeks at a time, the jump is buried in the wandering, and no alarm comes. A reader would have seen "improvement" around weeks 13 to 16, when nothing happened, and missed the real one.
| How much assessor drift is left in the numbers | A +0.10 SD jump caught within 4 weeks | False alarms per week | Weeks to read normal growth |
|---|---|---|---|
| None: audio re-scored, coach effect modelled | 99% | 0.4% | 10 |
| 0.02 SD residual | 86% | 0.9% | 12 |
| 0.05 SD, "mitigated" | 41% | 1.3% | 23 |
| 0.10 SD | 21% | 1.6% | 45+ |
| Untrained coaches, +0.35 ramp | 27% | 13–45% | not readable |
Read the last column. With drift measured and removed, the trend is a 10-to-12-week statement. With a "mitigated" 0.05 SD left in, it is 23 weeks: the team's half year. The cadence is decided by drift control, not by sample size. And drift cannot be tuned away; it has to be measured. That means: the app times the read and draws the passage, every child's audio is banked, a Monday calibration clip is scored by every coach, and 5 to 10% of clips are re-scored blind by someone who does not know the coach. The re-score budget Muzzammil was asked to price is what buys the fortnight.
06Zones: why "which part of ICT" is harder than "ICT"
A zone is one sixth of the schools, so one sixth of the sample, so bands about 2.5 times wider: a zone-week band is ±0.16 to 0.19 SD, a zone-fortnight ±0.12 to 0.14, and a zone trend takes 10 to 15 weeks. That part the team had right.
There is a nastier problem the call did not see. Each coach's schools all sit in one zone. So "Zone 3 scores high" and "the nine coaches in Zone 3 score generously" are the same number; nothing in the data can tell them apart. With a constant assessor effect of 12% of variance, that puts a floor of about ±0.23 SD on any zone level at any cadence, forever, unless coaches sometimes test each other's schools. One visit in ten by a neighbouring coach, plus a quarterly swap of two schools, breaks the tie. It is the price of any zone number.
Shrinkage (borrowing strength from the region to steady a zone) tightens a zone band by 4 to 35% but hides a genuinely lagging zone for months, so it is shown beside the direct number, never instead of it. Clusters of ten schools are ±0.30 SD after a whole quarter. They are not measurements.
07The good news hiding in the bad: diagnosis is fast
You asked for granularity for a reason: to know whether children are stuck at letter sounds, decoding, fluency or comprehension, so the lesson plans, coaching and home exercises aim at the right stage. That is a question about a level, not a trend, and levels are cheap. The share of Grade 3 children below 30 words a minute, or who cannot do a two-digit subtraction, is known to ±0.03 SD for ICT monthly and ±0.05 SD for a zone, from month one. The battery records each subtask separately for exactly this. So the RDF can carry two different kinds of statement from the start: "where children are stuck" monthly by zone, and "which way ICT is moving" fortnightly from week 10, each labelled for what it is.
08What the quiz can and cannot do
The quiz is our biggest stream of child data, so the natural hope is that it fills the gaps. The swarm read the live database this afternoon.
Three consequences. The quiz level is inflated by who takes it (children with a phone, stronger children): +0.26 to +0.76 SD in ability, hidden by the ceiling. A quiz-only trend reads 22 to 38% too steep because weaker children drop out first. And as a helper to sharpen the coach sample, it is worth 5% if it correlates 0.3 with the coach measure, 29% at 0.7; today it correlates about 0.24, so 3%. Teacher authoring also fell to a third in week two (672 to 221 teachers).
What it is good for now: a check on the lesson's SLOs, the coach's pre-visit brief that Ali asked for, and an intervention (children go on to the video library). What would let it carry real signal later: a fixed block of 6 to 8 calibrated questions per grade and subject inside every quiz, real SLO codes saved, links scoped to a class, and a tap-your-name join to the roster. That is a second-quarter build. The trend does not wait for it.
09Could teachers or parents test everyone?
The team's "you'd have to test everyone" sounds impossible until you notice we own a channel to 1,600 active teachers and thousands of parents' phones, and a reading-assessment voice-note feature already wired into the bot. Volume is not the problem: teachers running a one-minute read-aloud for five random children a week would produce thousands of reads.
The problem is that it does not happen. Our own teacher-run reading assessment in Punjab collapsed from 177 starts a month to 10; ICT's legacy mini-assessment was completed for 0.4% of the slots it opened. At the 5 to 10% participation this platform has actually sustained, a zone number from that channel is ±0.29 to 0.45 SD, worse than useless. And teacher scoring is inflated; the trend survives a constant inflation, but any inflation that changes over time reads as learning. So: a completion-only pilot in one zone in the second quarter, with the coach's own draw landing on the same children the same week to measure the inflation. It joins the model at 40 to 50% class participation, not before.
10Putting it together: what we can say, and when
| Unit · cadence | Band, sampling only | With drift or unmodelled coach | What can be said | Publish? |
|---|---|---|---|---|
| ICT · week | ±0.08 SD | ±0.11 | Visits, completion, absences, drift chart. No learning number. | Ops panel only |
| ICT · fortnight | ±0.04 | ±0.10 | "Over W weeks, Grade 3 and 5 fluency has moved b ± band words a minute per week; this is / is not yet distinguishable from zero." Readable from week 10–12 at normal growth if drift ≤0.02–0.05. | Yes, from week 8–12, gated on drift |
| ICT · month | ±0.03 | ±0.10 | Level, and the subskill profile: where children are stuck. | Yes |
| ICT · quarter | ±0.05 | ±0.10 | Quarter-on-quarter change; smallest detectable 5.2 points (fresh) or 3.3 (panel). | Yes |
| Zone · week | ±0.16–0.19 | ±0.28 | Nothing. | Never |
| Zone · fortnight | ±0.12–0.14 | ±0.23–0.25 | Nothing until coaches cross zones. | Internal only |
| Zone · month | ±0.05 relative | ±0.24 | Subskill profile as a band, after cross-visits. | Band only |
| Zone · quarter | ±0.10 | ±0.23 | Zone trend by grade, season-adjusted. | Yes |
| Cluster (10 schools) | ±0.30 after a quarter | — | Nothing. | Never |
The publishing rule that follows: a band of ±0.05 SD or tighter is a point with its band; 0.05 to 0.15 is shown as a band only; wider is suppressed. Never school ranks, week-on-week differences, quiz-derived levels, or any sentence that says the coaching caused the movement. Cause stays with the independent study.
11Scorecard: the call, claim by claim
| What was said | Status | In one line |
|---|---|---|
| Week-on-week differences are noise | Right | Needs 115,000+ tests a week. The fast object is a slope, never a difference. |
| "You have to test everyone in the class" | Wrong lever | Effective sample is capped by schools reached ÷ ICC. Whole-class adds under 2× per school. |
| Repeat children reduce the sample | Half right | True for a level, false for change; about 10% on a pooled slope either way. |
| The instrument is irrelevant | Wrong | Ladders are blind inside rungs and at the ceiling (60% of Grade 5). Modest cost if growth were even; total cost where it is not. |
| Zone trends need a quarter or more | Right | 10 to 15 weeks, and zone levels need cross-visits to exist at all. |
| "Even with the same kids there will be noise" | Right, under-weighted | The noise that matters is the assessor, and it is twenty times the signal. |
| Trust nothing before half a year | Depends on one thing | 10 to 12 weeks with drift measured and removed; 23 weeks if it is left in. |
12The architecture, in one table
| Piece | What it is |
|---|---|
| Instrument | Five minutes a child: 60-second Urdu read on a server-drawn passage (Grade 3 at least 100 words, Grade 5 at least 150) + three comprehension questions; 60-second maths items per minute; floor branch to letter sounds. ASER rung derived from the same read, never tapped. Four equated forms per grade, form id logged. English in the second quarter. |
| Sampling | Server-drawn and logged before any name is shown. One grade per visit, alternating 3 and 5. No school twice in a fortnight; at least 30 distinct schools per zone per fortnight, and the count of distinct schools published with every number. Coach cross-visits one in ten, plus a quarterly swap of two schools. From week 5, one of five slots to a returning child in roster-complete schools. |
| Quiz | SLO-coverage and engagement panel plus the pre-visit brief. Second quarter: anchor block, SLO codes, class-scoped links, tap-your-name join. Enters the estimator only after its correlation with the coach measure is known on 300+ children. |
| Population channel | Second-quarter completion-only pilot of a teacher-run read-aloud in one zone, anchored to the coach's draw on the same children. Joins at 40–50% participation. |
| Estimator | One statistical model refitted every Sunday: scores standardised within grade; school effects; a coach effect identified by cross-visits, the Monday clip and blind re-scores; per-coach drift; passage-form effects; the school calendar; a trend term within each season. The published band always includes the drift uncertainty, never sampling alone. A pre-agreed alarm rule so fortnightly refreshes do not pile up false alarms. |
| Reporting | Weekly ops panel; fortnightly ICT slope with band from week 8–12; monthly level and subskill profile; quarterly zone trends with a methods box. |
| Quality | Certification on six gold clips before day one; Monday clip; blind re-score 10% for four weeks then 5%, at least two clips per coach per week; per-coach drift chart to supervisors only; a public zero-stakes statement. First trend published only at measured drift ≤0.05 SD. |
13Four decisions only you can take
- Coach cross-visits, one in ten. A change from the call. Without it, no zone number exists. Decide it for ICT now; make it a condition for porting later, since AEO regions will find it harder.
- One returning child in five slots from week 5, roster-complete schools only. A change from "five new children until the quarter ends". Costs the trend nothing; it is the only way to learn r before the second-quarter panel is designed on a guess.
- Fund the audio bank and blind re-score from day one. Fix the Urdu tokenizer bug first (bd-r0rmm), then two re-scorers on retainer for about 115 clips a week falling to 58. This is what buys the fortnight. Without it the honest cadence is quarterly and the team's position stands.
- Adopt the publishing rule as policy before the first number exists.
14Ninety days
| Weeks | Owner | Deliverable | Gate |
|---|---|---|---|
| 0–2 | Sabeena, Sameer | Four Urdu forms per grade with comprehension; maths forms; floor-branch lists; six gold clips per grade | Forms mapped to ICT lesson-plan SLOs |
| 0–2 | Ali, Asma | Five-day dry run, 8–10 coaches across three zones: minutes per visit, completion, absences | Median visit ≤27 min; 70% complete five children; else four children, never a shorter battery |
| 0–2 | Ahwaz | Distinct schools per week from the visit log; server rota; cross-visit schedule | Baseline published |
| 0–2 | Muzzammil | Two re-scorers on retainer; cost line; certification protocol | Cost in the Doc |
| 0–2 | Haroon | Zero-stakes statement; RDF fortnightly writing rule; sign-off on the four decisions | Decided |
| 3–4 | Ali, Asma, Ahwaz | Certify all 55 coaches; launch all six zones; Monday clip and 10% re-score from day one; ops panel only. Estimate clustering, SD, form effects, coach share and drift from the first fortnight; re-run the simulation | 70% of visits carry five clean tests; re-score agreement ≥80%; observations not down |
| 5–8 | Ahwaz, Ali; Haroon, Sameer | Return slot live; forms equated; week 8: first fortnightly learning paragraph (expected: "not yet distinguishable from zero"). Quiz anchor bank and tap-your-name join in the background | Measured drift ≤0.05 SD |
| 9–13 | Ahwaz; Ali, Asma, Haroon | Week 10–12: ICT slope readable at normal pace; first r estimate from ~300 return pairs; first zone bands if coach effects are identified; quarter report with methods box. Second-quarter decisions on evidence; other regions two to three weeks behind | Slope band excludes zero, or the report says why not |
Build order: (1) the clean-test floor: server-side logged draw, in-app timer, form bank with ids, audio banked; (2) the drift pipeline: certification screen, Monday clip, blind re-score UI, per-coach chart; (3) the Sunday model job with the banding rule and the RDF panel files.
15What to hold onto
- Weekly is the collection grain; fortnightly is the reporting grain for the ICT trend; monthly for diagnosis; quarterly for zones.
- Precision is bought with schools reached and weeks pooled, not children per class.
- The assessor, not the sample, sets the clock. Bank audio, re-score blind, model the coach.
- Every number above is a planning value (clustering, SD, growth rate, drift, distinct schools per week) and gets replaced by the first fortnight's real data before anything is published. If Urdu growth in Grades 3 and 5 turns out below 0.005 SD a week, the quarter is the floor for ICT too.
- If five children do not fit in thirty minutes, test four. Never shorten the battery.