Weekly Learning Signal, ICT

1. How we got here

This page assumes you were not in any of the earlier discussions. Four moments matter.

Who said what on the call

2. Three ideas the rest depends on

A. How we measure the size of a change

Picture every Grade 3 child in ICT lined up from weakest to strongest reader. One standard deviation (SD) is roughly the gap between a typical child and a child ahead of 84 in 100. Researchers use it so reading, maths and different tests can be compared on one scale. Every number on this page is in SD.

B. The number of schools matters more than the number of children

Children in the same school resemble each other: same teachers, same neighbourhood, same books. Once you have tested five children in a school, a sixth mostly repeats what the first five told you about that school. The measure of this is the within-school similarity (statisticians call it the ICC). In the Rawalpindi study it was 0.30 to 0.38 for a single grade: about a third of the differences between children in one grade are differences between their schools. This page uses 0.35.

So "test the whole class" is the expensive way to get more precision. The cheap ways are to reach more schools and to add up more weeks. Across all of ICT, one round of testing can never be worth more than about 1,100 independent children, however many children per school are tested.

C. Draw one line through many weeks instead of comparing two

Coaches will run about 900 tests a week across about 155 schools (measured from their visits over the last six weeks). Even comparing the same schools over time, any single week's ICT average is fuzzy by about ±0.05 SD, because it depends on which children happened to be tested. A week of learning is about 0.012 SD. So comparing this week with last week is hopeless: the fuzz is several times larger than the change.

But the fuzz is random: some weeks come out a little high, some a little low. Put ten weeks on a chart and draw the best straight line through them, and the ups and downs largely cancel. The tilt of that line, the trend, is known far better than any single week.

3. What we can say, and when

The two columns have different conditions. For ICT as a whole, everything below needs coaches' scoring to be checked and corrected (section 6). For zones, it needs that and one more thing: the correction has to be applied coach by coach, because each zone has its own six to twelve coaches.

Numbers are 95% ranges on the plan as decided: five children per coach visit, Grades 3 and 5, about 900 tests a week. Zone ranges run from the busiest zone (Urban-I, about 33 schools reached a week) to the quietest (Nilore, about 18). Every input that is not yet measured locally is listed in the footnote and is replaced by real data after the first fortnight.

4. The quarterly goal: which children to test

Three designs are on the table. Five new: every visit tests five children never tested before (decided on 15 September for the first quarter, proposed by Muzzammil). Three returning plus two new: three children tested about a quarter earlier and two new (the August design, favoured by Haroon). Four new plus one returning: a middle option this study adds.

Why returning children help

If Ayesha reads 40 words a minute in September, their November score will be close to "40 plus a term of learning", because strong readers stay strong and weak readers stay weak. How closely one score predicts the next is a correlation between 0 and 1. In US reading tests, scores about three months apart correlate at 0.87 to 0.94; in Kenya, 0.77 one month apart. No Urdu figure exists yet, so this page assumes 0.75, and the first quarter should measure it. When the same child is tested twice, everything that makes Ayesha Ayesha cancels out, and what remains is their change. When two different children are compared, the difference between the children swamps the change.

Why the design hardly changes the two-weekly trend

A returning child is only re-tested about a quarter after the first test, so within the first quarter all three designs produce almost the same data. And a line through ten weeks has already averaged away most of the "which child happened to be picked" noise that returning children remove. The design matters for comparing one quarter with the next, which is the main goal.

5. The test itself: EGRA and EGMA, not ASER

ASER sorts a child into one of five boxes: nothing, letters, words, paragraph, story (and a similar ladder for maths). EGRA (reading) and EGMA (maths) count: how many words the child reads correctly in one minute, how many sums they answer correctly in one minute, how many comprehension questions they get right.

That is why the choice of test matters even when results are expressed in SD: a test can only show change that its scale can register. ASER is excellent for counting how many children are far behind, and poor for seeing children move. EGRA and EGMA see both. An ASER level can still be read off the same recording if we ever want to compare with ASER reports; it is a by-product, not the main measure.

The coach records one voice note per block. AI marks the recording and the coach checks the marks on a pre-filled form before anything is saved. In that study's test on 187 real recordings, the AI's word count landed within five words of the enumerator's for 74 to 97% of children. The whole test is not yet proven to fit five minutes; the Islamabad pilot will time every child. If five children do not fit into 30 minutes, test four children rather than drop a subject.

The cost of the finer test is that timed reading takes care to score. When reviewers re-listened to the Rawalpindi recordings, they disagreed with 8% of the timed-reading counts where the enumerator had recorded the 60-second count, and with about half where the child finished early or was stopped and the count had to be worked out. Where a reviewer wrote down a corrected number, the enumerator's score was the higher one in 70 of 95 cases. Keeping the scoring honest is the subject of section 6.

6. Keeping the scoring honest

What drift is. A coach who marks a little more generously as the weeks pass, perhaps from getting used to the children or from wanting to see progress, produces scores that rise even when children have not improved. It creeps slowly and in one direction, so on a chart it looks exactly like learning.

How big it can be

• Kenya, 2023. Testers were randomly assigned to children, so the effect is clean. Which tester a child happened to get explained 12% of the maths score, and recorded scores rose by 0.35 SD over the testers' first seven working days. The authors could not tell growing leniency apart from testers getting better at spotting what weak children know.
• Rawalpindi, May 2026 (Akademos enumerators, 2,233 children, 28 enumerators). On untimed maths, enumerators clearly differed: 6 to 18% of the score depended on who tested the child, close to Kenya. On reading, no clear difference showed between two enumerators testing the same class, but only about 370 children allowed that check, so a Kenya-sized effect cannot be ruled out. One analysis found scores rising over an enumerator's first week by about as much as in Kenya; two stricter checks did not. So drift here is neither proven nor ruled out. Separately, a change to the tablet form in the middle of fieldwork moved maths scores by 0.9 SD, a reminder that any change to the test itself also shows up as a fake change.
• United States. In one large reading-fluency study, which examiner tested a child explained 16% of the score, three times as much as which school the child attended.

What the world does about it

Every major assessment has a second person re-mark a random slice of the work without seeing the first mark, and compares the two. America's national assessment (NAEP) re-marks 5 to 25% of answers, TIMSS about 200 answers per question per country, and PISA 100. EGRA surveys double-score at least 150 children and only hire testers who agree with a gold-standard score at least 90% of the time, often using recordings of real children. In the United States, reading tests re-scored from recordings agree with the live marking on 97 to 99% of words; the main live error is the stopwatch, a few seconds on a 60-second read. What we did not find is a programme that runs the whole loop this page proposes: coaches test, a slice of recordings is re-marked blind, and each tester's habit is measured and removed. Every part is standard practice; the combination is new.

How it would work here

1. Every test is recorded, one voice note per block, which the EGRA five-minute design already requires.
2. The AI marks every recording first, before the coach sees anything, and that first mark is stored. The AI marks every coach's children the same way week after week and does not become generous, so it works as a constant ruler. Comparing each coach's final marks with the AI's on the same recordings shows, for every coach every week, whether their marking is drifting. On the Rawalpindi recordings the better AI model differed from the enumerators by about as much as a second human listener did.
3. Independent human listeners re-score about 5 to 10% of recordings blind, seeing neither the coach's mark nor the AI's. This checks both the coaches and the AI, for example whether the AI marks Punjabi- or Pothwari-accented Urdu more harshly. The AI model version is pinned; if it ever changes, a fixed set of recordings is re-scored to measure the jump.
4. "Modelling out" a coach's habit means: for each coach, average the gap between their marks and the re-scores on the same recordings ("Coach A marks 3 words a minute higher"), and subtract it from that coach's scores. Recompute every month so drift is caught as it happens. With about 20 re-scored recordings per coach per quarter, each coach's habit is known to about ±0.13 SD, which averages out to about ±0.02 SD for ICT and ±0.04 to 0.05 SD for a zone.
5. What recordings cannot fix is how the test was given: which child sat down, help given before the recording started, how comfortable the child felt. The controls are: the server picks the child and the coach confirms the name; a scripted cue phrase starts each block; the AI checks every recording for prompting during the timed minute, a story read once instead of twice, or a timer problem; and a small number of cross-visits, where a coach tests in a neighbouring coach's school, as a spot check.

Does this solve the zone problem?

Mostly, yes. 51 of our 55 coaches work in one zone only, and each zone has 6 to 12 coaches. Without correction, a zone's average mixes how well its children read with how generously its coaches mark, which puts an uncertainty of ±0.20 to 0.28 SD on any comparison between zones. Scoring every recording with the same ruler, checked by one shared pool of human listeners, marks every zone the same way and cuts that to ±0.04 to 0.05 SD. What remains is differences in how coaches give the test, which the protocol checks and cross-visits cover.

A zone's own change from quarter to quarter is less affected, because a coach who is always equally generous cancels out when their own schools are compared with themselves. Only a coach whose generosity changes, which is drift, distorts a zone's trend.

7. What the lesson quiz can add

The lesson quiz works like this: a teacher records a lesson, Rumi writes eight multiple-choice questions on that lesson's learning outcomes (SLOs), and children answer on a parent's phone. From 6 to 26 September, 1,251 teachers wrote 2,565 quizzes and 5,628 children completed 16,773 of them. The data were re-checked on 26 September to answer three questions.

Does a quiz score say something stable about a child?

One quiz on its own says little. A child's score on one quiz predicts their score on a different quiz a few days later at a correlation of 0.31 within the same subject (0.27 across subjects), so looking within one subject barely helps. Maths is highest at 0.37 and Urdu lowest at 0.17. The reason is length: eight questions is a short test, so luck plays a large part in any one score. A third of sessions score a perfect 8 out of 8, which also hides differences between strong children.

Several quizzes averaged say much more. Reliability is the share of a score that reflects the child rather than luck. Measured on the 308 children who have taken four or more quizzes in one subject:

So the average of about five quizzes in one subject is a stable measure of where a child stands in their class. Today only 5% of quiz-takers have that many; the share grows as teachers use the quiz more.

Are the SLOs stored, and can quizzes be grouped by skill?

Yes. Version 1 of this page said otherwise; that was wrong. Every quiz stores its SLOs in English and Urdu, with the level taught and a quote from the lesson, and every question is tied to one of them. Two thirds of quizzes also link to a curriculum code through the lesson plan the teacher downloaded around the lesson. That code shows what was planned, which is not always what was taught.

Each quiz's SLOs are worded differently, but many describe the same skill. Grouping them by meaning within each subject and grade gives 364 skills that appear in five or more quizzes. The largest are converting mixed and improper fractions (68 quizzes), the meanings of Urdu words (60), and numerator and denominator (54). Two quizzes on the same skill predict each other better than two on different skills (0.43 against 0.30).

That makes a class-level score per skill possible now: 110 teacher-and-skill combinations already have two or more quizzes with at least five children each. This is what Ali asked for on the call, for the coach's pre-visit brief. A per-child score per skill is too thin today, because a child answers about two questions per skill per quiz.

Can the quiz replace the coach test for ICT or zone numbers?

No. Quiz-takers are the children with access to a parent's phone who chose to finish, not a random sample of the class. A quiz average describes the children who took it, not the class, the zone or ICT. The coach test, with children drawn at random, measures level and change. The quiz is the frequent, skill-by-skill signal that tells teachers and coaches what to reteach. Retakes of the same quiz (28% of completed sessions, and mostly the same day) raise scores by about 13 points through practice, so they are kept out of any learning number.

8. A change is not yet impact

Everything above tells us whether learning moved and by how much. Children learn every quarter whether or not we help, so "Grade 3 reading rose 0.15 SD this quarter" is not by itself evidence that our work caused it. Three comparisons turn a quarterly change into evidence of impact, and all three are available from this data:

• Against the normal pace. The yearly impact studies and published benchmarks tell us how fast children in similar schools learn without extra support. A quarter that runs clearly faster than that pace is a signal.
• Staggered starts. When a new piece of support (a phonics push, a new lesson-plan series) reaches some zones or schools before others, the ones that started later are the comparison for a quarter.
• Same quarter, last year. Once a year of data exists, each quarter can be compared with the same quarter a year earlier, which removes the effect of the school calendar.

The yearly impact studies stay the official measure of impact. This system shortens the time it takes to learn whether things are working from a year to a quarter, and gives an early direction within the quarter.

9. What the team said, what Haroon said, what the evidence shows

10. Decisions to be made

1. Which children to test. Five new, four new plus one returning, or three returning plus two new; and when to switch. Recommendation in section 4.
2. Blind re-scoring. Who the independent listeners are (a small in-house team or scorers on retainer), how many recordings per coach they re-score, and the budget. Muzzammil was asked on the call to price exactly this.
3. Cross-visits. Whether coaches occasionally test in a neighbouring coach's school, as a check on what recordings cannot catch (section 6).
4. The instrument details, owned by Sabeena and Sameer: the stories, letter names or sounds, vowel marks, the made-up words, and the maths items. These are listed in the EGRA five-minute study.
5. What the RDF publishes: the trend with its range every two weeks for ICT, the where-children-are-stuck picture monthly, zone results quarterly, and a rule that no number is shown without its range.
6. The quiz: whether to add a small common block of questions to every quiz so results can be linked across lessons (section 7).
7. How impact is judged each quarter: which comparison from section 8 is used, and who signs it off.