The field form runs on NIETE's sandbox WhatsApp number as /observe2live on sandbox. A coach fills it in while the lesson happens, picks the lesson plan, adds photos, seals it and sends the recording, then checks the moments found in it and the plan's steps. From there the visit ends exactly as /observe does: the debrief with the teacher, the coach's own card, the teacher's report, and the visit marked complete. This version adds that end, which Riffat's 4 October test found missing. Click through a whole visit below, or try it on a phone.
The sandbox holds test accounts and test schools only. The number you message from needs a coach or school-leader account there; if the planner shows no school, add one from its menu. A coach never appears in their own teacher list. The plan list shows the plans the teacher got from the bot, so a test teacher may have none; "Upload new" adds any other plan. A sealed record can't be changed, by design, so practise on a practice lesson. Each recording is analysed once: send a new one for each visit, not one sent before.
Everything in the phone comes from the code running on the sandbox, run here in your browser with a made-up lesson, plan and debrief: the wording, the checks, the plan list, the levels, the brief, the debrief, the report and the moment the visit is marked complete. The model calls are not run; fixed answers stand in for them: the moments in the recording, the grading of the plan's steps, the debrief's transcript, the feedback to the coach, and the notes and text of the teacher's report. The coach's card and the teacher's report are drawn by the sandbox's own renderers when this page is built. Use Next, the arrow keys, or the buttons in the phone; tap an image to see it full size.
This is an earlier step, as it was.
Riffat ran /observe2 on the sandbox number on 4 October and found the visit stopping halfway. Everything below was fixed and tested on the sandbox the same evening, with a whole visit run from WhatsApp to "complete". Riffat's two earlier visits stay as they were; the fixes apply to new visits.
| Found | What the build does now |
|---|---|
| After the check, ask for the debrief recording, as /observe does | Submitting the check now hands the visit to /observe's own end: "Debrief now or later?", the brief as the debrief guide, the debrief recording, the coach's card, then the teacher's report, which the coach sees first. As in /observe, the visit counts as complete only when the debrief is done and the report sent. "Later" keeps it in the /observe2 menu under "Complete debriefs" and "Send reports". |
| Apart from the form, everything stays as /observe: the start and the end | Checked step by step against /observe. Scheduling, rescheduling and cancelling a visit say /observe2 in their messages. A visit started from the schedule is tied to its teacher as /observe ties it. If the teacher can't be matched to a recording, Rumi asks who was observed, as /observe does. The report is /observe's report, filled from the coach's levels, the plan's result and the photos. |
| Photos from the gallery, not only the camera | A step of its own before the seal: take photos now, upload saved photos from the gallery or files, or none. Up to ten. On WhatsApp on a computer the camera button does nothing; "Upload saved photos" works there. |
| A recording that got a generic voice note back (the second visit) | The first file sent was the first visit's recording again, so it was refused, correctly. But the refusal left the form unable to take another recording, so the 11-minute file that followed went to Rumi's ordinary chat, which answers with a voice note. Now a refused repeat leaves the form waiting, and any recording of two minutes or more sent while a form waits goes to that form. |
| Anything else dropped? | A lesson plan added after the recording arrived is now graded too; it used to be graded before the plan was there. "Someone else" after the report preview now shows a fresh preview for the new teacher; this was broken in /observe as well. |
The go-live call with Ahwaz, Rifat, Annie and Ali, and what the sandbox build now does about each point. 16 coaches start on Tuesday 6 October; the others stay on /observe.
| Asked | What the build does now |
|---|---|
| Fidelity is missing from the new form | It is back, as in /observe. The coach picks the lesson plan in the form, before the seal. After the recording, the plan's steps are rated by the grader /observe uses. In the check, each step comes already rated: done as planned, done another way, partly done, not done, or couldn't tell. The coach changes any step they saw differently, and the score is worked out again from their answers. The screen says how many steps count toward the score: homework is listed, but not counted. Without a plan, the check goes straight to the levels. |
| Ask for the lesson plan inside the form, and let the coach check it is the right one | A new screen after Part 2 lists the teacher's most recent plans from the bot, newest first, named as /observe's list names them, with "Upload new" at the end. The seal screen names the plan picked. The coach goes back to change it, until the seal. |
| A plan the bot didn't give the teacher (asked after the call) | "Upload new" adds it the ways /observe takes one: photos of the paper plan (one per page), a PDF or Word file, or the plan typed in. After the recording it is read the way /observe reads an uploaded plan, checked that it is a lesson plan, turned into steps and graded by the same grader. Several photos are read in order as one plan. If it can't be read, or isn't a lesson plan, the brief says so. |
| No cap of three photos for coaches | Up to ten. The photo line says which ones help: the board, the materials, children using them, groups at work, and the board at the end. |
| The wording must explain itself | The questions the call found unclear are reworded. "What happened at the switch?" is now "How did the switch go?", and "Ready before being told" is now "Children knew what came next". Ahwaz's own wording pass replaces ours when it arrives. |
| Make the debrief more reflective (Ali) | The brief opens with two questions for the teacher, and asks one about the moment it offers. It also says how much of the plan was taught. |
| Record when the coach moves to Part 2 | Already stored: when the form was opened, when each part was saved, and when it was sealed, all by the server's clock. So the time to fill the form can be compared with /observe. |
| Keep a record when the coach changes a level | Already stored: the level the rule proposed sits beside the coach's final level. |
| Kept as they are | One thing to work on first (the team is reviewing the list). No indicator codes in front of the coach. Levels still come from what the coach counted and the moments confirmed. Only "When you sat with a group" allows more than one answer. |
The questions from 30 September, and what the sandbox build does about each.
| Asked | What the build does |
|---|---|
| Can a coach change answers as they go? | Yes. A WhatsApp form keeps its answers while it is open, and nothing is sent until the coach taps "Part 1 done", "Part 2 done" or "Seal and send". Each of those saves the part on the server, so if the form closes, only the part in progress is lost, and reopening it carries on where the record stands. How long a phone keeps a form open is part of the phone test below. |
| Can WhatsApp's own recorder stay on while the form is filled in? | We don't rely on it. Meta documents nothing about it, and a coach would only find out after the lesson that it had stopped. The form asks for the phone's own voice recorder app, which is made to keep recording in the background. The phone test checks both. |
| Where does the recording go, and what guides the coach? | Into the same chat, after the seal. The chat says so three times: in the steps sent with the form, in the message right after the seal ("stop the recorder, then Share, WhatsApp, this chat"), and in the reply when it arrives. The recording finds the sealed record by itself. |
| One form or two? | One: Part 1, Part 2, then the seal, from the first minute of the lesson to the last. |
| 15 + 15 minutes, or 20 + 20? | The coach says how long the period is (30, 35 or 40 minutes) and the form splits it in half. A 40-minute period has Part 1 from minute 0 to 20; the form says so on its first screen. |
| The routines question is hard to follow | It is now two plain questions: "Did the class switch activity?", and if yes, "How did the switch go?", with four answers a coach can see, from "Took long, no clear signal" to "Children knew what came next". Every other question was rewritten the same way (below). |
Each question describes something the coach can see, and each answer carries a line saying what to look for. Left: version 3 of this page. Right: live on the sandbox since 2 October, and on the screens in the walkthrough.
| Version 3 of this page | Live on the sandbox |
|---|---|
| Who got to answer?Only children with hands upA quiet child picked onceQuiet children picked oftenPicked often, different ways | Who did the teacher ask?Only children with hands upThe teacher asked only children who had a hand up.One child without a hand upOnce, the teacher asked a child who had not put a hand up.Several without hands upMore than once, the teacher asked children who had not put a hand up.Other ways to include everyoneFor example: names picked at random, talking to a partner first, or everyone writing an answer. |
| Pair or group work?No pair or group workGroups, but each worked aloneGroups had to combine workRoles, and they reported backThe room can't seat groups | Did children work together?No, alone or as a whole classNo pair or group task in this part.Sat together, worked aloneChildren sat in pairs or groups, but each did their own work.Made one thing togetherThe task needed the group to work together, e.g. one chart or one answer per group.Had roles and reported backEach child had a job, and the group told the class what they did.The room can't seat groupsFixed benches, or no space to sit together. |
| When you sat with a groupThey took turns, no repliesOne child built on anotherBuilt on each other oftenArgued it out, agreed answer | When you sat with a groupTick everything you sawEach spoke, no one repliedChildren spoke in turn but did not answer each other.One child replied to anotherOnce, a child answered or added to another child's idea.Replied to each other oftenTwo or more times, children answered or added to each other's ideas.Disagreed, then agreedChildren gave reasons, questioned each other and agreed on one answer. |
| Books, board or materials?None, though they'd helpOnly the teacher used themChildren used themChildren explained with themNothing suits this topic | Who used books or materials?Books, the board, charts, cards or objects.Nobody, though they would helpNo book, board, chart or objects were used, though they would have helped.Only the teacherThe teacher showed or wrote; the children did not touch them.The children used themChildren used books, cards or objects, or wrote on the board.Children explained with themA child used a material to show or explain their thinking.Nothing suits this topicNo material would help with this topic. |
| Did the class change activity?YesNo | Did the class switch activity?For example: from listening to the teacher to writing, or from the whole class into groups.YesNo |
| How did the change go?No routine; things stalledRoutines, slow to settleClear routine, quick recoveryChildren knew before told | How did the switch go?If the class switched more than once, pick what happened most often.Took long, no clear signalThe teacher didn't clearly say it was time to change. Children kept chatting or wandering, and the lesson stopped.Signal given, but took longThe teacher said it was time to change, but many children took more than a minute or had to be called back.Changed within a minuteThe teacher said it was time to change, and nearly every child was on the new task within about a minute.Children knew what came nextThe children already knew the routine and moved to the next task on their own, without being told one by one. |
| Anything to report?Nothing to reportPraise that sounded sarcasticRidicule or humiliationA wrong fact left standingThe recording failed | Anything to report?Something that should never happen in a classroom, or a recording that failed.Nothing to reportPraise that sounded mockingWords like 'well done' said in a way that made fun of a child.A child was shamed or mockedAny ridicule, humiliation or threat towards a child.Wrong fact left uncorrectedThe teacher said or wrote something wrong and did not correct it.The recording failedThe recorder stopped, or nothing was recorded. |
| Lesson planUsed a lesson planHad one, didn't follow itNo plan for this lesson | Was there a lesson plan?Followed a lesson planHad one, didn't follow itNo plan for this lesson |
| Step | What the server does |
|---|---|
| /observe2 | Opens today's visit planner with a marker, so the Start that follows is known to be an /observe2 visit. Coaches and school leaders only; anyone else is told it is for coaches. |
| Start | Ties the visit to the teacher as /observe does, creates the visit's record, and asks how long the period is. |
| Period | Saves it, and sends the form with its own link to the record. |
| Part 1, Part 2 | Checks the answers (children who spoke can't outnumber children present; first-time speakers can't outnumber those still quiet after Part 1) and saves each part with the server's time. A form opened again carries on where the record stands. |
| Lesson plan | Lists the teacher's most recent plans from the bot, the same list /observe offers and named the same way, and saves the pick with what the grader needs to find the plan's steps. "Upload new" stores photos or a file of the plan after the reply, or keeps typed text. The plan is locked with the seal. |
| Photos | Opens the camera picker or a file picker, saves how many photos came, and stores them after the reply as JPEG, like every classroom photo. A photo step changed while photos are still being stored keeps only the newer choice. |
| Seal | Locks the record once, with the time; the database refuses any later change to what was sealed. Then sends the recording steps to the chat. |
| Recording | Joins the coach's open record (same coach, same teacher) before transcription starts. While a form waits, any recording of two minutes or more goes to it. A recording analysed before is refused, and the form keeps waiting for the right one. No photo or lesson-plan questions: the form has them. |
| Moments and the plan | One model call finds the moments in the transcript, each with its minute and words. At the same time, /observe's grader rates the plan's steps against the same transcript: a listed plan's own steps, or an added plan read and turned into steps the way /observe does it. The coach's classroom photos go to the grader as /observe's do. If that grading fails, the moments and the check go on without it. They are stored with the levels they add up to on their own, which no screen shows. A transcript without timings is refused before the model runs, with its own message to the coach. |
| Check | Opens when both the seal and the moments exist. Each screen is saved as it is submitted. The plan's steps are re-scored from the coach's answers by /observe's scorer, and kept beside the recording's grading. The levels are added up by the fixed rule, and Submit happens once. |
| Hand-over | After Submit, saves the visit the way /observe saves a submitted form: the levels become the report's sections, the plan's result its fidelity section, the photos its classroom photos, and the brief this visit's debrief guide. Never over a cancelled visit. Then asks "Debrief now or later?" with /observe's own message. |
| Brief | Built from the checked record, with no model, and sent as the debrief guide when the coach taps "Debrief now". A level the rule only fills in "for now" is described by what was seen, not by that level's name. |
| Debrief, card, report | /observe's own steps, unchanged: the debrief recording is transcribed and the coach gets feedback on the conversation as a card; the report is drawn, shown to the coach, and sent to the teacher on "Send now". When the debrief is done and the report sent, the visit is marked complete. |
On 2 October the whole visit was run on the sandbox number from WhatsApp, with a three-and-a-half-minute recorded test lesson that follows a Grade 4 maths plan: /observe2, the planner, both parts, the plan picked from the list, the seal, the recording, 12 moments found, the plan's steps graded at 64% (3 done, 3 partly, 1 not done, out of 7 that count), one step corrected by the coach (71%), the 17 levels and the brief. Three things it showed were fixed the same day: a single refused answer now shows its own message; the brief counts the partly done steps; and the plan screen says which steps count and names each step's part of the lesson. Later the same day a second whole visit added a plan the bot hadn't given the teacher, a PDF attached through "Upload new". After the recording, its 8 steps were read from the PDF and checked the way /observe checks an uploaded plan (6 of the 7 that count done, 86%; homework listed but not counted), and the brief carried the result. Both forms passed Meta's own validation and are published on the sandbox account. On 30 September the earlier build passed 16 of 16 checks through the live endpoints, including the database refusing a change to a sealed record. On 4 October, after the fixes, a whole visit ran on the sandbox number from WhatsApp to the end: both parts, the plan, two photos uploaded as files, the seal, a repeat of an earlier recording refused with the form still waiting, then the new recording, 11 moments, the plan's steps (71%), the levels, "Debrief now", the brief, a debrief recording, the coach's card, the report preview, "Someone else", and the report sent, with the visit marked complete. "Someone else" had stopped producing a new preview, in /observe too; that was fixed the same evening.
A recording hears every question, every piece of praise and every wrong answer, with the minute it happened. A coach at the back of the room sees what no recorder can: how many different children spoke, whose hands were up, whether a group really worked together, whose hands were on the materials. So each side does what it is good at. During the lesson the coach answers plain questions about what they see, in two parts. Rumi finds the moments it heard and quotes them. Neither has to guess at the other's part.
A coach who has seen Rumi's draft cannot un-see it. Today every coach starts from Rumi's draft. Most of it goes back as it arrived, and what does change mostly goes up: since mid-August the coach's final ends five points above Rumi on average. In Rawal Town two staff observers watching the same lessons went the other way, 81 changes down and none up, and Rumi's two drafts of those lessons disagreed with each other. The data can't say who is right, because none of it was collected independently. So the coach's record is locked, with the time, before anything from the recording is shown.
The new rubric makes level 3 something you can count: "at least two open questions, each answered by one child." So Rumi's job becomes finding and quoting those moments, a task anyone can check, rather than picking a level, the task on which the same lesson got different answers on different runs. The coach says yes or no to each moment, a fixed rule adds up the yeses, and the level arrives filled in. Rumi's own score is kept for the evals team and never shown to the coach.
The coach writes down what they see as it happens, Rumi says what it heard, the coach confirms, the rule adds up. Time is saved in the adding up, not by pre-filling what the coach saw.
Seventeen indicators are too many to hold in mind while watching a class, and on 29 September Ahwaz settled that coaches don't have to: "They should just capture the evidence. That is their job while doing the observation. The rubric-based ratings come at the end, when all the evidence has been collected." So the form never names an indicator, a code or a level while the lesson is on. Its questions follow the lesson: how many children spoke, who got to answer, what happened when children worked, the books and materials, the switches between activities. The coach answers what they see, and the rule decides which indicator each answer feeds.
The four moments from version 2 survive in one place: after the seal, Rumi groups what it heard under them, and the teacher's profile can use the same grouping. The pocket card is dropped, because a coach needs nothing in hand. The card below is for the team: it shows where each of the 17 sits.
Every threshold is read straight off the level 3 and level 4 descriptors in the workbook. "You saw" rows take the level from the coach's answers in the two parts. "Rumi heard" rows take it from the moments the coach confirmed, counted by a fixed rule. The coach can change any level, with a reason.
| Indicator | Evidence | How the level is reached | The four levels, as the coach reads them |
|---|---|---|---|
| سوال Asking: When the teacher asks, who answers, and how? | |||
| The teacher's questions core five C1 · Quality Questioning | Rumi heard, you confirm | Level 3 at two or more open questions, each answered by one child. Level 4 also needs a "why" question with a follow-up. hole Exactly one open question answered by one child has no level. | No real questions asked · Yes/no or chorus answers only · Open questions, one child each · Plus a 'why' and a follow-up |
| Who got to answer C3 · Equitable & Inclusive Participation | you saw | Your answers to "Who did the teacher ask?" in the two parts; the higher counts. | Only raised hands answered · A quiet child picked once · Quiet children picked often · Varied ways, silent child in |
| Children's reasoning core five D1 · Depth of Student Reasoning | Rumi heard, you confirm | Level 3 when two or more children explain why. hole Exactly one child reasoning has no level. | One-word answers only · Longer answers, no reasoning · 2+ children explained why · A child went beyond the ask |
| How many children spoke core five D2 · Participation Breadth | you saw | Your numbers: children who spoke in Part 1, plus those speaking for the first time in Part 2. 3 to 5 is level 2; 6 or more, with someone speaking in each part, is 3; two or more new voices in Part 2 make it 4. Rumi's count of voices is shown beside it as a floor. hole "Spread across the lesson" is not defined; the rule reads it as at least one child speaking in each part. | 1-2 children, or none · 3 to 5 children · 6+ across the lesson · 6+, and new voices later |
| Children's own questions D5 · Student Questioning & Curiosity | Rumi heard, you confirm | Level 3 at one question about the content from a child. Level 4 at two, one going beyond the lesson. | No child asked anything · Only 'which page' questions · A question about the content · 2+, one beyond the lesson |
| غلطی Wrong answers: When a child gets it wrong, what happens next? | |||
| When an answer was wrong core five C2 · Responsive Re-explanation & Feedback | Rumi heard, you confirm | Level 3 when the teacher gives the reason on two or more wrong answers. Level 4 when a child fixes it and the teacher tries a new way. | Wrong answers ignored · Marked wrong, little reason · Explained why, twice or more · Child fixed it; new way shown |
| Safe to be wrong core five C6 · Positive & Supportive Learning Environment | Rumi heard, you confirm | Level 3 at two or more pieces of praise for effort or thinking. Ridicule you report forces 1. hole Exactly one piece of praise for effort has no level. | Flat or dismissive · Praise only for right answers · Praised effort, 2+ times · Mistakes used, disagreement OK |
| Trying when unsure D4 · Student Confidence & Risk-Taking | Rumi heard, you confirm | Level 3 when two or more children try while openly unsure. hole Exactly one unsure attempt has no level. | No one tried · Tried only when sure · 2+ tried while unsure · Tried unasked, kept going |
| کام Working: When children work, what do they do? | |||
| Children's choices C4 · Student Agency & Voice | Rumi heard, you confirm | Level 3 from one real choice that the teacher acted on. | Teacher decided everything · Choices on small things only · A real choice, acted on · Children led part of it |
| Switching activities C5 · Classroom Management | you saw | Your answers to "Did the class switch activity?" and "How did the switch go?". Only parts with a switch count, and the weaker one decides. No switch in either part is N/A. | Took long, no clear signal · Signal given, but took long · Changed within a minute · Children knew what came next |
| Group work C7 · Student Grouping & Collaboration | you saw | Your answers to "Did children work together?" in the two parts; the higher counts. "The room can't seat groups" is N/A. | No pair or group task · Sat together, worked alone · Made one thing together · Had roles and reported back |
| Books and materials C8 · Use of Manipulatives & Learning Aids | you saw | Your answers to "Who used books or materials?" in the two parts; the higher counts. "Nothing suits this topic" is N/A. | None used, though needed · Only the teacher used them · Children used them · Children explained with them |
| Children answering each other D3 · Student-to-Student Interaction | you saw | What you ticked when you sat with a group, in either part; the highest tick counts. With no group work, Rumi judges from what it heard near the recorder. hole A single exchange between children has no level. | No child spoke to another · Spoke, but no one replied · Replied to each other 2+ times · Disagreed, then agreed |
| سمجھانا Explaining: How well does the teacher explain? | |||
| Accuracy of content F1 · Content Accuracy | you pick | No counting rule yet (Section F is unreviewed): you pick from Rumi's moments. A wrong fact you report forces 1. | Wrong content left standing · A slip not corrected · All accurate · Accurate, misconceptions met |
| Subject words F2 · Use of Academic Language | you pick | No counting rule yet: you pick from Rumi's moments. | Everyday words only · Terms used, not explained · Terms used and defined · Children made to use them |
| Explaining why F3 · Depth of Explanation | you pick | No counting rule yet: you pick from Rumi's moments. | 'Do it this way' only · Some why, mostly memorising · Explained why it works · Children reasoned it out |
| Links to real life F4 · Relevance & Connections | you pick | No counting rule yet: you pick from Rumi's moments. | No links to life or subjects · Link mentioned, not explained · Link explained clearly · Children made their own links |
Six rows are asked in both parts, so the rule needs to know which part counts when they differ. For who got to answer, group work, materials and children answering each other, the question is whether it happened at all, so the higher part counts. For switching activities, the question is whether the class settled each time, so the weaker switch counts. How many children spoke has its own rule: everyone from Part 1, plus those speaking for the first time in Part 2. These are our placeholders until Ahwaz's team decides (question 22).
A fixed rule has to give every count a level, and the rubric leaves some without one. Four rows (C1, C6, D1 and D4) have the same gap: level 2 describes "none of it", level 3 needs "at least two", and exactly one instance has nowhere to go. Take D1, depth of student reasoning. Level 2 is "every response restates what was given"; level 3 is "at least two students give a response showing reasoning". One child reasoning is neither. D3 has the same gap for a single exchange between children, and D2 never defines "spread across the lesson"; with two parts, our rule reads it as at least one child speaking in each. The rule scores the gaps 2 for now and marks them. That is a placeholder, not a decision (question 4). Ahwaz's principle from the call, "one level of evidence corresponds to one level in the rubric only", is exactly why each gap needs a level.
Ahwaz confirmed this half on 29 September as drafted: Rumi's moments with "yes, this happened" or "no, or not like this", then the coach's pick kept or changed with the first pick saved, then the debrief built from that. The debrief is the part the workbook left out and, Haroon told him, what worries coaches most: "how does this all feed into my debrief, because I've got to have that conversation with the teacher." We designed that link ourselves; Ahwaz's answer was "Sure, sure."
The workbook says: no section total, no overall score, no percentage. The sections measure different things, and a 1 to 4 scale can't be averaged meaningfully. On 25 September the product side agreed to move teacher reports to section percentages, and on 24 September the ICT team said heads and teachers keep asking for scores. Since 29 September, on NIETE's sandbox, every observation score a teacher or head sees is a band read off the percentage (Excellent, Good, Average, Below average, Needs support), and a head's STEPS view takes its ratings from coach visits only: S from Section F, T from Sections B and C pooled by points, E from Section D. Both assume a percentage exists, and the 17 have none. Our suggestion is a profile: each indicator's level in the same plain words, grouped by the four moments, with the priority and the one thing to try, and no number that adds rows together. What teachers and heads see, including how a profile becomes an S, T and E, is for Ali's and Asma's teams to decide with Ahwaz (question 12).
The workbook, the protocol, the field form, the pilot plan and the calls were written at different times, and they don't agree. Each line is either something the build needs settled, or something that would confuse coaches if we copied it as written. The call on 29 September settled four of them.
| What one source says | Where | Status | In plain words |
|---|---|---|---|
| "THE VISIT IS TWO 15-MINUTE SEGMENTS"; "unclear how splitting the class into two segments is helpful" | Protocol; a reviewer's comment on it (22 Sep) | Settled 29 Sep | Two parts. Ahwaz: few children are expected to speak while the teacher explains, and more once they practise, so the parts are compared. Version 2 had merged them; version 3 restores them. |
| A seating chart with a tick per child who speaks: "the bridge between DC and the coach" | Protocol, section 2; field form block 1 | Settled 29 Sep | Dropped. Ahwaz: "It's not that important, as long as they are capturing evidence in some method." How many children spoke now comes from two numbers in each part; Rumi's count of voices is the floor. |
| Six off-task sweeps serve B7 | Protocol; field form block 2 | Settled 29 Sep | Dropped. B7 is not among the 17, Ali had suggested dropping them, and Ahwaz agreed on the call. |
| Provisional scores for "the coach-owned rows", five minutes, before leaving | Protocol | Settled 29 Sep | No scores before the seal. Ahwaz: coaches "should just capture the evidence… The rubric-based ratings come at the end." The coach records evidence during the lesson, picks what to work on first, and the levels are added up after the seal. |
| A light independent tally of questions, praise, wrong answers and children's questions, "RUN ALWAYS" | Protocol; field form block 5 | Narrowed 29 Sep | Ahwaz wants evidence written down as it happens and accepted the form's questions, which count children who spoke but leave questions, praise and wrong answers to the recording. Whether the tally strips can go is question 1. |
| The coach scores "the seven coach-owned rows: B5, B6, C5, C8, C9, C10, D3" before leaving | Observation protocol v7; field form v7, block 7 | Out of date | Four of the seven are no longer in the framework: B5 and B6 are Section B, which was dropped, and C9 and C10 were excluded. Only C5, C8 and D3 remain. The coach records evidence for six rows instead, confirmed on the call (question 2). |
| "Every row says DC (scoreable from the audio) or HITL (needs a person in the room)" | Start here tab | Not true in the file | All 17 rows say DC, including C5, C8 and D3, which the protocol says need the coach. |
| "The core five: C1, C2, C6, D1, D2" | Start here tab | Conflict | The visit brief and protocol say "core six", and the 23 September call spoke of six core indicators. The sixth is never named (question 9). |
| "The DC draft arrives... confirm or change the drafts" | Start here tab; protocol | Conflict | The pilot plan (24 September) says Rumi's ratings are never shown to coaches, only its evidence. We follow the pilot plan. |
| Coaches use a blank form "for 1 to 1.5 weeks", then go back to a pre-filled one | 25 Sep meeting with Qurat-ul-ain and Rifat | Conflict | On 23 September the evidence collection was to stay blank "in perpetuity", with only the adding-up pre-filled (question 11). |
| "No section total, no overall score and no percentage" | Start here tab | Conflict | On 25 September teacher reports were agreed to move to section percentages; on 24 September the ICT team said heads and teachers keep asking to see scores; and since 29 September sandbox shows every observation score as a band read off the percentage (question 12). |
| "Photographs are optional (Sept 2026 decision)" | Protocol, section 6 | Contradicts itself | The same protocol later says the recorder-failure fallback is "a second reason the photo shot list is not optional." |
| "Tier B falls from 21 of 27 scoreable to 18"; "do not attempt 27 indicators by hand"; "FICO v4.3" | Protocol | Stale | Written for a 27-indicator version. The coach-facing text needs rewriting for 17 before anything is printed or quoted to coaches. |
| Section F "NOT REVIEWED in this pass: rubric text carried over from v1" | Section F tab | Open | F's levels don't follow the new conventions. Level 1 of F1 means "stated something wrong", not "not observed", and no level 3 is a count. So no rule can add F up (question 14). |
| "Every level 4 begins 'Beyond level 3' and names ONE additional behaviour" | Start here tab | Minor | C1, C2, C6, C7, D3 and D5 each name two behaviours at level 4. The rule needs both, which makes level 4 harder to reach than the convention implies. |
| Every level 3 "carries a number you can count to" | Start here tab | Gap | Five rows (C1, C6, D1, D3, D4) give exactly one instance no level, and D2's "spread" is undefined. Ahwaz's principle on the call, one level of evidence to one level, means each gap needs closing ("How each indicator gets its level", question 4). |
| The visit table's rows "T+0 to T+1 day", "T+1 to T+2 days", "Within 5 days"; the incident log and capture note rows | Protocol | Empty | These cells are blank in the file (downloaded 28 September, unchanged on 30 September). What happens between the seal and the conversation isn't written down anywhere except the pilot plan. |
| Roll out to "some coaches, at random"; then "why not all"; then "we can do all" | 23 Sep call | Undecided | The pilot plan names 16 coaches who pass a calibration week first. Not discussed on 29 September (question 10). |
| Go live on Monday 28 September | 23 Sep call | Superseded | On 29 September the next step became a sandbox build for Ahwaz to try. The pilot plan still freezes the rubric after a calibration week, "no changes after this" (question 10). |
| D4 was "flagged as overlapping" D1 and its level 4 reworded | Excluded tab; comments by Qurat-ul-ain, Abdul Waheed, Ali | Check | Ali suggested excluding D4 "as currently phrased". The reworded D4 is in Section D. Is the overlap settled? |
| C8's level 4: "students use resources purposefully to explain their thinking" | Section C; Ali's comment on the excluded C8 | Check | Ali suggested a hands-on reframing of level 4; it isn't reflected. Fine if deliberate. |
Grouped by what they hold up, each with its status after the call on 29 September and the answer we would go with if we had to decide today. Questions 21 to 24 are new from the call.
Statuses as of 5 October. Where the sandbox build or the 2 October call has answered one, it says so.
Answered settled on the call · Narrowed the call moved it; a short confirmation is left · Open still needs an answer · New raised by the call
/observe today, Rumi shows a pre-visit brief built from that teacher's own recordings: what's working, where support helps most, and the trend. A coach who reads Rumi's view of the teacher before walking in isn't starting blank. Should coach visits on the new form hide it, at least for the pilot?These are ours to make. None of them waits on the impact team.
/observe there, and the portal's Observations page is read-only today. Audio clips are the one thing WhatsApp can't carry. We'd go: WhatsApp, with a link to the portal for the clips.This page assumes you were not in any of the earlier discussions. Fourteen moments matter.
| When | What happened |
|---|---|
| 27 Aug – 15 Sep | The scoring-honesty work on today's FICO. On 2,788 live lessons scored on the current form (37 indicators, 148 marks), 93% of teachers landed in one 20-point band, and level 4 was given in 5% of judgements. Fourteen indicators barely move and hand every teacher about 26 percentage points whatever they do. A shorter 26-indicator version was built on sandbox. On 15 September Haroon kept the 37-indicator form in production and held the new one back. |
| 14 Sep | The same lesson, different advice. The sandbox scorer scored the same 42 lessons three times each. Any two runs of the same lesson named a different "one thing to work on" 64% of the time. |
| 16 Sep | Ahwaz's team opens the workbook "FICO ICT – Observation Framework": 17 indicators in three sections, an observation protocol, and a one-page field form. Taimoor, Qurat-ul-ain, Ali and Abdul Waheed commented on it between 16 and 24 September. |
| 18 Sep | The "wobble". Sameer scored the same transcripts ten times over. On D2, eight of the ten runs came back split between level 1 and level 2. |
| 22 Sep | Rawal Town. Two staff observers independently observed the same three teachers. Across 210 judgements they moved Rumi's pre-filled score down 81 times and up none. The two pre-filled forms, made from two recordings of the same lessons, disagreed with each other on 12 indicators. Across all coaches the direction is the opposite (see below). |
| 23 Sep | The call this page follows from (Model Sovereignty + OSS, 83 minutes). Ahwaz presented the design: coaches record evidence on a blank form; Rumi scores the same lesson in the background; the pairs become the ground truth for testing models. Haroon took on turning the field form into a WhatsApp form, aiming for Monday 28 September. |
| 24 Sep | The pilot plan. 21 teachers, 16 coaches, 50 lessons, about 12 weeks. In it, the coach's provisional scores lock before any output from Rumi, and Rumi's own scores are never shown to coaches. |
| 25 Sep | Two meetings. With Qurat-ul-ain and Rifat, the protocol was flagged as too heavy for coaches. Agreed: the form coaches finally use must be streamlined, and teacher reports should show percentages. At the evals alignment, the 17 indicators were locked as the benchmark set, to be final by mid-October after the Islamabad pilot. Section B (lesson-plan fidelity) may come back. |
| 28 Sep | Versions 1 and 2 of this page. Version 1 turned the workbook's paper form into WhatsApp screens. Haroon found it hard to follow without training, so version 2 dropped the paper and asked eight plain questions after the lesson. |
| 29 Sep | Haroon and Ahwaz walk through version 2 (the last 20 minutes of an hour-long call). Settled: evidence is written down during the lesson, in two parts, with counts as numbers; no seating chart and no off-task sweeps; coaches aren't told which indicator they are scoring; everything after the seal stands as drafted. Version 3 of this page listed every change; notes on the call, with who said what, are in the project folder. |
| 30 Sep | Version 3 of this page set out the design after the call: the form filled in during the lesson, in two parts, with counts as numbers. |
| 30 Sep | Built. The form, the check and the brief went live on NIETE's sandbox WhatsApp number as /observe2 the same day, with every question reworded in plain, observable words. The evals team's first coach-and-recording pairs come from the sandbox tests. |
| 2 Oct | The go-live call with Ahwaz, Rifat, Annie and Ali. Go ahead with 16 coaches from Tuesday 6 October. Fidelity comes back, exactly as in /observe; the lesson plan is picked inside the form; no cap of three photos for coaches; the wording must explain itself, and Ahwaz reworks it; the "Work on first" list goes back to the team. |
| 2 Oct | Built and tested the same day. The plan pick, the plan check, the new wording, ten photos and a more reflective brief, on the sandbox, then a whole visit run through the sandbox number from WhatsApp. |
| 4 Oct | Riffat's test on the sandbox. The visit stopped after the check: no debrief recording was asked for, so no coach card and no report. Photos could only be taken there and then, not picked from the gallery. On a second visit a recording got a generic voice note back. Also asked: make everything except the form work exactly as /observe does, the start and the end. |
| 4 Oct | Fixed and tested the same evening. /observe2 now ends as /observe does (debrief, coach card, teacher's report, then complete), photos have their own step with upload from the gallery, and a recording sent while a form waits always reaches the form. A whole visit was run through the sandbox number to "complete". |
Positions are taken from the workbook comments and the meeting records. Where the transcript's speaker labels were unreliable, we give the position without a quote.
| Person | Position |
|---|---|
| Ahwaz, impact team | Coaches should record evidence, not judge. "The only mistake that they can make is in counting." Rumi's help belongs in adding the evidence up, and in pointing the coach to evidence they missed. Run the same lesson through both channels, measure the gap, and later correct scores from teachers' own recordings by it. The framework will keep improving: "we are not claiming that this is the perfect one." On 29 September: evidence should be written down as it happens, in two segments, with counts as numbers; coaches should capture evidence without being told which indicator it feeds; the seating chart and off-task sweeps are "not that important". |
| Haroon | A pre-filled form is poor ground truth, because coaches rarely change it. Calibration should happen in the flow of work, not by sitting coaches in a room with recordings. Committed to wiring the field form into WhatsApp. Asked when, if ever, pre-filling returns, since Rumi exists partly to save coaches time. His summary, which Ahwaz agreed with: "the time reduction comes in the aggregation, not in the evidence collection." On 29 September: "I am sold on what you've done in the framework thing." He agreed to every change above, and expects to use the method in every region if coaches take to it quickly. |
| Salman, evals | Needs pairs of a human score and a model score for the same lesson to test scoring accuracy; that work is parked until this data arrives. Asked why the form isn't given to all coaches, to get the data faster. |
| Sameer, impact team | Found that coaches rarely change what the model pre-fills, and measured how much the model's own scores wobble between runs. |
| Ali, programme | Coaches are not used to working this way; wants the current protocol alongside for comparison. Fidelity is no longer measured on this form, so the off-task sweeps may not be needed; a Time on Task indicator would use them. D4 overlaps D1. C9 and C10 should go (they did). On 2 October: start with a mix of experienced and newer coaches across the regions, since experienced coaches go beyond the form on judgement and newer ones need its structure; and make the debrief more reflective, so the teacher reflects too. |
| Qurat-ul-ain and Rifat | Coaches already juggle thirteen or fourteen manuals; a protocol this detailed will meet resistance. Teachers find raw scores confusing and would understand percentages. On 2 October: the lesson plan picked in the same form, with a chance to confirm or change it; more photos than three, with a line on what to photograph; and "Work on first" reviewed by the team as broader growth areas. |
| Abdul Waheed, NIETE | The visit asks too much of coaches: tallying and timed sweeps divide their attention from the moments that make feedback useful. Section E is also coming and adds more to the same visit. |
| Taimoor, NIETE | Either every indicator should be observable by both Rumi and the coach, or split them cleanly: audio indicators Rumi scores often, visual ones the coach scores on visits. "Right now we are somewhere in between." |
| Asma, ICT team | Heads and teachers keep asking to see FICO data. "We have to give them something," because they have been given a view from the start. |
Read from the NIETE database on 28 September, for the 30 days from 29 August to 27 September. A coach visit is a lesson a coach recorded and reviewed; a teacher's own recording is scored by Rumi alone.
| Claim | Who, when | Status | In plain words |
|---|---|---|---|
| About 2,000 coach visits in the past month | 23 Sep call | Right | 2,376 started, 2,289 forms submitted, 1,883 fully finished (form, debrief and report). |
| About 800 lessons a day scored by Rumi alone | 23 Sep call | Right lately | 829 and 801 per school day in the last two weeks, up from 228 at the end of August. |
| About 400 coach visits a day | 23 Sep call | Too high | About 110 per school day. One coach visit for every seven lessons teachers record themselves. |
| One teacher in four is seen by a coach, three in four by Rumi alone | 23 Sep call | Right, week by week | 24–25% of teachers each week. Over 30 days, 70% of the teachers reached had a coach visit. |
| Give a coach a pre-filled form and "90% don't even touch it" | 23 Sep call | Half right | Wrong per form: 77% have at least one change, and every coach with five or more forms has made changes. Right per score: 86% of Rumi's scores and 98% of its evidence notes go back as filled. Untouched forms come back a median 2.7 minutes after arriving. |
| Coaches mark Rumi's scores down, never up | Rawal Town, 22 Sep | Out of date | True for the first weeks (186 down, 24 up, July to mid-August) and for the two staff observers in Rawal Town. Since mid-August, 6,500 changes up and 1,526 down. |
| Coaches do three observations a day | 23 Sep call | Right, on the days they observe | A median of 3 on days a coach observes. Across all school days, including days with none, 2.1. Between 22 and 31 of the 55 coaches reach 12 in a week. |
| The debrief is a 10 to 15 minute conversation | The workbook | Not today | Today's recorded debrief comes a median five minutes after the form and lasts about four minutes. |
Two numbers the database stores about edits are wrong and were not used: every submitted form reports 27 "text changes", and rows marked not applicable are counted as rescored. Both come from how today's form is filled and read back; they are logged for fixing. The figures above compare Rumi's frozen draft with the coach's final version directly.