In international education, most literacy programs focus just on reading classes in the early grades. Additionally, these programs often use a coach, paid for by donors, to travel periodically to schools to support teachers in implementing the reading program. This seems logical because it is familiar.
However, as a researcher, I know to always test your assumptions. This focus on just foundational literacy programs was initially driven by simplicity, funding limitations, and the pressure of quick results. But can a literacy program exist in isolation from everything else going on in schools?
These two blogs will explore this idea. The first blog tells a story that frames how teachers are influenced by their peers in schools. The second blog is about data and evidence.
"What Works?" Even the Right Question to Ask?
I'll answer this question by way of a story.
A Long Time Ago in a School Far, Far, Away . . .
Over twenty years ago (!), I was a volunteer teacher at Chizongwe Technical Secondary School, a rural public secondary school in Zambia's Eastern Province. The school followed the old colonial boarding school model, with teachers and pupils living on campus.
The Teacher Attendance Problem
Although most teachers were within a couple of hundred yards of their classrooms, their classroom attendance was low, probably averaging about 50%, leaving the pupils sitting in class all day waiting and hoping for their teachers to turn up and instruct them. Concerned about this, I approached the headmaster to explain the issue. He called a faculty meeting and told the teachers to improve their attendance. The next day, most of the teachers were teaching in their classrooms. On the second day, about 70% of the teachers attended. Over a few days, teacher classroom attendance returned to about 50% — where it had been before my conversation with the headmaster.
The Environment Wins Every Time
I worked at the school for two years. Over time, I realised that this behavioural norm of teacher classroom attendance was pervasive and even influenced my conduct. If I had an "important" personal chore to do, such as going to the Post Office, I felt it could be prioritised over my classroom teaching and attendance. If I set the pupils work to complete, could I prioritise getting the errand done rather than being in class? Several times, I did. This behaviour is something I would never have dreamed of repeating when teaching in the UK or the USA. However, it was socially acceptable at Chizongwe.
The Exception
There was one teacher at the school in Zambia who was different. He had almost perfect classroom attendance and was the go-to for the pupils to arrange extra-curricular activities. I found it hard to keep up with him; I looked up to him, and I collaborated with him as much as he was willing. He was the bright spot amongst a faculty who, at times, seemed almost uninterested in the well-being and education of their pupils. This teacher, my old friend George Chileya, was the exception. I asked him once why he behaved the way he did. He replied:
"I don't know any other way."
What distinguished George from his peers? It was simply who he was.
These exceptional teachers are often framed in the context of positive deviance research. This research examines the behaviour and practices of these exceptions and suggests that others should adopt similar practices. But this misses the point. This type of research examines symptoms but not their causes. George behaves a certain way because of his personality. It's who he is. George cannot be replicated. He's a one-off.
Why Most Literacy Programs Have Impact
Impact evaluations or Randomised Controlled Trials (RCTs) provide a tidy answer to the question, "Did this literacy program have an impact on learning?" You get a single number, an effect size or average increase in correct words per minute. But that's a little too tidy. These outcomes ignore the reality that programs are implemented in schools and classrooms, and that implementation varies greatly. Most of the time, only a few teachers effectively implement the program and improve learning outcomes. These teachers are like George; they're the exception. So, when we measure the impact of a reading program, we're just looking at the stars. Even literacy programs with statistically significant impact and a "decent" effect size have, in truth, impacted the learning of very few children in the program.
Focus on the "Normal" Majority of Teachers
In truth, the normal teachers are the other teachers. The ones at Chizongwe with the poor attendance. What we should be spending much more time on is understanding why programs have very little influence on the instructional quality of most teachers.
Why "What Works?" Is Never the Right Question to Ask
So, even literacy programs with statistically significant impact and a "decent" effect size have, in truth, impacted the learning of very few children in the program. Literacy programs focus on the individual teacher, not understanding the influence the environment has on the teacher. Impact evaluations mask where something does not work and why. We don't do this because it is unintuitive and not a good story to tell a funder.
People in Glass Houses Shouldn't Throw Stones
By trade, I am a researcher and evaluator. I've done dozens of program evaluations. I should probably be called out for presenting lots of statistically significant effect sizes over the years, then writing about their limitations. But my observation and frustration have always been how generalised impact (including A/B testing) has not really helped us improve programming at scale. If I'm calling out issues with my own work and career, that means something — at least to me.
In the next blog, we'll look at the data and explore the challenge of literacy programs running in isolation from what I would call the Whole School Model. We'll focus on the normal, average teacher — not George.