Pandemic Learning Loss: Part 2
Last week I discussed the problem of pandemic learning loss. This week my focus is on solutions: What can be done, what the challenges are, and whether we're making progress. The data on progress illustrate some classical misuses of statistics. (And, a classical political problem: You risk hurting your own cause by airing questionable data.)
Recap
—In 2020-2021, American students at every grade level made less progress than their peers did in previous years. This "pandemic learning loss" is greater among economically disadvantaged students, African American students, Hispanic students, and low-performing students.
—Pandemic learning loss is a serious problem, but its seriousness may be overstated. (a) The pandemic exacerbated rather than created academic weaknesses. (b) Learning didn't stop during the pandemic. (c) The true extent of learning loss may be overestimated (owing to unusual testing conditions, highly stressed students, etc.)
Challenges
2021-22 was supposed to be the comeback year, but numerous challenges made it difficult to address learning loss and other problems:
—Omicron emerged mid-year, causing shifts back to virtual or hybrid instruction for varying periods of time.
—Schools experienced shortages of teachers and other staff, owing to a combination of people leaving the profession (at least temporarily), people getting laid off, and new, federally funded positions not getting filled. Meanwhile, educators who stayed put experienced more than the usual stress.
—Students exhibited more disciplinary problems and poorer attendance than usual, owing to increased stress and difficulties adjusting to school re-openings, new rules, mid-year changes to the rules, etc.
Strategies
In Appendix A, I briefly describe where we can find guidance on addressing pandemic learning loss. Here I discuss three types of strategies that can help, along with some statistical context for each:
1. Academic strategies: The example of tutoring.
Studies show that tutoring can boost the performance of students from all backgrounds and grade levels. Although best delivered by a teacher, tutoring can be carried out effectively by anyone who receives guidance on how to do it. The strongest effects are observed when it's intensive (e.g., three or more sessions per week), conducted during the school day, capped at three to four students per tutor, and accompanied by informal assessments of student progress. Consistent with these findings, the vision of of Secretary of Education Miguel Cardona is to "give every child that fell behind during the pandemic at least 30 minutes per day, three days a week, with a well-trained tutor who is providing that child with consistent, intensive support."
This school year, most districts have increased the extent of tutoring they offer, thanks in part to federal support through the 2021 ARP Act (more on that later). Another exciting development is that on February 25, a bipartisan group of Senators introduced the PATHS to Tutor Act. If passed, this act will support partnerships between schools of education and community organizations, providing opportunities for teacher candidates to tutor students in high-needs schools, while offering compensation and/or clinical hours for the tutoring work.
Statistically speaking, what seems most promising to me is that according to a large (and methodologically strong) meta-analysis conducted in 2020, the most optimal tutoring approaches raise student performance on standardized tests by as much as half a standard deviation on average. This extent of improvement would offset, or nearly offset, what students tend to "lose", according to most studies.
2. Mental health strategies: The example of counseling.
The pandemic has been emotionally trying for everyone, including students. Both research and informal observation tell us that over the past two years, students have experienced more than usual amounts of stress, depression, suicidality, isolation, and other mental health challenges. Counselors can help students deal with these issues. In addition, depending on the grade level of the students, school counselors may provide academic support, assistance with college preparation, guidance on healthy behaviors and relationships, and outreach to individual students who experience trauma.
Studies show that counseling benefits students' mental health, academic performance, graduation rates, college admission, and so on, particularly when a comprehensive school counseling framework is used. In such a framework, counselors act proactively to disseminate preventative programs at the school and classroom level rather than waiting for problems to arise and then addressing them.
In spite of the demonstrated benefits of counseling, here are three statistics which indicate that American students are underserved:
—Only 30 states require public schools to hire a counselor.
—Minimum student-counselor ratios are only mandated in 16 states and range from 250:1 to 750:1.
—Nationally, the average student-counselor ratio is 424:1.
Economically disadvantaged students, and students of color, are disproportionately underserved by school counselors. Meanwhile, nearly 2% of American students attend a school that employs a police officer but not a counselor.
Secretary of Education Cardona's vision is for every student to have "access to a mental health professional – whether through their school or through a community-based organization" and for every high school to have "at least one career counselor", but these goals have been more difficult to achieve than those for tutoring. Although most schools who don't have a counselor want to hire one, supply has not kept pace with demand. Counselors, like other educational professionals, have been less eager than usual to work in schools this academic year, owing to concerns about the lingering presence of COVID-19, the prospect of spending at least some time counseling students virtually, and other pandemic-related challenges. In addition, some administrators are reluctant to hire counselors owing to budgetary constraints. Even if they've received federal funding through the 2021 ARP Act, they may not want to use the money for hiring, since it's unclear what will happen when ARP funds run out in 2024. (By contrast, administrators tend to see tutoring as a better use of the funds, because even a couple of years of expanded tutoring could have enduring benefits for students.)
3. Technology strategies: The example of internet access.
Contributors to pandemic learning loss include lack of (or difficulty accessing) reliable internet connections, as well as inadequate family support for using internet-based instructional platforms. These problems are especially prevalent among economically disadvantaged students who are, in turn, disproportionately students of color.
Better internet access, and more support for internet-based instruction, would have helped prevent pandemic learning loss, particularly when instruction was partly or completely conducted online. Studies have linked internet support to better academic performance (although benefits are not necessarily observed if families are provided internet access but no guidance). In addition, remote learning can be effectively supported by means of technologies that don't rely on the internet. For example, the non-profit organization Learning Equality has developed a platform that allows students to do schoolwork on their devices at home, then bring the devices to school periodically and sync with the teacher's server (e.g., a laptop), where the teacher can see their progress, provide feedback, and assign new work.
Funding
Support for the kinds of strategies I've discussed (intensive tutoring, counselors, educational technology, etc.) is coming from many sources.
For example, the American Rescue Plan (ARP) Act, passed in March of 2021, included over 122 billion dollars of emergency relief to P-12 schools, as well as separate lines of support for special populations (e.g., students with disabilities and homeless students) as well as college students. The educational goals of ARP were to reopen or keep schools open, combat learning loss, and support student mental health. As of now, all 50 states, as well as Washington DC and Puerto Rico, have received ARP money and used it to reduce class sizes, improve ventilation in school buildings, provide staff with personal protective equipment, hire new staff, establish tutoring initiatives, increase internet access, and develop programs that address learning loss.
Progress
I believe that American educators and other stakeholders have the determination, skills, and resources necessary to end pandemic learning loss. Many people are working to make that happen. But it's too soon to know how successful we've been, because students are still experiencing the effects of the pandemic, and we don't expect these effects to be fully addressed in the short run.
Unfortunately, there's been a rush among well-meaning experts to present data showing that new strategies are working. I think this is dangerous. When the data are preliminary, or otherwise weak, you risk losing further support for what might actually turn out to be helpful.
Following are two particularly troubling examples of this rush:
1. In February, I read about a study showing that virtual tutoring had a "modest" impact on reversing pandemic learning loss. That sounded encouraging, so I looked at the actual study. There I discovered that the researchers reported positive but statistically insignificant effects of tutoring on middle school math and reading.
You read that right. Statistically insignificant. In other words, not significant. The researchers (and journalists who wrote about the study) were enthusiastic about the findings, because the tutoring was low-cost (i.e., provided by college volunteers) and low-dosage (i.e., two half-hour tutoring sessions per week). But the significance game cannot be played this way. If a finding isn't statistically significant, it doesn't count. (In Appendix B, I explain, in plain English, what statistical significance means.)
We know that tutoring helps students. But when stakeholders (teachers, admins, non-profits, etc.) try to get further support for tutoring, it won't help to have studies floating around that praise tutoring on the basis of nonsignificant effects. Studies like that are just fodder for some ideologue to say: "Look, the findings weren't actually significant. Don't waste tax dollars on tutoring."
2. On March 11, the White House released a statement on the anniversary of the American Recovery Plan (ARP) Act that included the following two bulleted items:
—"Schools have gone from 46% open before ARP to 99% safe and open today"
—"ARP led to record growth in local education jobs that are critical to meeting students’ academic and mental health needs"
The ARP Act is a major legislative achievement and greatly beneficial for American education. But these two bullet points don't hold up well under scrutiny.
Regarding the first item, ARP deserves some credit for that 99% stat, but school re-openings are attributable in part to other factors too, including the decline of the delta variant, the availability of vaccines to students, and (related to those developments) ongoing pressure from parents and other stakeholders.
Regarding the second item, ARP deserves much credit for education-related hiring, but the data are hard to evaluate. During the pandemic, individuals left jobs and, partly independent of that, positions were shut down. Since ARP came on line, jobs have been filled, and positions have been opened or re-opened, though not necessarily filled. It's difficult to tease apart these different phenomena and identify exactly what ARP has done (other to put a lot of cash into district hands).
ARP is not a bipartisan bill. All of the Democratic Senators, and none of the Republicans, voted for it. For all the good it has done, references to 99% of schools being open, or to "record growth" in local education jobs, are just easy targets for Republicans who want to say that ARP was a bad idea.
To be fair, Republican legislators want to end pandemic learning loss too. They simply disagree with Democrats about what to do and how much to spend on it. I have a strong feeling that in the run-up to the 2024 presidential election, you'll hear Republicans complaining about ARP costing too much and doing too little. We need better stats to counter those complaints.
A suggestion
We will overcome pandemic learning loss, thanks to the dedication of educators and other stakeholders who want the best for our students. But I think we should also be careful how we frame concepts like learning loss and gain.
In recent decades, student achievement has been increasingly defined in terms of performance on state-mandated achievement tests, and so, not surprisingly, pandemic learning loss is defined in terms of lower-than-expected performance on those tests. However, the full extent of what we want students to get out of school isn't captured by achievement test scores. There are limitations to what the stats can tell us.
Teachers and their students are living through one of the most significant events of the 21st century so far. In my opinion, they need more time to pause and reflect on the meaning of the pandemic, for society and for each of them as individuals. This would lead to much learning, and many opportunities to foster a sense of community. But…those achievement tests are on everyone's minds, because student performance continues to have consequences for teachers, administrators, districts, etc...
My suggestion is that we should reframe pandemic learning loss, and the reversal of that loss, as something broader than just changes in test scores. We should call for a moratorium on testing (other than the relatively informal, formative assessments that teachers use to check on student progress). We should prioritize student mental health, and a sense of community, and use the time formerly spent on testing and test preparation to focus on instruction.
Appendix A: Guidance on reversing pandemic learning loss
We know a lot about coping with pandemic learning loss, thanks to studies on reversing the effects of other kinds of disruptions (summer break, Hurricane Katrina, etc.), studies that address achievement gaps, and studies that focus specifically on the effects of the pandemic.
Fortunately for busy educators, the results of these studies do get distilled into useful, practical guidance. Some guidance can be found in comprehensive documents (see here, for example), while some is available through expert clearinghouses. The best example I've seen is the EdResearch for Recovery Project, overseen by the Annenberg Institute at Brown University and Results for America. EdResearch for Recovery asks leading researchers from around the country to provide evidence briefs of relevance to the post-pandemic recovery of American education. Learning loss is one of the topics addressed, and researchers frame their recommendations in concrete, practical terms.
Finally, complementing a solid and growing research base, we also have opinions – a lot of opinions – on reversing pandemic learning loss. These opinions are grounded in personal experience rather than data. Most of these opinions make good sense (engage students, reach out to families, provide individualized support, etc.), although they seem like best practices under any circumstances rather than pandemic-specific.
Appendix B: Statistical significance
Statistical significance is a mathematical concept. What follows is a plain-English explanation, using the example of tutoring research.
Researchers study samples and generalize to populations. If you want to know whether tutoring helps 2nd graders with their reading skills, you can't study the entire population of 2nd graders. Rather, you work with a sample and generalize your findings to that population. Let's say, for purposes of this discussion, you decide to examine reading progress among 100 2nd graders who received tutoring for six months versus 100 2nd graders who did not. Your goal is to draw conclusions from this sample of 200 students that apply to all 2nd graders.
When we look at the results from a sample, we may notice patterns such as differences between means or correlations among variables. For example, let's say you find that the average amount of reading progress is greater among the 2nd graders who were tutored than among those who weren't. Can you trust this finding? After all, you only sampled a total of 200 students. Would you still expect to find an effect of tutoring across the entire population of 2nd graders?
To answer this question, you would run statistical analyses on the reading progress data you've gathered. The purpose of these analyses would be to determine whether or not the difference between the tutored and non-tutored groups is significant. Why is this the purpose? Because a significant effect, by definition, is one that's strongly expected to occur in the population. In other words, if your two groups of 2nd graders differed significantly in reading progress, then you would consider it highly like to see that difference across the entire population of 2nd graders.
Why do I say "highly likely"? Well, you can't be 100% certain that what you found with your 200 students would be observed in the population, because you didn't study the entire population. Statistical tests can only give you more or less confidence about what might happen if you had done so. So, we can think of a significant effect as one we expect to occur in the population, with a degree of confidence that approaches - but never quite reaches - 100%. There's always at least a tiny chance of a fluke – i.e., an effect observed in the sample but not in the population.
Researchers choose the cut-off for what would constitute a fluke. That value is called an alpha value, and it's often set at .05. An alpha value of .05 means a 5 in 100 chance that a finding observed in a sample is a fluke and would not be observed in the population.
Once an alpha value is chosen, statistical tests are run to determine something called a p value, which is the actual, calculated chance that a finding observed in a sample is a fluke. If the p value is smaller than the alpha value, we conclude that the finding is significant.
Now you can see how the game is played. If the 2nd graders who were tutored show greater reading progress on average than their peers, and you run a statistical comparison between their means that yields a p value of .03, then you would say that the difference is significant, because .03 is less than .05. Your statistical test is telling you is that there's only a 3 in 100 chance that the result is a fluke and therefore wouldn’t be found in the population.
(I haven't told you how statistical tests calculate p values, because the math is very involved and differs from test to test. Nonetheless, once you’ve obtained a p value, it's relatively easy to interpret.)
That .05 I mentioned is an arbitrary cut-off for significance. A p value of .03 doesn't imply much more confidence about the population than a p value of .07 would. But using a cut-off is part of the rules of the game. Regardless of whether you choose an alpha value of .05, or .001, or something else, if you compare two means and obtain a p value that's larger than the alpha, you have to say that the difference between means is not significant, and you can't conclude that the difference would be observed in the population. The sample means may differ, but that’s just one sample; you can't assert that you'd see this difference in the population.
There are lots of controversies around the use and misuse of significance testing. It's not the only game in town, and in recent years it has begun to lose favor among some scientists. But most research in fields like social science and education still rely heavily on it. My concern about the tutoring study discussed in the newsletter is simply that a nonsignificant difference between means was presented as a genuine difference, and that's not how mainstream science works at the moment.
(Those of you who are deeply versed in stats might take issue with some of my discussion here. However, describing statistical concepts in plain English is like translating poetry from one language to another: something inevitably gets lost in the translation. I've never seen a simple narrative description of significance that was completely faithful to the math. If you can create one, I'll post it to one of these newsletters and offer you "significant" praise.)