COVID and "Learning Loss": Test Scores Should Not Be Our Immediate Concern

I'll get back to Lakewood in a bit, but I want to take a minute and talk about a new report out yesterday from JerseyCAN and others about learning loss and COVID-19.

Regular readers know I've had my issues with JerseyCAN over the years: too many times, their analyses have missed the mark, often because of a lack of understanding about things like what test scores actually represent and how they should be used to direct K-12 education policy. 

I might make similar complaints about this latest report. Using a convenience sample's outcomes on one test to extrapolate outcomes on another for an entire population is inherently problematic. Further, changes in proficiency rates on the same test are often artifacts of the process of setting those rates, and not indications of any changes in actual student learning. There's also a whole problem of equating "proficiency" across grade levels that tends to get pushed aside in these discussions. Plus the easy way we accept "grade level" as some sort of absolute when it's really a social construct...

But in the end, none of that really matters, because what the report is showing is almost certainly correct: student learning has suffered during the pandemic, and the losses are almost certainly greater for students of color and those in economic disadvantage. 

How could this be otherwise? There is a clear digital divide along racial and ethnic lines in this country; when learning moves on-line, the effects will reflect that divide. And, as I've shown repeated, access to in-person schooling has also been racially unequal, a reflection of the inequalities in New Jersey's school funding system. These factors are undoubtedly combining to disadvantage students of color and students in economic disadvantage. 

I suppose there may be some value in presenting these losses as changes in test outcomes, as there are plenty of policymakers in the state who act as if test scores are the only measure by which we can evaluate the quality of schooling. But in all honesty, the report is telling us something we already knew: the pandemic has been bad for students, and worse for some than others.

So if JerseyCAN had stopped there, I'd really have very little problem with their report -- at least, the problems I'd have would be confined to technical details. But they took upon themselves to tell the state what it must do given their findings:

JerseyCAN recommends that stakeholders and policymakers consider the following solutions to help accelerate student learning for our students.

• Urgently prioritize the adoption and statewide implementation of extensive summer programming so that we can stem the COVID slide now and further stop more students from falling behind;
• Adopt and implement personalized, research-based solutions for accelerating student learning like high-dosage tutoring;
• Allow parents to exercise their choice to retain or hold back their child, if desired, to provide additional time to students for learning and the provision of social and emotional supports;
• Incentivize all districts to adopt high-quality instructional materials that are aligned to statewide assessments, which canprovide teachers and parents with ongoing information about student academic growth and that can project proficiency on NJSLA; and
• Administer statewide assessments in Spring 2022 that are comparable to those administered in Spring 2019 to establish a new baseline from which to measure student growth moving forward and to also enable comparisons to pre-pandemic statewide proficiency.

Let's take these one at a time.

• I am all for high-quality summer programming. Summer slide is a real issue, especially for economically disadvantaged children. But if our metric for determining that "loss" is test scores, we're running the risk of creating programs that sit students in front of screens all summer long doing test prep. As an educator, it is my opinion that this is the last thing kids need after this very difficult year. How about a lot of physical activity and socialization, music, art, free reading, exploratory learning -- what those of us in the education research field refer to technically as "fun"! In fact, we could do that, and add access to counseling and health services, family engagement programs, early childhood education... golly, if only there was a model of schooling that did this...
• Tutoring can be good -- but you have to remember a few things. Tutors need training. Good tutoring is not cheap. Tutoring is a supplement for high-quality curriculum and instruction, not a replacement. Again, if all we're doing is getting lower-paid workers to sit in classrooms with kids while they drill on test-prep instruction, we're missing an opportunity to help kids address the trauma they've experienced this past year.
• Expanded grade retention is probably one of the most overhyped policies that has been pushed over the past couple of decades. There is good evidence that the games Florida has played with retaining kids was behind its unearned status as an educational "miracle." While I am certainly sympathetic to the idea of giving parents more of a say in the decision to retain their child, and while I do believe there are times when it is warranted, pushing retention immediately after a cohort has shown learning losses doesn't make a lot of sense: if everybody suffered, why hold only some kids back? Besides, are our schools really ready for a mass mixing of age groups? 
• Everyone is for high-quality materials. But if the primary goal of those materials is to increase test scores, they will run the risk of emphasizing only those skills needed to pass the test, which can narrow the curriculum and constrain instruction. I am not against using test scores judiciously as a data point in helping to assess a child's learning, but state tests are simply not a practical, nor particularly useful, way to inform student instruction -- especially when teachers can't see the exam.
• There are always items put in different administrations of standardized tests to help equate scores. And, to the extent possible, proficiency rates should be equivalent between different years. So, sure, we can use test outcomes to help determine where we stand. But what really matters is this: What are we going to do with those results?

This gets to the heart of my concerns about the premise behind JerseyCAN's report, which I see manifesting in the statements of other education policy stakeholders. We are coming through an unprecedented crisis in modern American history. Children are hurting. Families have been traumatized. Schools have suffered losses in their staffs. We haven't faced this kind of trauma since World War II. So what are these folks proposing to help kids?

Testing. Analyzing test scores. Instruction based on test outcomes in two areas of the curriculum. Tutoring in support of that test-centered instruction. Then... more testing.

Look, I like a good run at the data as much as the next guy. I've said consistently testing has its place. But our kids are hurting, and the last thing we should be doing is trying to recreate the test-obsessed pedagogy we were using before the pandemic. Further, we shouldn't be going back to the test-and-punish systems we used to hold schools and educators "accountable," which was doing nothing to address the root causes of unequal educational opportunity, like inequitable school funding.

We should instead be focused on getting kids to love school -- especially if they weren't loving it before the virus hit. We should get kids and families access to the health and emotional supports they need so students can arrive at the schoolhouse door ready to learn. We should be investing in our school infrastructure to make our buildings safe and healthy. We should be expanding things like arts education, a proven strategy for boosting student engagement and learning.

A less stunted, more comprehensive view of schooling is required at this time. Testing can be part of it, but students, not test scores, should be the focus.

Now, back to Lakewood...