Debunking David Brooks on Education

Let's give Jack Ciattarelli a break for a bit and talk about NY Times columnist David Brooks (link to gifted article).

When Democrats are at their best, they are performing one job: reducing inequality and making American life more just. That’s what Franklin Roosevelt did with the New Deal and Lyndon Johnson tried to do with the war on poverty. That’s what Bill Clinton and Barack Obama did with their education reform policies. Both Clinton and Obama ran as education outsiders and change agents. In between those presidencies, Democrats worked with George W. Bush on the No Child Left Behind Act, which passed with a majority of 384-45 in the House and 91-8 in the Senate. No Child Left Behind was all about bringing accountability to America’s schools.

Because of those reform efforts, student achievement test scores in reading, math and most other academic subjects shot upward between the mid-1990s and about 2013.... [emphasis mine]

OK, stop right there. Even Brooks himself, in this very article, admits that test scores rise and fall due to many factors, including many that are outside of school: "Student outcomes are rarely just about what happens in the schools." So how can he possibly make such a sweeping causal claim about the effects of No Child Left Behind? 

I won't try to write a comprehensive summary of the research on NCLB here; suffice to say there's a good bit of work on the subject, and the conclusions present a complicated picture. Yes, there's some evidence of gains due to NCLB in some subjects, but not in others. And those gains might have to do with "teaching to the test," which is problematic. Schools did seem to change some of their practices in the wake of NCLB, but some of those changes may not be desirable. 

My point here isn't to make a claim about whether NCLB is good or bad; it's to remind us that changes in test scores are not, by themselves, good barometers of whether or not a particular policy was effective. Brooks, however, seems to be uninterested in such cautions:

George W. Bush had earlier warned of the “soft bigotry of low expectations,” but in the age of equity, schools moved to ease rigor and standards for poorer kids. Many schools stopped assigning whole books and started assigning short passages. What the education writer Tim Daly calls the education depression had begun.

Is it true? Did, sometime shortly after 2013, schools and, by extension, states move to lower their standards?

We can test this empirically—again, with some cautions. But let's talk about how accountability works in our schools first.

Under federal law and since NCLB was enacted in 2001, all states have had to test their students in Grades 3 through 8 in math and reading. The tests, however, vary from state to state, so there is no common scale on which to assess average student performance. However, the National Center for Education Statistics "maps" the results of state tests on to the "gold standard" national test: the National Assessment of Educational Progress (NAEP).

In the simplest terms: NCES looks at the percentage of students a state deems "proficient" on their own test. They then look at the NAEP data and determine what that test's score is for the same percentage of students who took the NAEP. That allows us to look at the differences in proficiency between states: which states set the bar at a relatively high NAEP score, and which set the bar lower.

Because NCES does this every time the NAEP is administered, it also allows us to assess how standards have changed over time. I took this data and averaged it across all states for each year of the NAEP (weighted by student enrollment). There are some cautions involved in this: for one, state tests might emphasize different content at different times, so increases in mapped scores might not reflect more "rigor." We're also assuming the NAEP is a consistent measure of student learning across time (a pretty reasonable assumption, IMO). But I'd argue this is our best estimate of whether state standards have, on average, moved higher. Have they?

Over the last two decades, the definition of "proficiency" on state tests has, on average, become more rigorous, not less.

This shouldn't surprise anyone who remembers that decade and the Common Core wars. By 2015, most states had moved to more difficult tests, and proficiency rates, as defined by those state tests, had decreased. It wasn't that the kids were performing worse; it was that the tests were harder because the standards were more challenging.

So, no, Mr. Brooks, states did not move "to ease rigor and standards for poorer kids." If anything, they did exactly the opposite.

I could spend a week rebutting everything in Brooks piece, but I'll confine myself to one more thing:

We’ve now had 12 years of terrible education statistics. You would have thought this would spark a flurry of reform activity. And it has, but in only one type of people: Republicans. When it comes to education policy, Republicans are now kicking Democrats in the butt.

Schools in blue states like California, Oregon and Washington are languishing, but schools in red states like Mississippi, Alabama, Tennessee and Louisiana, traditional laggards, are suddenly doing remarkably well. Roughly 52 percent of Mississippi’s Black fourth graders read at grade level, compared with only 28 percent in California. Louisiana is the only state where fourth-grade achievement levels have returned to prepandemic levels. An Urban Institute study adjusted for the demographics of the student bodies found that schools in Mississippi are educating their fourth graders more successfully in math and reading than schools in any other state. Other rising stars include Florida, Texas and Georgia.

One of these days I'll get to why those Urban Institute adjusted scores need to be approached with great caution.* But let's focus instead on Brooks's claim about red vs. blue states, and go back to the NAEP scores. Here are Grade 8 reading scores for the seven states Brooks cites.

Remember: Brooks's claim is that, sometime around 2013, Democratic states abandoned accountability and saw their scores "languish," while Republican states resisted and saw their students "do remarkably well."

Well, that's certainly not true for Alabama. The "miracle" state of Mississippi, which pundits claim has shown massive gains due to its use of the "science of reading," actually has a worse score than it had in 2003. California was always a laggard, but has moved up in relative position since 2003. There's no doubt Oregon and Washington have seen declines since 2017, but much of that is post-pandemic: it is quite possible different states were affected in different ways by Covid-19.

Which is the point here: even Brook's cherry-picked example is full of caveats; the data just doesn't support his sweeping statement. And yes, I do mean "cherry-picked." Does it strike you as odd that Brooks chose three Pacific coast examples to represent all blue states? What if, instead, he chose New Jersey, Connecticut, and Massachusetts? 

It's quite a different picture now: even with pandemic learning loss, these blue states maintain their relative position. Of course, I'm just looking at one grade and one test... but, again, that's the point. You can't make sweeping generalizations about the efficacy of particular policies by cherry-picking the examples you want. 

Do I think the NAEP has its place in education policy debates? Yes, I do. But what Brooks is doing here is really no better than HHS Secretary Robert F. Kennedy, Jr. "making the proof" that links Tylenol to autism. Pundits who wade in education policy should do better.

* For now, I'll just say that the definition of "poverty" should change when the cost of living varies so much between states. This is especially true when your poverty measure (free and reduced-price lunch eligibility) is becoming increasingly unreliable. Urban's adjusted scores do not take this into account. It's a problem, admittedly a complex one. More later.