The Real Story of New York's Schools Can't Be Told In Test Scores

What is it about New York and test scores?

Test score releases always generate plenty of heat now matter where you go. Politicians and advocates and stakeholders all furiously send out their press releases right after the scores are posted, confident the "latest data proves" their particular agenda is "getting results," even though "we have a lot of work to do," and [INSERT CLICHE HERE].

But New York, in particular, seems to love debating the meaning of their scores. Maybe it's because there is a big education research community in place in and around the city. Maybe it's because parent advocates are particularly well organized, teachers unions are particularly vocal, and wealthy interests are particularly engaged in education issues (golly, I wonder why...). Maybe it's because the debates over mayoral control and charter school expansion are actually battles in larger political wars, especially the interminable power struggles between Albany and NYC.

Whatever the reason, this last score release by NYSED is once again being presented with little to no context, and little to no understanding of how tests work. Granted, I'm the first to admit I'm barely a few feet above sea level on the climb to the right of Mount Stupid:

But I know enough about standardized tests to know that it's pointless to compare proficiency rates from year-to-year on completely different tests. But don't take my world for it; ask the guy who literally wrote the book on testing:

Meanwhile, the researchers we spoke with said any comparisons between this year’s test and last year’s are inherently flawed. The state acknowledged Friday that the tests are not directly comparable because this year’s tests were shorter, for instance, and untimed.

“The state said in their release that the tests are not comparable,” said Dan Koretz, a professor of education at Harvard. “That’s sort of the end of the story. They’re not comparable.” [emphasis mine]

And yet the Kings of Mount Stupid -- aka the Wall Street Journal editorial board -- would have you believe these results cry out for the immediate, rampant expansion of charter schools:

According to new state testing data, citywide student proficiency increased this year on average by 7.6 percentage points in English and 1.2 percentage points in math to 38% and 36.4%, respectively. Some have attributed the city’s gains, which mirror those statewide, to shorter and easier tests.

Yet strikingly, proficiency at charter schools this year jumped 13.7 percentage points in English and 4.5 percentage points in math to 43% and 47%, respectively. In other words, charter students have improved by two to four times as much as the citywide average.

Yes, one year of noisy, biased, crappy fill-in-the-bubble test scores is all the Journal needs to turn over NYC's schools to Eva Moskowitz and her ilk. And there's no need to account for student population differences, or spending differences, or attrition, or suspension rates, or unobserved differences, or a whole host of other factors. Just shut down Tweed once and for all -- the tests prove we have to!

There's been talk that the state monkeyed around with test scoring this year to jack up the proficiency rates. Lord knows there have been been plenty of shenanigans in the history of New York's test score reporting. I'm not convinced as of yet, however, that the state has done anything more than attempt some basic year-to-year equating...

But that's actually pretty bad by itself. Why would you even try to equate two tests that are inherently different in form? Why not just admit you're setting a new base, and next year's noisy, biased, crappy results will show us...  well, honestly, next to nothing -- it's only one year.

Look, as I've said many times before: I think it's a good idea to have some sort of testing regime for system accountability purposes. We need to set some standards and see if we're meeting them if we're ever going to make a case for the reforms we need (equitable and adequate funding, elimination of child poverty, desegregation, and so on -- you know, stuff that actually works). School funding litigation has relied on test results to argue cases for funding reform, and that's a good thing.

Of course, we could get all the data we need for these purposes with a lot less cost and a lot less intrusion. But we'd still have to provide the appropriate context; only then can we tell the real story of our schools and our policies.

In that spirit, let me offer a few graphs that look over time at some of the things going on in New York State. This isn't even close to comprehensive, but it's a start. All of it is based on a new dataset we're going to be unveiling soon (so, yes, this post is basically a chance for me to give it a test run - sue me).

1) New York City is not the state's leader in charter school proliferation.

The data I have here is a bit lagged: 2014 means the spring of the 2013-14 school year. Still, it's fairly clear that NYC, even with its high-profile expansion of charter schools, is well behind cities like Albany, Buffalo, and Rochester when it comes to charter sector share. I have some reason to believe these upstate cities actually have greater charter shares*, but even on its face, NYC hasn't begun to match many Upstate cities in charter proliferation.

Can the NYC charter sector confidently claim at this point that it will keep its aggregate advantage over the city's public, district schools when 20 to 30 percent of the population is in charters? And what will be the consequences for the district? We know Buffalo's and Albany's district schools have paid a steep fiscal price for their high level of charter proliferation. Can we be certain the same won't happen in NYC?

Certain reformy types who cast themselves as "sector agnostics" will tell you that NYC is where chartering, like a bucket of chicken, has been "done right." I'd argue NYC hasn't even done chartering yet -- not like what's to come. Before the Moskowitz et al. are allowed to do whatever they want, maybe someone should explain how Gotham is going to avoid the problems charters have caused Upstate.

2) Upstate NY cities have seen a huge shift in the demographic characteristics of their students.

I wrote about this a few years ago, but it's still amazing when I look at it. The three big Upstate NY cities have seen a massive amount of white flight from their schools over the past two decades. Syracuse is especially shocking, but Buffalo and Rochester aren't far behind. NYC, by contrast, has had a fairly stable percentage of white students.

As this story from the Hechinger Report notes, the death of manufacturing Upstate has had severe consequences for children:

While gentrifying large cities like New York have grown increasingly prosperous in recent years, smaller urban centers like Syracuse have seen large increases in poverty. 

“Syracuse is now number one in the nation in terms of the percentage of black and Hispanic families living in high-poverty neighborhoods,” said Sharon Contreras, outgoing superintendent of Syracuse City School District. “I know kids can overcome poverty, but when students are living in abject poverty in segregated communities with poor health care and housing, that makes the work for our teachers all the more challenging.”

Which leads us to...

3) Upstate NY cities have seen a dramatic rise in childhood poverty relative to NYC.

Historically, student-aged poverty in NYC has been lower than that found Upstate, although the levels were close in the early 2000s. After the Great Recession, however, there was a significant split: age 5 to 17 poverty rose at a much slower rate in NYC than it did in Rochester, Syracuse, and Buffalo. And I'm not even counting the many smaller cities Upstate, like Utica and Albany, that have also been hit hard.

Before anyone starts making comparisons between NYC and the rest of the state, they ought to at least acknowledge these realities. I'm not saying the 5 boroughs don't have their problems with childhood poverty; I am saying Upstate has been slammed especially hard, and that's a real problem with any simple test score comparison.

One more thing:

4) The per pupil spending gap between NYC and Upstate districts has been changing.

I'm reluctant to post this because I have to attach so many caveats to it. This is purely descriptive data, adjusted for geographic and yearly differences in education costs. It's noisy and I'm only including the Big 5 districts; however, the big inequities in New York State education funding have as much to do with smaller cities as anything else. New York State has all kinds of "stealth inequities" built into its state funding system, and there's reason to believe that's led to some districts engaging in "inefficient" spending patterns.

In other words: the topic of New York State spending is far more complex than I can get into here. But I think it's important to point out that even as poverty has been on the rise in all Upstate cities, education spending, relative to NYC, has not. Syracuse in particular has taken a big hit over the last few years, even as its poverty rates have skyrocketed. Again, this data is noisy and the drop off over the past two years may be an anomaly; however, it does follow several years of declines. 

The point here is that if we buy into the assumption that children in poverty require more resources in their schools -- an assumption upheld by a boatload of evidence --  the large uptick in poverty Upstate should have been followed by a large uptick in resources. But we really haven't seen that since 2009, when Upstate childhood poverty began its sharp climb.

There's so much more I could add to this. But my goal isn't to try to tease out why test scores are coming out the way they are in New York; it's to show that simply pointing to differences in proficiency rates from one year to the next is bad enough, but it's made worse when you don't provide any context.

New York is a big, populous state with enormous differences in wealth. Many of its regions have undergone profound transformations over the last two decades. So simple claims of NYC "catching up" with the rest of the state are mostly meaningless, as are claims about charter schools "beating the odds." There's a much more complex tale to be told, and proficiency rate changes over a year just aren't adequate to the job.

Mercifully quiet this time around.

ADDING: Leonie Haimson gives us a NY test score history lesson:

Then there's the lack of any historical context. The paper of record has a lamentable record of failing to report on the well-documented evidence of inflated test score gains that occurred from 2003-2009, until the state itself admitted what had happened and re-calibrated the cut scores in 2010.  Their unshakeable credulity led to a front page story on August 3, 2009  -- a little more than seven years ago to the day, recounting the big jump in student achievement and giving credit to the Bloomberg reforms.  Like now, they refused to explain the multiple sources of evidence to the contrary, including the fact that the NAEPs showed only modest gains over the same time period.  My argument with the Times editors even made the Village Voice .   As Wayne Barrett wrote,

The Times front page piece last week -- headlined "Gains on Tests in New York Schools Don't Silence Critics" -- failed to quote any real critics, but gave Klein six self-promoting paragraphs. It did bury a single questioning quote from two academics not known as critics of the test scores in the thirty-fourth paragraph, but the top of the story trumpeted success scores that would have silenced any critic. If, that is, they were true. 

Two days after the article ran, the NY Senate voted to renew Mayoral control.  A few months later, Bloomberg was re-elected to a third term.   Sure enough, when the test bubble burst in 2010  all the gains were shown to be illusory. Even after that, though, in 2011 a writer for the NY Times Magazine reported that "since 2006, the city's elementary and middle schools have seen a 22-point increase in the percentage of students at or above grade level in math (to 54 percent) and a 6-pont increase in English (to 42 percent)." 

These statistics were completely fabricated of course, provided by DOE to the reporter, and somehow neither the reporter nor any editors had bothered to  check them.  It turned out the DOE had made up the data by re-adjusting the cut scores to where the state had previously put them, essentially rewriting history as though test score inflation and deflation had never occurred. 

All this is to say: If reporters at the NY Times and other media outlets are prepared to point out the unjustified claims promoted by public officials and some advocacy groups last week, that is good; but they might try also provide some context to explain the larger reasons for skepticism.

Remember:

* Matching charters to their sending districts is complex work, especially in urban areas that don't have highly concentrated core cities. There are, for example, charter schools in Lakawana just south of Buffalo. My method matches them to that suburban town, even though it's very likely much of their population is coming from the city itself.

Caveat regressor.