Joel Klein: As Excellent As He Says He Is? Part III

Joel Klein is one of America's foremost proponents of corporate education "reform." He asserts that public education is in such a crisis that it constitutes a national security threat - a threat that can be dealt with by buying technology from the firm he runs for Rupert Murdoch, Amplify. He also believes that the "reforms" he and Mayor Michael Bloomberg instituted in New York City - including mayoral control, school closings, charter expansion, and test-based teacher evaluations - should be replicated across the country, based on what he terms the "compelling" improvements made under his tenure.

But does the record support his claims? What is the legacy of Joel Klein in New York City? I'm looking at the evidence to see if the facts support Klein's boasts. Here's the series so far:

Part I: Joel Klein has no problem twisting the facts to suit his ends. Has he done the same thing when crowing about his "success" in New York?

Part II: When you break down national test scores by student subgroups, Klein's "success" in New York isn't very impressive; in fact, it's downright disappointing.


Once again, here are Joel Klein's claims about New York City's improvements on national tests under his tenure as chancellor:

Finally, on the federal tests — the National Assessment of Educational Progress — which are taken on a sampled basis every couple of years, the city showed improvement on each of the four tests — with significant improvement on three of them. 

Indeed, in the fourth grade, the city’s students today are about a year ahead of where they were in reading and math when Bloomberg started.

Now there are two things to consider here: first, is what Klein is saying true? Second, if it is true, does it matter? In other words, even if New York City's students made "significant improvement" on the NAEP, can those improvements be attributed to Klein and Bloomberg's policies?

In Part II of this series, I related an analysis of NYC's NAEP scores by Leonie Haimson and Elli Marcus that broke down the student population into several different subgroups and compared NYC with nine other large cities around the country. From this view, NYC's progress on the test is not very impressive; some groups, like non-poor 8th Graders, actually lost ground under Klein.

But what if we looked at the total average scores on the NAEP? Compared to other big cities, did NYC get better gains? Using the NAEP Data Explorer, let's take a look. The data tool allows us to compare NYC to the nine other large cities for which we have NAEP data going back to 2003.

We'll start with 4th Grade, where Klein claims students have made impressive gains:

Well, NYC started off as one of the better large cities, and ended there as well. Yes, there were some gains... but not really any better than any other city:

Was there a gain? Yes. Is that gain any more impressive than the other major cities? No, not really. It's important to understand that even poor old Cleveland made gains on this test; that suggests the gains New York City had can be attributed to national-level factors, and not to any specific policies the city implemented. Considering how mediocre NYC has been compared to its national sister cities, it makes no sense to me that Klein can claim "compelling" evidence for the success of his policies.

Let's look at 4th Grade reading:

Yes, some gains -  but anemic gains, as compared to other cities. NYC is not in Charlotte's league on overall scores, and not in Atlanta's league when it comes to gains.

I've got the charts for 8th Grade below, and they even more unimpressive. Yes, NYC made some gains, but they are even weaker, in comparison to other cities, than the 4th Grade gains.

Let's stop here and acknowledge something important: we don't even know if these gains are educationally significant. Yes, according to some, a 10-point gain on the NAEP is very roughly equal to a year's worth of learning (I have a problem with this assertion, but we'll save that discussion for another day). But demographic and other changes could be as responsible for score changes in New York as anything else. And there may be issues with the test itself and the way it's scored that lead us to believe the gains are smaller than what the test scores report.

This is a long discussion for another day. What clear from both my analysis and the paper by Haimson and Marcus is this: under Joel Klein and Michael Bloomberg, New York City's NAEP scores did not increase as much as in many other major cities in the United States.

Now, Klein may argue that he NYC shouldn't be compared to Boston or San Diego; a more apt comparison would be to the other cities in New York State. I'll tackle this topic next; until then, here are the 8th Grade score comparisons:

"Compelling"? Meh...

This isn't going so well...