Mythbusting Chris Cerf at @NJEA: Poverty

Chris Cerf, New Jersey's very reformy Commissioner of Education, spoke on Friday at the NJEA convention. And the commish made it clear that he is gravely, deeply worried that he is being misunderstood:

The idea that anybody thinks that poverty doesn't matter is inconceivable. Of course it matters. Of course it is a huge predictor of academic and educational success. There's no doubt. But it is also true that schools matter. And schools matter a great deal. And to take the premise that poverty matters all the way to its most aggressive conclusion which is: "Hey, you know, what are you going to do about it?  Because it is such a determinate"... I think is neither a leap that the facts suggest not morality allows. 

So Chris Cerf was staking his case on the role of schools and poverty; he was trying to convince a room full of teachers that poverty is important, but that it can't be a distraction from implementing his reformy policies.

To bolster his claim, Cerf then showed one of the most mendacious, deceptive graphs I've ever seen used in a public policy debate*:

And here's what he said as it was shown:

And let me... each dot represents a school in New Jersey. On one axis is the poverty rate - free and reduced price lunch - one axis is performance. You've got a bunch of schools who have very low poverty and quite poor perfromance. And vice versa. And no one's saying there is not a predictive effect to this. All we are saying is, recognizing the influence of poverty, there is a continuum of success, and we should continue to strive to move our instruction, to move our schools, to move our educational offerings, to a point where we move children who are faced with an unfair set of challenges or burdens, higher and better than we are today.

OK, the Commissioner was speaking extemporaneously, so let's cut him some slack on that last sentence. But let's also have Bruce Baker explain why this graph is so utterly, shamelessly deceptive:

The title conveys the intended point of the graph – that if you look hard enough across New Jersey – you can find not only some, but MANY higher poverty schools that perform better than lower poverty schools.

This is a bizarre graph to say the least. It’s set up as a scatter plot of proficiency rates with respect to free/reduced lunch rates, but then it only includes those schools/dots that fall in these otherwise unlikely positions. At least put the others there faintly in the background, so we can see where these fit into the overall pattern. The suggestion here is that there is not pattern.

The apparent inference here? Either poverty itself really isn’t that important a factor in determining student success rates on state assessments, or, alternatively, free and reduced lunch simply isn’t a very good measure of poverty even if poverty is a good predictor. Either way, something’s clearly amiss if we have so many higher poverty schools outperforming lower poverty ones. In fact, the only dots included in the graph are high poverty districts outperforming lower poverty ones. There can’t be much of a pattern between these two variables at all, can there? If anything, the trendline must be sloped up hill? (that is, higher poverty leads to higher outcomes!)

Let me break down the good professor a little more for you (pay attention, Commissioner!):

There is an unquestioned correlation between poverty and educational outcomes as measured by test scores; not even Commissioner Cerf is going to try to argue with that. Granted, there are some schools that "beat the odds": they score better than what we would expect (and, sometimes, worse than what we would expect).

But it is either innumerate, deceptive, or both to suggest that the mere existence of outliers "proves" that the correlation between poverty and test scores can be overcome - especially with policies that do not address poverty itself.

Baker continues:

Note that the graph doesn’t even tell us which or how many dots/schools are in each group and/or what percent of all schools these represent. Are they the norm? or the outliers?

So, here’s the actual pattern:

Hmmm… looks a little different when you put it that way. Yeah, it’s a scatter, not a perfectly straight line of dots. And yes, there are some dots to the right hand side that land above the 65 line and some dots to the left that land below it.

BUT THE REALITY IS THAT FREE/REDUCED LUNCH ALONE EXPLAINS ABOUT 2/3 OF THE VARIATION IN PROFICIENCY RATES ACROSS SCHOOLS!

What Cerf did on this slide is essentially take the statistical "noise" - the variations that occur in any test due to random errors out of anyone's control - and use it as a justification for downplaying the effects of poverty on test scores.

Are there schools that consistently "beat the odds"? Almost certainly - although quite often, they are getting better test scores because the data we have isn't "finely-tuned" enough to let us see differences in student characteristics that may exist. Still, there probably are schools that consistently do a better job than others with equivalent student populations. We should identify these schools and study their practices - but we should also be honest about whether their results are replicable. In many cases - say, with the high-flying charters of Newark - they are not.

But this is where Cerf becomes, frankly, incoherent. Because even as he makes the case that poverty matters, he continues to insist we can overcome it with policies that do not address poverty itself:

That's not the same as saying poverty doesn't matter. Frankly I find that perspective to be offensive and very difficult for anyone to argue. 

And yet you showed that graph above, Commissioner...

The real question: what do you do with that fact? What do you do with it? Do you look for solutions or interventions or changes in funding or changes in instructional practice or changes in organization of a classroom or changes of who's in our classrooms? You don't just give up on the basis of some outside, extrinsic force.

Well, here's what Cerf, Chris Christie, and the NJDOE have done about it:

They've taken away the right of poor, urban districts to self-determination - just because they happen to be urban, poor, and full of people of color:

Jersey City, Paterson, Newark, and now Camden have lost any right to determine their own school policies. Why? Because they are full of beautiful, deserving children who happen to come from poor families. And what has decades of state control and the failure of the Christie administration to fully fund their schools as is required by law done for them?

Paterson

Trenton

It looks to me like "giving up" is exactly what the Christie administration has done. And yet Commissioner Cerf has the audacity to claim that his critics (including us "lowest forms of life") are saying we shouldn't do anything about reforming schools because we think poverty is intractable.

Let me be clear, Commissioner: I think there is quite a bit we could do to make these schools better, starting with fully funding SFRA, making the facilities in poor urban centers better than those at New Jersey's wealthiest private schools (you know, the ones where Chris Christie sends his own kids), and returning democracy and local control to all of our public schools. I suspect this would do much more to raise teacher quality and morale than imposing your innumerate evaluation system, which evidence suggests will punish teachers who work in these schools.

But hand-wringing about poverty while disenfranchising local parents isn't a strategy for confronting poverty; it's a cop-out. And abusing data in the service of this cop-out isn't helping anyone.


* To be clear: I don't know if this was the exact graph used - but if not, it was nearly identical. NJDOE has used this same graph before; if the one Cerf showed at NJEA differed, it was only in minor and cosmetic changes. If and when NJDOE releases Cerf's slide show, I will put the exact graph up here. Until then: I can report, without hesitation, that this graph was almost exactly identical, if not a perfect replica.