I will protect your pensions. Nothing about your pension is going to change when I am governor. - Chris Christie, "An Open Letter to the Teachers of NJ" October, 2009

Sunday, November 10, 2013

Mythbusting Chris Cerf at @NJEA: Poverty

Chris Cerf, New Jersey's very reformy Commissioner of Education, spoke on Friday at the NJEA convention. And the commish made it clear that he is gravely, deeply worried that he is being misunderstood:
The idea that anybody thinks that poverty doesn't matter is inconceivable. Of course it matters. Of course it is a huge predictor of academic and educational success. There's no doubt. But it is also true that schools matter. And schools matter a great deal. And to take the premise that poverty matters all the way to its most aggressive conclusion which is: "Hey, you know, what are you going to do about it?  Because it is such a determinate"... I think is neither a leap that the facts suggest not morality allows. 
So Chris Cerf was staking his case on the role of schools and poverty; he was trying to convince a room full of teachers that poverty is important, but that it can't be a distraction from implementing his reformy policies.

To bolster his claim, Cerf then showed one of the most mendacious, deceptive graphs I've ever seen used in a public policy debate*:


And here's what he said as it was shown:
And let me... each dot represents a school in New Jersey. On one axis is the poverty rate - free and reduced price lunch - one axis is performance. You've got a bunch of schools who have very low poverty and quite poor perfromance. And vice versa. And no one's saying there is not a predictive effect to this. All we are saying is, recognizing the influence of poverty, there is a continuum of success, and we should continue to strive to move our instruction, to move our schools, to move our educational offerings, to a point where we move children who are faced with an unfair set of challenges or burdens, higher and better than we are today.
OK, the Commissioner was speaking extemporaneously, so let's cut him some slack on that last sentence. But let's also have Bruce Baker explain why this graph is so utterly, shamelessly deceptive:
The title conveys the intended point of the graph – that if you look hard enough across New Jersey – you can find not only some, but MANY higher poverty schools that perform better than lower poverty schools.
This is a bizarre graph to say the least. It’s set up as a scatter plot of proficiency rates with respect to free/reduced lunch rates, but then it only includes those schools/dots that fall in these otherwise unlikely positions. At least put the others there faintly in the background, so we can see where these fit into the overall pattern. The suggestion here is that there is not pattern.
The apparent inference here? Either poverty itself really isn’t that important a factor in determining student success rates on state assessments, or, alternatively, free and reduced lunch simply isn’t a very good measure of poverty even if poverty is a good predictor. Either way, something’s clearly amiss if we have so many higher poverty schools outperforming lower poverty ones. In fact, the only dots included in the graph are high poverty districts outperforming lower poverty ones. There can’t be much of a pattern between these two variables at all, can there? If anything, the trendline must be sloped up hill? (that is, higher poverty leads to higher outcomes!)
Let me break down the good professor a little more for you (pay attention, Commissioner!):

There is an unquestioned correlation between poverty and educational outcomes as measured by test scores; not even Commissioner Cerf is going to try to argue with that. Granted, there are some schools that "beat the odds": they score better than what we would expect (and, sometimes, worse than what we would expect).

But it is either innumerate, deceptive, or both to suggest that the mere existence of outliers "proves" that the correlation between poverty and test scores can be overcome - especially with policies that do not address poverty itself.

Baker continues:
Note that the graph doesn’t even tell us which or how many dots/schools are in each group and/or what percent of all schools these represent. Are they the norm? or the outliers?
So, here’s the actual pattern:
Hmmm… looks a little different when you put it that way. Yeah, it’s a scatter, not a perfectly straight line of dots. And yes, there are some dots to the right hand side that land above the 65 line and some dots to the left that land below it.
BUT THE REALITY IS THAT FREE/REDUCED LUNCH ALONE EXPLAINS ABOUT 2/3 OF THE VARIATION IN PROFICIENCY RATES ACROSS SCHOOLS!
What Cerf did on this slide is essentially take the statistical "noise" - the variations that occur in any test due to random errors out of anyone's control - and use it as a justification for downplaying the effects of poverty on test scores.

Are there schools that consistently "beat the odds"? Almost certainly - although quite often, they are getting better test scores because the data we have isn't "finely-tuned" enough to let us see differences in student characteristics that may exist. Still, there probably are schools that consistently do a better job than others with equivalent student populations. We should identify these schools and study their practices - but we should also be honest about whether their results are replicable. In many cases - say, with the high-flying charters of Newark - they are not.

But this is where Cerf becomes, frankly, incoherent. Because even as he makes the case that poverty matters, he continues to insist we can overcome it with policies that do not address poverty itself:
That's not the same as saying poverty doesn't matter. Frankly I find that perspective to be offensive and very difficult for anyone to argue. 
And yet you showed that graph above, Commissioner...
The real question: what do you do with that fact? What do you do with it? Do you look for solutions or interventions or changes in funding or changes in instructional practice or changes in organization of a classroom or changes of who's in our classrooms? You don't just give up on the basis of some outside, extrinsic force.
Well, here's what Cerf, Chris Christie, and the NJDOE have done about it:


They've taken away the right of poor, urban districts to self-determination - just because they happen to be urban, poor, and full of people of color:


Jersey City, Paterson, Newark, and now Camden have lost any right to determine their own school policies. Why? Because they are full of beautiful, deserving children who happen to come from poor families. And what has decades of state control and the failure of the Christie administration to fully fund their schools as is required by law done for them?

Paterson

Trenton

It looks to me like "giving up" is exactly what the Christie administration has done. And yet Commissioner Cerf has the audacity to claim that his critics (including us "lowest forms of life") are saying we shouldn't do anything about reforming schools because we think poverty is intractable.

Let me be clear, Commissioner: I think there is quite a bit we could do to make these schools better, starting with fully funding SFRA, making the facilities in poor urban centers better than those at New Jersey's wealthiest private schools (you know, the ones where Chris Christie sends his own kids), and returning democracy and local control to all of our public schools. I suspect this would do much more to raise teacher quality and morale than imposing your innumerate evaluation system, which evidence suggests will punish teachers who work in these schools.

But hand-wringing about poverty while disenfranchising local parents isn't a strategy for confronting poverty; it's a cop-out. And abusing data in the service of this cop-out isn't helping anyone.


* To be clear: I don't know if this was the exact graph used - but if not, it was nearly identical. NJDOE has used this same graph before; if the one Cerf showed at NJEA differed, it was only in minor and cosmetic changes. If and when NJDOE releases Cerf's slide show, I will put the exact graph up here. Until then: I can report, without hesitation, that this graph was almost exactly identical, if not a perfect replica.

6 comments:

Unknown said...

Wear your 'lowest form of life'TM as a badge of honor.

Giuseppe said...

Finland has a child poverty rate of 4% while that of the US is 22%. The US, the richest and most powerful nation on earth with the biggest economy and yet we tolerate this level of poverty and inequality. Oh wait, just blame teachers and their unions, works like a charm every time.

Russ Walsh said...

Great post again, Jazzman. We must keep calling these reformers to account.

alm said...

The point of the Cerf graph, as I understand it, is that while lunch rate is broadly predictive of school achievement, it isn't dispositive - there are schools that break that trend, just not all that many.

You call those data points 'noise' -- can you be more clear on what you mean?
If you're saying that those schools are measurement error, you'd expect to see them bubble up one year, and disappear the next. And that would be an empirical question - are there some schools that consistent 'beat the odds' year over year?

If you're saying that those schools are noise in the sense that they aren't replicable or otherwise outliers that should be discarded, I think you need to source that. Having looked at this data before, certainly some of those schools are selective magnets or exam schools (which explains why they beat the odds - selection effects).

But what makes them noise, in your view?

alm said...

*consistently, yikes.

Duke said...

Thx all.

alm, we're back to burden of proof again. It is more than reasonable to think a good deal of what is in those circles is noise. I have left open the possibility that some is not... but it is on Cerf to give evidence something else is going on beside random testing effects.

He hasn't.