I will protect your pensions. Nothing about your pension is going to change when I am governor. - Chris Christie, "An Open Letter to the Teachers of NJ" October, 2009

Tuesday, August 16, 2016

Useless Testing "Gap" Analyses and the Newspapers That Love Them

There is so much wrong with this "analysis" by the The Bergen Record of test scores and other outcomes for the 31 "Abbott" districts -- the (mostly) disadvantaged New Jersey school districts that received additional aid for years (although inconsistently) under a series of court rulings. Among other problems:

  • As Julia Sass Rubin points out, the authors don't include the charter schools, which are a large portion of the student enrollments in many districts.
  • They don't account for changes over time in student characteristics.
  • They assume the proficiency rates consistently measure "proficiency" across time, a huge assumption that, to my knowledge, has never been assessed by the state.
  • They assume the Abbotts got "extra aid," even though, for years, the amount the districts received was not what the state's own law says they need to provide an adequate education (and that amount isn't close to enough to expect equalized school outcomes anyway).
I could go on, but I want to make a larger point about so-called "gap" analyses, and why any attempt, like The Record's, to judge an education policy's efficacy based on "gaps" is fundamentally flawed.

And I'm going to keep this so simple it can be drawn out in Sharpies (click to enlarge):

Let's say you have two schools: one in a wealthier area, one is a less-wealthy place. The advantaged school starts off with a higher proficiency rate than the disadvantaged school. As time goes on, the disadvantaged school improves -- but so does the advantaged school

After several years, both schools are performing better than they did previously. But the gap isn't any smaller. Is that a "failure" on the part of the disadvantaged school? Are the state's education policies "failures" because the gap didn't shrink? Would the policies be "successful" if the advantaged school didn't improve so the "gap" shrunk?

Now, The Record's analysis does note the graduation rate "gap" has closed between the Abbotts and the rest of the state. Why would that be?

You can't have a graduation rate over 100 percent, right? So there's a "ceiling" effect: because so many suburban districts were already graduating almost all of their students, they really couldn't improve. But more disadvantaged districts, with lower graduation rates, could and did improve (why they did is an interesting question I'll get to soon). 

So there was a chance for the "gap" to shrink in graduation rates that wasn't available for proficiency rates (few schools, even in the 'burbs, have nearly 100 percent proficiency) or SAT scores. Which makes The Record's piece largely meaningless.

As Ajay Srikanth and I pointed out, there is a great deal of high-quality evidence that shows school funding reform in New Jersey led to substantial improvements in outcomes. Our review matches the evidence found by the Education Law Center. This comports with a large and growing body of high-quality evidence from around the nation that shows equitable and adequate funding can and does make a meaningful difference in student and school outcomes.

The Record has done some fine education journalism over the years. This, unfortunately, was a serious misstep -- as is most "gap" analysis. Reach out to me next time, folks, and we'll see if we can't come up with something a bit better.

Yeah, on second thought, DON'T mind it...

ADDING: Both Bruce Baker and Matt DiCarlo wrote nice pieces a few years ago about how NJDOE totally messed up its "gap" analysis.

1 comment:

StateAidGuy said...

I don't have the time right now to do an analysis of the Record piece, but I agree, the study is flawed.

The authors don't make an attempt to directly compare the Abbotts with poor-non-Abbotts.

Nobody ever said that paritization of funds would produce paritization of results, but if the theory of Abbott is correct, there should be higher academic performance in the Abbott districts than equally poor non-Abbotts who spend 50-60% as much and don't get Pre-K.



Yes, the Abbotts have increased their graduation rates, but so has the entire country, including states with consistently regressive school funding.

Also, most of the increase in Abbott grad rates has taken place in the last few years, during which the Abbotts have had to make cuts.

This is also contrary to Abbott theory.