Useless Testing "Gap" Analyses and the Newspapers That Love Them

There is so much wrong with this "analysis" by the The Bergen Record of test scores and other outcomes for the 31 "Abbott" districts -- the (mostly) disadvantaged New Jersey school districts that received additional aid for years (although inconsistently) under a series of court rulings. Among other problems:

• As Julia Sass Rubin points out, the authors don't include the charter schools, which are a large portion of the student enrollments in many districts.
• They don't account for changes over time in student characteristics.
• They assume the proficiency rates consistently measure "proficiency" across time, a huge assumption that, to my knowledge, has never been assessed by the state.
• They assume the Abbotts got "extra aid," even though, for years, the amount the districts received was not what the state's own law says they need to provide an adequate education (and that amount isn't close to enough to expect equalized school outcomes anyway).

I could go on, but I want to make a larger point about so-called "gap" analyses, and why any attempt, like The Record's, to judge an education policy's efficacy based on "gaps" is fundamentally flawed.

And I'm going to keep this so simple it can be drawn out in Sharpies (click to enlarge):

Let's say you have two schools: one in a wealthier area, one is a less-wealthy place. The advantaged school starts off with a higher proficiency rate than the disadvantaged school. As time goes on, the disadvantaged school improves -- but so does the advantaged school. 

After several years, both schools are performing better than they did previously. But the gap isn't any smaller. Is that a "failure" on the part of the disadvantaged school? Are the state's education policies "failures" because the gap didn't shrink? Would the policies be "successful" if the advantaged school didn't improve so the "gap" shrunk?

Now, The Record's analysis does note the graduation rate "gap" has closed between the Abbotts and the rest of the state. Why would that be?

You can't have a graduation rate over 100 percent, right? So there's a "ceiling" effect: because so many suburban districts were already graduating almost all of their students, they really couldn't improve. But more disadvantaged districts, with lower graduation rates, could and did improve (why they did is an interesting question I'll get to soon). 

So there was a chance for the "gap" to shrink in graduation rates that wasn't available for proficiency rates (few schools, even in the 'burbs, have nearly 100 percent proficiency) or SAT scores. Which makes The Record's piece largely meaningless.

As Ajay Srikanth and I pointed out, there is a great deal of high-quality evidence that shows school funding reform in New Jersey led to substantial improvements in outcomes. Our review matches the evidence found by the Education Law Center. This comports with a large and growing body of high-quality evidence from around the nation that shows equitable and adequate funding can and does make a meaningful difference in student and school outcomes.

The Record has done some fine education journalism over the years. This, unfortunately, was a serious misstep -- as is most "gap" analysis. Reach out to me next time, folks, and we'll see if we can't come up with something a bit better.

Yeah, on second thought, DON'T mind it...

ADDING: Both Bruce Baker and Matt DiCarlo wrote nice pieces a few years ago about how NJDOE totally messed up its "gap" analysis.