Don't Buy Into Reformy Frameworks

I agree with quite a bit in Dana Goldstein's review of Steve Brill's new corporate "reform" manifesto, Class Warfare. Unfortunately, Goldstein's review is undermined by a big error:

Brill and the accountability crowd are correct to note that high-performing teachers are consistently able to raise the test scores of even the poorest children. Research shows that an improvement of one standard deviation in teacher quality leads to approximately two to four points of gain for a student on a 100-point test in reading or math. Five years of great teachers in a row, therefore, could raise a student’s test scores by ten to twenty points.

What's funny is that just today, Larry Ferlazzo has put out a list of posts debunking this very myth. Here's an excerpt from the piece at the top of Ferlazzo's list, written by Matt DiCarlo:

The first thing to keep in mind is that most of these are just extrapolations. The researchers didn’t follow a group of low-performing students over five years, assigning some to five consecutive great teachers and then measuring the outcome. Instead, they took the average one-year gains among students of “top teachers” (however defined), and then determined how many of these one-year gains are equivalent to the average aggregate achievement gap. 

[...] 

On a similar note, the “X consecutive teachers” argument depends on the assumption that a teacher’s effect is “persistent” – that it does not diminish over time (see here for a great technical discussion of how this applies to Sanders and Rivers). So, for example, those students who gained 10 percentile points in one year with an “effective” teacher – going from, say, the 30th to the 40th percentile – would, if assigned to another one the next year, get to the 50th percentile, to the 60th the year after that, and so on. The assumption is that, each year, students start where they left off, and the effect of their previous teacher remains intact. 

In contrast, there ample evidence that a teacher’s effect on test scores “decays” rather quickly over time, and there is still little idea of how to best account for this phenomenon (though recent advances are very interesting). Since only part of a teacher’s effect persists –students don’t retain a great deal of what they learn – it’s a bit implausible to take one-year gains and project them out across several years. The “X consecutive teachers” argument kind of treats test score gains like weight gains – you can just add them up – and this belies the complex, transitory nature of teaching and learning. [emphasis mine]

Here's Diane Ravitch, quoted by Valerie Strauss:

This is akin to saying that baseball teams should consist only of players who hit over .300 and pitchers who win at least twenty games every season; after all, such players exist, so why should not such teams exit. The fact that no such team exists should give pause to those who believe that almost every teacher in almost every school in almost every district might be a superstar if only school leaders could fire at will.

So, no: there is no evidence that a particular teacher's ability to create a one-year gain in test scores can be extrapolated to five times the gain over five years.

We need to get this stuff right.