TFA doesn't welcome critics? I'm simply shocked...
You can follow this weekend's festivities via Twitter: @TFA25FactCheck. But be prepared: if you haven't recalibrated your B.S. detector, it might just explode:
Apparently, this slide was shown as "proof" that the reformy reforms in Washington D.C. -- including innumerate teacher evaluation -- are leading to gains so large in the nation's capitol that they must be rendered in bright yellow bars.
Too bad whoever made this forgot a few things:
- Test score gains are not necessarily equivalent across different tests. In other words: we don't know if gaining 10 points on the Grade 4 reading test is at all equivalent to gaining 10 points on the Grade 8 math test.
- Test score gains are not necessarily equivalent across different parts of a score distribution. In other words, going from a 230 to a 240 is not equivalent to going from a 260 to a 270: it might be much harder to gain 10 points from one starting point than it is from another.
So combining the scale scores of different tests at different starting points and then comparing them is pretty much worthless.
- It is pointless to compare test score gains without accounting for changes in student populations. We know D.C. has seen substantial demographic changes; you can't just slap up scores that correlate to student characteristics without acknowledging these changes.
- Test score changes are not, by themselves, proof that particular policies are successful. Look at the top of this slide: "How the DC Public Schools Changed Everything to Get, Grow, and Keep Great Teachers and Principals." Is the person who put this up seriously suggesting a few teacher policy choices are the cause of the test score gains? That it couldn't possibly be a host of other factors? Really?
OK, I wasn't there. Maybe this was a simple descriptive introduction, leading up to a sophisticated analysis with proper controls for student population changes and scale differences. Maybe this was used as an example of how not to use NAEP data to make a case for a particular policy intervention. Maybe TFA is going to have a weekend full of serious policy discussions, and not engage in some really shameless data manipulation to push their particular agenda.
ADDING: Tonight's menu:
Mis-NAEPery a la Baker.
Mis-NAEPery a la Polikoff.
Mis-NAEPerty a la DiCarlo.