More "One Study Does Not a Policy Make"

Bruce Baker has weighed in on the big study, reported in the New York Times, that attempts to tie teacher ratings using standardized test scores to students' future earnings:

First and perhaps most importantly, just because teacher VA scores in a massive data set show variance does not mean that we can identify with any level of precision or accuracy, which individual teachers (plucking single points from a massive scatterplot) are “good” and which are “bad.” Therein exists one of the major fallacies of moving from large scale econometric analysis to micro level human resource management.

Second, much of the spin has been on the implications of this study for immediate personnel actions. Here, two of the authors of the study bear some responsibility for feeding the media misguided interpretations. As one of the study’s authors noted:

“The message is to fire people sooner rather than later,” Professor Friedman said. (NY Times)

This statement is not justified from what this study actually tested/evaluated and ultimately found. Why? Because this study did not test whether adopting a sweeping policy of statistically based “teacher deselection” would actually lead to increased likelihood of students going to college (a half of one percent increase) or increased lifelong earnings. Rather, this study showed retrospectively that students who happened to be in classrooms that gained more, seemed to have a slightly higher likelihood of going to college and slightly higher annual earnings. From that finding, the authors extrapolate that if we were to simply replace bad teachers with average ones, the lifetime earnings of a classroom full of students would increase by $266k in 2010 dollars. This extrapolation may inform policy or future research, but should not be viewed as an absolute determinant of best immediate policy action.

Everyone who opines on this needs to read all of Bruce's post.  I'll have some further thoughts later as I work my way through the study. For now, though:

This was one study in one city which found a very small economic gain (about $250/year) that correlates to higher test scores that MAY have to do with teacher quality. The authors admit the data they used could only link test scores to earnings for people only as old as 28. Think about that; only age 28. Those of you who, like me, are more advanced in age may want to ask yourself how much you think your 6th Grade math teacher impacts your current salary (I don't even remember her name).

This is awfully weak tea, especially if we are talking about upending a system of teacher compensation and job security that has been in place for decades. It would be foolish beyond belief to call for using test scores in teacher evaluations based on this one very limited study (and yet...).

We know that VAM, based on test scores, will falsely identify bad teachers as good, and good teachers as bad; the study does not refute this at all. So I see no way that using VAM will make teaching a more attractive career for high-quality candidates. And I have yet to see how using VAM solves the (overblown, in my opinion) problem of dismissing "bad" teachers when even its adherents admit it really is only useful for dismissing the lowest of the low: the teachers any good principal could identify through observations anyway.

Again: more later this week.