Unless I’m missing something, TNTP does not present any basic characteristics of their survey sample, to say nothing of trying to compare them with those of the larger DCPS group (the same was true of the districts in the “Irreplaceables” report). This is always required (even with random samples), but it’s especially important given that TNTP is an advocacy organization, one which is very well-known in DCPS (for example, it was founded by former Chancellor Michelle Rhee), and places many teachers in the district. Attitudes toward them can be very contentious, and it’s certain that at least some people decided for or against completing the survey based in part on their views of TNTP and/or their policy stances.As far as I'm concerned, the survey in this report is useless. But I still stand by the linking of "Evaluation System" and "Compensation" - read more below.
I was going to write a long takedown of TNTP's report on Washington D.C.'s schools, but Gary Rubinstein thankfully beat me to the punch (emphasis mine):
The first explanation in that last paragraph is contradicted by one of the points reformyists never want to discuss: an "effective" teacher isn't always "effective" with all students. Some teachers may be great with gifted kids; some may be great with special needs kids. Some may be terrific with Hispanic children because they understand the culture and language; some may only shine with affluent kids in the suburbs. It's basically a manifestation of the psychological concept of "goodness of fit": we perform better or worse depending on whether our environment suits us.
Reformyists tell us, however, that if we measure student "growth," and not absolute performance, we remove any bias in measuring a teacher's effectiveness caused by assigning that teacher low-performing students. Leave aside that there's no proof that this is the case, it still doesn't address the issue that the "high-performing" teachers in D.C.'s schools may not be as great if they were teaching a different set of students.
And, if that's true, what would lead us to believe they really are "irreplaceable"?
A few more things: look at this chart from page 13 (click to enlarge):
The chart purports to show that D.C.'s compensation system, which features merit pay bonuses, isn't a big factor in why teachers leave. Leave aside the dubious practice of using rankings to make that case (let's also leave aside the really awful visual design to present the data). Notice the rankings for "Evaluation System"; D.C.'s "high performers" cite it more than any other district as a reason to leave. Well, if evaluation is tied to compensation, isn't one just a proxy for another? Isn't ranking "evaluations" high as a reason to leave pretty much the same as ranking "compensation" highly when the two are linked?
And shouldn't we be worried that the evaluation system ranks so high as a reason to leave teaching in D.C.?
Another concern: it's been well documented here and elsewhere how faulty teacher evaluation systems are when using Value Added Modeling. But the other big part of D.C.'s system is the use of teacher observations. And the way those observations are used is fatally flawed, because they imply a phony sense of precision.
To go over this quickly once again, here's an example from the DCPS website of how a teacher obtains her observation score in D.C.'s IMPACT system:
Notice how all of the scores from observations are only significant to one digit - 1, 2, 3, or 4 - but they are expressed in two digits with a ".0" added to each. This is absolutely phony: it implies a two-digit precision when none exists. There is only one significant figure in each observation; therefore, the overall averaged score must be only one significant figure as well.
This is an abuse of mathematics designed to meet a policy end, because IMPACT does not work unless it is able to make fine distinctions between teachers. But it doesn't, and that throws the entire validity of TNTP's study into question, because we're not even sure the "high performers" who were rated solely on their observations are really "high performers" at all.
Last thought: the Washington Post rushed in to embrace this study and tout Michelle Rhee's "reforms." But nowhere did they - or, for that matter, TNTP - show that these practices are improving student achievement. I thought that was what this was supposed to be all about, right?
Rubinstein points to the work of G.F. Brandenburg, the truth-telling prophet of D.C.'s schools:
Click through and view the ugly truth. You can be sure the Washington Post didn't.I spent some tedious hours looking at publicly-available data on DC-CAS standardized test scores that all public school students in grades 3-8 and 10 must take every year, in particular looking at the ‘pass’ rates for various sub-groups like whites, blacks, hispanics, poor and non-poor kids (as measured by whether they are eligible for free or reduced-price lunch), and so on.In every single case, when I compared the scores of the historically higher-achieving group with the scores of the lower-achieving group, I find that in DCPS, all of the schemes of Rhee, Kamras, Fenty, Henderson, Gray and a few powerful billionaires have been a complete and utter failure. Even on their own terms.
Washington Post Editorial Board