More "What Research?!"

As I've moaned before, I hate it when anyone says "research shows..." and then refuses to cite the research.

I mean, I really hate it:

Teachers are the No. 1 in-school factor affecting student success. Research says a highly effective teacher can help students achieve as much as an additional year's worth of academic gains over one school year compared with a less effective teacher. 

[...] 

Research shows that there are both outstanding teachers and ineffective teachers across all levels of experience. According to two recent studies, the vast majority of seniority-based layoffs result in better teachers leaving classrooms and less-effective teachers staying.

Jeez Louise. What do I have to do? Guess what research you're talking about? You couldn't give me a clue?

Let's break it down:

Teachers are the No. 1 in-school factor affecting student success.

A favorite of the reformy crowd, because it's so mendacious. Yes, teachers are the largest in-school factor, but only about 20% of achievement outcomes can be attributed to the school; this has been confirmed in many studies. The largest determinants of student achievement occur outside of school or are simply error.

Research says a highly effective teacher can help students achieve as much as an additional year's worth of academic gains over one school year compared with a less effective teacher. 

This is a variant on the "three great teachers" myth. Every time I've seen "research" that avers something like this, it turns out to be a projection of a limited gain over a longer timespan than that of the actual study, or a hypothetical based on the largest possible effect in a study with no accounting for error in measurement.

Of course, I don't know if that's true here. Why? Because they won't even tell me what friggin' study they're referring to!

Research shows that there are both outstanding teachers and ineffective teachers across all levels of experience.

I have no doubt that there are ineffective teachers who've taught for 25 years; the questions are: 1) how many are there, and 2) can we identify them with a high degree of accuracy? Did this "research" address these points? Who knows?

According to two recent studies, the vast majority of seniority-based layoffs result in better teachers leaving classrooms and less-effective teachers staying.

Well, even though they don't cite them, at least I know what they're talking about here, because these are favorites of The New Teacher Project:

Boyd, Donald; Lankford, Hamilton; Loeb, Susanna; and Wyckoff, James (2010). “Teacher Layoffs: An Empirical Illustration of Seniority v. Measures of Effectiveness.” The Urban Institute, National Center for the Analysis of Longitudinal Data in Education Research (CALDER).  

Goldhaber, Dan and Theobold, Roddy (2010). “Assessing the Determinants and Implications of Teacher Layoffs.” Center for Education Data & Research, University of Washington-Bothell.

We'll take these in order. The first study does not talk at all about a "vast majority" of layoffs; it looked at  two grade levels (4th and 5th) in New York City for one year. Again: this wasn't a measurement of an actual policy that was implemented; it is an illustration of what might have happened if NYC had implemented a different policy.

And that creates two problems. The first is obvious: if you fire teachers by test scores, you shouldn't be surprised if the test scores of the teachers you kept are higher than the ones you let go (duh). What you can't judge is how accurate those scores were in the first place. The Value-Added Model (VAM) scores were based on differing amounts of test data, up to four years. Well, what if you keep a lousy first-year teacher who happened, through error, to have great scores?

We know the error rates on the tests are very high; making high-stakes decisions based on them is like rolling dice. Do our teacher-authors here think that's a good idea?

Which brings us to our second problem: since this is only an illustration, we have no idea what other effects this policy will have on a teaching corps. What will this do to morale? How will it affect teachers who aren't subject to standardized testing? The study avoids overtly confronting this problem, but implicitly acknowledges it:

Value-added is currently only feasible for the portion of teachers who teach in tested grades and subjects, often math and ELA in grades 4 through 8, thus limiting their applicability. In addition, as described above, we know little about the extent to which value-added measures employing standardized achievement tests capture other important dimensions of teaching. While these issues should be considered in the application of value-added, we should not lose sight of the main point. Informing teacher layoffs with measures of effectiveness, while not perfect, does offer the potential to meaningfully improve the quality of instruction in some classrooms. [emphasis mine]

The "potential"; even cheerleaders of this system won't go beyond that.

As to the Goldhaber & Theobold study, let's check in with Bruce Baker:

Hill and Roza offer as evidence that their proposed strategies have been evaluated in terms of productive efficiency and cost effectiveness, three links to related “studies”. One of those links points to a paper by Dan Goldhaber and Roddy Theobald, regarding potential costs/benefits of Seniority based versus “Quality-Based” layoffs. On the one hand, the paper does not yield any decisive guidance for short term budget planning and on the other hand, suffers the circular logic I’ve discussed on numerous occasions on my blog regarding measuring the effectiveness of the policy by the same measure used to implement the policy (e.g. did firing teachers with low value added scores leave us with more teachers with high value added scores).  The central conclusion of the paper(s) is that “Finally, simulations suggest that a very different group of teachers would be targeted for layoffs under an effectiveness-based layoff scenario than under the seniority-driven system that exists today.” This is hardly surprising, and of limited usefulness for informing state or local leaders on how to handle personnel decisions in tough budgetary times, or the expected benefits or downsides of such policies, and fails to address such basic issues as the costs of putting into place a system that might be used for making such decisions. I provide a hypothetical discussion of this topic in an earlier blog post.

Do you understand now why it bugs me so much that these op-eds won't cite the research they use to come to their conclusions? The entire point of peer-reviewed research is to subject it to scrutiny to see if it holds up; how can we do that if we don't even know what it is? I'm not asking newspapers to provide academic rebuttals; I'm just asking they tell us where these assertions of "fact" come from.

As it is, I find the argument these teachers make incredibly weak. We know that experience matters in teaching; no one disputes this for at least the first several years. We know we can accurately quantify experience far, far better than VAM-based "effectiveness." Given the inaccuracies involved in VAM, the fact that layoffs also take into account certifications and specialization, and how little we know about the confounding effects of "quality-based" layoffs, where is the "moral obligation" to do this?

Here's a guess: I won't post links, but the Facebook profiles of two of the authors (h/t SBS) show them to be fairly young (for the record, I'm under 10 years at my school). I imagine this whole issue hits them a little closer to home than some other teachers. I'll be frank: I'm really not very open to listening to appeals to "moral obligations" until we spell out what's at stake for everyone involved. In this case, we have younger teachers possibly retaining their jobs over older ones.

They have every right to make their pitch, and they have every right to try to convince us all that what they are proposing is in the best interest of children (again, I don't think they succeed). But let's refrain from exclusively claiming the moral high ground when our own interests are also at issue, shall we?

(Boy, my troll is gonna love that last paragraph...)