Bradford Believes In the Merit Pay Fairy

Derrell Bradford is angry that the adults are telling him the Merit Pay Fairy isn't real:

The recent decision by New York City to end its teacher bonus program has been used as “evidence … that merit pay doesn’t ‘work.’” (see here)  The NYC decision was based on a RAND Corporation-Vanderbilt study (see report here) that concluded that “narrow” pay-for-performance programs that consist only of financial incentives have no effect on student achievement.   Before drawing sweeping conclusions like “merit pay doesn’t work,” we should take a closer look at the program and the report.

The NYC program largely included only school-wide bonuses.  These bonuses were distributed by committees that displayed “a strong preference for egalitarian distribution plans,” so that each teacher received about the same reward of $3,000.  Individual teacher performance was considered by only 31% schools, and even in these schools about 75% of staff received the same award.   The study concluded that  “almost one-half of teachers … indicated that the bonus was not enough to motivate extra effort” and that “the lack of results might be due to the limited motivational power of the bonuses.”

Poor Derrell: he so desperately wants to believe in the Merit Pay Fairy that he will deny pretty much any evidence that she doesn't exist. You can almost see the tears well up in his little eyes as he stammers: "But.. but... this study isn't the right study! This merit pay plan is the wrong merit pay plan! We just need to tweak it! Everyone, clap harder or the Merit Pay Fairy will die!"

The sad fact, Derrell, is that we looked for the Merit Pay Fairy in Tennessee: she wasn't there. She wasn't in Texas, either. Corporate reformers stayed up all night in Chicago waiting for her; she never showed. She wasn't in Michigan. We even tried looking for the Merit Pay Fairy in 18th Century England; sorry, mate.

Now, she's failed to show up in NYC, breaking Derrell's heart. But, like Linus in the pumpkin patch, that doesn't mean Derrell will ever stop believing:

Consistent with what RAND/Vanderbilt suggested in these studies, there are examples of merit pay models that use a broader approach that combines financial incentives with professional development, a proper evaluation system and concrete steps to facilitate implementation and teacher buy-in.  The Denver Public Schools (DPS) PROCOMP plan has existed since 1999 with notable success (see here).  Initially a pilot program “focused on developing a direct link between student achievement and teacher compensation,” two-thirds of DPS teachers are now enrolled in the permanent program where teachers can opt in and forego the traditional step/seniority-based compensation model.

It's interesting that Derrell won't draw "sweeping" conclusions about about the NYC plan, but he will draw them about Denver's PROCOMP. Because the evidence is hardly a ringing endorsement for merit pay: (p.70)

First, other district reforms were implemented concurrent with ProComp. Educational reforms do not operate independent of all other things going on in a school district. DPS welcomed new superintendent Michael Bennett during summer, 2005 – several months before the ProComp ballot measure, yet several months after the execution of the ProComp joint agreement. Soon afterward DPS initiated the “Denver Plan” comprehensive initiative toward improving student achievement (into which ProComp was integrated). ProComp and the Denver Plan share similar timelines; as such, the analyses reported above cannot definitively attribute effects to one or the other. 

In the same way that particulars of the district context introduce confounds to causal attribution, so too do state and federal policy contexts. The timeframe examined in this study was significantly influenced by the increased role of testing and accountability ushered in by the federal No Child Left Behind Act. Colorado initiated several reforms that could have influenced outcomes over the time period examined here... [emphasis mine]

This, in fact, is the largest problem with studying merit pay plans: too many confounding variables. That's why the NYC study was so important: it randomly assigned schools to the program. I'm not expert enough to state this conclusively, but there's a very good chance this is the best experimental design we've had to evaluate merit pay so far.

Unlike Derrell's other example:

More broadly, a Stanford study of the Teacher Advancement Program (TAP) that encompassed 151 schools in 10 states found that merit pay programs that are part of comprehensive reform plans increased student performance over similar non-TAP schools.  Likewise, a study of six Teacher Incentive Fund sites implementing the TAP system and covering 134 schools in six states found “preliminary indicators showing increased student achievement, wide stakeholder support, improvements in recruitment and retention, and positive changes in school cultures.”  (see link to both here ) [emphasis mine]

Again: confounding variables: there's a lot more going on at TAP schools than merit pay. Even then, another Mathematica study found that TAP schools in Chicago didn't show any improvement in state test scores, so it's not as if the data is conclusive.

Further, the Stanford study - which was, more accurately, a graduate thesis (not that there's anything wrong with that) -  uses a method called "synthetic control matching" to create "hypothetical" schools to judge the TAP schools against. Put another way: it uses statistics to create schools that don't really exist.

I am in no way, shape, or form qualified to judge this method: it may be perfectly valid. But even if it is, the results aren't exactly overwhelming:

There are plenty of implications here for the broader field of school improvement. But please don't go thinking that this study is somehow a counterweight to the big merit-pay study from last month, on Nashville's POINT program. POINT was a "pure" experiment, with the only major difference between treatment and control groups being the performance-pay element. 

By contrast, in TAP schools, teachers also get group-based professional development, individual feedback keyed to an evaluation framework, and opportunities to take on additional roles in schools and to be compensated for them, all features that didn't apply in Nashville. So use caution in trying to compare these studies. [emphasis mine]

So, the gain isn't consistent across models, and we have no idea how much of the gain is attributable to merit pay as opposed to many other changes.

This is not a lot to hang the Merit Pay Fairy's tiara on. But Derrell is determined to clap harder:

The bottom line is that the demise of a narrow and flawed pay-for-performance program should not end the discussion on merit pay.  B4K believes that merit pay should be a key component of any comprehensive education reform plan.   It must be tied into evaluation system that rates teachers fairly and transparently, and integrates professional development into the process.   It should reward superior performance, augment promotion to positions of greater responsibility and incentivize effective teachers to teach in high-needs schools and subject areas.  And, not least, it should help to attract and retain the highest quality teachers.

Yes, and it should also get those stubborn grass stains out of your whites, burn fat without exercise, and leave your breath minty fresh. But I'm afraid, Derrell, that those of us back on planet Earth have to consider what a real merit pay system would look like.

And it ain't pretty. The implementation of the type of merit pay Derrell touts is inevitably tied to Value-Added Modeling based on standardized tests scores. This is a wildly flawed method that will inevitably misidentify teachers.

Further, over and over again we hear from the conservative governors pushing corporate reform that "we don't have any more money." If that's true, any merit pay scheme will be predicated on the idea of taking money from a "bad" teacher and giving it to a "good" one. Need I spell out what a recipe for disaster this is going to be? At the very least, parents are going to be up in arms about how their children are assigned to teachers who have or haven't earned merit pay.

And getting any sort of buy-in during this era of teacher bashing is nearly impossible:

Policy makers often complain about the difficulty of changing teachers’ behavior. This is not surprising in an environment where teachers have few guarantees that their efforts to change will be rewarded. Change is slow and difficult work, and it is tempting for teachers to ignore the policy de jour since the next regime change is likely to, once again, change the rules of the game. Education has a reputation for constant policy change; educa- tional mandates come and go before it is really possible for meaningful change to take place (Cohen 1988; Cuban 1993; Tyack and Cuban 1995). 

The inability of school districts to credibly commit has been largely ignored in the research literature but may be a significant factor in the failure of many policies to positively affect practice. While this study only looks at the Minneapolis case, it has implications for education policy more generally. The broader question is this: Can a public school district credibly commit to policies that, in order to be effective, require a long-term financial and political commitment? The answer to this question may very well be a resounding, “No.” [emphasis mine]

In other words: this is the latest scam. And, given the corporate reform movement's track record, there's just no reason to trust it, especially when folks like Derrell are spending millions on ads that bash our unions.

Let me break in with a personal opinion: like charter schools, I do think it's possible that merit pay has its place. I've argued for a long time that the structure of schools is too flat; if we're going to make an analogy to business, I think most managers would say that a principal shouldn't have so many direct reports. Tying "master teacher" status to pay may be a good thing. Rewarding excellence - with a buy-in from teachers and their unions - may have it's place.

But let's be clear: if you made a list of stuff to do to really address the achievement gap, merit pay would be at the bottom. And it's funny Derrell wants to use Colorado as an example, because Bruce Baker clues us in, while testifying in a case there, as to what should be at the top:

Baker discussed at length the higher cost of educating at-risk students and said when those costs are taken into account, Colorado districts with large numbers of such students are at a noticeable disadvantage. He also said funding gaps can account for 60 percent of achievement gaps in reading and 46 percent of math achievement gaps. [emphasis mine]

We see the two worlds of reform very clearly here. Bruce Baker, the adult in the room, is telling us that there are no magic wands. You want to address the achievement gap? Pay up. You need to get more resources to the kids who need it the most. I would only add that those resources need to be accompanied by a push to eliminate poverty, give people the opportunity to have good jobs with living wages, build infrastructure, provide health care, and eliminate racism.

Derrell, however, lives in a happy, shiny world were all problems will be magically wiped away with a few more charter schools, some vouchers, and merit pay. He believes this so much that he keeps staying up past his bedtime every night, waiting for the Merit Pay Fairy to wave her wand and transform underperforming schools into Neverlands.

And he's very, very cross when anyone tries to explain to him that it's time to stop believing in the Merit Pay Fairy and finally grow up.

Yo, Derrell - don't hold yer breath waitin' fer me, OK pal?