I will protect your pensions. Nothing about your pension is going to change when I am governor. - Chris Christie, "An Open Letter to the Teachers of NJ" October, 2009

Saturday, August 6, 2011

Bradford Believes In the Merit Pay Fairy

Derrell Bradford is angry that the adults are telling him the Merit Pay Fairy isn't real:
The recent decision by New York City to end its teacher bonus program has been used as “evidence … that merit pay doesn’t ‘work.’” (see here)  The NYC decision was based on a RAND Corporation-Vanderbilt study (see report here) that concluded that “narrow” pay-for-performance programs that consist only of financial incentives have no effect on student achievement.   Before drawing sweeping conclusions like “merit pay doesn’t work,” we should take a closer look at the program and the report.
The NYC program largely included only school-wide bonuses.  These bonuses were distributed by committees that displayed “a strong preference for egalitarian distribution plans,” so that each teacher received about the same reward of $3,000.  Individual teacher performance was considered by only 31% schools, and even in these schools about 75% of staff received the same award.   The study concluded that  “almost one-half of teachers … indicated that the bonus was not enough to motivate extra effort” and that “the lack of results might be due to the limited motivational power of the bonuses.”
Poor Derrell: he so desperately wants to believe in the Merit Pay Fairy that he will deny pretty much any evidence that she doesn't exist. You can almost see the tears well up in his little eyes as he stammers: "But.. but... this study isn't the right study! This merit pay plan is the wrong merit pay plan! We just need to tweak it! Everyone, clap harder or the Merit Pay Fairy will die!"

The sad fact, Derrell, is that we looked for the Merit Pay Fairy in Tennessee: she wasn't there. She wasn't in Texas, either. Corporate reformers stayed up all night in Chicago waiting for her; she never showed. She wasn't in Michigan. We even tried looking for the Merit Pay Fairy in 18th Century England; sorry, mate.

Now, she's failed to show up in NYC, breaking Derrell's heart. But, like Linus in the pumpkin patch, that doesn't mean Derrell will ever stop believing:
Consistent with what RAND/Vanderbilt suggested in these studies, there are examples of merit pay models that use a broader approach that combines financial incentives with professional development, a proper evaluation system and concrete steps to facilitate implementation and teacher buy-in.  The Denver Public Schools (DPS) PROCOMP plan has existed since 1999 with notable success (see here).  Initially a pilot program “focused on developing a direct link between student achievement and teacher compensation,” two-thirds of DPS teachers are now enrolled in the permanent program where teachers can opt in and forego the traditional step/seniority-based compensation model.
It's interesting that Derrell won't draw "sweeping" conclusions about about the NYC plan, but he will draw them about Denver's PROCOMP. Because the evidence is hardly a ringing endorsement for merit pay: (p.70)
First, other district reforms were implemented concurrent with ProComp. Educational reforms do not operate independent of all other things going on in a school district. DPS welcomed new superintendent Michael Bennett during summer, 2005 – several months before the ProComp ballot measure, yet several months after the execution of the ProComp joint agreement. Soon afterward DPS initiated the “Denver Plan” comprehensive initiative toward improving student achievement (into which ProComp was integrated). ProComp and the Denver Plan share similar timelines; as such, the analyses reported above cannot definitively attribute effects to one or the other. 
In the same way that particulars of the district context introduce confounds to causal attribution, so too do state and federal policy contexts. The timeframe examined in this study was significantly influenced by the increased role of testing and accountability ushered in by the federal No Child Left Behind Act. Colorado initiated several reforms that could have influenced outcomes over the time period examined here... [emphasis mine]
This, in fact, is the largest problem with studying merit pay plans: too many confounding variables. That's why the NYC study was so important: it randomly assigned schools to the program. I'm not expert enough to state this conclusively, but there's a very good chance this is the best experimental design we've had to evaluate merit pay so far.

Unlike Derrell's other example:
More broadly, a Stanford study of the Teacher Advancement Program (TAP) that encompassed 151 schools in 10 states found that merit pay programs that are part of comprehensive reform plans increased student performance over similar non-TAP schools.  Likewise, a study of six Teacher Incentive Fund sites implementing the TAP system and covering 134 schools in six states found “preliminary indicators showing increased student achievement, wide stakeholder support, improvements in recruitment and retention, and positive changes in school cultures.”  (see link to both here ) [emphasis mine]
Again: confounding variables: there's a lot more going on at TAP schools than merit pay. Even then, another Mathematica study found that TAP schools in Chicago didn't show any improvement in state test scores, so it's not as if the data is conclusive.

Further, the Stanford study - which was, more accurately, a graduate thesis (not that there's anything wrong with that) -  uses a method called "synthetic control matching" to create "hypothetical" schools to judge the TAP schools against. Put another way: it uses statistics to create schools that don't really exist.

I am in no way, shape, or form qualified to judge this method: it may be perfectly valid. But even if it is, the results aren't exactly overwhelming:
There are plenty of implications here for the broader field of school improvement. But please don't go thinking that this study is somehow a counterweight to the big merit-pay study from last month, on Nashville's POINT program. POINT was a "pure" experiment, with the only major difference between treatment and control groups being the performance-pay element. 
By contrast, in TAP schools, teachers also get group-based professional development, individual feedback keyed to an evaluation framework, and opportunities to take on additional roles in schools and to be compensated for them, all features that didn't apply in Nashville. So use caution in trying to compare these studies. [emphasis mine]
So, the gain isn't consistent across models, and we have no idea how much of the gain is attributable to merit pay as opposed to many other changes.

This is not a lot to hang the Merit Pay Fairy's tiara on. But Derrell is determined to clap harder:
The bottom line is that the demise of a narrow and flawed pay-for-performance program should not end the discussion on merit pay.  B4K believes that merit pay should be a key component of any comprehensive education reform plan.   It must be tied into evaluation system that rates teachers fairly and transparently, and integrates professional development into the process.   It should reward superior performance, augment promotion to positions of greater responsibility and incentivize effective teachers to teach in high-needs schools and subject areas.  And, not least, it should help to attract and retain the highest quality teachers.
Yes, and it should also get those stubborn grass stains out of your whites, burn fat without exercise, and leave your breath minty fresh. But I'm afraid, Derrell, that those of us back on planet Earth have to consider what a real merit pay system would look like.

And it ain't pretty. The implementation of the type of merit pay Derrell touts is inevitably tied to Value-Added Modeling based on standardized tests scores. This is a wildly flawed method that will inevitably misidentify teachers.

Further, over and over again we hear from the conservative governors pushing corporate reform that "we don't have any more money." If that's true, any merit pay scheme will be predicated on the idea of taking money from a "bad" teacher and giving it to a "good" one. Need I spell out what a recipe for disaster this is going to be? At the very least, parents are going to be up in arms about how their children are assigned to teachers who have or haven't earned merit pay.

And getting any sort of buy-in during this era of teacher bashing is nearly impossible:
Policy makers often complain about the difficulty of changing teachers’ behavior. This is not surprising in an environment where teachers have few guarantees that their efforts to change will be rewarded. Change is slow and difficult work, and it is tempting for teachers to ignore the policy de jour since the next regime change is likely to, once again, change the rules of the game. Education has a reputation for constant policy change; educa- tional mandates come and go before it is really possible for meaningful change to take place (Cohen 1988; Cuban 1993; Tyack and Cuban 1995). 
The inability of school districts to credibly commit has been largely ignored in the research literature but may be a significant factor in the failure of many policies to positively affect practice. While this study only looks at the Minneapolis case, it has implications for education policy more generally. The broader question is this: Can a public school district credibly commit to policies that, in order to be effective, require a long-term financial and political commitment? The answer to this question may very well be a resounding, “No.” [emphasis mine]
In other words: this is the latest scam. And, given the corporate reform movement's track record, there's just no reason to trust it, especially when folks like Derrell are spending millions on ads that bash our unions.

Let me break in with a personal opinion: like charter schools, I do think it's possible that merit pay has its place. I've argued for a long time that the structure of schools is too flat; if we're going to make an analogy to business, I think most managers would say that a principal shouldn't have so many direct reports. Tying "master teacher" status to pay may be a good thing. Rewarding excellence - with a buy-in from teachers and their unions - may have it's place.

But let's be clear: if you made a list of stuff to do to really address the achievement gap, merit pay would be at the bottom. And it's funny Derrell wants to use Colorado as an example, because Bruce Baker clues us in, while testifying in a case there, as to what should be at the top:
Baker discussed at length the higher cost of educating at-risk students and said when those costs are taken into account, Colorado districts with large numbers of such students are at a noticeable disadvantage. He also said funding gaps can account for 60 percent of achievement gaps in reading and 46 percent of math achievement gaps. [emphasis mine]
We see the two worlds of reform very clearly here. Bruce Baker, the adult in the room, is telling us that there are no magic wands. You want to address the achievement gap? Pay up. You need to get more resources to the kids who need it the most. I would only add that those resources need to be accompanied by a push to eliminate poverty, give people the opportunity to have good jobs with living wages, build infrastructure, provide health care, and eliminate racism.

Derrell, however, lives in a happy, shiny world were all problems will be magically wiped away with a few more charter schools, some vouchers, and merit pay. He believes this so much that he keeps staying up past his bedtime every night, waiting for the Merit Pay Fairy to wave her wand and transform underperforming schools into Neverlands.

And he's very, very cross when anyone tries to explain to him that it's time to stop believing in the Merit Pay Fairy and finally grow up.

Yo, Derrell - don't hold yer breath waitin' fer me, OK pal?


CommutingTeacher said...

Thank you! You have clearly and definitively said what the rest of us have scattered through our heads. Now, to get the rest of the public to see this.

Duke said...

Thx, CT.

Teacher Mom said...

Brilliant, absolutely brilliant!

Here teachers are already incentivized to teach in tougher districts and needier students. On average teachers in NJ make more in urban districts. We in the ghetto call it "combat pay." When I first came to the inner city I made $5,000 more a year with fewer steps than in the suburbs. I have to admit I work much harder in the inner city with fewer "results." (There are days where I would willingly take a pay cut to go back.)

I had a 98% pass rate in the burbs. In the city I'm at about 60% up from 30% my first year there. I ask people all the time, "Do you think I somehow became worse when I came to the city?" No, the kids there just have far more to overcome before school even starts. So what happens when those "great" teachers are "incentivized" to come from the burbs and scores still don't go up? What then?

Kristina said...

The Vanderbilt study, a very much anticipated study also found merit pay not to improve student outcome on tests.

Duke said...

Thx TM. Glad you got the blog going again.

Kristina, the Tennessee link is to a USA Today piece about the Vanderbuilt study, but I didn't label it as such.

I'll redo this piece in a more concise form later and include a direct link. Thx for reading!

Unknown said...

TAP was developed by the Milken foundation and the "studies" Bradford quoted were done by their paid staff on NIET. More on that after this:
Do you remember junk-bond king Michael Milken and his bro Lowell? They were involved in bilking investors of millions on Wall Street in the 1980's. Michael went to federal prison in the days when the Justices Dept investigated Wall Street Fraud. When he was released in the early 1990's, he had an awakening, so to speak. His deep social consciousness led him to save poor kids by founding his education foundation. You know, a not-for-profit think-tank. The model seems to follow that of tobacco and oil companies. As written about in "Merchants of Doubt", these corp. purchased well known scientists to spread doubt about the independent scientific evidence on climate change and health effects of tobacco.

He and his bro have been selling their snake oil for years. You have to hand it to the Milkens, they can spot a tiny opening in the market and drive a truck through it.

Hence, TAP and NIET. I have a copy of NIET's first research study on TAP effects done by the Milkens bought and paid-for researchers. The data analysis is garbage. They failed to run any statistical analysis on scores and reported only percentage changes- a no-no for the type of research design. My take is they ran the analysis and there were no effects so they reported it to show effect of TAP.

I'll send it to you if you or anyone else wants to see it. It's loaded with B-school jargon.

In light of this, we'd have to question the veracity of all of NIET's analyses and review independent studies of TAP effects. Unfortunately, they've convinced MANY state legislators that they are the real deal in edu-merit pay reforms.

Duke said...

God stuff, jcg.

The Stanford study had at least SOME peer-review as a grad thesis, although I don't think it went fully through the ringer. But the design with synthetic control is awfully suspect to me.

The stuff you're talking about is garbage, straight up. The ed world is full of bought and paid for pseudo-research like this.