Clapping Harder For the Merit Pay Fairy

UPDATE BELOW

Earlier this week, I wrote about the death of the Merit Pay Fairy in Newark, New Jersey.

Hey, Jazzman, you bum -- I ain't dead yet!

Back in 2012, Newark began an experiment in teacher merit pay, fueled by funds from Facebook's Mark Zuckerberg. Teachers were promised up to $20 million over three years in extra incentive pay -- but in the first year, only $1.4 million was disbursed, and most of that appears to have comes from other teachers, who had their pay docked because they were deemed "ineffective."

Merit pay, in other words, was little more than a broken promise to the teachers of Newark right from the start. A survey of Newark teachers in the first year found a large majority did not see the compensation system as "reasonable, fair, and appropriate." (p. 24) It's not a surprise, therefore, that this past month both the teachers union in Newark, the NTU, and the district's administration decided that the program was not worth continuing. 

But some reformy folks believe in merit pay the same way some children believe in fairies: they don't want to acknowledge the evidence that shows, even in the most generous reading, that the benefits of merit pay are very small and likely are not indicative of true increases in student learning. Like Peter Pan, these true believers hope against hope that fairies can be brought back to life simply by clapping harder:

In 2012, Newark Public Schools did something remarkable. The district reached an agreement with the Newark Teachers Union that would fundamentally shift how teachers are not just evaluated but paid. 

Then-Gov. Chris Christie and Randi Weingarten, the President of the American Federation of Teachers, announced the groundbreaking deal together on national television. At the time, my organization, the National Council on Teacher Quality, called the contract “a model to which other districts should aspire.”

That's from Kate Walsh, president of NCTQ, an organization that has previously held up Newark Public Schools (NPS) as an exemplar of teacher evaluation, claiming the district was "getting results" from its system:

The district gave the evaluation system a chance to work. While Newark saw students’ achievement decline initially after the new evaluation system was implemented, the district persevered and student achievement rose to the level it had been before and, in English, exceeded previous levels. (p.12)

It's hard to be more vague than that -- how was student achievement measured? What was the improvement? Most importantly: how do we know if the teacher evaluation system was affecting results?

As Bruce Baker and I pointed out in our review of NPS "reforms," plenty of other districts with similar demographics were showing similar growth in student achievement, without things like merit pay. In addition, Newark has not seen the same demographic shifts over time many comparison districts have.

You must account for this stuff if you're going to make a causal claim about merit pay in Newark. Alas, Walsh still seems wholly uninterested in digging into these details; for example:

The district would also start to use their dollars in the same way that other employers do. Pay would become a strategic tool to attract the best teachers to where they were most needed. Teachers who were ineffective would no longer receive an annual raise. Teachers who were rated as highly effective earned a healthy $5,000 bonus. Even better, high performing teachers who were either able to teach subjects that were hard for the district to staff, particularly in the lowest performing schools, could earn even more, up to $12,500 a year.

In the first year of the contract, Newark had about 3,200 teachers. How many qualified for the highest bonus, $12,500? Only eleven. Is Walsh really trying to make the case this small disbursal made a significant difference in teacher quality in Newark? She continues:

The results spoke for themselves. After five years of implementation, 96 percent of highly effective teachers chose to stay in Newark and 49 percent of ineffective teachers were voluntarily leaving the district—exactly the sort of pattern schools need to see but rarely do. Accordingly, the district has higher student enrollment now than at any other time in recent history, suggesting parents gained a renewed confidence in the district.

First of all, we have no way of knowing whether these teacher attrition and retention rates are significantly better than they would be in the absence of the merit pay scheme. We don't know how they compare to similar districts' rates. We don't even know how they compare to rates before merit pay in Newark. Again, it's completely unwarranted to make any sort of causal claim without at least some attempt to compare these rates to a counterfactual.

Second, the notion that there's any evidence that shows student enrollment has increased because of "renewed confidence" due to merit pay is absurd on its face: "I was going to move my family, but now that Newark has teacher merit pay, we're staying!" Maybe the city's child population is simply increasing. Is Walsh so enamored with merit pay she's willing to make wild stretches like this?

Apparently, she is:

Where once compensation was used as a strategic deployment of resources to ensure the district can fill its vacancies, keep its best teachers, and ensure the most vulnerable students have access to them, soon there will be nothing but raises based on years of experience, and requiring teachers to spend precious time and money earning another degree they more than likely do not need. 

Research shows over and over again that advanced degrees do not make teachers more effective with the exception of math and science. 

First of all, there is a large body of evidence that shows teacher experience correlates with effectiveness. And while gains in effectiveness are strongest in the first few years, gains do persist up through the third decade of a teacher's career. Tying compensation to experience is hardly a policy without evidence to support it.

Next: "the exception of math and science" is a very big exception. Is Walsh prepared to offer a bonus only to math and science teachers with advanced degrees? And is she really sure French teachers don't benefit from degrees in French, or music teachers don't benefit from degrees in conducting, or that teachers in many other subjects don't benefit from gaining expertise through earning an advanced degree in that subject?

I've been looking at the research on advanced degrees and teaching for some time now, and the conclusion I've come to is that it is highly limited. Most studies don't account for alignment of teaching subject and degree concentration; in other words, the results are likely skewed because they don't separate getting a degree in what you teach from getting any degree.

These studies also usually don't account for variations in the quality of the degree-granting programs: crappy on-line programs are lumped in with rigorous degrees from research universities. In addition, the student outcomes are almost always measured by test scores, which limits the teachers studied to tested grades (3-8) and only two subjects (math and English).

Walsh's sweeping statement is simply not justified. Further, she ignores the reality that NPS must compete with other districts that offer masters pay bumps to attract qualified teaching candidates. Is the district suppose to ignore this reality? Especially because there is no evidence whatsoever that NPS has attracted better candidates to its teaching staff than other districts?

As I said in the last post: there was supposed to be an ongoing study of merit pay in NPS. But that study ended after a single year. We have no evidence whatsoever that Newark attracted better teaching candidates, improved student outcomes, or raised teacher effectiveness by using merit pay.

But this lack of evidence isn't stopping Walsh from clapping harder:

In 2012, Newark Public Schools took bold steps to create a compensation system that would help to attract and keep the best teachers. The district used resources strategically to ensure the most vulnerable students had access to the best teachers, an accomplishment that many districts struggle to achieve. With this new contract, instead of being a leader in strategic compensation, Newark becomes a district that takes a one-size-fits-all approach to its teachers, to the detriment of its students and teachers alike.

Even by current reformy standards, this statement is way over the top. We have no evidence the best teachers in Newark went to the neediest students. We have no evidence Newark was better at teacher allocation than districts that didn't implement merit pay. We have no evidence Newark is now a "one-size-fits-all" district. We have no evidence merit pay was a benefit to Newark's teachers and students.

What we do know is that the majority of Newark teachers didn't think the system was fair. That, by itself, is enough to declare Newark's merit pay experiment dead -- even if some folks keep clapping for it.



UPDATE: After I posted this blog, I went back and looked at the teacher survey again, which is part of a report commissioned by the American Institutes for Research. AIR is an excellent research organization, and they produce high-quality work. That said, there are a few oddities in their Newark report:

In response to a set of questions about their knowledge of the current evaluation process, 83 percent of teachers and 99 percent of school leaders reported that they have a clear understanding of the evaluation process. In addition, in response to a set of questions about the fairness of the evaluation process, 72 percent of teachers and 92 percent of school leaders reported that the evaluation process is fair, which is larger than the 30 percent reported fairness by teachers in an evaluation of 25 districts in New Jersey (Firestone, Nordin, Shcherbakov, Kirova, & Blitz, 2014) and the 39 percent reported fairness by teachers in 10 districts in Arizona (Ruffini, Makkonen, Tejwani, & Diaz, 2014). [p.20]

Let's set aside the Arizona survey, which I haven't yet read, and just focus on the New Jersey one. That report, which I'm well-acquainted with as it came out of Rutgers (where I got my PhD and current teach part-time), did not have a sample of representative districts. It was, instead, an evaluation of a pilot program of teacher evaluation conducted in 25 districts across the state. It had a low response rate (39 percent), but more important, it was conducting a survey after the state had imposed a new law on districts, TEACHNJ, forcing them to rework their evaluation systems.

I can tell you as a teacher who lived through that time: TEACHNJ was not popular with many working teachers in the state. So it shouldn't be surprising the popularity of the new system was so low. Unlike Newark in its first year, there wasn't a whole bunch of money promised from an outside source going to these districts.

My point here is that the comparison is, at best, strained. And AIR really should have spelled out more clearly the limits of the comparison of the two reports.

Here's another oddity from the AIR report:

Figure 2 shows that the retention rates among teachers rated “effective” and “highly effective” exceed 90 percent, whereas retention rates among teachers rated “partially effective” and “ineffective” are 72 percent and 63 percent, respectively. In contrast, the most recent results from the national 2012–13 Teacher Follow-Up Survey indicate that 84 percent of public school teachers are retained, on average (Goldring, Taie, & Riddles, 2014).

Golding et al. does find 84.3 percent of teachers stayed in their positions -- but that's all teachers, not "effective" ones. There's simply no way to know, based on this report, whether Newark did any better in retaining its better teachers thanks to merit pay.

Again, it's fine to include this data point, but a little more context is probably in order.

The Merit Pay Fairy Dies in Newark

One of the long-running characters on this blog is the Merit Pay Fairy.

Hey, youse bums -- get back to the teachin' already!

The Merit Pay Fairy lives in the dreams of right-wing think tanks and labor economists, who are absolutely convinced that our current teacher pay system -- based on seniority and educational attainment -- is keeping teachers from achieving their fullest potential. It matters little that even the most generous readings of the research find practically small effects* of switching to pay-for-performance systems, or that merit pay in other professions is quite rare (especially when it is based on the performance of others; teacher merit pay is, in many contexts, based on student, and not teacher, performance). 

Merit pay advocates also rarely acknowledge that adult developmental theory suggests that rewards later in life, such as higher pay, fulfill a need for older workers, or that messing with pay distributions has the potential to screw up the pool of potential teacher candidates, or that shifting pay from the bottom of the teacher "quality" distribution to the top -- and, really, that's what merit pay does -- still leaves policymakers with the problem of deciding which students get which teachers.

Issues like these, however, are at the core of any merit pay policy. Sure, pay-for-performance sounds great; it comports nicely with key concepts in economic theory. But when it comes time to implement it in an actual, real-world situation, you've got to confront a whole host of realities that theory doesn't address.

Which is what seems to have happened in Newark:

In 2012, Newark teachers agreed to a controversial new contract that linked their pay to student achievement — a stark departure from the way most teachers across the country are paid. 

The idea was to reward teachers for excellent performance, rather than how many years they spent in the district or degrees they attained. Under the new contract, teachers could earn bonuses and raises only if they received satisfactory or better ratings, and advanced degrees would no longer elevate teachers to a higher pay scale. 

The changes were considered a major victory for the so-called “education reform” movement, which sought to inject corporate-style accountability and compensation practices into public education. And they were championed by an unlikely trio: New Jersey’s Republican governor, the Democratic-aligned leader of the nation’s second-largest teachers union, and Facebook founder Mark Zuckerberg, who had allocated half of his $100 million gift to Newark’s schools to fund a new teachers contract. 

“In my heart, this is what I was hoping for: that Newark would lead a transformational change in education in America,” then-Gov. Chris Christie said in Nov. 2012 after the contract was ratified. 

Seven years later, those changes have been erased. 

Last week, negotiators for the Newark Teachers Union and the district struck a deal for a new contract that scraps the bonuses for top-rated teachers, allows low-rated teachers to earn raises, and gives teachers with advanced degrees more pay. It also eliminates other provisions of the 2012 contract, which were continued in a follow-up agreement in 2017, including longer hours for low-performing schools. [emphasis mine]

I blogged about that 2012 contract many times as it was being negotiated. It was never popular; only 37 percent of the membership approved it, thanks to low turnout in the voting. The teachers were promised up to $20 million in extra funding, over three years, dedicated to merit pay, all coming from Mark Zuckerberg's famous donation to Newark's schools. But the actual disbursement was far less: only $1.4 million in the first year. And much of that appears to have come from teachers who were denied regular annual raises due to "poor performance."

More senior teachers with advanced degrees had the option of not participating in merit pay; only about 20% chose to enter the system, putting to rest any idea merit pay was popular among teachers who had a choice. In addition, the percentage of "highly effective" teachers was much higher in the pool of teachers who opted out of merit pay than those who were in the system.

This pretty much destroys the notion that "better" teachers are clamoring for merit pay; in Newark, many doubted the system would work to their benefit. 

This trepidation can be found in a survey of Newark teachers by the American Institutes for Research, which was conducted in the first year of the contract. Over 40 percent of teachers both in and out of the merit pay system believed merit pay would hurt collaboration (p. 25); 60 percent believed the system ignored important aspects of their teaching (p. 24). Nevertheless, over 70 percent believed their pay system was fair and appropriate at their school. My reading of these results is that there was concern over the system, but teachers were willing to give it a shot.

Those of us who follow Newark's schools know what happened next: a mass revolt against the district's administration, which reported directly to the governor at the time, Chris Christie. A mayoral election where state control of schools was the key issue. The return of local control of schools after two decades. And now, the end of Newark's merit pay experiment.

One curious thing about the AIR report: it was labeled as "Year One":

NPS [Newark Public Schools] commissioned American Institutes for Research (AIR) to conduct an evaluation of the implementation and impact of the NPS/NTU contract and associated initiatives. The three-year evaluation focuses on a variety of outcomes (e.g., educator perceptions, teacher retention, teacher effectiveness, and student achievement) associated with the four contract components. In the first year of the evaluation, the period to which this report corresponds, the evaluation team used qualitative and quantitative techniques to assess the implementation of the contract components and to examine the association between the new evaluation and compensation systems (i.e., Components 1 and 2) and teacher retention. This report presents findings related to educator perceptions, as captured by teacher and school leader surveys administered in spring 2015, after two years of contract implementation (i.e., as of the 2014–15 school year) and teacher retention after one year of contract implementation (i.e., through the 2013–14 school year).1 The AIR evaluation team plans to examine the contract’s impact on teacher effectiveness and student achievement in 2016 and 2017, respectively. [emphasis mine]

Guess what? There was ever a follow-up study of Newark's merit pay system. For whatever reason, AIR never published any further reports on whether merit pay helped improve student learning, improved effective employee retention, or improved the quality of new teaching candidates in NPS.

You would think, given all the hoopla over this contract at the time, that a thorough study of how merit pay played out in Newark would have been a top priority for the state, the district, and all of the folks who were connected to the Zuckerberg donation. Alas, we'll never know how this hyped contract affected the district. We'll never know if merit pay in Newark -- perhaps the most high-profile implementation of teacher pay-for-performance in the United States -- actually worked.

Were I cynical, I'd think that the folks behind this system didn't really want to know whether merit pay works. I'd think they want to keep merit pay a policy based on theory, not evidence. Because if Newark turned out like all the other merit pay experiments, it would show, at best, practically small improvements in student outcomes. And the price for that tiny improvement would be a chaotic and impractical system of teacher evaluation and compensation that was always doomed to fail.

Good thing I'm not cynical...

For those of you who weren't with me during the early, snarkier days of this blog: the idea for the Merit Pay Fairy came from a scene in a play by Christopher Durang:

“You remember how in the second act Tinkerbell drinks some poison that Peter is about to drink in order to save him? And then Peter turns to the audience and he says that ‘Tinkerbell is going to die because not enough people believe in fairies. But if all of you clap your hands real hard to show that you do believe in fairies, maybe she won’t die.’ So, we all started to clap. I clapped so long and so hard that my palms hurt and they even started to bleed I clapped so hard. Then suddenly the actress playing Peter Pan turned to the audience and she said, ‘That wasn’t enough. You did not clap hard enough. Tinkerbell is dead.’ And then we all started to cry.”

It really doesn't matter how many deaths the Merit Pay Fairy dies -- some folks just keep clapping harder:

Shavar Jeffries, who led the Newark school board in 2012 and is now president of the national advocacy group Democrats for Education Reform said he is happy to see teachers get more money under the new agreement. But he said it is disappointing that teachers’ performance will no longer automatically influence their pay — a disconnect, he argued, that many families do not support.

“There’s almost no parent in the city of Newark,” he said, “who thinks that there shouldn’t be a relationship between pay and whether you’re actually doing a good job for babies each and every day in the classroom.”

I don't live in Newark, but I think I'm safe in making this statement: there is definitely no parent in Newark who doesn't want a good teacher for their own kid. The question, then, is what are we doing to ensure that every child in Newark -- and, for that matter, every community -- has a well-trained, competent, effective teacher in their classroom.

Taking away money from "bad" teachers and giving it to "good" ones does little to improve the overall effectiveness of the teaching corps. What would help is raising the base pay for teachers so as to close the compensation gap they suffer compared to other professions; that way, we could attract the best possible candidates into the profession and increase the chances that every child has an effective teacher.

What won't help is imposing unfeasible pay-for-performance systems that, time after time, fail to deliver meaningful improvements. Clap as hard as you want -- the Merit Pay Fairy is dead in Newark, and the prognosis in other communities is not good.

* I'm adding this note in anticipation of a counter-argument for merit pay I've seen before: if you include studies in other countries, the effect size of merit pay grows. But these other countries have such different contexts for teaching and worker pay that applying their results to the U.S. is a highly dubious proposition. Further, the effect size is still very small -- again, even under the most generous interpretation of the results. When effects are so small, we are justified in questioning whether the variation is related to the construct; in other words, are there real improvements in learning, or are teachers just slightly pumping up scores through test prep?

No, I'm not going to debate this on Twitter. Write an article or blog post and I'll respond.