Clapping Harder For the Merit Pay Fairy

UPDATE BELOW

Earlier this week, I wrote about the death of the Merit Pay Fairy in Newark, New Jersey.

Hey, Jazzman, you bum -- I ain't dead yet!

Back in 2012, Newark began an experiment in teacher merit pay, fueled by funds from Facebook's Mark Zuckerberg. Teachers were promised up to $20 million over three years in extra incentive pay -- but in the first year, only $1.4 million was disbursed, and most of that appears to have comes from other teachers, who had their pay docked because they were deemed "ineffective."

Merit pay, in other words, was little more than a broken promise to the teachers of Newark right from the start. A survey of Newark teachers in the first year found a large majority did not see the compensation system as "reasonable, fair, and appropriate." (p. 24) It's not a surprise, therefore, that this past month both the teachers union in Newark, the NTU, and the district's administration decided that the program was not worth continuing. 

But some reformy folks believe in merit pay the same way some children believe in fairies: they don't want to acknowledge the evidence that shows, even in the most generous reading, that the benefits of merit pay are very small and likely are not indicative of true increases in student learning. Like Peter Pan, these true believers hope against hope that fairies can be brought back to life simply by clapping harder:

In 2012, Newark Public Schools did something remarkable. The district reached an agreement with the Newark Teachers Union that would fundamentally shift how teachers are not just evaluated but paid. 

Then-Gov. Chris Christie and Randi Weingarten, the President of the American Federation of Teachers, announced the groundbreaking deal together on national television. At the time, my organization, the National Council on Teacher Quality, called the contract “a model to which other districts should aspire.”

That's from Kate Walsh, president of NCTQ, an organization that has previously held up Newark Public Schools (NPS) as an exemplar of teacher evaluation, claiming the district was "getting results" from its system:

The district gave the evaluation system a chance to work. While Newark saw students’ achievement decline initially after the new evaluation system was implemented, the district persevered and student achievement rose to the level it had been before and, in English, exceeded previous levels. (p.12)

It's hard to be more vague than that -- how was student achievement measured? What was the improvement? Most importantly: how do we know if the teacher evaluation system was affecting results?

As Bruce Baker and I pointed out in our review of NPS "reforms," plenty of other districts with similar demographics were showing similar growth in student achievement, without things like merit pay. In addition, Newark has not seen the same demographic shifts over time many comparison districts have.

You must account for this stuff if you're going to make a causal claim about merit pay in Newark. Alas, Walsh still seems wholly uninterested in digging into these details; for example:

The district would also start to use their dollars in the same way that other employers do. Pay would become a strategic tool to attract the best teachers to where they were most needed. Teachers who were ineffective would no longer receive an annual raise. Teachers who were rated as highly effective earned a healthy $5,000 bonus. Even better, high performing teachers who were either able to teach subjects that were hard for the district to staff, particularly in the lowest performing schools, could earn even more, up to $12,500 a year.

In the first year of the contract, Newark had about 3,200 teachers. How many qualified for the highest bonus, $12,500? Only eleven. Is Walsh really trying to make the case this small disbursal made a significant difference in teacher quality in Newark? She continues:

The results spoke for themselves. After five years of implementation, 96 percent of highly effective teachers chose to stay in Newark and 49 percent of ineffective teachers were voluntarily leaving the district—exactly the sort of pattern schools need to see but rarely do. Accordingly, the district has higher student enrollment now than at any other time in recent history, suggesting parents gained a renewed confidence in the district.

First of all, we have no way of knowing whether these teacher attrition and retention rates are significantly better than they would be in the absence of the merit pay scheme. We don't know how they compare to similar districts' rates. We don't even know how they compare to rates before merit pay in Newark. Again, it's completely unwarranted to make any sort of causal claim without at least some attempt to compare these rates to a counterfactual.

Second, the notion that there's any evidence that shows student enrollment has increased because of "renewed confidence" due to merit pay is absurd on its face: "I was going to move my family, but now that Newark has teacher merit pay, we're staying!" Maybe the city's child population is simply increasing. Is Walsh so enamored with merit pay she's willing to make wild stretches like this?

Apparently, she is:

Where once compensation was used as a strategic deployment of resources to ensure the district can fill its vacancies, keep its best teachers, and ensure the most vulnerable students have access to them, soon there will be nothing but raises based on years of experience, and requiring teachers to spend precious time and money earning another degree they more than likely do not need. 

Research shows over and over again that advanced degrees do not make teachers more effective with the exception of math and science. 

First of all, there is a large body of evidence that shows teacher experience correlates with effectiveness. And while gains in effectiveness are strongest in the first few years, gains do persist up through the third decade of a teacher's career. Tying compensation to experience is hardly a policy without evidence to support it.

Next: "the exception of math and science" is a very big exception. Is Walsh prepared to offer a bonus only to math and science teachers with advanced degrees? And is she really sure French teachers don't benefit from degrees in French, or music teachers don't benefit from degrees in conducting, or that teachers in many other subjects don't benefit from gaining expertise through earning an advanced degree in that subject?

I've been looking at the research on advanced degrees and teaching for some time now, and the conclusion I've come to is that it is highly limited. Most studies don't account for alignment of teaching subject and degree concentration; in other words, the results are likely skewed because they don't separate getting a degree in what you teach from getting any degree.

These studies also usually don't account for variations in the quality of the degree-granting programs: crappy on-line programs are lumped in with rigorous degrees from research universities. In addition, the student outcomes are almost always measured by test scores, which limits the teachers studied to tested grades (3-8) and only two subjects (math and English).

Walsh's sweeping statement is simply not justified. Further, she ignores the reality that NPS must compete with other districts that offer masters pay bumps to attract qualified teaching candidates. Is the district suppose to ignore this reality? Especially because there is no evidence whatsoever that NPS has attracted better candidates to its teaching staff than other districts?

As I said in the last post: there was supposed to be an ongoing study of merit pay in NPS. But that study ended after a single year. We have no evidence whatsoever that Newark attracted better teaching candidates, improved student outcomes, or raised teacher effectiveness by using merit pay.

But this lack of evidence isn't stopping Walsh from clapping harder:

In 2012, Newark Public Schools took bold steps to create a compensation system that would help to attract and keep the best teachers. The district used resources strategically to ensure the most vulnerable students had access to the best teachers, an accomplishment that many districts struggle to achieve. With this new contract, instead of being a leader in strategic compensation, Newark becomes a district that takes a one-size-fits-all approach to its teachers, to the detriment of its students and teachers alike.

Even by current reformy standards, this statement is way over the top. We have no evidence the best teachers in Newark went to the neediest students. We have no evidence Newark was better at teacher allocation than districts that didn't implement merit pay. We have no evidence Newark is now a "one-size-fits-all" district. We have no evidence merit pay was a benefit to Newark's teachers and students.

What we do know is that the majority of Newark teachers didn't think the system was fair. That, by itself, is enough to declare Newark's merit pay experiment dead -- even if some folks keep clapping for it.



UPDATE: After I posted this blog, I went back and looked at the teacher survey again, which is part of a report commissioned by the American Institutes for Research. AIR is an excellent research organization, and they produce high-quality work. That said, there are a few oddities in their Newark report:

In response to a set of questions about their knowledge of the current evaluation process, 83 percent of teachers and 99 percent of school leaders reported that they have a clear understanding of the evaluation process. In addition, in response to a set of questions about the fairness of the evaluation process, 72 percent of teachers and 92 percent of school leaders reported that the evaluation process is fair, which is larger than the 30 percent reported fairness by teachers in an evaluation of 25 districts in New Jersey (Firestone, Nordin, Shcherbakov, Kirova, & Blitz, 2014) and the 39 percent reported fairness by teachers in 10 districts in Arizona (Ruffini, Makkonen, Tejwani, & Diaz, 2014). [p.20]

Let's set aside the Arizona survey, which I haven't yet read, and just focus on the New Jersey one. That report, which I'm well-acquainted with as it came out of Rutgers (where I got my PhD and current teach part-time), did not have a sample of representative districts. It was, instead, an evaluation of a pilot program of teacher evaluation conducted in 25 districts across the state. It had a low response rate (39 percent), but more important, it was conducting a survey after the state had imposed a new law on districts, TEACHNJ, forcing them to rework their evaluation systems.

I can tell you as a teacher who lived through that time: TEACHNJ was not popular with many working teachers in the state. So it shouldn't be surprising the popularity of the new system was so low. Unlike Newark in its first year, there wasn't a whole bunch of money promised from an outside source going to these districts.

My point here is that the comparison is, at best, strained. And AIR really should have spelled out more clearly the limits of the comparison of the two reports.

Here's another oddity from the AIR report:

Figure 2 shows that the retention rates among teachers rated “effective” and “highly effective” exceed 90 percent, whereas retention rates among teachers rated “partially effective” and “ineffective” are 72 percent and 63 percent, respectively. In contrast, the most recent results from the national 2012–13 Teacher Follow-Up Survey indicate that 84 percent of public school teachers are retained, on average (Goldring, Taie, & Riddles, 2014).

Golding et al. does find 84.3 percent of teachers stayed in their positions -- but that's all teachers, not "effective" ones. There's simply no way to know, based on this report, whether Newark did any better in retaining its better teachers thanks to merit pay.

Again, it's fine to include this data point, but a little more context is probably in order.