Blogging Resumes

It's been a while...

I won't get into the details, but suffice to say I had to take some time off from blogging. Too much stuff on my plate, and I was running the risk of doing it all badly. Something had to give.

During the time off, I thought about shutting the blog down. I've got a place to do my Jersey-specific work, another place to do descriptive analysis and other similar research, another place to preach the gospel of equitable and adequate school funding, and a bunch of other things in the works for 2020 -- including a few partners some of you will probably be shocked to see me working with (heh, heh)...

So why keep blogging? It's way more time-intensive than you probably think, it doesn't get me any accolades, it certainly doesn't make me new friends (usually it's quite the opposite). Many times blogging has distracted me from doing paid work, or work that would up my cred in academic circles, or work around the house that keeps my marriage happy.

So why keep blogging?

The simplest answer is that I think there are analyses to be conducted or data to be crunched or points to be made that aren't appropriate for an academic journal, or a formal policy brief, or an op-ed in a news site. A blog is a junk drawer for a policy nerd like me: a place to throw all the stuff that doesn't really go anywhere else.

But it's also a place to shoot down bad faith arguments, poor analysis, and general nonsense. And we have plenty of that in the K-12 space these days. It's also a place to break down needless complexity, which is pervasive within the education policy world, especially since the economists took over... Kidding! (Sort of...)

And I've found that this blog has been a good place to explain my own work, in a voice that some may find more accessible. I was right in the middle of that when I stopped; I'm going to pick up where I left off next.

After that... who knows? As Booker T says, time is tight. So let's make the most of it. See you in a bit, and Happy New Year.

The State of New Jersey's Teachers, Part 2: A Failure To Achieve Diversity

I'm breaking down my new report for the New Jersey Policy Perspective on New Jersey's teachers in a series of blog posts:

Part 1: Teachers, Aging, & Pensions

* * *

This graph is from the "short" version of my new report on New Jersey's teachers:

About 1 in 5 students in New Jersey is a white female -- but two-thirds of NJ's teachers are white females. Our teaching corps looks nothing like our student population.

Before I dive deeper into the data, let me first answer the obvious question: Why should we care? Does it matter that our teachers are overwhelmingly white women? 

In fact, we have more and more evidence that it does, and I share a short summary of this evidence in my report. Among the studies I cite:

Colette N. Cann, 2013: This article has an excellent review of the research on teacher-student racial alignment:

In the area of race matching, Dee (2004) examined how the racial background of teachers contributes to the academic performance of students— particularly Black students. He found that Black and White students both performed better with teachers of the same race possibly because same-race-matching of students and teachers provided positive role models for students and reduced racist teacher practices that adversely affected student performance. 

Eddy and Easton-Brooks (2011) looked at whether having an African American teacher increased the mathematics performance of Black students from kindergarten through fifth grade. Using data from 1,200 students in the Early Childhood Longitudinal Study, kindergarten through fifth grades (ECLS-K-5), they found that racial matching mattered to mathematics scores for Black youth (although they didn’t find any differences by gender, socio-economic status, or racial composition of the school). 

Other studies have shown that there is a statistically significant and important relationship among the race of the teacher, the race of the student, and academic performance. Easton-Brooks et al. (2010), as cited in Eddy and Easton-Brooks (2011), found that the performance of Black youth in reading was higher if students had at least one Black teacher between kindergarten and fifth grade.

Egalite, Kisida, and Winter, 2015:“We find small but significant positive effects when black and white students are assigned to teachers of their own race/ethnicity in reading and when black, white, and Asian/Pacific Islander students are assigned to teachers of their own race/ethnicity in math.

We also examine the effects of race matching by students' prior performance level, finding that lower-performing black and white students appear to particularly benefit from being assigned to a race-congruent teacher."

Lindsay and Hart, 2017: “We find consistent evidence that exposure to same-race teachers is associated with reduced rates of exclusionary discipline for Black students. This relationship holds for elementary, middle, and high school grade ranges for male and female students, and for students who do and do not use free and reduced-price lunch. Although we find reductions in referrals for a number of different types of offenses, we find particularly consistent evidence that exposure to same-race teachers lowers office referrals for willful defiance across all grade levels, suggesting that teacher discretion plays a role in driving our results."

Look, I don't think it's a good idea for black students to only have black teachers, and white students only have white teachers, and girls only have women teachers, and so on. But I do think we've got to have a teaching workforce that looks like our student body. Even if we didn't have all this evidence that teacher diversity helps student achievement, it's enough to say that teacher diversity is important because we want our schools to reflect the best values of our society -- and that includes prizing people of all different backgrounds.

But in New Jersey, we're not doing a very good job at creating a diverse workforce for our schools. Let's start with gender:

4 out of 5 New Jersey teachers are women. To those who still contend teaching is a well-paid profession (we'll get to this topic next), let me ask you this: why aren't men flocking to the profession? I mean, if it pays so well, and has such awesome job security, you'd expect the teaching corps to be getting more male, not less. But exactly the opposite is happening.

And, again: it's not just women who dominate teaching, it's white women.

There's been a very small decline in the percentage of teachers who are white; still, over 4 in 5 teachers are white as of 2017. How the other races are represented in the teaching corps is interesting:

Let me first note that race and ethnicity in the teacher workforce is defined by seven categories: White, Black, Hispanic, Asian, Native American, Pacific Islander/Native Alaskan, and Two or More Races. The number of teachers in these last three categories is so small I didn't include them in this analysis.

Over the last two decade, the percentage of Asian teachers has gone up very slightly. The proportion of Hispanic teachers has grown a bit more substantially. But the percentage of Black teachers, which was only 9 percent anyway in 1997, has gone actually down. 

This is discouraging, and we need to figure out why. Is the barrier to entry too great? Are there increasingly better opportunities than teaching for black college graduates? Are school districts just not hiring as many black teachers? We probably can't get to the answers through data analysis; we're going have to start doing some qualitative research into this question (and I'm going to have to start digging more through the literature to see what we already know).

Let's finish by looking at the entire student population of New Jersey, and compare it to the teacher population. Here are the students as of 2017:

And here are the teachers:

Notice how very few men of color are teachers; it's really remarkable, considering how many Black and Hispanic male students we have. 

A reporter asked me the other day whether I thought the outreach programs at teacher training institutes, especially universities, could help. I said it couldn't hurt... but let's be honest: not many talented young people of color will seriously consider a teaching career if they have other, more lucrative prospects in the labor market. 

Why would we expect graduates of color to react to economic forces differently than white college graduates? Don't they have the same student loans to pay off? Don't they have the same ambitions to lead happy, prosperous lives? Attracting more people of color and men to teaching starts with making the profession itself more attractive -- and that includes good compensation.

So let's talk about teacher pay in New Jersey next...

The State of New Jersey's Teachers, Part 1: Teachers, Aging, & Pensions

I've been blogging for a long time, and teaching for even longer than that. One thing I've picked up from doing both is that people learning differently. Some students like learning in groups with their peers; some like one-on-one time with their teachers; some like working things out for themselves.

In the same way, when I write about education policy, I've noted that some folks like to read a long, academic policy brief, some like a short burst of tweets, and some -- believe it or not -- like to read to read the informal prose of a blog.

So, in the same way I provide multiple ways to learn for my students, I'm going to start blogging and tweeting more about my research work. Which means, this week, we're going to talk about New Jersey teachers.

* * *

I have a new report out today published by the New Jersey Policy Perspective: New Jersey's Teacher Workforce, 2019. This is actually the second in a series of pieces on Jersey education policy, the first being Bruce Baker's report on school funding from earlier this year (which I synopsized for NJPP).

One of the reasons I wanted to do this report is that so much focus has been on the plight of teachers over the last year, especially in states like West Virginia or Oklahoma or Arizona, where school funding is wholly inadequate. But there's been less focus on teachers in states like New Jersey, which has been a leader both in school funding equity and in student achievement.

When I started this, I was actually surprised about how little we really know about the Garden State's teachers. Who are they? What is their educational and racial/ethnic background? How have the demographics of the teacher corps changed?

So the report starts there -- but I also wanted to take a good hard look at how teachers are paid. For years, when Chris Christie was governor, we heard over and over again that teachers had a sweet deal: hefty raises for a job where the hours and days at work are less than other workers. But is that really true? What does the evidence show? Are New Jersey's teachers making big bucks compared to similar workers?

Further: how do salaries vary across the state? Do less advantaged districts have the ability to recruit and retain good teachers by offering competitive salaries? And what about the vaunted teacher retirement and health care benefits we're always hearing about? Are they really that lucrative?

Reasonable people can differ about how to approach the data to shed some light on these questions -- but I think I've come up with some defensible answers. Let's start by looking first at the age of the folks who sit behind the big desk...

Like all the graphs I'm going to show, this comes directly from the report, although formatting may be a little different. This shows the mean (aka average) age of teachers in NJ. It's a small but fairly steady decline. Why?

This might be my favorite graph in the whole report... but it does take a bit of explaining. What I did here was take all the teachers in 1997, and put them into "bins," based on their age. The bins are 5 years "wide": all the teachers in 1997 who were between age 20 and 24 go in one bin, then all the teachers who are age 25 to 29 in the next, and so on.

I did that for 1997 because that was the earliest year for which I had good data. Then I did it for five years later, in 2002. And again in 2007, and so on, all the way to 2017. What you get at the end is a graph that shows how the distribution of teachers by age has changed over the last two decades.

Here's the takeaway: Two decades ago there were a lot of teacher in their 50's who have since retired. We now have many teachers who won't retire for another couple of decades, at least. Which means that we don't have as many teachers entering the pension system as we will 20 years from now...

Which means that now is the time to strengthen our pension system, before the next wave of teacher retirees arrives.

We'll talk more about pension and their importance as this series continues. For now, let's see how teachers' ages vary across different types of NJ schools:

Again, some explanation is in order. New Jersey's schools districts are classified into different "District Factor Groups" (DFGs). DFG-A and DFG-B are the most disadvantaged communities: high levels of poverty, low property values, etc. Some of these districts were party to the Abbott lawsuits, which led to additional funding for them. But some were not; they haven't always had the same access to resources as the Abbotts.

DFG-CD/DE/FG districts are less disadvantaged; for lack of a better term, let's call them "working class" communities. The most affluent districts are DFG-GH/I/J. Then there are the charters schools, which, in NJ, are de facto autonomous school districts, accountable only to the state. "Special Services" districts serve children with profound special education needs. Finally, Vo-Tech districts, which generally fall along county lines, provide vocational and technical training for high school students.

The most obvious thing to notice in this graph is how young the charter school teachers are compared to all the other publicly funded schools. As I've noted many times, this creates a serious problem: charter schools may well be "free riding" on the other schools' wages. Charters can offer lower wages because they know their teachers won't stay long, bolting to better-paying districts as soon as they can.

But the charters can only get away with this because their young teachers know that, if they can stick it out a few years, better paying jobs in district schools will be available. The more I study this, the more indisputable the evidence becomes.

But the other thing here that stands out is how much older the vo-tech teachers are than everyone else. I really don't know why that may be... but considering how much emphasis state legislators have been putting on vo-tech lately, we probably should take a closer look at this. Because...

The vo-tech teachers are older, but they're less a little less experienced, on average. That suggests to me these schools are may be drawing from a pool of folks who worked into private sector for a while, then entered teaching. What does it take to recruit these people into vo-tech schools? We'll see in a bit.

But the big story is still the charters: the teachers there have far less experience than all the other schools. Again: do we have a free riding problem?

Let's talk about race, gender, and New Jersey's teachers next...

Clapping Harder For the Merit Pay Fairy

UPDATE BELOW

Earlier this week, I wrote about the death of the Merit Pay Fairy in Newark, New Jersey.

Hey, Jazzman, you bum -- I ain't dead yet!

Back in 2012, Newark began an experiment in teacher merit pay, fueled by funds from Facebook's Mark Zuckerberg. Teachers were promised up to $20 million over three years in extra incentive pay -- but in the first year, only $1.4 million was disbursed, and most of that appears to have comes from other teachers, who had their pay docked because they were deemed "ineffective."

Merit pay, in other words, was little more than a broken promise to the teachers of Newark right from the start. A survey of Newark teachers in the first year found a large majority did not see the compensation system as "reasonable, fair, and appropriate." (p. 24) It's not a surprise, therefore, that this past month both the teachers union in Newark, the NTU, and the district's administration decided that the program was not worth continuing. 

But some reformy folks believe in merit pay the same way some children believe in fairies: they don't want to acknowledge the evidence that shows, even in the most generous reading, that the benefits of merit pay are very small and likely are not indicative of true increases in student learning. Like Peter Pan, these true believers hope against hope that fairies can be brought back to life simply by clapping harder:

In 2012, Newark Public Schools did something remarkable. The district reached an agreement with the Newark Teachers Union that would fundamentally shift how teachers are not just evaluated but paid. 

Then-Gov. Chris Christie and Randi Weingarten, the President of the American Federation of Teachers, announced the groundbreaking deal together on national television. At the time, my organization, the National Council on Teacher Quality, called the contract “a model to which other districts should aspire.”

That's from Kate Walsh, president of NCTQ, an organization that has previously held up Newark Public Schools (NPS) as an exemplar of teacher evaluation, claiming the district was "getting results" from its system:

The district gave the evaluation system a chance to work. While Newark saw students’ achievement decline initially after the new evaluation system was implemented, the district persevered and student achievement rose to the level it had been before and, in English, exceeded previous levels. (p.12)

It's hard to be more vague than that -- how was student achievement measured? What was the improvement? Most importantly: how do we know if the teacher evaluation system was affecting results?

As Bruce Baker and I pointed out in our review of NPS "reforms," plenty of other districts with similar demographics were showing similar growth in student achievement, without things like merit pay. In addition, Newark has not seen the same demographic shifts over time many comparison districts have.

You must account for this stuff if you're going to make a causal claim about merit pay in Newark. Alas, Walsh still seems wholly uninterested in digging into these details; for example:

The district would also start to use their dollars in the same way that other employers do. Pay would become a strategic tool to attract the best teachers to where they were most needed. Teachers who were ineffective would no longer receive an annual raise. Teachers who were rated as highly effective earned a healthy $5,000 bonus. Even better, high performing teachers who were either able to teach subjects that were hard for the district to staff, particularly in the lowest performing schools, could earn even more, up to $12,500 a year.

In the first year of the contract, Newark had about 3,200 teachers. How many qualified for the highest bonus, $12,500? Only eleven. Is Walsh really trying to make the case this small disbursal made a significant difference in teacher quality in Newark? She continues:

The results spoke for themselves. After five years of implementation, 96 percent of highly effective teachers chose to stay in Newark and 49 percent of ineffective teachers were voluntarily leaving the district—exactly the sort of pattern schools need to see but rarely do. Accordingly, the district has higher student enrollment now than at any other time in recent history, suggesting parents gained a renewed confidence in the district.

First of all, we have no way of knowing whether these teacher attrition and retention rates are significantly better than they would be in the absence of the merit pay scheme. We don't know how they compare to similar districts' rates. We don't even know how they compare to rates before merit pay in Newark. Again, it's completely unwarranted to make any sort of causal claim without at least some attempt to compare these rates to a counterfactual.

Second, the notion that there's any evidence that shows student enrollment has increased because of "renewed confidence" due to merit pay is absurd on its face: "I was going to move my family, but now that Newark has teacher merit pay, we're staying!" Maybe the city's child population is simply increasing. Is Walsh so enamored with merit pay she's willing to make wild stretches like this?

Apparently, she is:

Where once compensation was used as a strategic deployment of resources to ensure the district can fill its vacancies, keep its best teachers, and ensure the most vulnerable students have access to them, soon there will be nothing but raises based on years of experience, and requiring teachers to spend precious time and money earning another degree they more than likely do not need. 

Research shows over and over again that advanced degrees do not make teachers more effective with the exception of math and science. 

First of all, there is a large body of evidence that shows teacher experience correlates with effectiveness. And while gains in effectiveness are strongest in the first few years, gains do persist up through the third decade of a teacher's career. Tying compensation to experience is hardly a policy without evidence to support it.

Next: "the exception of math and science" is a very big exception. Is Walsh prepared to offer a bonus only to math and science teachers with advanced degrees? And is she really sure French teachers don't benefit from degrees in French, or music teachers don't benefit from degrees in conducting, or that teachers in many other subjects don't benefit from gaining expertise through earning an advanced degree in that subject?

I've been looking at the research on advanced degrees and teaching for some time now, and the conclusion I've come to is that it is highly limited. Most studies don't account for alignment of teaching subject and degree concentration; in other words, the results are likely skewed because they don't separate getting a degree in what you teach from getting any degree.

These studies also usually don't account for variations in the quality of the degree-granting programs: crappy on-line programs are lumped in with rigorous degrees from research universities. In addition, the student outcomes are almost always measured by test scores, which limits the teachers studied to tested grades (3-8) and only two subjects (math and English).

Walsh's sweeping statement is simply not justified. Further, she ignores the reality that NPS must compete with other districts that offer masters pay bumps to attract qualified teaching candidates. Is the district suppose to ignore this reality? Especially because there is no evidence whatsoever that NPS has attracted better candidates to its teaching staff than other districts?

As I said in the last post: there was supposed to be an ongoing study of merit pay in NPS. But that study ended after a single year. We have no evidence whatsoever that Newark attracted better teaching candidates, improved student outcomes, or raised teacher effectiveness by using merit pay.

But this lack of evidence isn't stopping Walsh from clapping harder:

In 2012, Newark Public Schools took bold steps to create a compensation system that would help to attract and keep the best teachers. The district used resources strategically to ensure the most vulnerable students had access to the best teachers, an accomplishment that many districts struggle to achieve. With this new contract, instead of being a leader in strategic compensation, Newark becomes a district that takes a one-size-fits-all approach to its teachers, to the detriment of its students and teachers alike.

Even by current reformy standards, this statement is way over the top. We have no evidence the best teachers in Newark went to the neediest students. We have no evidence Newark was better at teacher allocation than districts that didn't implement merit pay. We have no evidence Newark is now a "one-size-fits-all" district. We have no evidence merit pay was a benefit to Newark's teachers and students.

What we do know is that the majority of Newark teachers didn't think the system was fair. That, by itself, is enough to declare Newark's merit pay experiment dead -- even if some folks keep clapping for it.



UPDATE: After I posted this blog, I went back and looked at the teacher survey again, which is part of a report commissioned by the American Institutes for Research. AIR is an excellent research organization, and they produce high-quality work. That said, there are a few oddities in their Newark report:

In response to a set of questions about their knowledge of the current evaluation process, 83 percent of teachers and 99 percent of school leaders reported that they have a clear understanding of the evaluation process. In addition, in response to a set of questions about the fairness of the evaluation process, 72 percent of teachers and 92 percent of school leaders reported that the evaluation process is fair, which is larger than the 30 percent reported fairness by teachers in an evaluation of 25 districts in New Jersey (Firestone, Nordin, Shcherbakov, Kirova, & Blitz, 2014) and the 39 percent reported fairness by teachers in 10 districts in Arizona (Ruffini, Makkonen, Tejwani, & Diaz, 2014). [p.20]

Let's set aside the Arizona survey, which I haven't yet read, and just focus on the New Jersey one. That report, which I'm well-acquainted with as it came out of Rutgers (where I got my PhD and current teach part-time), did not have a sample of representative districts. It was, instead, an evaluation of a pilot program of teacher evaluation conducted in 25 districts across the state. It had a low response rate (39 percent), but more important, it was conducting a survey after the state had imposed a new law on districts, TEACHNJ, forcing them to rework their evaluation systems.

I can tell you as a teacher who lived through that time: TEACHNJ was not popular with many working teachers in the state. So it shouldn't be surprising the popularity of the new system was so low. Unlike Newark in its first year, there wasn't a whole bunch of money promised from an outside source going to these districts.

My point here is that the comparison is, at best, strained. And AIR really should have spelled out more clearly the limits of the comparison of the two reports.

Here's another oddity from the AIR report:

Figure 2 shows that the retention rates among teachers rated “effective” and “highly effective” exceed 90 percent, whereas retention rates among teachers rated “partially effective” and “ineffective” are 72 percent and 63 percent, respectively. In contrast, the most recent results from the national 2012–13 Teacher Follow-Up Survey indicate that 84 percent of public school teachers are retained, on average (Goldring, Taie, & Riddles, 2014).

Golding et al. does find 84.3 percent of teachers stayed in their positions -- but that's all teachers, not "effective" ones. There's simply no way to know, based on this report, whether Newark did any better in retaining its better teachers thanks to merit pay.

Again, it's fine to include this data point, but a little more context is probably in order.

The Merit Pay Fairy Dies in Newark

One of the long-running characters on this blog is the Merit Pay Fairy.

Hey, youse bums -- get back to the teachin' already!

The Merit Pay Fairy lives in the dreams of right-wing think tanks and labor economists, who are absolutely convinced that our current teacher pay system -- based on seniority and educational attainment -- is keeping teachers from achieving their fullest potential. It matters little that even the most generous readings of the research find practically small effects* of switching to pay-for-performance systems, or that merit pay in other professions is quite rare (especially when it is based on the performance of others; teacher merit pay is, in many contexts, based on student, and not teacher, performance). 

Merit pay advocates also rarely acknowledge that adult developmental theory suggests that rewards later in life, such as higher pay, fulfill a need for older workers, or that messing with pay distributions has the potential to screw up the pool of potential teacher candidates, or that shifting pay from the bottom of the teacher "quality" distribution to the top -- and, really, that's what merit pay does -- still leaves policymakers with the problem of deciding which students get which teachers.

Issues like these, however, are at the core of any merit pay policy. Sure, pay-for-performance sounds great; it comports nicely with key concepts in economic theory. But when it comes time to implement it in an actual, real-world situation, you've got to confront a whole host of realities that theory doesn't address.

Which is what seems to have happened in Newark:

In 2012, Newark teachers agreed to a controversial new contract that linked their pay to student achievement — a stark departure from the way most teachers across the country are paid. 

The idea was to reward teachers for excellent performance, rather than how many years they spent in the district or degrees they attained. Under the new contract, teachers could earn bonuses and raises only if they received satisfactory or better ratings, and advanced degrees would no longer elevate teachers to a higher pay scale. 

The changes were considered a major victory for the so-called “education reform” movement, which sought to inject corporate-style accountability and compensation practices into public education. And they were championed by an unlikely trio: New Jersey’s Republican governor, the Democratic-aligned leader of the nation’s second-largest teachers union, and Facebook founder Mark Zuckerberg, who had allocated half of his $100 million gift to Newark’s schools to fund a new teachers contract. 

“In my heart, this is what I was hoping for: that Newark would lead a transformational change in education in America,” then-Gov. Chris Christie said in Nov. 2012 after the contract was ratified. 

Seven years later, those changes have been erased. 

Last week, negotiators for the Newark Teachers Union and the district struck a deal for a new contract that scraps the bonuses for top-rated teachers, allows low-rated teachers to earn raises, and gives teachers with advanced degrees more pay. It also eliminates other provisions of the 2012 contract, which were continued in a follow-up agreement in 2017, including longer hours for low-performing schools. [emphasis mine]

I blogged about that 2012 contract many times as it was being negotiated. It was never popular; only 37 percent of the membership approved it, thanks to low turnout in the voting. The teachers were promised up to $20 million in extra funding, over three years, dedicated to merit pay, all coming from Mark Zuckerberg's famous donation to Newark's schools. But the actual disbursement was far less: only $1.4 million in the first year. And much of that appears to have come from teachers who were denied regular annual raises due to "poor performance."

More senior teachers with advanced degrees had the option of not participating in merit pay; only about 20% chose to enter the system, putting to rest any idea merit pay was popular among teachers who had a choice. In addition, the percentage of "highly effective" teachers was much higher in the pool of teachers who opted out of merit pay than those who were in the system.

This pretty much destroys the notion that "better" teachers are clamoring for merit pay; in Newark, many doubted the system would work to their benefit. 

This trepidation can be found in a survey of Newark teachers by the American Institutes for Research, which was conducted in the first year of the contract. Over 40 percent of teachers both in and out of the merit pay system believed merit pay would hurt collaboration (p. 25); 60 percent believed the system ignored important aspects of their teaching (p. 24). Nevertheless, over 70 percent believed their pay system was fair and appropriate at their school. My reading of these results is that there was concern over the system, but teachers were willing to give it a shot.

Those of us who follow Newark's schools know what happened next: a mass revolt against the district's administration, which reported directly to the governor at the time, Chris Christie. A mayoral election where state control of schools was the key issue. The return of local control of schools after two decades. And now, the end of Newark's merit pay experiment.

One curious thing about the AIR report: it was labeled as "Year One":

NPS [Newark Public Schools] commissioned American Institutes for Research (AIR) to conduct an evaluation of the implementation and impact of the NPS/NTU contract and associated initiatives. The three-year evaluation focuses on a variety of outcomes (e.g., educator perceptions, teacher retention, teacher effectiveness, and student achievement) associated with the four contract components. In the first year of the evaluation, the period to which this report corresponds, the evaluation team used qualitative and quantitative techniques to assess the implementation of the contract components and to examine the association between the new evaluation and compensation systems (i.e., Components 1 and 2) and teacher retention. This report presents findings related to educator perceptions, as captured by teacher and school leader surveys administered in spring 2015, after two years of contract implementation (i.e., as of the 2014–15 school year) and teacher retention after one year of contract implementation (i.e., through the 2013–14 school year).1 The AIR evaluation team plans to examine the contract’s impact on teacher effectiveness and student achievement in 2016 and 2017, respectively. [emphasis mine]

Guess what? There was ever a follow-up study of Newark's merit pay system. For whatever reason, AIR never published any further reports on whether merit pay helped improve student learning, improved effective employee retention, or improved the quality of new teaching candidates in NPS.

You would think, given all the hoopla over this contract at the time, that a thorough study of how merit pay played out in Newark would have been a top priority for the state, the district, and all of the folks who were connected to the Zuckerberg donation. Alas, we'll never know how this hyped contract affected the district. We'll never know if merit pay in Newark -- perhaps the most high-profile implementation of teacher pay-for-performance in the United States -- actually worked.

Were I cynical, I'd think that the folks behind this system didn't really want to know whether merit pay works. I'd think they want to keep merit pay a policy based on theory, not evidence. Because if Newark turned out like all the other merit pay experiments, it would show, at best, practically small improvements in student outcomes. And the price for that tiny improvement would be a chaotic and impractical system of teacher evaluation and compensation that was always doomed to fail.

Good thing I'm not cynical...

For those of you who weren't with me during the early, snarkier days of this blog: the idea for the Merit Pay Fairy came from a scene in a play by Christopher Durang:

“You remember how in the second act Tinkerbell drinks some poison that Peter is about to drink in order to save him? And then Peter turns to the audience and he says that ‘Tinkerbell is going to die because not enough people believe in fairies. But if all of you clap your hands real hard to show that you do believe in fairies, maybe she won’t die.’ So, we all started to clap. I clapped so long and so hard that my palms hurt and they even started to bleed I clapped so hard. Then suddenly the actress playing Peter Pan turned to the audience and she said, ‘That wasn’t enough. You did not clap hard enough. Tinkerbell is dead.’ And then we all started to cry.”

It really doesn't matter how many deaths the Merit Pay Fairy dies -- some folks just keep clapping harder:

Shavar Jeffries, who led the Newark school board in 2012 and is now president of the national advocacy group Democrats for Education Reform said he is happy to see teachers get more money under the new agreement. But he said it is disappointing that teachers’ performance will no longer automatically influence their pay — a disconnect, he argued, that many families do not support.

“There’s almost no parent in the city of Newark,” he said, “who thinks that there shouldn’t be a relationship between pay and whether you’re actually doing a good job for babies each and every day in the classroom.”

I don't live in Newark, but I think I'm safe in making this statement: there is definitely no parent in Newark who doesn't want a good teacher for their own kid. The question, then, is what are we doing to ensure that every child in Newark -- and, for that matter, every community -- has a well-trained, competent, effective teacher in their classroom.

Taking away money from "bad" teachers and giving it to "good" ones does little to improve the overall effectiveness of the teaching corps. What would help is raising the base pay for teachers so as to close the compensation gap they suffer compared to other professions; that way, we could attract the best possible candidates into the profession and increase the chances that every child has an effective teacher.

What won't help is imposing unfeasible pay-for-performance systems that, time after time, fail to deliver meaningful improvements. Clap as hard as you want -- the Merit Pay Fairy is dead in Newark, and the prognosis in other communities is not good.

* I'm adding this note in anticipation of a counter-argument for merit pay I've seen before: if you include studies in other countries, the effect size of merit pay grows. But these other countries have such different contexts for teaching and worker pay that applying their results to the U.S. is a highly dubious proposition. Further, the effect size is still very small -- again, even under the most generous interpretation of the results. When effects are so small, we are justified in questioning whether the variation is related to the construct; in other words, are there real improvements in learning, or are teachers just slightly pumping up scores through test prep?

No, I'm not going to debate this on Twitter. Write an article or blog post and I'll respond.

What's Really Happening In Camden's Schools?

This latest series on Camden's schools is in three parts:

Part I

Part II

Part III (this post)


I want to wrap up this series of posts about Camden's schools with a look at the latest CREDO report, which the supporters of recent "reforms" keep citing as proof of those reforms' success.

Long time readers know the CREDO reports, issued by the Center for Research on Education Outcomes at Stanford University, have been perhaps the best known of all research studies on the effectiveness of charter schools. The reports, which are not peer-reviewed, look at the differences in growth in test scores between charter schools and public district schools, or between different school operators within the charter sector. CREDO often issues reports for a particular city's or state's charter sector; they last produced a statewide report for New Jersey in 2013.

I and others have written a great deal over the years about the inherent limitations and flaws in CREDO's methodology. A quick summary:

-- The CREDO reports rely on data that is too crude to do the job properly. At the heart of CREDOs methodology is their supposed ability to virtually "match" students who do and don't attend charter schools, and compare their progress. The match is made on two factors: first, student characteristics, including whether students qualify for free lunch, whether they are classified as English language learners (in New Jersey, the designation is "LEP," or "limited English proficient"), whether they have a special education disability, race/ethnicity, and gender.

The problem is that these classifications are not finely-grained enough to make a useful match. There is, for example, a huge difference between a student who is emotionally disturbed and one who has a speech impairment; yet both would be "matched" as having a special education need. In a city like Camden, where childhood poverty is extremely high, nearly all children qualify for free or reduced-price lunch (FRPL), which requires a family income below 185 percent of the poverty line. Yet there is a world of difference between a child just below that line and a child who is homeless. If charter schools enroll more students at the upper end of this range -- and there is evidence that in at least some instances they do -- the estimates of the effect of charter schools on student learning growth very likely will be overstated.

-- CREDO's use of test scores to match students and measure outcome differences is inherently problematic. The second factor on which CREDO makes student matches is previous student test scores. Using these as a match is always problematic, as test outcomes are prone to statistical noise. For now, I'll set aside some of the more technical issues with CREDO's methods and simply note that all tests are subject to construct-irrelevant variance, a fancy way of saying that scores can rise not because students are better readers or mathematicians, but simply because they are better test takers. If a charter school focuses heavily on test prep -- and we know many of the best-known ones do -- they can pump up effect sizes without increasing student learning in a way that we would consider meaningful.

-- The CREDO reports translate charter school effects into a "days of learning" measure that is wholly unvalidated. I've been going on about this for years: there is simply no credible evidence to support CREDO when they make a claim about a charter school's students showing "x number of days more learning" than a public district school's students. When you follow CREDO's citations back to their original sources, you find they are making this translation based on nothing. It's no wonder laypersons with little knowledge of testing often misinterpret CREDO's results.

Again, I and others have been writing about these limits of the CREDO studies for years. But the Camden "study" has some additional problems:

First, it's not really a "study" -- it's a Powerpoint slideshow that is missing some essential elements that should be included in any credible piece of research. Foremost of these is a description of the variables. In previous reports, CREDO at least told its readers how the percentages of FRPL, special education, and LEP students varied between charter and public district schools. But they don't even bother with this basic analysis here. And it's important: if the charter sector is taking proportionally fewer of the students who are more challenging to educate, they may be creating a peer effect that can't be scaled up.

Second, free and reduced-price lunch (FRPL) is an even less accurate measure of student socio-economic status when a district has universal enrollment in the school meal program. If a student's family knows she will automatically receive a free meal, they will have less incentive to fill out an application. We've seen significant declines in the percentage of FRPL students in some districts that have moved to universal enrollment, indicating this is a real phenomenon.

In contrast, New Jersey charter schools get more funding when they enroll a FRPL-eligible student. They have an incentive to get a student's family to fill out an application that the district does not. Did CREDO account for this? They don't say.

Third, the switch in tests in 2015 complicates any test-based analysis. There are at least two reasons for this: first, the previous test scores of students in both the charter and public district groups go back to 2014, when the NJASK was the test. As Bruce Baker and I showed in our analysis of Newark's schools, there is evidence that schools made a sudden shift in their relative standing on test outcomes when the switch in tests occurred. Shifts this fast are almost certainly not due to changes in student instruction; instead, they occur because some students were more familiar with the new format of the test than others. This, again, puts the matching process in doubt.

The second reason is related to the first: some schools likely took longer to acclimate to the new test than others. Their relative growth in outcomes, therefore, will probably shift in later years. Did that happen in Camden? Let's look at some of the CREDO report's results:

As I explained in earlier posts, there are three types of publicly-funded schools in Camden: Camden City Public Schools, denoted here as "TPS" for "Traditional Public Schools"*: independent charter schools; and renaissance schools, which are operated by charter management organizations (CMOs), and are supposed to take all students in a neighborhood catchment zone (but don't -- hang on...).

Note here that CCPS schools made a leap in relative growth between 2015 and 2016. Do you think it's because the schools got so much better in one year? Or is it more logical to believe something else is going on? A more likely explanation is that Camden students were not well prepared for the change in test format in 2015, but then became more familiar with it in 2016. It's also possible the charter students were better prepared for the new test in 2015, but lost that advantage in 2016, when their relative growth went down.

This is all speculation... but that's the point. Sudden shifts in test score outcomes are likely due to factors other than better instruction. Making the claim, based on sudden shifts in outcomes, that any particular sector of Camden's school system is getting better results due to their practices is a huge leap -- especially when we now know something important about the renaissance schools...

Because renaissance schools are not enrolling all students in their neighborhoods, their students are different from CCPS students in ways that can't be captured by the data. Here, again, is the State Auditor in his renaissance school report:

• The current enrollment process has limited the participation of neighborhood students in renaissance schools. Per N.J.S.A. 18A:36C-8, renaissance schools shall automatically enroll all students residing in the neighborhood of a renaissance school. Instead, the district implemented a centralized enrollment system in which families must opt in if they prefer to attend a renaissance school. This process has left the district with fewer than half of neighborhood students being enrolled in their neighborhood renaissance school. 

• [...]

• The current policy could result in a higher concentration of students with actively involved parents or guardians being enrolled in renaissance schools. Their involvement is generally regarded as a key indicator of a student’s academic success, therefore differences in academic outcomes between district and renaissance students may not be a fair comparison.

As I said in the last post, it is very frustrating that the State Auditor gets this, but people who proclaim to have expertise in education policy do not. Let me state this as simply as I can:

A "study" like the Camden CREDO report attempts to compare similar students in charters and public district schools by matching students based on crude variables. Again, these variables aren't up to the job -- but just as important, students can't be matched on unmeasured characteristics like parental involvement. Which means the results of the Camden CREDO report must be taken with great caution.

And again: when outcomes suddenly shift from year-to-year, there's even more reason to suspect the effects of charter and renaissance schools are not due to factors such as better instruction.

One more thing: any positive effects found in the CREDO study are a fraction of what is needed to close the opportunity gap with students in more affluent communities. There is simply no basis to believe that anything the charter or renaissance schools are doing will make up for the effects of chronic poverty, segregation, and institutional racism from which Camden students suffer.

Now, there are some very powerful political forces in New Jersey that do not want to acknowledge what I am saying here. They want the state's residents and lawmakers to believe that the state takeover of Camden's schools, and subsequent privatization of many of those schools, has led to demonstrably better student outcomes -- so much better that upending democratic, local control of Camden's schools was worth it.

Remember: the takeover and privatization of Camden's schools was planned without any meaningful local input. From 2012:

CAMDEN — A secret Department of Education proposal called for the state to intervene in the city’s school district by July 1, closing up to 13 city and charter schools. 

[...] 

The intervention proposal, which was obtained by the Courier-Post, was written by Department of Education employee Bing Howell. 

He did not respond to a phone call and email seeking comment. 

Howell serves as a liaison to Camden for the creation of four Urban Hope Act charter schools. He reports directly to the deputy commissioner of education, Andy Smerick.

Howell’s proposal suggests that he oversee the intervention through portfolio management — providing a range of school options with the state, not the district, overseeing the options. He would be assisted by Rochelle Sinclair, another DOE employee. Both Howell and Sinclair are fellows of the Los Angeles-based Broad Foundation. [emphasis mine]

A California billionaire paid for the development of a "secret" (that's the local newspaper's word, not mine) proposal to wrest democratic, local control of schools away from Camden and develop a "portfolio" of charter, renaissance, private, and public schools. This proposal fit in nicely with the plans for Camden's redevelopment, which, as we are now learning, included a series of massive tax breaks for corporations with ties to the South Jersey Democratic machine.

The same forces that are now trying to justify this tax giveaway are the same forces that pushed forward a radical transformation of Camden's schools. They would have us all believe that this transformation is as "successful" as their tax schemes.

But in both cases they are relying on the flimsiest of evidence, badly interpreted and devoid of any meaningful context. The case for educational "reform" in Camden is as weak as the case for corporate tax incentives in Camden.

Camden's families deserve what so many suburban families in New Jersey have: adequately funded and democratically, locally controlled schools. Small, dubious bumps in student growth found in incomplete "studies" are not an acceptable substitute.

That's all, for now, about Camden. We'll move on to another state next...


* I don't use "TPS" because I think it's a loaded term: the word "traditional" can carry all sorts of unwarranted negative connotations. CCPS schools are properly defined as "public district schools."

Camden, Charter Schools, and a Very Big Lie

This latest series on Camden's schools is in three parts:

Part I

Part II (this post)

Part III 

Let's get back to the deeply flawed editorial from this week's Star-Ledger that I wrote about yesterday. In that post, I explained how "creaming" -- the practice of taking only those students who are likely to score high on standardized tests -- is likely a major contributor to the "success" of certain charter schools.

Charter school advocates do not like discussing this issue. The charter brand is based on the notion that certain operators have discovered some special method for getting better educational outcomes from students -- particularly students who are in disadvantaged communities -- than public district schools. But if they are creaming the higher-performing kids, there's probably nothing all that special about charters after all.

It's important to understand this debate about charters and creaming if you want to understand what's happening now in Camden's schools.

Because Camden was going to be the proof point that finally showed the creaming naysayers were wrong with a new hybrid model of schooling: the renaissance school. These schools would be run by the same organizations that managed charter schools in Newark and Philadelphia. The district would turn over dilapidated school properties to charter management organizations (CMOs); they would, in turn, renovate the facilities, using funds the district claimed it didn't have and would never get.

But most importantly: these schools would be required to take all of the children within the school's neighborhood (formally defined as its "catchment"). Creaming couldn't occur, because everyone from the neighborhood would be admitted to the school. Charter schools would finally prove that they did, indeed, have a formula for success that could be replicated for all children.

Well, guess what?

EXECUTIVE SUMMARY
CITY OF CAMDEN SCHOOL DISTRICT July 1, 2015 to February 28, 2018 

[...]

• The current enrollment process has limited the participation of neighborhood students in renaissance schools. Per N.J.S.A. 18A:36C-8, renaissance schools shall automatically enroll all students residing in the neighborhood of a renaissance school. Instead, the district implemented a centralized enrollment system in which families must opt in if they prefer to attend a renaissance school. This process has left the district with fewer than half of neighborhood students being enrolled in their neighborhood renaissance school.

That's from a report from the State Auditor that was released earlier this year -- a report ignored by many in the NJ the press, including the Star-Ledger (the Courier Post and NJ Spotlight ran stories on the lack of oversight for renaissance schools, but didn't address the problems with the neighborhood enrollments).

Understand, the SL played a pivotal role in spreading the news that renaissance schools would enroll every student within their catchments. Here, for example, is an editorial from 2012 [all emphases mine]:

The campus will grow one grade level at a time, serving every kid in the neighborhood — including those learning English, or with special needs.

In real time, only snarky teacher-bloggers expressed any skepticism. But the SL continued to assure Camden's families that the renaissance schools would accept all students in the neighborhood; here's a piece from 2014:

District officials said the renaissance schools serve specific neighborhoods, where all students within that neighborhood are guaranteed enrollment.

2015:

According to the district, renaissance schools differ from conventional charter schools in that they guarantee a seat to every student living in its local neighborhood, and that they contract with the local school district.

This exact phrasing was used in an SL piece just a month later; apparently, the newspaper couldn't come up with new ways to assure residents every local student would have a seat. Here's yet another piece from 2017, where the SL gave South Jersey political boss George Norcross space to assure Camden's parents that every neighborhood child would get a seat at their renaissance school:

Renaissance schools are neighborhood schools that serve students in a defined catchment area, guaranteeing enrollment for any student living in that neighborhood. In other words, a child's fate is not left to a lottery.

Now, if anyone at the SL had read the Urban Hope Act, which created renaissance schools, they'd know what Norcross wrote in the pages off their newspaper simply wasn't true:

1. If there are more students in the attendance area than seats in the renaissance school, the renaissance school shall determine enrollment by a lottery for students residing in the attendance area. In developing and executing its selection process, the nonprofit entity shall not discriminate on the basis of intellectual or athletic ability, measures of achievement or aptitude, status as a handicapped person, proficiency in the English language, or any other basis that would be illegal if used by a school district.

This directly contradicts Norcross, and the repeated reports in the SL. But, hey, what did the newspaper know back in 2017, before the Auditor's report? Maybe they thought it was a good idea to give a powerful political figure the benefit of the doubt; maybe every neighborhood kid really was getting into a renaissance school, no matter what the actual law said.

But then, in 2019, the Auditor's report was released, and all doubt was erased: the renaissance schools were not enrolling all neighborhood students. The previous reporting was false. How embarrassing...

Surely, from now on when the SL writes about renaissance schools, they will acknowledge the promise of a guaranteed seat for all students within those schools' catchments was broken. Surely, they will admit their previous reporting was inaccurate, and apologize for getting the story wrong. If not that, at least they will demand to know why the promises the district and the state made to Camden's families were now being broken.

Won't they?

South Jersey political boss George Norcross also deserves credit for using his political weight to push these reforms in Camden. Just because he’s defending a corrupt tax incentives program doesn’t mean he’s not doing good elsewhere. He helped push through a new law that allowed nonprofit charter operators to run neighborhood schools, but also forced them serve every student who walks through the door.

Technically, that's true -- the problem is that not every student from the neighborhood -- who were all promised a seat -- is allowed to walk through the door of their local renaissance school.

Again, from the Auditor's report:

In the 2016–17 enrollment lottery, 461 students were accepted to renaissance schools. Of these students, 247 (54 percent) resided in the neighborhood of their renaissance school. In the 2017–18 enrollment lottery, 838 students were accepted to renaissance schools. Of these students, 387 (46 percent) resided in the neighborhood of their renaissance school. Overall, less than half of students accepted to renaissance schools (49 percent) through the enrollment lottery process for the 2016–17 and 2017–18 school years were from the renaissance school’s neighborhood. 

All neighborhood students who submitted applications by the deadline for the 2016–17 lottery were accepted in their neighborhood renaissance school; however, 47 students who applied by the deadline for the 2017–18 lottery had to be placed on their neighborhood renaissance school’s wait list. As of October 2017, there were 195 students on the wait list for their neighborhood renaissance school.[emphasis mine]

 The Auditor also explains why this matters:

The current policy could result in a higher concentration of students with actively involved parents or guardians being enrolled in renaissance schools. Their involvement is generally regarded as a key indicator of a student’s academic success, therefore differences in academic outcomes between district and renaissance students may not be a fair comparison.

This is a reality some of us have been trying to explain to outlets like the Star-Ledger for years. But for whatever reason, it appears the paper would rather use the weaseliest of words than admit they've been wrong all along. From last week's editorial:

This addressed a common knock on charters: that they self-select their students, by keeping out the poorest kids or those with special needs.

Those typical criticisms don’t apply in Camden. The so-called “renaissance schools” under charter management take the same, or more of the poorest and special ed kids as the district schools. 

See how they've moved the goalposts? Before, every kid in the neighborhood got a seat; now, the kids are the same...

Except, as the Auditor points, out, it's likely they aren't. The very act of enrolling your child in a renaissance school is likely a marker that you are a more "actively involved parent." We know, thanks to a great deal of high-quality research (see the lit review here) that parents rely on their social networks to help them make decisions in school "choice" systems, and that different parents have different networks. It's not at all a stretch to think the students in renaissance schools differ from other students on characteristics that can't be shown in the data. 

In other words: the renaissance schools may very well be creaming. Why is the State Auditor capable of getting this simple point, but the Star-Ledger editorial board isn't?

I'll talk more about these "unobserved" student differences and why they matter in my next post. For now, we need to understand this:

When the people of Camden were told that every child in a renaissance school's catchment would be enrolled, they were lied to. I'm using the passive voice deliberately here because who exactly did the lying -- and who simply transmitted this very big lie -- is open to debate.

But I would think that journalists -- whose primary function is to deliver the truth to their readers -- would, of all people, not want to perpetuate falsehoods when confronted with the facts. How sad that New Jersey's largest newspaper has such low standards, and such little regard for their readers.

We'll talk about the latest "study" on Camden schools' effectiveness next.

 Star-Ledger Editorial Board

ADDING: Over the years, the Star-Ledger opinion section has been remarkably inept when it comes to writing about education:

• They blamed teacher seniority when an award-winning teacher in Camden was fired -- except she never was.
• They tried to show the failure of Camden's schools by pointing to the low proficiency rate at Camden Street School -- expect that school was in Newark, and hosted programs for that district's most cognitively impaired students.
• They said a group of Newark teachers told "lies" about a contract negotiation -- except what those teachers actually said was, in fact, accurate.
• They gave an anti-tenure superintendent space to tell stories about her staff -- except her own board said they weren't true (she was later terminated by that same board).
• They misrepresented the views of union leaders -- even when those leaders were quite clear in their answers to direct questions.
• They engaged in some particularly nasty language when describing the grassroots opposition to school leadership in Newark -- including making the accusation that local activists "don't seem to give a damn about the children."
• They made fun of a union official's weight. Yes, they did.

Let me be clear about something: over the years, the Star-Ledger has had some excellent reporters on the education beat, including Jessica Calefati, Peggy McGlone, and Adam Clark. And, of course, the great Bob Braun worked there for years.

But the opinion section has been, and remains, a mess. If you're a public school teacher and you pay to read this dreck, you should really ask yourself: "Why?"