Charter Schools, An Exchange: Part II

In Part I of this series, I invited Dmitri Mehlhorn, a well-known advocate for charter schools and other education "reforms," to comment on charter schools and make the case for his point of view. What follows is my reply; I'd urge you to read Dmitri's essay first to get the full context of my reply.

ADDING: Here is an index for the entire exchange:

Part I: Mehlhorn's opener.

Part II: My reply.

Part III: Mehlhorn's response.

Part IV: My second reply.

Part V: Mehlhorn's second response.

Part VI: My final reply.

* * *

First of all, Dmitri, thank you for agreeing to this exchange. I'm going to try to keep the tone civil here -- but I'm afraid I'm also going to be rather blunt.

Your essay is, in many ways, a perfect example of "reform" thinking. Almost everything you say is factually correct; however, you are so selective in your citations, and so lacking in context when discussing them, that you arrive at conclusions that simply are not warranted.

These are the points I'm going to address from your essay:

• The CREDO studies provide little evidence of a practical positive effect of charter schools on student learning in the aggregate. They also provide little evidence that the charter sector as a whole has been improving over the last six years.
• There is little evidence to support any contention that the limited econometric research into a few charter school chains can be generalized to the entire sector.
• The lottery study you cite (I assume Angrist, Cohodes, Dynarski, Pathak, & Walters, 2013), like all charter lottery studies, is limited to the population of students who enter the lottery. Further, "being a charter school" is NOT the only treatment.
• The research I assume you cite from Mathematica on Achievement First and Uncommon shows they serve different student populations than their feeder schools, with fewer students with special needs (SWSN) and fewer Limited English Proficient (LEP) students.
• The Texas charter study you cite only studied districts with comparatively low charter penetration. Other studies have shown a pernicious effect of charter proliferation on host districts.
• Doug Harris himself has urged caution in generalizing the changes in New Orleans to the country at large. He has also strongly cautioned about a process of charterization that excludes community input.

The CREDO Studies

Dmitri, you claim that the 2015 CREDO study "puts the burden of proof squarely on those who would attack the efficacy of charter schools." First of all, the burden of proof remains on those who make an affirmative case for charter proliferation, and it always will. It is impossible to reply to: "Prove that more charters WON'T make things better."

You catalog some of the objections to the CREDO methodology, but you dismiss them far too easily. As Bruce Baker has shown, how you classify student economic disadvantage matters when it comes to test score outputs. As I have shown, the differences between special education populations can be quite significant. These things do, in fact, matter a great deal.

Next, it's impossible to make claims about the improvement of the overall charter sector when the charters CREDO studied changed. As you note, the 2009 and 2015 CREDO studies looked at different samples of states and schools. One thing I noticed locally about the 2015 study was that it included Newark, NJ, but not Camden, Trenton, Jersey City, or other towns in my state with large charter populations (the NJ CREDO study showed charters in these towns had no effect, or a negative effect). And Newark was not included in the 2009 study, nor was it part of the 16-state growth comparisons in the 2013 study.

Put simply: you can't claim that the sector as a whole is improving when you aren't evaluating the same sample in different studies.

Next, as Maul & McClelland explain, the "significance" of the CREDO results is statistical, and not necessarily practical. When you have large sample sizes, as in the CREDO studies, any difference is going to be statistically "significant." But is it a difference that actually matters?

One of my great frustrations is the continuing use of "days of learning" as an interpretation of effect sizes. Some people I respect think this is fine; I don't. Not only does "days of learning" imply a precision that can't be justified; it also is being applied to tests that are not "vertically scaled." A layperson is likely going to think that a student who is "180 days ahead" of another is learning the next grade's material; the tests cannot make that determination, however, because they don't test next year's content.

Let's, instead, look at the CREDO effect sizes without translation.

Here are the reported aggregate effect sizes for the three CRDEO studies you cite, including the 2015 study which really shouldn't be compared to the other two. A legitimate critique of making a graph like this would be to point out that the y-axis could be changed to any scale I want to make the bars look bigger or smaller. What I've done here, then, is set up a relevant comparison: the bracket is the difference in standard deviations between a typical school with average family income in the 90th percentile, and a school with that income in the 10th percentile.

The results are clear: even if we accept the 2015 CREDO results as typical for the sector, the practical effect of charter schools does not come close to closing the income gap advantage.

In summary: according to the CREDO studies, charter school effects are practically quite modest, and any notion that the sector is improving rapidly needs to be tempered by the realization that family income has a far more profound affect on test scores than systems of school governance.

The Econometric Research

I wouldn't attempt here to encapsulate all the research being done by admittedly brilliant economists like Josh Angrist on charter schools. I will freely admit they have found, in a significant number of cases, a real charter effect. But their research has been quite limited in its scope. KIPP gets studied quite a bit (which, apparently, they pay for). A few other chains have been looked at... but what about these guys?

NOTE: I have replaced the graph originally in this post with a more accurate one. See here for details.

This is from Bruce Baker, based on a dataset we've been assembling. KIPP is actually one of the smaller players in the charter sector. Where's the peer-reviewed research on K12, a rather dicey operation? How about some NBER papers on Charter Schools USA, which is making money hand-over-fist? How about a study of UNO, which has had its share of troubles? Maybe Academica, a big player in the free-for-all that is the Florida charter school sector? Or White Hat, at the center of Ohio's increasingly troubled charter sector?

If we're really going to be serious about evaluating the effects of charter schools, why aren't we looking at these guys? Nobody in their right mind would ever make the case that just because KIPP has shown some positive effects, these guys must show them as well.

But let's take a closer look at the those KIPP studies anyway...

The Charter Lottery Studies

I'll go back to Bruce Baker again, who's written the definitive piece on the limitations of the so-called charter lottery studies. These studies are limited to 1) the schools that are over-subscribed and must have a lottery, and 2) the students who actually bother to enter the lottery. Again: no one should try to generalize the findings of these studies to the entire charter sector.

The other issue with most of these pieces of research is that they are surprisingly vague about what exactly causes a charter management organization like KIPP to see gains compared to feeder public schools. Again, Angrist is a brilliant researcher -- but I'm surprised when I read work like this at how quickly he glosses over the characteristics of the schools themselves (and, for that matter, how little documentation there is of the qualitative methods used to arrive at the descriptions of the schools' practices).

KIPP

So what do we know about KIPP? One thing is for sure: they get a lot of extra money in philanthropic contributions. They also have a relatively inexperienced staff, which allows them to offer more competitive salaries in many cases than their feeder districts, as they don't have to pay more experienced teachers' higher salaries. For this extra money, their staff can increase the school day and the school year. Matt DiCarlo puts it well:

If you do the simple math, NYC's oversubscribed charters offer, on average, 31 percent more time in school, which is the equivalent of roughly 56 days (using the 6.5 hour day of regular public schools). If we compare this with the overall impacts that I converted to “days of learning," we see that, on average, the estimated additional days of learning charters provided in reading (40) was lower than the actual days they offered (56). In math, it was higher (70 versus 56), but not by much. In other words, interpreting the finding that students pick up the equivalent of 40 or 70 extra "days of learning" is a bit different when you consider that the students were actually in these schools for the equivalent of 50-60 extra days (on average). [emphasis mine]

In addition: KIPP only takes those students who wish to go to KIPP, a key feature in any "choice" system. Even a Mathematica study admits: "Unlike local district schools, KIPP’s late entrants also tend to have higher prior achievement levels and fewer males than the rest of the KIPP student body," even as the authors argue that peer effects do not explain all of KIPP's advantage.

I would agree: no one thing explains why KIPP gets the results it gets. But more resources, a longer day and year, a self-selected student body, a focus on test prep... it would be astonishing if KIPP didn't get better results. But does this, by itself, justify a call for rampant charter expansion, particularly if the vast majority of the students won't attend KIPP schools with all of their resource advantages?

One more thing:

This is from the latest Mathematica report on KIPP (p. xviii):

Several factors may explain the trends in KIPP middle school impacts, including changes in the number and composition of schools in the sample, the relative performance of newer versus older schools, and changes over time in the effectiveness of existing KIPP schools as the network has expanded. Overall, KIPP’s student achievement impacts decreased during a time of high growth in the network, although they rebounded somewhat during the i3 scale-up period (Figure ES.5).[emphasis mine]

If we're going to rely on the Mathematica studies, we'd best consider all of what they say. KIPP's expansion has accompanied a decrease in its effect -- an effect that is still only a fraction of the effect of family income of test outcomes.

One thing we ought to figure out soon is whether the KIPP model can be scaled up any more than it currently is. KIPP has admitted repeatedly that it won't educate all of the children in a school district. Maybe the charter chain believes it can't get enough teachers to churn so it can keep salary costs relatively low. Maybe it doesn't believe it can scale up its philanthropic giving as it scales up its size.

I've said it before and I'll say it again: KIPP gets decent results. They run some good schools; good for them, they should be proud. But these lottery studies aren't nearly enough evidence to justify the dismantling of public school districts, as some would have us do. We should instead be asking why all schools can't pay their teachers more to work longer days and school years if that is part of what is helping to increase test-based outcomes.

Attrition

There's only one study I know of by Mathematica that looked at the attrition rates of Uncommon and Achievement First. The study is very limited in its methods: only five schools were sampled (three AF, 2 Uncommon), all were in New York City, and the study only looked at attrition between Grades 6 and 7 (p. 4-5). In addition:

The baseline proportion of students in special education or with LEP is significantly lower at this sample of AF/Uncommon middle schools compared with both district-wide levels and levels observed in feeder elementary schools. (p. 4)

I am always surprised when charter supporters get indignant about this. Of course the student populations are different. Of course there aren't as many LEP students or SWSN. In a choice system, it only makes sense that parents would "choose" a school that meets their child's needs, and it only makes sense that schools would specialize so that they weren't redundant with each other.

McDonalds doesn't sell tacos (last I checked), and Taco Bell doesn't sell hamburgers. Markets will push providers to differentiate themselves to meet different customer desires. When I looked at New Jersey charter student populations, I saw the same trend that Mathematica found: the charters were serving fewer SWSN and LEP students. The charters, at least so far, are differentiating themselves in the "market" by not serving as many of these students.

Is this a good thing?

As I've noted, the Uncommon school in New Jersey has a substantial rate of cohort attrition compared to its feeder district, Newark. It is not unreasonable to question whether this is affecting test scores as cohorts move through the school.

I also recently noted that Uncommon, the former employer of our future SecEd, John King, has high suspension rates compared to the local public districts in all the regions where it exists.

High suspension rates seem to be a defining feature of NYC charter schools, in contradiction to stated USDOE policy. But that's not surprising: charters have been at the forefront of the "no excuses" school discipline movement. Again, in a "choice" system, some families may look at high suspension rates as an acceptable price to pay to enroll their child in a school they consider academically superior, or even just safer. At least two questions arise, however:

1) Can charters exist as currently constituted only if they can rely on the local district schools to take all students, including the ones for whom charters are not a good "fit"?

2) Why do we accept charters that promote submissive attitudes in lower-income minority students, but not in the suburbs for more affluent children?

Pernicious Effects of Charterization

The Texas study you cite, Dmitri, is an excellent piece of research. But the latest data used in the study is from 2002-03, and at that time no district in Texas had charter penetration of more than 12 percent (p. 128). In fact, the vast majority of districts had charter penetration under 5 percent.

What happens, however, when charter enrollment hits a tipping point? When, as in Newark, charters start capturing upwards of 30 percent of the students enrolled, with even greater percentages on the horizon? What happens to the local school district then?

Bifulco and Reback (2011) find that charter proliferation had a significant negative impact on local school district finances in New York State. Arsen and Ni (2012) found similar negative effects in Michigan. Moody's also found charters negatively affect a district's finances. A report to the Pennsylvania Legislature by Ed Fuller and others at Penn State found charter payments were having a serious impact on local district budgets. PA actually saw one of its districts, Chester-Upland, go into a fiscal tailspin thanks to charter school payments. There are also similar reports from Tennessee.

This makes perfect sense: charter schools are redundant systems of school governance. Having multiple school leaders, multiple administrations, multiple service providers and so on is inherently inefficient. This can't be simply brushed aside: charterization can seriously hurt a local district's ability to educate the students who chose not to attend charters.

I must say I find it fascinating to see folks clamoring for more charters without making sure the children who stay in public district schools aren't hurt by charter expansion. Where is the urgency on behalf of these students? Who is writing the op-eds in Education Post and The 74 to speak for them?

New Orleans

Dmitri, I won't relitigate the debate on New Orleans and charter proliferation here. Instead, I'll let the man you cite, Doug Harris -- a highly skilled researcher and someone I've found to be quite reasonable in interviews -- speak for himself. From The 74, hardly an outlet for critics of charter schools:

Let me answer that by adding one more thing to what I was saying earlier about the conditions of New Orleans. One of the conditions in New Orleans was it was a very low performing district. It was a disaster by almost any measure. When I first got here I thought some of that was just hyperbole because everyone kept saying it, but now I've seen more and more objective evidence of that. The FBI had an office within the school board because there was so much corruption to investigate that they needed to have offices within the school board offices. I just got my hands on a report after many years of trying that was the Council of Great City Schools evaluating the human resources management of the district. The litany of problems in the district was just appalling, the way it operated. The outcomes were also poor as a result. It was the second lowest performing district in the second lowest performing state in the country. 

If you're in a district that already functions pretty well, maybe this is not necessarily a good idea. Here you had a district that was not functioning well by any measure. I think people tend to assume that all urban districts are terrible because the scores are low but it really is true that a lot of that is driven by the demographics and the socioeconomics of the districts. That's real. There are some very good schools in urban areas and there are some districts that probably do a pretty good job. You don't want to throw that out just because of a probable misinterpretation, if they're not taking into account where students are starting. 

I would tell a district or somebody advocating it in a district to look very carefully at how the traditional district is operating before going down that path. 

I would also tell them all the conditions, all the things they have to do to make it work. I think people tend to think, oh we just need more charter schools. But Detroit is a case study in how that goes wrong. Detroit has a lot of charter schools — I think it's second or third in market share — but it's terrible. The system has so little coordination. [emphasis mine]

I'll say again what I've said many, many times: I am not against charter schools. I started my K-12 career working in a charter. There is a place for "choice" in American education. There are some good charter schools out there and they should be commended for their work.

But when I read you, Dmitri, gravely intone that someone like me, questioning the claims of the charter cheerleading industry, is on "untenable moral ground" because of my doubts... well, sorry buddy, but I think you're projecting more than a little. Because the real shaky moral ground is under the feet of those who pretend that suburban "choice" is the same as urban charter school "choice," and use that posturing to allow charters to grow without proper oversight.

Charter "choice" is not suburban "choice." Shuffling children around within the borders of their district into schools that have unequal access to resources and unequal commitments to educating all students is not the "choice" offered in the suburbs. Offering families either underfunded, crumbling, filthy public schools or charters that are not state actors and do not afford students and parents the same due process rights is not the "choice" offered in the suburbs. Requiring students to submit to excessive punishments for trivial infractions is not the "choice" offered in the suburbs.

Let me close this portion of our conversation by quoting, as I did a few days ago, from the noted education scholar Pedro Noguera:

I'm not against charter schools, let me be clear, I'm in favor of any good school that's good for kids. But some of the charter schools that are being held up as a model believe that their goal is to regiment, to completely control their students. To control how they sit, control their eye contact, control their movements in the hallway. Many of them have silence in the hallway and no talking in the lunch room. John King, the new commissioner of education of New York state, is held up as a real reformer because he founded a very successful charter school in Boston called Roxbury Prep and went on to found a network the called Uncommon Schools. And I would say that academically this school is far out-performing many public schools that are serving the same population of kids. So I would acknowledge that they are doing a much better job. I would also acknowledge that the model they use does not appeal to me.

I've visited this school, and I noticed that children are not allowed to talk in the hall, and they get punished for the most minor infraction. And when I talked with John King afterwards, I said, "I've never seen a school that serves affluent children where they're not allowed to talk in the hall." And he said, "Well, that might be true, but this is the model that works for us, we've found that this is the model that our kids need."

So I asked him, "Are you preparing these kids to be leaders or followers? Because leaders get to talk in the hall. They get to talk over lunch, they get to go to the bathroom, and people can trust them. They don't need surveillance and police officers in the bathroom." And he looked at me like I was talking Latin, because his mindset is that these children couldn't do that.

Unfortunately what is often driving these high-performing schools is the idea that the kids need to be broken. That the kids' culture needs to be taken away from them and replaced with something else, because they come in with deficits. They come in as damaged goods. And these schools believe that their job is to mold the kids into something else.

And when they succeed, they end up with kids who no longer want to be associated with their own families and their own communities. Because the education they've been given has led them to believe that the goal is to escape. To escape the neighborhood, escape the community, to go someplace else. And so rather than education being a resource to help families and help communities, instead it's being used to cream off those we think have the talent. And let them go someplace else. Meanwhile the community stays exactly as it was and in many cases deteriorates further. That's not the education that I endorse. [emphasis mine]

Dmitri, you say: "Knowingly sending a child to a bad school is morally bankrupt." I couldn't agree more. Which is why, before we allow charter schools free rein to expand however much they would like, we'd better make damn sure they are providing children with an education that truly is better, and does not harm those students who choose not to attend. Marginal test score gains at a few high-profile charter chains are nice, but they are in no way enough evidence to show that charters should be allowed to grow unchecked.

I look forward to your reply.

Mark