Charter Schools, An Exchange: Part IV

Dmitri Mehlhorn and I continue our exchange about charter schools.

Part I: Mehlhorn's opener.

Part II: My reply.

Part III: Mehlhorn's response.

 Leave your thoughts in the comments below. We'll go one more round before we finish this series. Again, my thanks to Dmitri for participating in this debate.

ADDING: Here is an index for the entire exchange:

Part I: Mehlhorn's opener.

Part II: My reply.

Part III: Mehlhorn's response.

Part IV: My second reply.

Part V: Mehlhorn's second response.

Part VI: My final reply.

* * *

Dmitri Mehlhorn starts off his latest chapter in this debate by quoting me:

“As Jazzman wrote to me: ‘Almost everything you say is factually correct; however … you arrive at conclusions that simply are not warranted.’”

Notice the ellipsis? Here’s what I said in its entirety:

“Your essay is, in many ways, a perfect example of "reform" thinking. Almost everything you say is factually correct; however, you are so selective in your citations, and so lacking in context when discussing them, that you arrive at conclusions that simply are not warranted.”

Reads a little bit different when you get the whole quote, doesn’t it?

In a way, I’m glad Mehlhorn did this: unwittingly, he has given a perfect demonstration of why the charter sector's arguments in favor of the rampant expansion of charter schools simply do not hold up. He cherry-picks his data and places it completely out of appropriate context. Then he mischaracterizes my arguments against his position by setting up straw men, stuffed with poor paraphrases of my words. In addition, he misinterprets the research he himself cites, creating the impression that charters in the aggregate are getting practically significant gains.

To top it all off, he brushes aside my concerns about the pernicious effects of charters, as if the real fiscal problems that come from having redundant systems of school management can be ignored if there are meager gains in test scores, which are crappy, noisy measures of learning to begin with.

I’m going to dismantle Mehlhorn's points one at a time; afterward, I’m going to summarize why I see the argument for charter expansion as little more than a distraction from doing what really needs to be done to address the growing inequality, chronic poverty, and endemic institutional racism from which our nation currently suffers.

Fair warning: I'm going to be blunt. That doesn't mean I don't appreciate Dmitiri's time and effort in having this exchange.

The Burden of Proof

As Ajay Srikanth points out in the comments below the last post, Mehlhorn has misinterpreted the use of “strict scrutiny” in his defense of charters. I do, however, agree with one point he makes: anyone who advocates for a particular policy should be able to give a “compelling public rationale” for it. What Mehlhorn fails to apprehend, however, is that the burden of proof is not on those who express skepticism about a policy; it’s on those who take an affirmative stance for it.

Let’s suppose I’m a state official, and Dmitiri comes to me with an application to start the Russell’s Teapot Charter School of Excellence. I look at his application and say: “Sorry, you haven’t shown me a ‘compelling public rationale’ for approving this program.”

It isn't sufficient for Dmitri to say simply say: “Well, you can’t prove my charter school WON’T be successful! And I’ve got parents lined up to enroll their kids! You MUST give me the charter if you can’t prove I WON’T have a great school!”

Would anyone think this was reasonable? Yet that’s exactly the position Mehlhorn takes: if someone can gather up a few parents who want something, they should get it, absent a compelling reason that they shouldn’t.

This is a terrible way to guide any public policy, let alone education. Charter school advocates are the ones who need to demonstrate why they should be allowed to run schools that are not state actors yet continue to receive public funding. And we certainly have enough evidence available at this point to make reasonably informed judgments about the role charter schools should play going forward; we should examine it and see if the arguments of the charter sector for widespread expansion hold up.

Of course, we have to interpret that evidence correctly. Unfortunately, Mehlhorn, like so many “reformers,” demonstrates he is either incapable or unwilling to do just that.

The CREDO Studies, Again

Mehlhorn once again cites the CREDO studies of charter school effects as evidence in favor of charter expansion. Here’s his argument (without any ellipses):

“Is it just some cities, or is it a national story? Let’s go back to Stanford University CREDO’s 2015 study of 41 urban regions. Jazzman is right that results vary by city. For instance, charters in Newark, Detroit, and DC do better than traditional schools; in Las Vegas, Fort Worth, and West Palm Beach, charters do worse. But what is the aggregate result for students who otherwise don’t have choices? What about for students like Sally, who is in the cohort that CREDO refers to as Hispanic ELL?  Nationally, across both well-run and poorly run city charters, charters improve literacy and numeracy for Sally’s cohort by 0.1 standard deviations over their matched peers in traditional schools.” [emphasis mine]

Notice that Mehlhorn asks what the “aggregate result” is for students; he then cherry-picks the highest performing subgroup’s results, and not the aggregate he himself called for.

The subgroup effect sizes in the 2015 CREDO report are found in Table 5, pages 17-18. Here, in its entirety, is the table:

I’m sorry to be blunt, but this is really shameless. Mehlhorn isolates the subgroup with the largest effect size from all the others, and doesn’t even acknowledge that for some other subgroups – whites and Native Americans* – the effects are negative. Are we supposed to just wish those away?

The aggregate effects of charter schools according to the CREDO report is 0.055 SDs in math and 0.039 SDs in reading -- that's what we should be talking about. But let’s, solely for the sake of argument, ignore Mehlhorn’s trickery. Is a 0.1 effect size practically significant?

Effect Sizes

Mehlhorn says 0.1 SDs is significant. He cites statistics on cancer deaths as evidence; unfortunately, he completely misunderstands their implications when he makes his analogy. The average age of cancer victims’ deaths and the standard deviations do not in any way describe the effects of a treatment on cancer: they are purely descriptive. An effect size, in contrast, looks at the effect of some intervention, like enrolling in a charter school.

Mehlhorn is closer to the mark in suggesting that an investment strategy that “beats the market” by 0.1 SDs has significance; here, the implied treatment would be investing in some way that maximizes gains. However, as I explained in my last post, statistical significance is not the same as practical significance. In any scenario, more returns on investment are preferable to fewer. But if my investment goal is to make a million dollars, getting $275 over $232 is not of any practical significance for me.

Let me suggest another analogy that might make things more clear for Mehlhorn: let’s suppose I really want to play in the NBA, but I’m too short. I’m only 5’10”**, the average height for males in the United States. But I hear about a pill I can take that will increase my size “significantly.” I buy it and take it, and then measure myself after the treatment period is over. It turns out I grew by 0.1 SDs.

The standard deviation for height is 3 inches; I grew less than one-third of an inch. Are the Knicks going to be calling?

When it comes to schooling, I constantly hear from “reformers” that charter schools are proving: "Poverty is not a final destination for these kids. Charter schools have shown that kids from impoverished backgrounds cannot be written off."

Mehlhorn himself engages in similar rhetoric: “Charter critics can decry residential segregation and ask for all schools to become good, but until that happens, it is simply immoral to cap charter schools and thus deny poor parents the same choice that wealthy parents already have.” [emphasis mine]

This was the entire point of the chart I showed in my first post, and that I present once again here:

The comparison you have made, Dmitiri, is between the test score gap created by inequitable distribution of wealth and the gains of charter schools. But even if we accede to your brazen cherry-picking of results in the CREDO study, it’s clear the effects of charter schools do not come anywhere close to providing urban parents with an educational experience that is equivalent to those in the suburbs.

Yes, the charters, according to CREDO, do marginally better in the aggregate. The questions we ought to ask, however, are:

•        What is the price being paid for this marginal increase?
•         Is this really the best we can do?

Days of Learning

Mehlhorn, despite my cautions in my last post, seems to be hung up on this idea that the correct way to express charter school gains is in added “days of learning.” As I said previously, smart researchers I respect think this is fine. I don’t, and the next time I argue with them about this, I’m going to pull out Mehlhorn’s post. It is a nearly perfect illustration of how badly a layperson can misinterpret the conversion from effect sizes in standard deviations to “days of learning”; I hold it up as proof that the conversion is so conceptually flawed that it should never be used in studies of educational interventions again.

Here, in full, is Mehlhorn’s argument based on “days of learning”:

“In education, the most commonly used benchmark converts standard deviations into days of learning. By this method, CREDO shows charters providing Hispanic urban ELL students with effectively a 40% longer school year (72 days). Over the course of a K-12 career, that’s the equivalent of adding more than 5 extra years of schooling. That’s enough extra schooling for a four-year college plus a one-year masters program. Jazzman protests that this translation is imprecise, and he is correct. We don’t know for sure exactly 72 extra days of learning. It might only be 50 (just free junior college for Sally), or it might be 100 (throw in a free Ph.D). Either way, these are not small potatoes.” [emphasis mine]

No. No, no, no.

The implication Mehlhorn makes here – one that, admittedly, is logical for a layperson – is that the “days of learning” gains translate into additional content. But that can’t possibly be right, because the tests used to judge the “days of learning” do not assess students’ abilities in any content beyond their current grade level.

To illustrate, let’s use the Common Core standards in math for grade 6. If a student was “a year ahead,” we’d expect that student would be learning content for grade 7. One of the standards in grade 7 math is for students to calculate the circumference of a circle given its diameter. It’s possible a sixth grader knows how to do this… but we’ll never know from the test, because the grade 6 test, if it’s aligned to the standards, doesn’t test grade 7 material. So the grade 6 test won’t ask a student to calculate the circumference, and we won’t know if she can.

It would be better if instead of using the phrase “days of learning” the CREDO authors used the phrase “days of drilling.” Because that’s all the test really can measure; again, if it’s aligned to the standards, it won’t measure a student’s knowledge of content taught and tested in the next year.

This was the point of the Matt DiCarlo post I linked to in my previous response: yes, high-performing charters do have more actual school days. The higher effect sizes are a product, in part, of that extra drilling. But it is completely unwarranted to extrapolate that these extra days are days spent learning new content; the tests can’t tell us whether that occurred or not. Consequently, the notion a K-12 charter student has the equivalent of a year’s worth of graduate education is without any merit, even if said in jest.

To further complicate matters: Mehlhorn links to a paper by Hill & Booth on the translation of effect sizes into “years of learning.” He claims: "In education, the most commonly used benchmark converts standard deviations into days of learning." The problem is that at no point in his source do the authors sanction what the CREDO studies do: breaking down that hypothetical year into days. One reason is that they show the different transitions from grade to grade (K-1, 1-2, 2-3, etc.) have different effect sizes (p. 3). In general, the transitions in earlier grades have much greater effect sizes than the later grade transitions; this makes equating “days of learning” in each grade indefensible.***

Look, I can understand the desire of researchers to translate their findings into terms comprehensible to laypeople. But “days of learning” just doesn’t work. What does work is what’s actually being measured: the proportion of questions answered correctly on a test. I can state with confidence that in a multi-day test with many items, the effect sizes found in the CREDO studies amount to, at most, a few more bubbles filled in correctly.

Is upending our entire education system worth that?

The Econometric Studies

I must admit, this statement of Mehlhorn’s was a big surprise:

“Jazzman concedes the research from Mathematica and others that students who attend high-performing charter schools – such as KIPP and Achievement First – have better outcomes than students who apply to those schools but don’t win admissions in their lotteries. Jazzman dismisses these results by saying that those are just the good charter schools.

If he’s right, charter restrictions should stay in place for all charters that have not been validated by robust research. Validated charters should expand. Here in Virginia, my daughter’s friend Sally will have the option to attend KIPP schools. That is a huge relief for me, and I look forward to Jazzman’s support.”

If what Melhorn is saying is that only those schools, such as KIPP, that have been shown in studies to get marginal gains in tests scores should be allowed to expand, his position is likely going to send shock waves throughout the charter sector. Once again, here’s a graph from Bruce Baker:

KIPP is around 5 percent of the entire charter sector. The plurality of charters are small-time operations, with one or two schools in their network. Is Mehlhorn really saying none should be allowed to expand?

Of course, KIPP has plenty of money to throw around, so they can afford to commission Mathematica to extensively study their schools. Too bad for the smaller operators, I guess…

Let’s once again go over the limitations of these studies:

• They are restricted to a few operators like KIPP who represent only a fraction of the charter sector.
• The “lottery” studies are only generalizable to the pool of students who enter the lottery, and only the schools that are oversubscribed.
• Most importantly: they measure their effects through noisy, crappy standardized tests in only two subjects.

What do we really know about the total educational experience of children who attend “high-performing,” oversubscribed charters? Do they get the full range and breadth of educational experiences that suburban children in well-resourced schools receive? Do they have opportunities for extra-curricular sports – I mean, aside from the truly awful charters that have become a de facto recruiting system for the NCAA? What about the arts? Lab sciences and history?

This is an area that has been grossly underreported in the literature, yet it’s probably more important than math and reading outcomes on standardized tests. As Jonathan Kozol has noted, these tests are largely seen as annoyances in affluent suburban districts. Yet it seems to me that KIPP and other CMOs base most, if not all, of their claims of success on these tests.

Wayne Au and others have documented extensively that the accountability measures imposed since No Child Left Behind – measures that, I would argue, have largely driven the expansion of charters – are narrowing the curriculum, particularly for children in schools deemed to be “failing.” Charter expansion, it seems to me, is playing right into this dynamic. Charters are fighting on the field of the standardized test, even though the test is, even by the current charter-lovin’ administration’s own admission, a pernicious influence.

The questions stand: what is the price being paid for these small gains in measures that are largely correlated to income? And will moving school governance and public funds away from state actors improve urban school outcomes enough to make a real difference in social mobility and income inequity?

I say no, and I turn to a study Mehlhorn has now cited twice for a little evidence: Angrist and Dynarski(2013).

Oversubscribed Boston Charter Schools

Again: one study in one city of only oversubscribed charters is not enough evidence to justify the radical policy shifts Mehlhorn seems to favor. This is such an obvious point I feel silly having to make it, but if Mehlhorn insists on generalizing Angrist & Dynarski to the entire charter sector, I have little choice but to restate what should be plain to him.

This said, let me restate that Josh Angrist and Susan Dynarski are two highly respected researchers and their work should be taken seriously. So let’s jump in and see what they and their team found:

There is no doubt that this limited group of students in this limited group of charter schools in this one city did better on their one state graduation requirement test. I don’t discount this at all, even as I question its generalizability. I do, however, think it is far too simplistic to simply say that the secret of the charter schools is their “no excuses” pedagogy and structure.

The authors, in a footnote on p.5, admit that per pupil expenses are larger at Boston’s oversubscribed charters than at the district schools. There is a problem in making comparisons because of grade level differences; however, that problem is further compounded by the fact that district schools usually have spending obligations that charter schools do not. As I said in my earlier post, high-performing charters have been shown to have a spending advantage in previous studies. Might that be the case in Boston?

The authors point out that the charters they are studying have a longer school day and a longer school year than the counterfactual public schools (p. 4). If that’s the cause of the improvement in test scores, fine… why, then, do we need charter schools? Why not just spend more on personnel to extend the school day and year? I’ve seen no evidence that unions would have an objection to this, so long as their members are paid for their extra work.

The authors also admit that peer effects could, in fact, account for some of the differences between charter and district school outcomes (p. 20). How large is the effect of charter enrollment on peer achievement? 0.13 SD – the same effect size Mehlhorn crows about from his cherry-picked subgroup in the CREDO report.****

I actually have an issue with how the authors define “peer effect”: I don’t think it should be limited merely to one’s cohort peers’ test scores, as cohorts tend to mix in high school courses and test score histories likely don’t fully describe which peers influence classroom interactions. But even with this issue aside, the authors do acknowledge a substantial body of research that shows peer effects are real (footnote 27, p. 21). Why, then, is Mehlhorn so anxious to downplay those effects, yet oversells charter effects of the same or even lesser magnitude?

Again, Mehlhorn himself has suggested an equivalency between charter school “choice” and elite suburban schools. So what does Angrist & Dynarski’s paper say about this? Do Boston’s charters create an experience for their students that rivals that of suburban schools?

This study shows they most definitely do not. Yes, the charters raise SAT scores, likely the result of all that extra time in school drilling. But there is no evidence these charters are able to propel their students into the top quartile of SAT scores (Table 5). There is a very small effect size when measuring these students’ ability to get a 3 on AP exams, and no effect on getting a 4 or 5 (Table 6), the usual standard for getting transcript credits at elite colleges. Charter attendance does not increase the likelihood of graduating from high school; in fact, it decreases the chance of on-time graduation (p. 15). And while charters seem to have a small, positive effect on college enrollment and persistence, the study shows that charter students have no advantage in enrolling in the most elite colleges (Table 8).

Let me restate this so as to be absolutely clear: yes, Boston’s oversubscribed charters do show a substantial gain on test-based outcomes, and that shouldn’t be discounted. But it is a massive, unwarranted leap to go from that conclusion to suggesting that urban charter schools are closing the income gap in education; they most certainly are not.

And we still come back to the most important question: are these gains, which are largely marginal, worth the radical policy of shifting America’s schools away from democratic control and toward a market-like system?

Other Stuff Mehlhorn Gets Wrong

I’ve already gone on way too long; however, there’s still much left in Mehlhorn’s post to debunk. Quickly:

- Yes, charters were the first choice in Newark for many families. But here’s what’s telling: as I showed in this policy brief, the “popular” schools in Newark, both charter and public, are some of the most segregated schools in the city. I can’t think of better evidence to show that what parents are really choosing in a “choice” system are their children's peers.

- If there is any evidence that Los Angeles has a better system for authorizing charters than other cities, I haven’t seen it. The CREDO report for LA puts charter gains at 0.07 SDs for reading and 0.11 SDs for math. Again: one-third of an inch.

- The notion that the nation should pattern its education policies on DC and Tennessee is utterly laughable.

- As to New Orleans: again, Doug Harris himself urged great caution in generalizing his preliminary results. Why Mehlhorn doesn’t listen to him, I have no idea. It's also worth noting that New Orleans students who were displaced by Katrina into public schools outside of the city showed substantial gains after their first year.

- When the United States installs social programs equivalent to Sweden's, I’ll entertain a conversation about whether anything they’ve learned about vouchers can be applied to our country.

Charteriness

So what’s the bottom line here? Dmitiri and I agree that charters, on average, show a gain in test scores; however (please don't put an ellipsis in place of this phrase, Dmitiri), while he thinks that gain matters, I’ve shown he’s just plain wrong. The gain is, in the aggregate, very small. In no way can it be compared to the advantage children in affluent school districts enjoy over their less advantaged peers. In no way can charters be said to be providing anything close to an equivalent educational experience as can be found in the leafy ‘burbs. In no way can anyone credibly make the case that charters are substantially closing the so-called “achievement gap.” Anyone who even infers that urban charter “choice” is equivalent to suburban choice is selling a lie.

Furthermore, the gains that charters show are all based on noisy, crappy, unvalidated standardized tests whose outcomes can largely be explained by economic status. Even the most ardent adherents of the value of these tests have begun to back away from embracing them as the public comes to realize they are poor measures of the complete education of a child.

So what price is being paid for these marginal gains on these poor measures? What costs does charterization exact from all of the stakeholders in American education so that students can make small, inconsequential gains on a series of bubble tests? Mehlhorn seems to think that so long as students who don’t enroll in charters maintain their scores on these same questionable measures, no one should worry about any other pernicious effects from expanding charters.

But life is more than a test, Dmitiri, and the downsides of charterization are too large to ignore:

- Because charters are not subject to the same standards of oversight and accountability as public district schools, there is a large and growing trail of fraud, waste, and abuse within the charter sector.

- Because charters set up redundant systems of school governance, they negatively impact public district budgets (see all my links in Part II). The impact may or may not show up in test scores; however, as I have shown in my previous work, funding inadequacy can negatively impact programs in the arts, physical education, and student support services. I have also presented evidence that charters themselves do not provide many of the support services at-risk and special education students may need to be successful in school. These negative effects likely don't show up in test scores; that doesn't mean they aren't a serious problem.

- Because charters have been found not to be state actors, they limit the rights of parents, students, and teachers (again, see Part II).

- Because many “high-performing” charters hire teaching staff from outside the communities where they are situated, teachers of color may suffer an inordinate consequence of charterization.

- Because many high-performing charters engage in “no excuses” pedagogy, they may be teaching their students to be compliant in a way that would not be tolerated in affluent suburban districts.

Can anyone seriously make the case that answering a few more questions correctly on a standardized test is worth all this?

I’ll say it yet again: I am not against charter schools per se. There may well be a place for “choice” within our school systems. I am certainly sympathetic to the notion that urban parents want a better education for their children. I do not think parents should be forced to send their children to schools that are unsafe or have disruptive environments.

But the evidence is increasingly clear: those charters that show gains appear to rely on a combination of increased funding, longer schools days and years, a test-focused curriculum, peer effects, and, in some cases, student selection, attrition effects, and decreased class sizes. There is no “secret sauce”: no special magic that can only be concocted by charter schools.

Many times, charter advocates make the case that they are free from “bureaucracy,” which allows them to “innovate” and “do more with less.” I see no evidence that this is true. We could put more money into all schools to lengthen the school day and year and reduce class sizes. We could desegregate our schools and our communities to spread peer effects more evenly. We could further narrow the curriculum in all schools and focus only on tested subjects (this would be a very bad idea). “Charteriness” is not a necessary precondition for any of these changes.

So why does the “reform” movement focus so much of its attention of charter school expansion? I contend it’s not so much that the “reformers” want to make money off of charters or bust unions (although, for some of them, these are nice sidebenefits). Rather, the “reformers,” no matter how well-intentioned some may be, are looking for a quick fix to problems that must be addressed within a context that is much larger than education policy.

Yes, we need good schools to have a just society. But social mobility is not social equity. Moving a few more students of color up the socio-economic ladder (and, again: we don't have much evidence charters are really doing this) does not address a core problem America faces: millions of people are doing necessary work yet falling further and further behind. They are not seeing any of the productivity gains this country has made over the last several decades; instead, they continue to do work that everyone admits must be done while making less and less.

Shuffling students around within their district borders into schools that get only slightly better test scores does nothing to address this core problem in American society. Creating a false narrative that school outcomes are the cause of inequity, and not the symptom, doesn’t help the problem; it perpetuates it.

If the best we can do for America’s most disadvantaged and deserving children is to build more schools under private control that largely teach a hidden pedagogy of compliance and get small gains on bubble tests, we have failed. We have failed because we are having the wrong conversation, and we are deluding ourselves into thinking we have equalized the lives of advantaged and disadvantaged children merely through changing how schools are governed.

It’s long past time to start being honest with ourselves about this. Mehlhorn ended with a quote by Upton Sinclair; let me end with it as well:

"It is difficult to get a man to understand something, when his salary depends upon his not understanding it."

* A technical note: I’m not even sure the category “Hispanic Students w/ELL Status” is consistently reported throughout the sample CREDO used. In New Jersey, school-level test scores are not reported for this group. Does CREDO’s sample disaggregate the student level data consistently across all states? The technical appendix doesn’t seem to say.

** I’m actually 6’ 1”. Because you really needed to know…

*** There’s also the question of whether Hill & Booth’s work, based on “vertically scaled” tests, can be applied to state tests that are not scaled this way. I’ll save that discussion, which I think is important, for another time.

**** To clarify: "Not surprisingly, given the positive selection of charter applicants, charter enrollment is associated with increases in peer achievement in the first post-lottery year: the (marginally significant) effect here is roughly 0.13 in each subject." (p. 20)