Charter Schools, An Exchange: Part IV

Dmitri Mehlhorn and I continue our exchange about charter schools.

Part I: Mehlhorn's opener.

Part II: My reply.

Part III: Mehlhorn's response.

 Leave your thoughts in the comments below. We'll go one more round before we finish this series. Again, my thanks to Dmitri for participating in this debate.

ADDING: Here is an index for the entire exchange:

Part I: Mehlhorn's opener.

Part II: My reply.

Part III: Mehlhorn's response.

Part IV: My second reply.

Part V: Mehlhorn's second response.

Part VI: My final reply.

* * *

Dmitri Mehlhorn starts off his latest chapter in this debate by quoting me:

“As Jazzman wrote to me: ‘Almost everything you say is factually correct; however … you arrive at conclusions that simply are not warranted.’”

Notice the ellipsis? Here’s what I said in its entirety:

“Your essay is, in many ways, a perfect example of "reform" thinking. Almost everything you say is factually correct; however, you are so selective in your citations, and so lacking in context when discussing them, that you arrive at conclusions that simply are not warranted.”

Reads a little bit different when you get the whole quote, doesn’t it?

In a way, I’m glad Mehlhorn did this: unwittingly, he has given a perfect demonstration of why the charter sector's arguments in favor of the rampant expansion of charter schools simply do not hold up. He cherry-picks his data and places it completely out of appropriate context. Then he mischaracterizes my arguments against his position by setting up straw men, stuffed with poor paraphrases of my words. In addition, he misinterprets the research he himself cites, creating the impression that charters in the aggregate are getting practically significant gains.

To top it all off, he brushes aside my concerns about the pernicious effects of charters, as if the real fiscal problems that come from having redundant systems of school management can be ignored if there are meager gains in test scores, which are crappy, noisy measures of learning to begin with.

I’m going to dismantle Mehlhorn's points one at a time; afterward, I’m going to summarize why I see the argument for charter expansion as little more than a distraction from doing what really needs to be done to address the growing inequality, chronic poverty, and endemic institutional racism from which our nation currently suffers.

Fair warning: I'm going to be blunt. That doesn't mean I don't appreciate Dmitiri's time and effort in having this exchange.

The Burden of Proof

As Ajay Srikanth points out in the comments below the last post, Mehlhorn has misinterpreted the use of “strict scrutiny” in his defense of charters. I do, however, agree with one point he makes: anyone who advocates for a particular policy should be able to give a “compelling public rationale” for it. What Mehlhorn fails to apprehend, however, is that the burden of proof is not on those who express skepticism about a policy; it’s on those who take an affirmative stance for it.

Let’s suppose I’m a state official, and Dmitiri comes to me with an application to start the Russell’s Teapot Charter School of Excellence. I look at his application and say: “Sorry, you haven’t shown me a ‘compelling public rationale’ for approving this program.”

It isn't sufficient for Dmitri to say simply say: “Well, you can’t prove my charter school WON’T be successful! And I’ve got parents lined up to enroll their kids! You MUST give me the charter if you can’t prove I WON’T have a great school!”

Would anyone think this was reasonable? Yet that’s exactly the position Mehlhorn takes: if someone can gather up a few parents who want something, they should get it, absent a compelling reason that they shouldn’t.

This is a terrible way to guide any public policy, let alone education. Charter school advocates are the ones who need to demonstrate why they should be allowed to run schools that are not state actors yet continue to receive public funding. And we certainly have enough evidence available at this point to make reasonably informed judgments about the role charter schools should play going forward; we should examine it and see if the arguments of the charter sector for widespread expansion hold up.

Of course, we have to interpret that evidence correctly. Unfortunately, Mehlhorn, like so many “reformers,” demonstrates he is either incapable or unwilling to do just that.

The CREDO Studies, Again

Mehlhorn once again cites the CREDO studies of charter school effects as evidence in favor of charter expansion. Here’s his argument (without any ellipses):

“Is it just some cities, or is it a national story? Let’s go back to Stanford University CREDO’s 2015 study of 41 urban regions. Jazzman is right that results vary by city. For instance, charters in Newark, Detroit, and DC do better than traditional schools; in Las Vegas, Fort Worth, and West Palm Beach, charters do worse. But what is the aggregate result for students who otherwise don’t have choices? What about for students like Sally, who is in the cohort that CREDO refers to as Hispanic ELL?  Nationally, across both well-run and poorly run city charters, charters improve literacy and numeracy for Sally’s cohort by 0.1 standard deviations over their matched peers in traditional schools.” [emphasis mine]

Notice that Mehlhorn asks what the “aggregate result” is for students; he then cherry-picks the highest performing subgroup’s results, and not the aggregate he himself called for.

The subgroup effect sizes in the 2015 CREDO report are found in Table 5, pages 17-18. Here, in its entirety, is the table:

I’m sorry to be blunt, but this is really shameless. Mehlhorn isolates the subgroup with the largest effect size from all the others, and doesn’t even acknowledge that for some other subgroups – whites and Native Americans* – the effects are negative. Are we supposed to just wish those away?

The aggregate effects of charter schools according to the CREDO report is 0.055 SDs in math and 0.039 SDs in reading -- that's what we should be talking about. But let’s, solely for the sake of argument, ignore Mehlhorn’s trickery. Is a 0.1 effect size practically significant?

Effect Sizes

Mehlhorn says 0.1 SDs is significant. He cites statistics on cancer deaths as evidence; unfortunately, he completely misunderstands their implications when he makes his analogy. The average age of cancer victims’ deaths and the standard deviations do not in any way describe the effects of a treatment on cancer: they are purely descriptive. An effect size, in contrast, looks at the effect of some intervention, like enrolling in a charter school.

Mehlhorn is closer to the mark in suggesting that an investment strategy that “beats the market” by 0.1 SDs has significance; here, the implied treatment would be investing in some way that maximizes gains. However, as I explained in my last post, statistical significance is not the same as practical significance. In any scenario, more returns on investment are preferable to fewer. But if my investment goal is to make a million dollars, getting $275 over $232 is not of any practical significance for me.

Let me suggest another analogy that might make things more clear for Mehlhorn: let’s suppose I really want to play in the NBA, but I’m too short. I’m only 5’10”**, the average height for males in the United States. But I hear about a pill I can take that will increase my size “significantly.” I buy it and take it, and then measure myself after the treatment period is over. It turns out I grew by 0.1 SDs.

The standard deviation for height is 3 inches; I grew less than one-third of an inch. Are the Knicks going to be calling?

When it comes to schooling, I constantly hear from “reformers” that charter schools are proving: "Poverty is not a final destination for these kids. Charter schools have shown that kids from impoverished backgrounds cannot be written off."

Mehlhorn himself engages in similar rhetoric: “Charter critics can decry residential segregation and ask for all schools to become good, but until that happens, it is simply immoral to cap charter schools and thus deny poor parents the same choice that wealthy parents already have.” [emphasis mine]

This was the entire point of the chart I showed in my first post, and that I present once again here:

The comparison you have made, Dmitiri, is between the test score gap created by inequitable distribution of wealth and the gains of charter schools. But even if we accede to your brazen cherry-picking of results in the CREDO study, it’s clear the effects of charter schools do not come anywhere close to providing urban parents with an educational experience that is equivalent to those in the suburbs.

Yes, the charters, according to CREDO, do marginally better in the aggregate. The questions we ought to ask, however, are:

•        What is the price being paid for this marginal increase?
•         Is this really the best we can do?

Days of Learning

Mehlhorn, despite my cautions in my last post, seems to be hung up on this idea that the correct way to express charter school gains is in added “days of learning.” As I said previously, smart researchers I respect think this is fine. I don’t, and the next time I argue with them about this, I’m going to pull out Mehlhorn’s post. It is a nearly perfect illustration of how badly a layperson can misinterpret the conversion from effect sizes in standard deviations to “days of learning”; I hold it up as proof that the conversion is so conceptually flawed that it should never be used in studies of educational interventions again.

Here, in full, is Mehlhorn’s argument based on “days of learning”:

“In education, the most commonly used benchmark converts standard deviations into days of learning. By this method, CREDO shows charters providing Hispanic urban ELL students with effectively a 40% longer school year (72 days). Over the course of a K-12 career, that’s the equivalent of adding more than 5 extra years of schooling. That’s enough extra schooling for a four-year college plus a one-year masters program. Jazzman protests that this translation is imprecise, and he is correct. We don’t know for sure exactly 72 extra days of learning. It might only be 50 (just free junior college for Sally), or it might be 100 (throw in a free Ph.D). Either way, these are not small potatoes.” [emphasis mine]

No. No, no, no.

The implication Mehlhorn makes here – one that, admittedly, is logical for a layperson – is that the “days of learning” gains translate into additional content. But that can’t possibly be right, because the tests used to judge the “days of learning” do not assess students’ abilities in any content beyond their current grade level.

To illustrate, let’s use the Common Core standards in math for grade 6. If a student was “a year ahead,” we’d expect that student would be learning content for grade 7. One of the standards in grade 7 math is for students to calculate the circumference of a circle given its diameter. It’s possible a sixth grader knows how to do this… but we’ll never know from the test, because the grade 6 test, if it’s aligned to the standards, doesn’t test grade 7 material. So the grade 6 test won’t ask a student to calculate the circumference, and we won’t know if she can.

It would be better if instead of using the phrase “days of learning” the CREDO authors used the phrase “days of drilling.” Because that’s all the test really can measure; again, if it’s aligned to the standards, it won’t measure a student’s knowledge of content taught and tested in the next year.

This was the point of the Matt DiCarlo post I linked to in my previous response: yes, high-performing charters do have more actual school days. The higher effect sizes are a product, in part, of that extra drilling. But it is completely unwarranted to extrapolate that these extra days are days spent learning new content; the tests can’t tell us whether that occurred or not. Consequently, the notion a K-12 charter student has the equivalent of a year’s worth of graduate education is without any merit, even if said in jest.

To further complicate matters: Mehlhorn links to a paper by Hill & Booth on the translation of effect sizes into “years of learning.” He claims: "In education, the most commonly used benchmark converts standard deviations into days of learning." The problem is that at no point in his source do the authors sanction what the CREDO studies do: breaking down that hypothetical year into days. One reason is that they show the different transitions from grade to grade (K-1, 1-2, 2-3, etc.) have different effect sizes (p. 3). In general, the transitions in earlier grades have much greater effect sizes than the later grade transitions; this makes equating “days of learning” in each grade indefensible.***

Look, I can understand the desire of researchers to translate their findings into terms comprehensible to laypeople. But “days of learning” just doesn’t work. What does work is what’s actually being measured: the proportion of questions answered correctly on a test. I can state with confidence that in a multi-day test with many items, the effect sizes found in the CREDO studies amount to, at most, a few more bubbles filled in correctly.

Is upending our entire education system worth that?

The Econometric Studies

I must admit, this statement of Mehlhorn’s was a big surprise:

“Jazzman concedes the research from Mathematica and others that students who attend high-performing charter schools – such as KIPP and Achievement First – have better outcomes than students who apply to those schools but don’t win admissions in their lotteries. Jazzman dismisses these results by saying that those are just the good charter schools.

If he’s right, charter restrictions should stay in place for all charters that have not been validated by robust research. Validated charters should expand. Here in Virginia, my daughter’s friend Sally will have the option to attend KIPP schools. That is a huge relief for me, and I look forward to Jazzman’s support.”

If what Melhorn is saying is that only those schools, such as KIPP, that have been shown in studies to get marginal gains in tests scores should be allowed to expand, his position is likely going to send shock waves throughout the charter sector. Once again, here’s a graph from Bruce Baker:

KIPP is around 5 percent of the entire charter sector. The plurality of charters are small-time operations, with one or two schools in their network. Is Mehlhorn really saying none should be allowed to expand?

Of course, KIPP has plenty of money to throw around, so they can afford to commission Mathematica to extensively study their schools. Too bad for the smaller operators, I guess…

Let’s once again go over the limitations of these studies:

• They are restricted to a few operators like KIPP who represent only a fraction of the charter sector.
• The “lottery” studies are only generalizable to the pool of students who enter the lottery, and only the schools that are oversubscribed.
• Most importantly: they measure their effects through noisy, crappy standardized tests in only two subjects.

What do we really know about the total educational experience of children who attend “high-performing,” oversubscribed charters? Do they get the full range and breadth of educational experiences that suburban children in well-resourced schools receive? Do they have opportunities for extra-curricular sports – I mean, aside from the truly awful charters that have become a de facto recruiting system for the NCAA? What about the arts? Lab sciences and history?

This is an area that has been grossly underreported in the literature, yet it’s probably more important than math and reading outcomes on standardized tests. As Jonathan Kozol has noted, these tests are largely seen as annoyances in affluent suburban districts. Yet it seems to me that KIPP and other CMOs base most, if not all, of their claims of success on these tests.

Wayne Au and others have documented extensively that the accountability measures imposed since No Child Left Behind – measures that, I would argue, have largely driven the expansion of charters – are narrowing the curriculum, particularly for children in schools deemed to be “failing.” Charter expansion, it seems to me, is playing right into this dynamic. Charters are fighting on the field of the standardized test, even though the test is, even by the current charter-lovin’ administration’s own admission, a pernicious influence.

The questions stand: what is the price being paid for these small gains in measures that are largely correlated to income? And will moving school governance and public funds away from state actors improve urban school outcomes enough to make a real difference in social mobility and income inequity?

I say no, and I turn to a study Mehlhorn has now cited twice for a little evidence: Angrist and Dynarski(2013).

Oversubscribed Boston Charter Schools

Again: one study in one city of only oversubscribed charters is not enough evidence to justify the radical policy shifts Mehlhorn seems to favor. This is such an obvious point I feel silly having to make it, but if Mehlhorn insists on generalizing Angrist & Dynarski to the entire charter sector, I have little choice but to restate what should be plain to him.

This said, let me restate that Josh Angrist and Susan Dynarski are two highly respected researchers and their work should be taken seriously. So let’s jump in and see what they and their team found:

There is no doubt that this limited group of students in this limited group of charter schools in this one city did better on their one state graduation requirement test. I don’t discount this at all, even as I question its generalizability. I do, however, think it is far too simplistic to simply say that the secret of the charter schools is their “no excuses” pedagogy and structure.

The authors, in a footnote on p.5, admit that per pupil expenses are larger at Boston’s oversubscribed charters than at the district schools. There is a problem in making comparisons because of grade level differences; however, that problem is further compounded by the fact that district schools usually have spending obligations that charter schools do not. As I said in my earlier post, high-performing charters have been shown to have a spending advantage in previous studies. Might that be the case in Boston?

The authors point out that the charters they are studying have a longer school day and a longer school year than the counterfactual public schools (p. 4). If that’s the cause of the improvement in test scores, fine… why, then, do we need charter schools? Why not just spend more on personnel to extend the school day and year? I’ve seen no evidence that unions would have an objection to this, so long as their members are paid for their extra work.

The authors also admit that peer effects could, in fact, account for some of the differences between charter and district school outcomes (p. 20). How large is the effect of charter enrollment on peer achievement? 0.13 SD – the same effect size Mehlhorn crows about from his cherry-picked subgroup in the CREDO report.****

I actually have an issue with how the authors define “peer effect”: I don’t think it should be limited merely to one’s cohort peers’ test scores, as cohorts tend to mix in high school courses and test score histories likely don’t fully describe which peers influence classroom interactions. But even with this issue aside, the authors do acknowledge a substantial body of research that shows peer effects are real (footnote 27, p. 21). Why, then, is Mehlhorn so anxious to downplay those effects, yet oversells charter effects of the same or even lesser magnitude?

Again, Mehlhorn himself has suggested an equivalency between charter school “choice” and elite suburban schools. So what does Angrist & Dynarski’s paper say about this? Do Boston’s charters create an experience for their students that rivals that of suburban schools?

This study shows they most definitely do not. Yes, the charters raise SAT scores, likely the result of all that extra time in school drilling. But there is no evidence these charters are able to propel their students into the top quartile of SAT scores (Table 5). There is a very small effect size when measuring these students’ ability to get a 3 on AP exams, and no effect on getting a 4 or 5 (Table 6), the usual standard for getting transcript credits at elite colleges. Charter attendance does not increase the likelihood of graduating from high school; in fact, it decreases the chance of on-time graduation (p. 15). And while charters seem to have a small, positive effect on college enrollment and persistence, the study shows that charter students have no advantage in enrolling in the most elite colleges (Table 8).

Let me restate this so as to be absolutely clear: yes, Boston’s oversubscribed charters do show a substantial gain on test-based outcomes, and that shouldn’t be discounted. But it is a massive, unwarranted leap to go from that conclusion to suggesting that urban charter schools are closing the income gap in education; they most certainly are not.

And we still come back to the most important question: are these gains, which are largely marginal, worth the radical policy of shifting America’s schools away from democratic control and toward a market-like system?

Other Stuff Mehlhorn Gets Wrong

I’ve already gone on way too long; however, there’s still much left in Mehlhorn’s post to debunk. Quickly:

- Yes, charters were the first choice in Newark for many families. But here’s what’s telling: as I showed in this policy brief, the “popular” schools in Newark, both charter and public, are some of the most segregated schools in the city. I can’t think of better evidence to show that what parents are really choosing in a “choice” system are their children's peers.

- If there is any evidence that Los Angeles has a better system for authorizing charters than other cities, I haven’t seen it. The CREDO report for LA puts charter gains at 0.07 SDs for reading and 0.11 SDs for math. Again: one-third of an inch.

- The notion that the nation should pattern its education policies on DC and Tennessee is utterly laughable.

- As to New Orleans: again, Doug Harris himself urged great caution in generalizing his preliminary results. Why Mehlhorn doesn’t listen to him, I have no idea. It's also worth noting that New Orleans students who were displaced by Katrina into public schools outside of the city showed substantial gains after their first year.

- When the United States installs social programs equivalent to Sweden's, I’ll entertain a conversation about whether anything they’ve learned about vouchers can be applied to our country.

Charteriness

So what’s the bottom line here? Dmitiri and I agree that charters, on average, show a gain in test scores; however (please don't put an ellipsis in place of this phrase, Dmitiri), while he thinks that gain matters, I’ve shown he’s just plain wrong. The gain is, in the aggregate, very small. In no way can it be compared to the advantage children in affluent school districts enjoy over their less advantaged peers. In no way can charters be said to be providing anything close to an equivalent educational experience as can be found in the leafy ‘burbs. In no way can anyone credibly make the case that charters are substantially closing the so-called “achievement gap.” Anyone who even infers that urban charter “choice” is equivalent to suburban choice is selling a lie.

Furthermore, the gains that charters show are all based on noisy, crappy, unvalidated standardized tests whose outcomes can largely be explained by economic status. Even the most ardent adherents of the value of these tests have begun to back away from embracing them as the public comes to realize they are poor measures of the complete education of a child.

So what price is being paid for these marginal gains on these poor measures? What costs does charterization exact from all of the stakeholders in American education so that students can make small, inconsequential gains on a series of bubble tests? Mehlhorn seems to think that so long as students who don’t enroll in charters maintain their scores on these same questionable measures, no one should worry about any other pernicious effects from expanding charters.

But life is more than a test, Dmitiri, and the downsides of charterization are too large to ignore:

- Because charters are not subject to the same standards of oversight and accountability as public district schools, there is a large and growing trail of fraud, waste, and abuse within the charter sector.

- Because charters set up redundant systems of school governance, they negatively impact public district budgets (see all my links in Part II). The impact may or may not show up in test scores; however, as I have shown in my previous work, funding inadequacy can negatively impact programs in the arts, physical education, and student support services. I have also presented evidence that charters themselves do not provide many of the support services at-risk and special education students may need to be successful in school. These negative effects likely don't show up in test scores; that doesn't mean they aren't a serious problem.

- Because charters have been found not to be state actors, they limit the rights of parents, students, and teachers (again, see Part II).

- Because many “high-performing” charters hire teaching staff from outside the communities where they are situated, teachers of color may suffer an inordinate consequence of charterization.

- Because many high-performing charters engage in “no excuses” pedagogy, they may be teaching their students to be compliant in a way that would not be tolerated in affluent suburban districts.

Can anyone seriously make the case that answering a few more questions correctly on a standardized test is worth all this?

I’ll say it yet again: I am not against charter schools per se. There may well be a place for “choice” within our school systems. I am certainly sympathetic to the notion that urban parents want a better education for their children. I do not think parents should be forced to send their children to schools that are unsafe or have disruptive environments.

But the evidence is increasingly clear: those charters that show gains appear to rely on a combination of increased funding, longer schools days and years, a test-focused curriculum, peer effects, and, in some cases, student selection, attrition effects, and decreased class sizes. There is no “secret sauce”: no special magic that can only be concocted by charter schools.

Many times, charter advocates make the case that they are free from “bureaucracy,” which allows them to “innovate” and “do more with less.” I see no evidence that this is true. We could put more money into all schools to lengthen the school day and year and reduce class sizes. We could desegregate our schools and our communities to spread peer effects more evenly. We could further narrow the curriculum in all schools and focus only on tested subjects (this would be a very bad idea). “Charteriness” is not a necessary precondition for any of these changes.

So why does the “reform” movement focus so much of its attention of charter school expansion? I contend it’s not so much that the “reformers” want to make money off of charters or bust unions (although, for some of them, these are nice sidebenefits). Rather, the “reformers,” no matter how well-intentioned some may be, are looking for a quick fix to problems that must be addressed within a context that is much larger than education policy.

Yes, we need good schools to have a just society. But social mobility is not social equity. Moving a few more students of color up the socio-economic ladder (and, again: we don't have much evidence charters are really doing this) does not address a core problem America faces: millions of people are doing necessary work yet falling further and further behind. They are not seeing any of the productivity gains this country has made over the last several decades; instead, they continue to do work that everyone admits must be done while making less and less.

Shuffling students around within their district borders into schools that get only slightly better test scores does nothing to address this core problem in American society. Creating a false narrative that school outcomes are the cause of inequity, and not the symptom, doesn’t help the problem; it perpetuates it.

If the best we can do for America’s most disadvantaged and deserving children is to build more schools under private control that largely teach a hidden pedagogy of compliance and get small gains on bubble tests, we have failed. We have failed because we are having the wrong conversation, and we are deluding ourselves into thinking we have equalized the lives of advantaged and disadvantaged children merely through changing how schools are governed.

It’s long past time to start being honest with ourselves about this. Mehlhorn ended with a quote by Upton Sinclair; let me end with it as well:

"It is difficult to get a man to understand something, when his salary depends upon his not understanding it."

* A technical note: I’m not even sure the category “Hispanic Students w/ELL Status” is consistently reported throughout the sample CREDO used. In New Jersey, school-level test scores are not reported for this group. Does CREDO’s sample disaggregate the student level data consistently across all states? The technical appendix doesn’t seem to say.

** I’m actually 6’ 1”. Because you really needed to know…

*** There’s also the question of whether Hill & Booth’s work, based on “vertically scaled” tests, can be applied to state tests that are not scaled this way. I’ll save that discussion, which I think is important, for another time.

**** To clarify: "Not surprisingly, given the positive selection of charter applicants, charter enrollment is associated with increases in peer achievement in the first post-lottery year: the (marginally significant) effect here is roughly 0.13 in each subject." (p. 20)

Charter Schools, An Exchange: Part III

UPDATE: I see Dmitri is retweeting all the odd-numbered parts of this series (his), but not the even-numbered parts (mine). OK... but do yourself a favor: read my responses. See for yourself if Dmitri's arguments hold up.

I feel quite confident that you'll find he doesn't have much cause to believe he got the better of me in this exchange. But you be the judge.



Dmitri Mehlhorn and I continue our debate about charter schools below. In case you missed it, here's Part I by Dmitri, and here's my reply in Part II. I'll have my reply in Part IV up within a few days; stand by...

I also asked Dmitri to write a short bio of himself just for this series:

"Dmitri Mehlhorn is a Senior Fellow with the Progressive Policy Institute. He writes frequently about education, including his own time as a public school student and parent. He is a co-founder and Board Member of StudentsFirst, which lobbies opposite teachers' unions on policies regarding choice and data. He is also a seed investor; however, his firm Vidinovo refrains from K-12 investments to avoid conflict of interest."

One last thing: Blogger is not very good at translating formatting from MS Word to HTML. I tried to clean it up, but if it looks a little odd, that's not on Dmitiri; it's on me. I'll keep trying to make it as readable as possible.

ADDING: Here is an index for the entire exchange:

Part I: Mehlhorn's opener.

Part II: My reply.

Part III: Mehlhorn's response.

Part IV: My second reply.

Part V: Mehlhorn's second response.

Part VI: My final reply.

* * *

Episode III: The Return of the Charters

Do charter schools help kids learn?

I believe yes, and Jersey Jazzman (Mark Weber) believes no. We mostly agree on the basic facts, and disagree about the conclusions. As Jazzman wrote to me: “Almost everything you say is factually correct; however … you arrive at conclusions that simply are not warranted.” 

So what now?

At this point, we need to review the burden of proof in education policy, and then we need to review the evidence.

In education policy debates, who should decide?

If Jazzman and I, as proxies for the broader policy debate, agree that charters deliver better results, but cannot agree on whether they are “better enough,” how should our impasse be resolved?

Jazzman is refreshingly transparent: “the burden of proof remains on those who make an affirmative case for charter proliferation, and it always will.” This answer, while representative of reform skeptics, seems unmoored from the principles of American public education.

Parents seek freedom to make choices for their kids. A recent poll of parents found that two thirds of parents say “public charter schools offer parents in low-income communities options for quality schools that would otherwise be inaccessible to them” — and the percentages are even higher for Black and Latino parents. Parents with resources buy houses near good schools, and deploy financial and social capital to navigate the school system. Parents facing residential segregation due to race and poverty do not have those luxuries. As a result, majorities of African American parents would not send their child to their closest public school if they had a choice. Nationally, more than one million applications are on charter wait lists (I don’t know how many parents that means, but even if every parent submitted 20 applications each, that’s still a lot of parents who have been denied their preference for their kids). In Jazzman’s home state, for instance, 75% of Newark’s kindergarten parents listed charter schools as their first choice.

This is personal. As I’ve written, my mom used her salary as a public teacher to buy me three years of private school, and then saved money to move to a better neighborhood. Those resources transformed my life trajectory, but not every family has them. Sally (a pseudonym) is an awesome kid from my daughter’s soccer league, whose parents have limited income and even less English. When the local school puts the wrong teacher in front of Sally, her parents will have no recourse. They might use buses to sneak Sally into a better zip code, but if they get caught they can go to jail. Without charters, they are stuck.

Let’s be blunt: policies that block charter growth in urban areas do not limit choices for affluent parents, but they overrule the wishes of many parents who face residential segregation. In constitutional law, a policy decision that has a disproportionate impact on blacks and Hispanics is subject to “strict scrutiny.” The evidence has to demonstrate a compelling public rationale for the policy, and the policy has to be crafted in the least-onerous possible fashion in terms of burdens on black and Hispanic families.

Given these dynamics, Jazzman needs to demonstrate a compelling goal that can only be achieved by blocking the growth of urban charters. This is the appropriate burden of proof as long as affluent parents exercise their own choices, and as long as there are wait lists full of disadvantaged families.

So what does the evidence actually say?

Fortunately for the purposes of clarity, Jazzman has already conceded enough to justify the elimination of policies that cap or prohibit charter growth. Let’s review his work.

(1) Jazzman says it’s just the good charter schools

Jazzman concedes the research from Mathematica and others that students who attend high-performing charter schools – such as KIPP and Achievement First – have better outcomes than students who apply to those schools but don’t win admissions in their lotteries. Jazzman dismisses these results by saying that those are just the good charter schools.

 
If he’s right, charter restrictions should stay in place for all charters that have not been validated by robust research. Validated charters should expand. Here in Virginia, my daughter’s friend Sally will have the option to attend KIPP schools. That is a huge relief for me, and I look forward to Jazzman’s support.

But is he right? Is it just a few good charters, or does a diverse charter sector also deliver? One of the studies he praised, Angrist and Dynarski (NBER 2013), concluded that at least in Boston, Jazzman is wrong. The NBER team did a deep dive into Boston charter high schools. Since studies show that high school interventions are usually “too late” to make a lifetime difference, this is a harsh condition for charter advocates. Against this tough hurdle, Angrist et al. conclude that charters deliver meaningful, statistically significant gains in life-altering outcomes like college attendance, in ways that are not attributable to peer effects or spending differences. In Boston, it’s not just a few brands, it’s the whole charter sector – and not by a little, but by a lot.

(2) Jazzman says the national averages are not “better enough”

The next fallback for Jazzman is to say that good charter cities like Boston are outliers, and that we should not extrapolate anything from the ever-more-frequent studies validating specific charter chains and cities. If we do that, Jazzman says the national results are “basically the same” as traditional schools (consistently better in a statistically significant fashion, but not “better enough”).

If he’s right, hurray! Great news for Sally. We agree that we should expand charters, but only using the same quality review system that authorizers use in cities like Los Angeles and Boston. Again, I look forward to Jazzman’s support here in Virginia.

But is he right? Is it just some cities, or is it a national story? Let’s go back to Stanford University CREDO’s 2015 study of 41 urban regions. Jazzman is right that results vary by city. For instance, charters in Newark, Detroit, and DC do better than traditional schools; in Las Vegas, Fort Worth, and West Palm Beach, charters do worse. But what is the aggregate result for students who otherwise don’t have choices? What about for students like Sally, who is in the cohort that CREDO refers to as Hispanic ELL?

Nationally, across both well-run and poorly run city charters, charters improve literacy and numeracy for Sally’s cohort by 0.1 standard deviations over their matched peers in traditional schools.

How big is one tenth of a standard deviation? Well, it depends upon judgment and context. I asked a physician about healthcare, and it turns out that the standard deviation of deaths due to cancer is 15 years. One tenth of a standard deviation improvement would extend cancer victims’ lives by 1.5 years. I asked an investor about the stock market, and it turns out that an investment of $100 in 2004 would be worth $232 at the end of 2014, but $275 if it beat the market by 0.1 standard deviations. That’s about 19% more money. Those differences are real.

In education, the most commonly used benchmark converts standard deviations into days of learning. By this method, CREDO shows charters providing Hispanic urban ELL students with effectively a 40% longer school year (72 days). Over the course of a K-12 career, that’s the equivalent of adding more than 5 extra years of schooling. That’s enough extra schooling for a four-year college plus a one-year masters program. Jazzman protests that this translation is imprecise, and he is correct. We don’t know for sure exactly 72 extra days of learning. It might only be 50 (just free junior college for Sally), or it might be 100 (throw in a free Ph.D). Either way, these are not small potatoes.


(3) Jazzman says it’s just skimming and suspensions

Finally, Jazzman argues that the superior results are the result of skimming and selective attrition. Correctly, he notes that CREDO’s data intermixes severe vs. non-severe SPED, as well as extreme vs. mild poverty. Students with severe SPED and in extreme poverty do not enter charter lotteries as frequently, thus charters may have an easier-to-teach cohort than their comparison traditional public schools. Additionally, although charters appear to have less overall attrition when dropouts and suspensions are included, charters do not always backfill seats opened up by departing students. Over time, this can make charters’ cohorts easier to teach. Finally, Jazzman notes that charter slots go to parents who applied, creating a more active parent community and thus again making the cohorts easier to teach. Although deep-dive studies of cities like Boston, and charters like Achievement First, rule out the idea that these so-called “peer effects” account for charters’ superior performance in those narrow areas, Jazzman argues that they explain the national gains found by CREDO.

If he’s right, we still get a policy that helps Sally: namely, we should expand charters only in those circumstances where charters demonstrate results above and beyond peer effects. To address Jazzman’s valid concerns, such charters should be required to backfill open seats; also, students facing severe SPED and extreme poverty should be recruited to enter charter lotteries. But this evidence does not charter caps and prohibitions. Those policies sacrifice Sally’s future to the speculative bureaucratic idea that her presence in a traditional school might somehow benefit some more-disadvantaged child, even as we admit that wealthier and whiter parents long ago refused to make such sacrifices of their own children.

But is he right? Jazzman’s claim that that charters succeed by skimming requires that, at some point, charters in a district must reach a “tipping point,” where they have skimmed so many good students that performance collapses in the remaining traditional public schools. Jazzman himself throws out a tipping point of 30%, which is a reasonable number given his theory.

So what happens when charters reach that threshold within a district? Jazzman hyperlinks to several studies, all of which relate to school finance rather than student outcomes. School finance is a different issue, relating to fixed costs, pensions, other overhead, declining enrollment (which can happen from charters but also for other reasons), and mismanagement. Even if Jazzman cited only studies of the highest quality, they don’t address the “peer effect” claim that is central to Jazzman’s critique of the CREDO numbers. To reiterate, the peer effect argument does NOT say that traditional schools will struggle to downsize their fixed costs as students leave to attend charters. (If that were the issue, we would expand charters and provide traditional schools with downsizing support as they shrink.) Rather, Jazzman’s claim is that as charters take the more motivated, easier kids, the remaining cohort will get harder and harder to teach. Rather than exploring fiscal issues, the test of Jazzman’s thesis comes from exploring student results. 

The results on this question are brutal for Jazzman’s claim. In Florida, traditional public school performance has improved as school choice has expanded. Ditto in Denver. Ditto in New York City, especially in the boroughs with the highest charter penetration. Ditto in the 2008 study I cited from Texas, which Jazzman agreed was well done. True, none of these examples meets Jazzman’s 30% threshold, but it should worry him that each result points in the opposite direction of what his theory predicts.

As for the tipping point, how does Jazzman explain away the studies of Washington, D.C., as summarized by Clinton Administration policy wonk David Osborne here? In D.C., two thirds of students attend a different school than their closest traditional school, and 44% of students attend charters. Both traditional and charter schools are improving dramatically, faster than any other city in the nation, and faster than any state other than Tennessee. And can we revisit the Doug Harris research regarding New Orleans? Even if we don’t use New Orleans as a template for the nation, the enormous gains from district-wide conversion to charters flatly disprove Jazzman’s theory that charter gains only happen by skimming.

* * *

There is a simple and powerful theory that explains all of this evidence. The theory is that government agencies work best when they put citizen choice ahead of bureaucratic monopoly. This is the lesson from the Nordic countries that were celebrated in the recent Democratic presidential debate, which have introduced competition into government agencies. For example, Sweden offers vouchers to enable parent choice, and local scholars there have shown that as independent schools gained scale, the entire system (independent and traditional) saw performance improve. The benefits of flexibility consistently outweigh the costs of duplicative overhead.


Ultimately, however, this theory is not as important as the parents of color, segregated residentially in urban areas, who seek choice. Their voices should arbitrate whether the evidence for charters is “good enough.” If we have to wait for teachers’ union officials, and the politicians beholden to them, to decide that charter results are “better enough” to justify further expansion? Well, as Upton Sinclair explained, those families will be waiting a long time.

Eva Moskowitz Crosses The Line

Looks like the pressure is finally getting to Eva Moskowitz.

She's had a good long run as the Queen of Reformyland. As Peter Greene notes in his quick biography, Moskowitz pretty much gets whatever she wants, whenever she wants it. Not only that: she has, up until now, been given a free pass for taking some pretty brazen actions. How do you think the press would react if a public school superintendent moved her district's calendar around so her students could go to the state capital and protest?

Unfortunately for Eva, all the attention she's brought to herself has inevitably led people to start taking a hard, critical look at her self-proclaimed successes. Leo Casey has a comprehensive piece out at Shanker Blog that casts serious doubt on Success Academy's disciplinary policies:

To provide the most complete picture possible of what is happening in both the Success Academy schools and regular New York City public schools, it was necessary to gather data from a number of different sources. Let us start with most recent dataset, for the year 2013-14, which was published late last spring as part of the New York State School Report Cards. According to the state data, in 2013-14, Success Academy Charter Schools had a total of 728 suspensions for a suspension rate of 11 percent, while the New York City public schools had a total of 9617 suspensions for a suspension rate of one percent. 

We know that the NYC public school data is understated, however, because (just as in the case of its report to the U.S. Education Department cited above) only the most serious suspensions are ever reported to the New York State Education Department. Upon request, the New York City Department of Education supplied the Shanker Institute with the total number of all suspensions for the 2013-14 school year. These data showed 53,504 suspensions; yielding an annual suspension rate of five percent.

From the standpoint of Success Academy, therefore, the most charitable reading of these numbers is that the charter school network suspended its students at more than double the rate of the New York City public schools, eleven percent to five percent. [emphasis mine]

And that's very charitable.

The worst blow to Moskowotz's empire, however, came earlier this month when John Merrow broadcast a pointed piece about SA's suspension policies on the PBS Newshour.



Moskowitz squirms like an engineer at Volkswagen throughout the piece; I almost felt sorry for her...

Until she pulled one of the worst stunts I've ever seen in my career as an educator: she released personally identifiable information about the behaviors of a child who was enrolled in her school.

Merrow's piece featured an interview with a young boy who apparently was enrolled at SA when he was six years old (he is now 10). In the story, both he and his mother claim he was suspended "...three or four times his first year for losing his temper." Merrow's interviews are on camera and the identities of both the boy and his mother are not concealed.

Moskowitz decided that the accusation of a 10-year-old must not be allowed to stand. As part of an eight page letter released on SA's website, Moskowitz describes, in great detail, the exact behaviors she alleges the boy engaged in during his enrollment. She includes the written accounts of two of her teachers who, again, go into great detail about the boy's behavior.

I'm not going to link to the letter SA appears to have released on October 20 (according to the URL), because I believe it is in incredibly poor taste and a serious ethical breach for any professional educator, let alone a school leader. Merrow's piece aired on October 12, well before Moskowitz's letter. Her letter makes clear she had already seen the broadcast when she lodged her complaint with the PBS Newshour. Which means Eva Moskowitz published information about the behaviors of a child in her school knowing full well his name had already been released to the public.

I'm no lawyer, but I think federal law is very clear on this. FERPA, the Family Educational Rights and Privacy Act, explicitly states that schools must have a parent's written permission to release information from a student's educational record. The USDOE offers guidance to parents regarding their rights for the release of student records:

What is an Education Record? 

Education records are records that are directly related to a student and that are maintained by an educational agency or institution or a party acting for or on behalf of the agency or institution. These records include but are not limited to grades, transcripts, class lists, student course schedules, health records (at the K-12 level), student financial information (at the postsecondary level), and student discipline files. The information may be recorded in any way, including, but not limited to, handwriting, print, computer media, videotape, audiotape, film, microfilm, microfiche, and e-mail. [emphasis mine]

Source: 34 CFR § 99.2 ?Education Records? and ?Record? 

It would be very hard to argue the accounts of the teachers were not part of this student's "discipline file." It would also be hard to argue this parent gave up her right to keep this information private when she exercised her First Amendment rights:

What must a consent to disclose education records contain? 

FERPA requires that a consent for disclosure of education records be signed and dated, specify the records that may be disclosed, state the purpose of the disclosure, and identify the party or class of parties to whom the disclosure may be made. 34 CFR § 99.30. As such, oral consent for disclosure of information from education records would not meet FERPA's consent requirements. 

In addition, the State of New York has reaffirmed the privacy rights of parents when it comes to school records. Again, I'm not a lawyer so I can't offer an opinion as to whether FERPA was violated here. But even if it was, there's probably not any recourse for the parent under federal law: the worst that could happen is that SA could be denied federal funds.

Something tells me that a school that can raise over $9 million in one night isn't going to worry too much about that...

But whether the law was broken isn't even the most important issue here. What Moskowitz did was an inexcusable lapse of judgment. Eva Moskowitz has put her need to protect her brand over the privacy of a child who, by her own account, has challenges in a school setting.

This is yet another problem with the "market reform" theory of education. How much money does any corporation spend to maintain its public image? How hard will they fight if they perceive that image is being threatened? How little reluctance do they show to go after a critic of their company or their products?

Schools, however, are not corporations (at least, not yet). Parent complaints are not threats to a brand; they are advocacy for a child. I'm not at all suggesting that school leaders don't have the right to defend themselves, either in court or in public. But it would have been more than enough for Moskowitz to say: "We dispute these allegations; however, we will not discuss any individual case publicly, as all parents and children have a right to privacy in school."

Not only would this have been less questionable legally and ethically: I'd wager it would have been better for Moskowitz in the eyes of the public. Her attacks on this boy -- and that's exactly what they are -- come off to me as petty, unthinking, and, worst of all, cold. And I can't believe I'm the only one who feels this way.

It's very strange that a woman who has worked so hard to cultivate her public image is willing to risk having it trashed just so she can win a PR fight with a 10-year-old boy. She must think the stakes are very high.

And that's the problem.

Well, someone still likes me...

Christie's Pension Plan Won't Work?! I'm So Shocked...

Well, duh:

A new report from a Wall Street rating agency warns that Gov. Chris Christie's seemingly dormant plan to overhaul government worker pension and health benefits to save the state billions of dollars could come at a risk to school districts if Christie's proposed reforms don't play out as envisioned. 

Moody's also raised questions about the plan as proposed, saying it doesn't explain how reduced health costs for municipalities will be used to balance out new expenses for school districts picking up the tab for teachers' retirements. 

That proposal, devised by a special pension commission and unveiled in February, would freeze the existing pension plan and shift workers onto less generous retirement and health care plans. While the state, which pays for school employees pension and health benefits, would continue to pay for the current system's existing debts, school districts would have to assume the costs of the new system and retirees' health care. [emphasis mine]

What anyone with half a brain knew when they read the plan was that if there were all these savings to be had with no costs to localities, chances are someone would have found them by now. This plan was never anything more than a scheme to shift costs away from the state, to the towns and cities, and eventually to the employees.

And it is predicated on the fantasy that public workers are not part of the labor market:

"A condition of the commission's proposal is that costs will only be shifted to the extent they can be offset by local benefits savings," Tom Healey, who heads Christie's commission, has said. But the commission has yet to provide details on exactly how the health plan changes would save money.  

If the savings doesn't pan out, the proposal could burden school districts that have few options but to raise taxes, cut costs, borrow money or spend their reserves to pay the tab for teacher pensions, Moody's said. 

"According to the proposal, districts would not be financially affected because any tax increases necessary on their side would be more than offset by tax reductions at the local government level, thus making it at least cost neutral," the report said. "There is, however, no mechanism currently in place or proposed to compel municipalities and counties to either share the savings or reduce their budgets in step with the savings." 

Moody's said it's also unclear "how the various local government entities would rebalance these shared savings over time as health care and pension costs rise." 

What did the commission think: that public workers would just happily accept worse benefits with no increase in pay to offset them? That they would come to the conclusion that somehow they were overpaid -- contrary to all evidence -- and allow their composition to be slashed? That there would always be a supply of teachers and cops and other public workers who would gladly make way less in total compensation than they could in the private sector?

And that they would do all of things against their self-interest simply because the burden shifted from the state to the towns?

What the commission essentially said was that public workers should pay more for their health insurance, so the government could take the extra money and put it into shoring up the pensions. That is equivalent, by any reasonable standard, to a pay cut. You knows what happens when you cut pay?

People who are qualified are less likely to do the job. They look out at the labor market and they say things like: "You know, I think I would really like to be a teacher or a fire fighter or a civil engineer or a social worker. But I'd also like a nice middle-class existence, and I'm going to have all these student loans, so..."

Tom Kean, Sr. knew this back all the way back in 1982. Of course, this was back in the days when there were Republican politicians who understood that the laws of supply-and-demand applied to all workers, not just those in the private sector. Some of them actually thought if we wanted well-qualified people in government, maybe we should pay them. But in today's cable pundit-informed America, this common sense idea is now considered radicalism -- even by many Democrats.

Look, I'll make the same deal I always make: give me back all the money I paid into the pensions, plus interest. Then give me all the money the state was supposed to put in, plus interest. I'll take my pile and go elsewhere, and I'll negotiate a new contract with my employer. In return, the state doesn't get to take any more of my money involuntarily and then attempt to change the rules later.

Yes, it'll mean higher property taxes. It will mean you'll have to pay me and my fellow public workers more cash up front, rather than use the power of the capital markets to save taxpayers money. And it'll mean the current pension will have to be funded out of the general fund of the state.

Sound like a good plan to you? Didn't think so.

We could, however, start taxing the wealthy at rates they were already paying before Chris Christie came into office. We can start negotiating better deals with the sharks who are devouring our money while managing the current pensions. We can start negotiating better deals with the health care providers and pharmaceutical companies. We could even consider self-insuring (if Daddy George lets his people in the Legislature even bring up the idea...).

Will that be enough? Probably not, but it would be a better start than continuing to push for the ludicrous plan that everyone, including Wall Street, knows will not work.

The commission failed. Time to move on. Everyone, except the most delusional people in Trenton, agrees.

Who are you calling delusional?!

Charter Schools, An Exchange: Part II

In Part I of this series, I invited Dmitri Mehlhorn, a well-known advocate for charter schools and other education "reforms," to comment on charter schools and make the case for his point of view. What follows is my reply; I'd urge you to read Dmitri's essay first to get the full context of my reply.

ADDING: Here is an index for the entire exchange:

Part I: Mehlhorn's opener.

Part II: My reply.

Part III: Mehlhorn's response.

Part IV: My second reply.

Part V: Mehlhorn's second response.

Part VI: My final reply.

* * *

First of all, Dmitri, thank you for agreeing to this exchange. I'm going to try to keep the tone civil here -- but I'm afraid I'm also going to be rather blunt.

Your essay is, in many ways, a perfect example of "reform" thinking. Almost everything you say is factually correct; however, you are so selective in your citations, and so lacking in context when discussing them, that you arrive at conclusions that simply are not warranted.

These are the points I'm going to address from your essay:

• The CREDO studies provide little evidence of a practical positive effect of charter schools on student learning in the aggregate. They also provide little evidence that the charter sector as a whole has been improving over the last six years.
• There is little evidence to support any contention that the limited econometric research into a few charter school chains can be generalized to the entire sector.
• The lottery study you cite (I assume Angrist, Cohodes, Dynarski, Pathak, & Walters, 2013), like all charter lottery studies, is limited to the population of students who enter the lottery. Further, "being a charter school" is NOT the only treatment.
• The research I assume you cite from Mathematica on Achievement First and Uncommon shows they serve different student populations than their feeder schools, with fewer students with special needs (SWSN) and fewer Limited English Proficient (LEP) students.
• The Texas charter study you cite only studied districts with comparatively low charter penetration. Other studies have shown a pernicious effect of charter proliferation on host districts.
• Doug Harris himself has urged caution in generalizing the changes in New Orleans to the country at large. He has also strongly cautioned about a process of charterization that excludes community input.

The CREDO Studies

Dmitri, you claim that the 2015 CREDO study "puts the burden of proof squarely on those who would attack the efficacy of charter schools." First of all, the burden of proof remains on those who make an affirmative case for charter proliferation, and it always will. It is impossible to reply to: "Prove that more charters WON'T make things better."

You catalog some of the objections to the CREDO methodology, but you dismiss them far too easily. As Bruce Baker has shown, how you classify student economic disadvantage matters when it comes to test score outputs. As I have shown, the differences between special education populations can be quite significant. These things do, in fact, matter a great deal.

Next, it's impossible to make claims about the improvement of the overall charter sector when the charters CREDO studied changed. As you note, the 2009 and 2015 CREDO studies looked at different samples of states and schools. One thing I noticed locally about the 2015 study was that it included Newark, NJ, but not Camden, Trenton, Jersey City, or other towns in my state with large charter populations (the NJ CREDO study showed charters in these towns had no effect, or a negative effect). And Newark was not included in the 2009 study, nor was it part of the 16-state growth comparisons in the 2013 study.

Put simply: you can't claim that the sector as a whole is improving when you aren't evaluating the same sample in different studies.

Next, as Maul & McClelland explain, the "significance" of the CREDO results is statistical, and not necessarily practical. When you have large sample sizes, as in the CREDO studies, any difference is going to be statistically "significant." But is it a difference that actually matters?

One of my great frustrations is the continuing use of "days of learning" as an interpretation of effect sizes. Some people I respect think this is fine; I don't. Not only does "days of learning" imply a precision that can't be justified; it also is being applied to tests that are not "vertically scaled." A layperson is likely going to think that a student who is "180 days ahead" of another is learning the next grade's material; the tests cannot make that determination, however, because they don't test next year's content.

Let's, instead, look at the CREDO effect sizes without translation.

Here are the reported aggregate effect sizes for the three CRDEO studies you cite, including the 2015 study which really shouldn't be compared to the other two. A legitimate critique of making a graph like this would be to point out that the y-axis could be changed to any scale I want to make the bars look bigger or smaller. What I've done here, then, is set up a relevant comparison: the bracket is the difference in standard deviations between a typical school with average family income in the 90th percentile, and a school with that income in the 10th percentile.

The results are clear: even if we accept the 2015 CREDO results as typical for the sector, the practical effect of charter schools does not come close to closing the income gap advantage.

In summary: according to the CREDO studies, charter school effects are practically quite modest, and any notion that the sector is improving rapidly needs to be tempered by the realization that family income has a far more profound affect on test scores than systems of school governance.

The Econometric Research

I wouldn't attempt here to encapsulate all the research being done by admittedly brilliant economists like Josh Angrist on charter schools. I will freely admit they have found, in a significant number of cases, a real charter effect. But their research has been quite limited in its scope. KIPP gets studied quite a bit (which, apparently, they pay for). A few other chains have been looked at... but what about these guys?

NOTE: I have replaced the graph originally in this post with a more accurate one. See here for details.

This is from Bruce Baker, based on a dataset we've been assembling. KIPP is actually one of the smaller players in the charter sector. Where's the peer-reviewed research on K12, a rather dicey operation? How about some NBER papers on Charter Schools USA, which is making money hand-over-fist? How about a study of UNO, which has had its share of troubles? Maybe Academica, a big player in the free-for-all that is the Florida charter school sector? Or White Hat, at the center of Ohio's increasingly troubled charter sector?

If we're really going to be serious about evaluating the effects of charter schools, why aren't we looking at these guys? Nobody in their right mind would ever make the case that just because KIPP has shown some positive effects, these guys must show them as well.

But let's take a closer look at the those KIPP studies anyway...

The Charter Lottery Studies

I'll go back to Bruce Baker again, who's written the definitive piece on the limitations of the so-called charter lottery studies. These studies are limited to 1) the schools that are over-subscribed and must have a lottery, and 2) the students who actually bother to enter the lottery. Again: no one should try to generalize the findings of these studies to the entire charter sector.

The other issue with most of these pieces of research is that they are surprisingly vague about what exactly causes a charter management organization like KIPP to see gains compared to feeder public schools. Again, Angrist is a brilliant researcher -- but I'm surprised when I read work like this at how quickly he glosses over the characteristics of the schools themselves (and, for that matter, how little documentation there is of the qualitative methods used to arrive at the descriptions of the schools' practices).

KIPP

So what do we know about KIPP? One thing is for sure: they get a lot of extra money in philanthropic contributions. They also have a relatively inexperienced staff, which allows them to offer more competitive salaries in many cases than their feeder districts, as they don't have to pay more experienced teachers' higher salaries. For this extra money, their staff can increase the school day and the school year. Matt DiCarlo puts it well:

If you do the simple math, NYC's oversubscribed charters offer, on average, 31 percent more time in school, which is the equivalent of roughly 56 days (using the 6.5 hour day of regular public schools). If we compare this with the overall impacts that I converted to “days of learning," we see that, on average, the estimated additional days of learning charters provided in reading (40) was lower than the actual days they offered (56). In math, it was higher (70 versus 56), but not by much. In other words, interpreting the finding that students pick up the equivalent of 40 or 70 extra "days of learning" is a bit different when you consider that the students were actually in these schools for the equivalent of 50-60 extra days (on average). [emphasis mine]

In addition: KIPP only takes those students who wish to go to KIPP, a key feature in any "choice" system. Even a Mathematica study admits: "Unlike local district schools, KIPP’s late entrants also tend to have higher prior achievement levels and fewer males than the rest of the KIPP student body," even as the authors argue that peer effects do not explain all of KIPP's advantage.

I would agree: no one thing explains why KIPP gets the results it gets. But more resources, a longer day and year, a self-selected student body, a focus on test prep... it would be astonishing if KIPP didn't get better results. But does this, by itself, justify a call for rampant charter expansion, particularly if the vast majority of the students won't attend KIPP schools with all of their resource advantages?

One more thing:

This is from the latest Mathematica report on KIPP (p. xviii):

Several factors may explain the trends in KIPP middle school impacts, including changes in the number and composition of schools in the sample, the relative performance of newer versus older schools, and changes over time in the effectiveness of existing KIPP schools as the network has expanded. Overall, KIPP’s student achievement impacts decreased during a time of high growth in the network, although they rebounded somewhat during the i3 scale-up period (Figure ES.5).[emphasis mine]

If we're going to rely on the Mathematica studies, we'd best consider all of what they say. KIPP's expansion has accompanied a decrease in its effect -- an effect that is still only a fraction of the effect of family income of test outcomes.

One thing we ought to figure out soon is whether the KIPP model can be scaled up any more than it currently is. KIPP has admitted repeatedly that it won't educate all of the children in a school district. Maybe the charter chain believes it can't get enough teachers to churn so it can keep salary costs relatively low. Maybe it doesn't believe it can scale up its philanthropic giving as it scales up its size.

I've said it before and I'll say it again: KIPP gets decent results. They run some good schools; good for them, they should be proud. But these lottery studies aren't nearly enough evidence to justify the dismantling of public school districts, as some would have us do. We should instead be asking why all schools can't pay their teachers more to work longer days and school years if that is part of what is helping to increase test-based outcomes.

Attrition

There's only one study I know of by Mathematica that looked at the attrition rates of Uncommon and Achievement First. The study is very limited in its methods: only five schools were sampled (three AF, 2 Uncommon), all were in New York City, and the study only looked at attrition between Grades 6 and 7 (p. 4-5). In addition:

The baseline proportion of students in special education or with LEP is significantly lower at this sample of AF/Uncommon middle schools compared with both district-wide levels and levels observed in feeder elementary schools. (p. 4)

I am always surprised when charter supporters get indignant about this. Of course the student populations are different. Of course there aren't as many LEP students or SWSN. In a choice system, it only makes sense that parents would "choose" a school that meets their child's needs, and it only makes sense that schools would specialize so that they weren't redundant with each other.

McDonalds doesn't sell tacos (last I checked), and Taco Bell doesn't sell hamburgers. Markets will push providers to differentiate themselves to meet different customer desires. When I looked at New Jersey charter student populations, I saw the same trend that Mathematica found: the charters were serving fewer SWSN and LEP students. The charters, at least so far, are differentiating themselves in the "market" by not serving as many of these students.

Is this a good thing?

As I've noted, the Uncommon school in New Jersey has a substantial rate of cohort attrition compared to its feeder district, Newark. It is not unreasonable to question whether this is affecting test scores as cohorts move through the school.

I also recently noted that Uncommon, the former employer of our future SecEd, John King, has high suspension rates compared to the local public districts in all the regions where it exists.

High suspension rates seem to be a defining feature of NYC charter schools, in contradiction to stated USDOE policy. But that's not surprising: charters have been at the forefront of the "no excuses" school discipline movement. Again, in a "choice" system, some families may look at high suspension rates as an acceptable price to pay to enroll their child in a school they consider academically superior, or even just safer. At least two questions arise, however:

1) Can charters exist as currently constituted only if they can rely on the local district schools to take all students, including the ones for whom charters are not a good "fit"?

2) Why do we accept charters that promote submissive attitudes in lower-income minority students, but not in the suburbs for more affluent children?

Pernicious Effects of Charterization

The Texas study you cite, Dmitri, is an excellent piece of research. But the latest data used in the study is from 2002-03, and at that time no district in Texas had charter penetration of more than 12 percent (p. 128). In fact, the vast majority of districts had charter penetration under 5 percent.

What happens, however, when charter enrollment hits a tipping point? When, as in Newark, charters start capturing upwards of 30 percent of the students enrolled, with even greater percentages on the horizon? What happens to the local school district then?

Bifulco and Reback (2011) find that charter proliferation had a significant negative impact on local school district finances in New York State. Arsen and Ni (2012) found similar negative effects in Michigan. Moody's also found charters negatively affect a district's finances. A report to the Pennsylvania Legislature by Ed Fuller and others at Penn State found charter payments were having a serious impact on local district budgets. PA actually saw one of its districts, Chester-Upland, go into a fiscal tailspin thanks to charter school payments. There are also similar reports from Tennessee.

This makes perfect sense: charter schools are redundant systems of school governance. Having multiple school leaders, multiple administrations, multiple service providers and so on is inherently inefficient. This can't be simply brushed aside: charterization can seriously hurt a local district's ability to educate the students who chose not to attend charters.

I must say I find it fascinating to see folks clamoring for more charters without making sure the children who stay in public district schools aren't hurt by charter expansion. Where is the urgency on behalf of these students? Who is writing the op-eds in Education Post and The 74 to speak for them?

New Orleans

Dmitri, I won't relitigate the debate on New Orleans and charter proliferation here. Instead, I'll let the man you cite, Doug Harris -- a highly skilled researcher and someone I've found to be quite reasonable in interviews -- speak for himself. From The 74, hardly an outlet for critics of charter schools:

Let me answer that by adding one more thing to what I was saying earlier about the conditions of New Orleans. One of the conditions in New Orleans was it was a very low performing district. It was a disaster by almost any measure. When I first got here I thought some of that was just hyperbole because everyone kept saying it, but now I've seen more and more objective evidence of that. The FBI had an office within the school board because there was so much corruption to investigate that they needed to have offices within the school board offices. I just got my hands on a report after many years of trying that was the Council of Great City Schools evaluating the human resources management of the district. The litany of problems in the district was just appalling, the way it operated. The outcomes were also poor as a result. It was the second lowest performing district in the second lowest performing state in the country. 

If you're in a district that already functions pretty well, maybe this is not necessarily a good idea. Here you had a district that was not functioning well by any measure. I think people tend to assume that all urban districts are terrible because the scores are low but it really is true that a lot of that is driven by the demographics and the socioeconomics of the districts. That's real. There are some very good schools in urban areas and there are some districts that probably do a pretty good job. You don't want to throw that out just because of a probable misinterpretation, if they're not taking into account where students are starting. 

I would tell a district or somebody advocating it in a district to look very carefully at how the traditional district is operating before going down that path. 

I would also tell them all the conditions, all the things they have to do to make it work. I think people tend to think, oh we just need more charter schools. But Detroit is a case study in how that goes wrong. Detroit has a lot of charter schools — I think it's second or third in market share — but it's terrible. The system has so little coordination. [emphasis mine]

I'll say again what I've said many, many times: I am not against charter schools. I started my K-12 career working in a charter. There is a place for "choice" in American education. There are some good charter schools out there and they should be commended for their work.

But when I read you, Dmitri, gravely intone that someone like me, questioning the claims of the charter cheerleading industry, is on "untenable moral ground" because of my doubts... well, sorry buddy, but I think you're projecting more than a little. Because the real shaky moral ground is under the feet of those who pretend that suburban "choice" is the same as urban charter school "choice," and use that posturing to allow charters to grow without proper oversight.

Charter "choice" is not suburban "choice." Shuffling children around within the borders of their district into schools that have unequal access to resources and unequal commitments to educating all students is not the "choice" offered in the suburbs. Offering families either underfunded, crumbling, filthy public schools or charters that are not state actors and do not afford students and parents the same due process rights is not the "choice" offered in the suburbs. Requiring students to submit to excessive punishments for trivial infractions is not the "choice" offered in the suburbs.

Let me close this portion of our conversation by quoting, as I did a few days ago, from the noted education scholar Pedro Noguera:

I'm not against charter schools, let me be clear, I'm in favor of any good school that's good for kids. But some of the charter schools that are being held up as a model believe that their goal is to regiment, to completely control their students. To control how they sit, control their eye contact, control their movements in the hallway. Many of them have silence in the hallway and no talking in the lunch room. John King, the new commissioner of education of New York state, is held up as a real reformer because he founded a very successful charter school in Boston called Roxbury Prep and went on to found a network the called Uncommon Schools. And I would say that academically this school is far out-performing many public schools that are serving the same population of kids. So I would acknowledge that they are doing a much better job. I would also acknowledge that the model they use does not appeal to me.

I've visited this school, and I noticed that children are not allowed to talk in the hall, and they get punished for the most minor infraction. And when I talked with John King afterwards, I said, "I've never seen a school that serves affluent children where they're not allowed to talk in the hall." And he said, "Well, that might be true, but this is the model that works for us, we've found that this is the model that our kids need."

So I asked him, "Are you preparing these kids to be leaders or followers? Because leaders get to talk in the hall. They get to talk over lunch, they get to go to the bathroom, and people can trust them. They don't need surveillance and police officers in the bathroom." And he looked at me like I was talking Latin, because his mindset is that these children couldn't do that.

Unfortunately what is often driving these high-performing schools is the idea that the kids need to be broken. That the kids' culture needs to be taken away from them and replaced with something else, because they come in with deficits. They come in as damaged goods. And these schools believe that their job is to mold the kids into something else.

And when they succeed, they end up with kids who no longer want to be associated with their own families and their own communities. Because the education they've been given has led them to believe that the goal is to escape. To escape the neighborhood, escape the community, to go someplace else. And so rather than education being a resource to help families and help communities, instead it's being used to cream off those we think have the talent. And let them go someplace else. Meanwhile the community stays exactly as it was and in many cases deteriorates further. That's not the education that I endorse. [emphasis mine]

Dmitri, you say: "Knowingly sending a child to a bad school is morally bankrupt." I couldn't agree more. Which is why, before we allow charter schools free rein to expand however much they would like, we'd better make damn sure they are providing children with an education that truly is better, and does not harm those students who choose not to attend. Marginal test score gains at a few high-profile charter chains are nice, but they are in no way enough evidence to show that charters should be allowed to grow unchecked.

I look forward to your reply.

Mark