Chartery AWESOMENESS!

Diane Ravitch points us to a recent paper by Atila Abdulkadiroğlu, Joshua Angrist, Peter Hull, and Parag Pathak that finds positive effects from charter school takeovers in New Orleans and Boston. No doubt, this will be presented as more evidence that "relinquishment"* is the new model for success in urban school systems.

I look at it, however, as more evidence that we are rushing headlong into a series of "reforms" without thinking fully about the consequences of our actions.

Let me start by saying this is a smart study written by some very smart people. I've just started a book by Angrist on econometrics, and he is obviously an accomplished economist whose work is and should be taken seriously.

In Angrist's telling, econometrics pretty much boils down to one central idea: ceteris paribus, or "hold other things constant." In other words, if you want to find out if a particular policy is going to work or not, you have to make that policy the only thing that changes between a treatment and control group. Econometrics is largely about using statistical tools to do just that.

In the case of this paper, the authors use a student matching technique, combined with a basic statistics technique called regression, to hold constant the differences in students who went to charter schools that took over public schools in New Orleans and Boston, and those who stayed in the public system. The paper finds there are test score gains for those who went to the charters.

It's been my experience that charter cheerleaders often rush to research like this and immediately claim victory. "See?" they exclaim. "We need to let charter proliferation thrive! The children can't wait another minute!" 

But it's also been my experience that if you press them as to why the charters in these studies are performing better on tests -- which is the central policy question in the debate over charter expansion -- they suddenly get all wishy-washy. "We do more for less!" is often the best you can get out of them. "We aren't tied to the bureaucracy!" is another "explanation." Sometimes you get a vague smack down of teachers unions.

But I have yet to hear a charter cheerleader give a cogent explanation as to why a gain found in any number of studies that is attributable to charter schools is replicable on more than a limited scale. And I'm afraid that studies like this one, as interesting and ingenious as it is, do little to get to this fundamental question.

I actually have a few nits to pick with this paper. First, like so much econometrically-based research in education, there isn't much context here for understanding the effect sizes in question. The outcomes are based standardized tests scores, which are measures with their own inherent limitations and flaws. The effect sizes are expressed in standard deviations; thankfully, the authors here avoid the indefensible practice of converting those into "x days of learning." But, as Kevin Welner has noted, showing a "statistically significant" gain isn't the same as showing a practical gain.

Abdulkadiroğlu et. al. report that "Attendance at RSD takeover charters is estimated to increase math and ELA scores by an average of 0.21σ and 0.14σ, respectively, per year enrolled." (p. 14)** Translating from  geek-speak, that's equivalent to moving from the 50th percentile to the 58th in math, and the 50th to the 56th in English language arts. 

In the conclusions, they state: "In practice, cleaning up the non-charter counterfactual substantially boosts our estimates of RSD takeover effects on math, from about 0.21σ to 0.36σ, while leaving the smaller ELA estimates largely unchanged." That raises the math percentile to the 64th percentile, up from the 50th. 

Sure, that's not anything to ignore. But it's hardly enough to close the "achievement gap" we hear so much about. Are we really prepared to rush into radically changing how we structure our schools on this basis alone? Hardly seems prudent to me.

Second, this is yet another paper that doesn't disaggregate its measures of student characteristics. Put another  way: it shoves kids who are probably quite different into the same box. For example, students are classified as qualifying for free or reduced-price lunch, or not. But as both Bruce Baker and I have harped on over and over, the distinction between free lunch eligible (130% of the poverty line or below) and reduced-price lunch eligible (130% to 185% of the poverty line) matters, especially when you're in a community like New Orleans where nearly all the students are in economic disadvantage. 

Same with special education: a student with a speech impediment is not equivalent to one with a profound cognitive impairment. I understand it's often hard for researchers to get data to do this, but at the very least it should be acknowledged as a methodological limitation.

Third: the very first thing I did when I opened up the paper, after reading the abstract, was look for any description of the student populations of the schools, and, critically, whether that population had changed after the charter school team took it over. I couldn't find it.

The authors acknowledge that students in economically disadvantaged communities are often quite mobile, and that's certainly true. But why these students might leave one school or another matters, and what the school looks like after they leave matters. Why?

We know that peer effect is real. We know that there are "successful" charter schools that engage in patterns of significant student attrition. And if charters are shedding students in an effort to improve the peer climate of the school, that's a problem: there aren't enough "non-disabled, non-poor, fluent English speaking females" to make charter proliferation a viable large-scale strategy for urban school improvement.

Which brings me to my biggest problem with this paper and, for that matter, much of the econometric education research I read these days. Because, too often, I see our brilliant economist friends jump to conclusions based on qualitative evidence that is, to be charitable, incomplete. Take page 5 of this paper:

The No Excuses model for urban education is characterized by extensive use of tutoring and targeted remedial support, reliance on data and teacher feedback, a curriculum focused on basic skills, high expectations from students and staff, and an emphasis on discipline and comportment.

Is it? Just say "No Excuses," and the Chartery AWESOMENESS kicks in, and everybody starts learning? 

That's certainly what the charter sector would like us to believe: if only we stop accepting "excuses," and set "high expectations," and get back to "basic skills," (even if stuff like the arts is especially good for low-income children), urban education will be fixed.  

This sort of thing reminds me of Peter Pan telling the audience to clap harder when Tinkerbell lays dying. Sure, believing that you can change the lives of urban students is important, but the obvious question is: now that you "believe," what's next?

Let me put this another way:

Here's a very much incomplete graph showing some of the reasons why the charters in this study, and the CREDO studies, and in other charter school studies, might get the effects they show. In blue I've got a host of reasons why "successful" charter schools might "succeed" that have nothing to do with their "charteriness."

If, for example, a charter gets a peer effect because its student population is different from the school to which it is compared, that's not particularly "chartery": we could conceivably set up public schools that did the same thing, and they would enjoy those same advantages when comparing test-based outcomes. Likewise, a test-prep curriculum is not the exclusive province of charter schools; we could drill all kids on bubble computer test prep, regardless of whether they're in charter schools or not, and likely get better test scores.

We know many "successful" charters spend more on their students, allowing for longer school days, smaller class sizes, and wrap around services. To their credit, Abdulkadiroğlu et. al. include some data on this in the paper -- but it's not really helpful unless we adjust for differences in student populations in the schools we're comparing.

My point here is that in a real ceteris paribus analysis that tests the proposition that "charter schools do more with less," we'd be holding everything I've got in blue constant so we could isolate the independent variable we really care about: Chartery AWESOMENESS. Then we could get a better sense if the effect of Chartery AWESOMENESS is like this:

Or, perhaps, more like this:

This is where we have to put our focus next. Because if the charter effects we're seeing are due to factors that have nothing to do with the Chartery AWESOMENESS of a school...

Why don't we just implement them, if we can, in all public schools?

Again: this is a smart study and we should pay attention to it. But I'd respectfully suggest to the economists who continue to produce these pieces that it's well past time they began to shift their research focus. Yes, some "successful" charter schools get gains.

The critical question now is: "Why?"

You mean "Chartery AWESOMENESS!" isn't explicit enough?

* Isn't it funny that, if your'e an academic, it's OK to state your academic credentials if you produce research that favors the charter sector's cheerleading; however, if you put your credentials on a report that stands opposite to the company line, you can be brought up on ethics charges. Golly, you'd almost think there was a double-standard or something...

** Don't make the mistake of thinking this "yearly" gain can be in any way extrapolated across the entire time a student is enrolled at a charter school; in other words, there's no reason to believe those gains will always be the same each year, so that in five years the math gains are over one standard deviation. That's just not how these things work; as the paper points out numerous times, gains are usually stronger in the first year, then start to plateau.