UPDATED: Charter School Reality: TN Edition

UPDATE BELOW

OK, look...

I am not "anti-charter school." I started my career in a charter. I think there is a good case to be made for some -- some -- level of charter proliferation in many communities. I certainly don't think I ought to be telling people in other cities and towns that they shouldn't be able to decide for themselves how much "choice" they want in their district.

I also don't think it's a good idea to close charters wholesale and displace students who may be doing well at them. I'll even admit there are some good charters out there, and we should look at them carefully and see if there are some replicable best practices we can take away from them.

But let's be clear: we are not going to be able to have a serious conversation about the role of charter schools in our communities if the charter cheerleading industry continues to engage in poorly reasoned argument.

Too many charter cheerleaders rely on facile explanations for their "successes," and they too often do not offer a serious and, frankly, honest account of how charters actually operate. They are actually shooting themselves in the foot: you can't expect the rest of us to think you are engaging in a good faith discussion when you keep making arguments that are, at best, incomplete and, at worst, outright deceptive.

I say all this because, over the last few weeks, I've been dragged into debates about charters in several different locations. And what I'm seeing, over and over again, is really troubling: the use of "data" not to illuminate the discussion about charters, but to instead obfuscate. In the place of serious analysis, we're getting more and more spin.

Case in point:

Its also important to take a critical look at charters using data because some people tell charter stories that aren’t as positive as mine. Many detractors use their own anecdotes to claim that charters should be abolished all together.  They claim that charters aren’t any better than traditional public schools and that their claims to overcome poverty should be equated to charlatans selling fools gold or snake oil. Even when individual charters are found to be effective, detractors claim that these effects should be discounted because charters serve lower percentages of impoverished students than do traditional public schools. They also claim that charters “skim” the best students out of the general school population, that is, take the best and leave the rest, and that their achievement can therefore be explained by their students starting at a higher academic level than their peers.

If these accusations are true, they would and should call into question the entire charter movement. The problem is that they each fall apart when we look at the hard data behind charter schools here in Tennessee. I’ll go deeper into each of these claims throughout this piece using data from two sources, the CREDO National Charter School Study and Tennessee’s Charter Schools Annual Report for 2012-2013, but here’s a brief summary of what we’ll find;

• Data shows charters in Tennessee serve a more impoverished and higher minority population than traditional public schools
• Data shows that charter school students in Tennessee start at a lower academic level than their peers, refuting the “skimming” claim
• Data shows that Tennessee charters outpace traditional public schools on both growth and academic achievement measures
• This all suggests that charters are making meaningful progress towards overcoming poverty and closing the achievement gap [italic emphasis mine]

One of the links above is to a post of yours truly, where I took on the claim of a teacher and charter booster in Memphis about his school. As I said: there's nothing wrong with bragging on your school. But when you start using data to make claims that you are superior to "failing" schools, you ought to be able to defend them. Which brings us to the four claims above.

Let's take them one at a time, shall we?

1) When charter schools claim to "serve a more impoverished and higher minority population," that claim should be judged against the population of students in the community where they exist, and not against the larger population.

This should be a no-brainer, but sadly, it's not. Too often, charter cheerleaders make the claim that they serve more children in economic disadvantage than, say, the entire state in which they reside. But who cares? If a charter in Memphis draws its population from the poorest neighborhoods in the city, it should be compared to the public schools in those nieghborhoods. Why would it make any sense to compare it to wealthier suburban and exurban schools?

Jon Alfuth, the author here, cites the TNDOE's Charter Schools Annual Report to make his claim that charters serve a larger proportion of economically disadvantaged (ED) students. But that report compares Tennessee's charters -- which overwhelmingly draw from Memphis and Nashville -- with the entire state. I'm sorry, but that's a transparently absurd comparison.

We'd be much better off looking at how charters compare to their neighboring public schools:

All data from the TNDOE. Here are the high schools in the area roughly bounded by I-40, I-240, I-55 and the Mississippi River -- the urban core of Memphis. These schools don't include affluent areas in Shelby County like Germantown and Collierville: if we're going to include those schools, we'd best settle in for a long conversation about segregation and its effects on poverty and academic outcomes, and leave the charter schools out of it.

The graph only includes schools that had valid test takers for English II, a high-school course for which Tennessee administers a statewide end-of-course exam. Do charters take in many ED students? Yes, they do -- but only compared to the rest of the state.

Do charters take in as many ED students as the other public schools in Memphis? Clearly, they do not.

Saying Tennessee charters "serve a more impoverished and higher minority population" is a statement that is, at best, incomplete. And it directly impacts Alfuth's next two claims.

2) When charter schools claim to serve students who "start at a lower academic level than their peers," that claim should be judged, again, against the population of students in the community where they exist, and not against the larger population. 

Alfuth makes his case based on the infamous CREDO report. Again: you should not compare charters, concentrated in a few urban centers with large areas of poverty, with the entire state. It's simply not an apt comparison, because the charters aren't "skimming the cream" off of the milk of the fattest cows (sorry). But that's exactly what the CREDO report does, right on page 22: it compares the starting point of charter students concentrated in a few impoverished areas against the entire state.

Alfuth teaches at Soulsville Charter School; as seen above, Soulsville has a significantly smaller proportion of students in poverty than most of the rest of the core of Memphis. One reason why, I suspect, is his school requires an application for admission. Sure, anyone can apply... once they download the application. Think about the social capital that one act alone requires. Then multiply that by a few dozen, and you'll start to get the sense of how any application process advantages those families that have the wherewithal to navigate a "choice" system.

I can't directly refute Alfuth's claim here, because I don't have the data. But the specious comparison of urban charters to the entire state is more than enough for us to cast doubt upon it.

3) When charters make claims to academic success, they should acknowledge the role that student population demographics play in creating an atmosphere for that success.

The CREDO study was lauded by many because it created a "virtual match" between demographically similar students in and out of charters. Which is nice as far as it goes... but does nothing to address the subject of replicability.

Peer effect is real: if your charter school has fewer ED students, it's more likely to get better academic results. A school like Soulsville enjoys peer effect because its student population is less economically disadvantaged than the schools surrounding it; we would, naturally, expect it to get better results.

But it is impossible to have every student attend a school that has below average poverty! You just can't replicate that: for every lower-ED school like Soulsville, there has to be a higher-ED school like MCS Prep.

Which is why it's critical to take student population attributes into account when judging the "charter effect" on student achievement.

4) When charters make claims to "closing the achievement gap," those claims should be judged in a way that takes into account the differences in their student populations.

I have to admit, I thought this statement of Alfuth's was pretty funny:

In both districts charter achievement ties or exceeds absolute district achievement in all subjects. In Shelby County, charter students do even better in high school, exceeding the absolute achievement of traditional public schools by 5 percent or more in every subject. And lets keep in mind that we’re also comparing charters to every non charter in the district, including some that don’t share the same student profiles as the charters in question. If we could remove these schools from the traditional population, charters would do even better by comparison. [underline emphasis mine]

Oh, I see: according to the charter cheerleaders, it's OK to judge charters against an entire state when comparing student population characteristics, but we really shouldn't compare them to an entire state when comparing test-based outcomes.

Cherry-pick much?

Tell you what: how about we compare charters to schools across the state, but take into account their Free-Lunch eligible populations (a proxy measure for economic disadvantage)?

A little explanation: these are all the schools that reported English II test takers in Tennessee. We've got the percentage at proficient or above on the y-axis, and the proportion who are FL on the x-axis. Regular readers of this blog know FL correlates strongly with test-based outcomes, which is why the points -- each one a school -- follow the pattern of the trendline I have in the middle of the graph.

I've marked the Tennessee charters that reported English II scores with red triangles. Notice that some are above the trendline, which is good: they are doing better than we would predict. But some are below; more importantly, the charter sector, as a whole, is not that far away from the trend for the entire state.

Soulsville, where Alfuth teaches, is beating prediction; good for them. I'm sure the extra time the school has in session helps, especially since the students probably don't have many good options for academic or personal growth outside of the school day. But Soulsville is not a huge outlier; it has hardly "closed the gap" with the most affluent schools in the state.

Again: Alfuth should be proud of Soulsville. The students there should be proud of their outstanding work. It is fantastic to hear that kids who grew up with so many disadvantages are able to better themselves and attend college. Bravo and brava to you all! And congratulations to your teachers, who are undoubtedly proud of you and all you've achieved.

But if we're going to have a conversation about how to lift up the lives of all children, we have an obligation to be honest. And, frankly, Alfuth's post suffers from deception through omission. 

Tennessee's charter sector -- indeed, America's charter sector --  is not overcoming poverty in a way that can be replicated for all students who suffer from economic disadvantage. All the evidence suggests we will never be able to help all students until we lift them all out of the crushing poverty that is a product of an economic system that has come off of the rails. 

There is no evidence that charter schools can be broadly replicated to overcome the effects of poverty.

If we could just agree on this one simple point, maybe we could start a conversation about helping all kids.

More charter school reality in a bit...

So which one of y'all is going to invite this Jersey boy down to Memphis?

UPDATE: Lots of comments. Let's roll:

Except the CREDO study by design, matches similar students using a VCR (virtual control record) process. 

So the CREDO data isn't comparing a poor charter school student in TN vs. a student who goes to school in a well to do suburb. It's simply not correct to state otherwise.

Which is why I didn't say it. The problem with the CREDO study is that it matches individual students in schools with different populations. Here's an example:

Suppose a kid goes to a charter with 50% of the students qualifying for free lunch. That charter sits in a district where the feeder schools have 90% of the students qualifying for free lunch. CREDO matches that charter kid with a kid in a 90% FL school. The charter kid does better. Great!

Now what?

Can you make a 50% FL school for every kid? It's impossible in this district: it's impossible for every student to attend a school with below average poverty. So if lower poverty rates among peers help kids learn -- and it's pretty clear that they do -- charter expansion isn't going to help much, is it?

In spite of that, the charters even on your own chart are above the trend line, meaning they perform above average on measures of achievement.

So do, some don't -- that's the point. As a sector, charters are not beating poverty. Not one charter with an over-60% FL population is beating any Tennessee schools with an under-30% FL population.

One of my favorite replies to this is when charteristas say: "Well, we need to close the bad charters and open more good ones!" Why, yes, how silly of the rest of us -- let's just do that, by gum! Every school should be above average!

Do a really have to spell out the problem with that line of thinking?

Trying to discredit schools that are working well for students, whether they carry a charter label or district school label, isn't going to do anything for our students.

As I said, right at the beginning of the post -- I am not anti-charter. But I really, really don't like it when bad arguments are made in favor of charters. If the charter cheerleaders want me to stop pointing out the holes in their logic and analysis, I'd suggest they start making their arguments air-tight.

1. The demographic data I used did in fact compare charters to non charters within districts, not the entire state as you claim. I tweeted you the table in case you missed it in the original story.

I saw it - it's irrelevant. There's plenty of variation within districts to justify not using them for demographic comparisons, and to, instead, look more locally.

Also, I need to plead ignorance on something: I don't know much about the Shelby County/Memphis school district restructuring (on the surface, it looks like quite a story). But if that table includes the wealthier suburban schools outside of Memphis, it's useless.

My method is far better: look at the zip codes nearby and compare the public schools within them to charters.

2. You're correct that I used the CREDO study on statewide data. That data does compare charter students to the average Tennessee student, that is, the student in the 50th percentile. No, its not perfect, but I will contend the massive difference in Tennessee between the average TN student and charter student still provides a useful comparison to illustrate the starting gap between charters and traditional public schools.

I'm sorry, but that's just dead wrong. The only way that comparison would be useful is to compare the charter feeder schools -- which, in Memphis, appear to be entirely high-poverty schools -- in the same way. If the "massive difference" between charters and the rest of the state matter, the same difference between the charter feeders and the state should matter just as much.

Funny how the CREDO report didn't address this, isn't it?

3. I do agree with you that we need to take student population demographics into account when considering charter replication of any individual school. But if we just dismiss them out of hand once we find that demographic data differs, we'll never know. 

I'm not dismissing anyone out of hand: as I said, Soulsville beats prediction, and they should be proud of that. But we have to be honest about why the school does as well as it does. Platitudes about "hard work" just don't cut it.

4. I'm curious why you chose only English II out of all EOC accountability subjects to compare achievement vs poverty.

It's all I had time for and seemed to make the most sense. But I'll readily concede there may be greater differences in other subjects. Whether those differences close the "achievement gap" is another matter.

5. Believe it or not I'm not a u-rah-rah charters at all cost supporter. I'm interested in what works above all, and if something doesn't (see my writing on school vouchers) I'll oppose it. But regardless of what data we're using, its clear that charters in Tennessee are achieving something out of the ordinary compared to both charter schools nationally and district schools here in Tennessee. We can debate as to the size or the reason of the effect, but hopefully we can at least acknowledge something meaningful is happening and see what we can learn from it.

Without question: something very meaningful is happening in Tennessee. Arne Duncan keeps holding the state up as an exemplar, even as it has led the way in test-based teacher evaluation and charter proliferation and TFAism and the latest reformy flavor-of-the-month. Some of that is coming undone, thankfully -- but not all of it.

The headline for the post in question is: "Data Doesn't Lie: TN Charters Are The Real Deal." Well, I used data, and what I found isn't really out of the ordinary.

In fact, it's quite ordinary.