More About Attrition Rates in Boston Charter Schools

The debate over Massachusetts's Question 2 -- a referendum on lifting the cap on the number of charter schools in the commonwealth -- rages on. When I last weighed in, I pointed out that the "successes" of Boston's charter sector could not fairly be compared to the "failures" of the public schools because the two sectors were educating fundamentally different students.

One indicator of this is the cohort attrition rate: the shrinkage in the size of a cohort that occurs because students leave a school, but are not replaced with new students entering.

Here's a school that "backfills"; in other words, as students leave because their families move, or they drop out, or they transfer, or whatever, they are replaced by new students entering the system.

This school doesn't backfill. What happens to the size of their cohorts (another way of saying "Class of x")? They shrink -- and that's what appears to be happening in Boston's charter schools.

A quick note: there are two kinds of charters in Massachusetts. "Commonwealth" charters are independent of their host districts, while "Horace Mann" charters are managed within the district. The  debate over Question 2 is really focused on Commonwealth charters. Kennedy Academy in the chart above is a Horace Mann charter; the others, in red, are Commonwealth charters. In Boston, Commonwealth charter high schools have far greater cohort attrition than the public district schools or the Horace Mann charters.

Furthermore, this attrition is part of a pattern across time:

Year after year, the cohorts enrolled in Boston's charter high schools shrink much more than the cohorts in the public district schools. Why? The simplest explanation is that the charters are not backfilling at the same rate as the district. This is a clear indication that the students in Boston's charters are not the same as the students in the district: their families are less mobile, and they are more likely to be a better "fit" with the philosophies of the charters than the kids who left.

This is one reason I contend you can't make a comparison between the achievement of the two groups of students, even if you use a randomized controlled trial design: the kids who attrit from the charters are likely quite different from the kids who remain.*

Now, a reader who comments over at Diane Ravitch's blog pointed out that there might be another explanation for all this: maybe the charters are retaining students for an extra year to get them "caught up." If that's the case -- and, certainly, some Boston Commonwealth charters do advertise the fact they will retain students if they think it's warranted -- then the cohorts would appear to be shrinking over time, because there would be more freshman than upperclassmen, even if none of the students left.

It's an interesting theory; however, I think there's plenty of evidence to show retention does not explain all, or even most, of the cohort shrinkage found in Boston's charter schools. Let's start with this:

This is the "intake" rate for Boston's charters and the Boston Public Schools (BPS) in 2016. According to the MA Department of Elementary and Secondary Education: "The Intake Rate measures the number of students that enroll in the state, a district, or school after the beginning of the school year..." In other words, who is taking more of the students who are coming into schools mid-year?

The answer is clear: BPS and the Horace Mann charters have much greater intake rates than the Boston Commonwealth charters. Granted, this doesn't tell us much about the intake that occurs between school years; however, it does again provide some evidence that the students in BPS and the charters are not the same.

Next, let's look at how the cohorts shrink in BPS:

As each cohort progresses from freshman to senior year, it shrinks. Some of that is due to dropping out; some is due to student mobility; some is due to transfers out of the system and into other public or private systems; some may even be due to retention.

There's a good case to be made that BPS has made substantial improvements in its ability to retain students through high school:

The senior class of 2005 was only 65% of its size in its freshman year; that's improved to 85%. So, now that we have some context, let's look at the same data for some Boston charter schools. We'll start with the largest of the high school charters, City On a Hill:

If COAH was retaining students in their freshman year, we would certainly expect a drop in students from freshman to sophomore years. But then we'd expect to see the number of students level off between Grade 10 and Grade 12. Except that's not the case; COAH's cohorts continue to shrink in their upperclass years.

Let me show it in numbers; here are the cohorts and their enrollments for grades 9 to 12:

Yes, the cohorts shrink significantly between Grade 9 and Grade 10, but then they shrink again between Grade 10 and Grade 11. And most also shrink between Grade 11 and Grade 12.

I find it very unlikely that retention explains this. And even if it did: is that really a good thing? How many years does it take to graduate from COAH? Five? Six? More?

Here's the cohort shrinkage for Match Charter:

Again, in most cohorts, the attrition that starts from Grade 9 to Grade 10 continues in the upper grades. I've posted some more examples below, but let's step back a bit and assess all this before we get to them.

Yes, the cohorts shrink for BPS -- but they shrink much more for the high school charters. And they shrink through the four years of high school, suggesting that retention isn't the sole explanation; if it were, students are being retained more than one year, which is hardly ideal.

I would argue this data is yet another piece of evidence that throws into doubt the claim that Boston's charter high schools can easily scale up their "successes." Even if they are retaining students, these charters schools are seeing their cohorts shrink substantially. Clearly, after gaining admission, many of their students are "voting with their feet" in years other than their freshman year -- and they are not being replaced by the students who are, allegedly, on waiting lists.

There's one other thing worth noting here -- something you'll miss if you don't pay attention to the vertical scales of my graphs:

As I pointed out before, the Commonwealth charter schools are a tiny fraction of the total Boston high school population. What happens if the cap is lifted and they instead enroll 25 percent of Boston's students? What about 50 percent? 

Let's suppose we ignore the evidence above and concede a large part of the cohort shrinkage in charters is due to retention. Will the city be able to afford to have retention rates that high for so many students? In other words: what happens to the schools budget if even more students take five or six or more years to get through high school? 

In a way, it doesn't really matter if the high schools get their modest performance increases through attrition or retention: neither is an especially innovative way to boost student achievement, and neither requires charter school expansion. If Boston wants to invest in drawing out the high school careers of its students, why not do that within the framework of the existing schools? Especially since we know redundant school systems can have adverse effects on public school finances?

I'll have more to say about Question 2 in a bit. For now, here are some more cohort enrollment patterns for Boston charter high schools:

Again: it would be very hard to explain all of these solely through retention. But even if we could: that still isn't an argument for raising the charter school cap.

More to come -- stand by...  

* Nerds, we'll talk about instrumental variable estimation and attrition soon, I promise.