Attrition in Denver Charter Schools

Earlier this month David Leonhardt of the New York Times wrote yet another column extolling the virtues of charter schools. I feel like a broken record when I say, once again, that education policy dilettantes like Leonhardt don't seem to understand that it requires more than a few studies showing a few charters in a few cities in a few select networks get marginally better outcomes on test scores to justify large-scale charter expansion.

There are serious cautions when it comes to the proliferation "successful" charters, starting with the fiscal impact on hosting districts as charters expand. We should also be concerned about the abrogation of student and family rights, the lack of transparency in charter school governance, the narrowing of the curriculum in test-focused charters, the racially disparate disciplinary practices in "no excuses" charters, and the incentives in the current system that encourage bad behaviors.

But let's set all that aside and look at the evidence Leonhardt presents to justify his push for more charters:

Unlike most voucher programs, many charter-school systems are subject to rigorous evaluation and oversight. Local officials decide which charters can open and expand. Officials don’t get every decision right, but they are able to evaluate schools based on student progress and surveys of teachers and families. 

As a result, many charters have flourished, especially in places where traditional schools have struggled. This evidence comes from top academic researchers, studying a variety of places, including Washington, Boston, Denver, New Orleans, New York, Florida and Texas. The anecdotes about failed charters are real, but they’re not the norm.

You'll notice that Leonhardt picks cities and states that uphold his argument while excluding others like Detroit, Philadelphia, and Ohio. In addition: I spent a lot of time last year explaining why the vaunted Boston charter sector isn't all it's cracked up to be. I've also documented the mess that is Florida's charter sector. I'll try to get to some of Leonhardt's other examples, but for now: let's talk about Denver.

I'll admit it's one region where I haven't spent much time looking at the charter sector. Leonhardt links to a study that shows some significant gains for charters... although I have some serious qualms about the methodology used in the report. I'm working on something more formal which addresses the issue, but for now (and pardon the nerd talk): I am increasingly skeptical of charter effect research that uses instrumental variables estimators to pump up effect sizes. So far as I've seen, the validity arguments for its use are quite weak -- more to come.

For now, however, let's concede the Denver charter sector does, in fact, get some decent test score gains compared to the Denver Public Schools. The question, as always, is how they do it. Do they lengthen their school day and school year? If so, that's great, but we could do that in the public schools as well. Do they provide smaller class sizes and tutoring? Again, great, but why do we need schools that are not state actors to implement programs like that?

What we want to find are reasons that we can attribute only to the governance structure of charters -- not to resource differences, not to student population differences, but to the inherent characteristics of charters themselves.

And one thing I've found, time and again, is that one of the characteristics of "successful" charters is that they engage in patterns of significant student cohort attrition.

Let me explain what's going on here: this is data for the DSST network, one of the more lauded groups of charter schools in Denver. We're looking at the "class" of each cohort that has come through the entire charter chain; in other words, how big the Class of 2014 was when they were freshman, then sophomores, then juniors, and then seniors. I've done the same with each class back to 2008.

See the pattern? As DSST student classes pass through the charter schools year-to-year, the number of students enrolled shrinks considerably. The Class of 2014, for example, is 62 percent of the size as seniors as it was when freshmen. The shrinkage ranges from 61 to 73 percent over the eight years on the graph.

Where do the kids who leave go? Many likely go back to the Denver Public Schools. Some of those likely drop out, which counts against DPS's graduation rate -- but not the charter schools'. In any case, they aren't being replaced, which I find odd considering how supposedly "popular" charters are.

Some make the case that the larger freshman classes are due to retention: the schools keep the kids for an extra year to "catch them up." Which I suppose is possible... but it raises a host of questions. Do public students have the same opportunities to repeat a grade? Are the taxpayers aware they are paying for this? Why is there still significant attrition between Grade 10 and Grade 11?

Let's look at some other Denver charters and their cohort attrition patterns. Here's KIPP, the esteemed national charter network:

They haven't been running high schools as long as DSST, but the patterns are similar. KIPP's history is as a middle school provider; here are their attrition patterns in the earlier grades:

KIPP's Grade 8 cohorts shrink from 73 to 84 percent of their size in Grade 5. Again: if they're so popular and have such long wait lists -- and if the DPS schools are so bad -- why aren't they backfilling their enrollments? Note too that much of the attrition is after Grade 6. Most Denver elementary schools enroll Grades K to 5. It doesn't appear as if many students come into KIPP looking to move on after only one year; most of the attrition is in the later grades. Why would kids be leaving in the middle of their middle school experience?

Another middle school provider moving into high school is STRIVE:

Grade 8 is between 56 and 80 percent the size of Grade 6. Let's look at one more: Wyatt Academy.

The last class we have data for shrank to 69 percent of its size in First Grade 1 by the time it got to Grade 8.

Let's be clear: cohort shrinkage occurs in DPS as well.

The last year for which we have data was an outlier: the Class of 2018 was 75 percent as big in Grade 8 as it was in Grade 5. For previous years, that figure ranges from 81 to 90 percent. The comparisons to the charters are admittedly tricky: the transition from Grade 5 to 6, for example, is sure to see students moving out of the area or into the private schools, both from DPS and the charters. 

But it's still striking to me that "popular" charters, which are allegedly turning away lottery losers, seem to lose more students proportionally than the "failing" DSP schools.

DPS has a large number of students leave their Grade 9 cohort before Grade 12. Many are dropouts, and that's a serious problem. But why does DPS get slammed for this while the charter high schools are declared "successful" even as they are losing at least as large a proportion of their students as the public high schools?

Again, this is tricky stuff. I'm certainly not going to declare that Denver's charter sector is getting all of its gains from pushing out the lower performers; we don't have nearly enough evidence to make that claim. But neither can we declare definitively, as Leonhardt does, that charter "...success doesn’t stem from skimming off the best." When you lose this many students, particularly in high school, you have to back up and take a more critical view of why some charters get the gains that they do.

One more thing: look at the y-axes on my graphs. The scale of Denver charter school enrollments is nothing like the scale found in DPS. Only recently has STRIVE come around to about 10 percent of DPS's enrollment per class. How can we be sure the gains they make, if any, can be sustained as the sector gets larger?

When charters shed this many kids, there has to be a system that catches them and enrolls them in school. A system that takes them at any time of year, no matter their background. A system that doesn't get to pick and choose which grades it will enroll and when. That system is the public schools; arguably, charters couldn't do what they do without it.

Before we declare charters an unqualified success, we ought to think carefully about whether factors like attrition play a part in helping them realize their test score gains, and what that means for the public school system.

I'll try to get to Denver more this summer. But let's get back to New Jersey next...