I will protect your pensions. Nothing about your pension is going to change when I am governor. - Chris Christie, "An Open Letter to the Teachers of NJ" October, 2009

Saturday, October 22, 2016

More About Attrition Rates in Boston Charter Schools

The debate over Massachusetts's Question 2 -- a referendum on lifting the cap on the number of charter schools in the commonwealth -- rages on. When I last weighed in, I pointed out that the "successes" of Boston's charter sector could not fairly be compared to the "failures" of the public schools because the two sectors were educating fundamentally different students.

One indicator of this is the cohort attrition rate: the shrinkage in the size of a cohort that occurs because students leave a school, but are not replaced with new students entering.
Here's a school that "backfills"; in other words, as students leave because their families move, or they drop out, or they transfer, or whatever, they are replaced by new students entering the system.

This school doesn't backfill. What happens to the size of their cohorts (another way of saying "Class of x")? They shrink -- and that's what appears to be happening in Boston's charter schools.


A quick note: there are two kinds of charters in Massachusetts. "Commonwealth" charters are independent of their host districts, while "Horace Mann" charters are managed within the district. The  debate over Question 2 is really focused on Commonwealth charters. Kennedy Academy in the chart above is a Horace Mann charter; the others, in red, are Commonwealth charters. In Boston, Commonwealth charter high schools have far greater cohort attrition than the public district schools or the Horace Mann charters.

Furthermore, this attrition is part of a pattern across time:


Year after year, the cohorts enrolled in Boston's charter high schools shrink much more than the cohorts in the public district schools. Why? The simplest explanation is that the charters are not backfilling at the same rate as the district. This is a clear indication that the students in Boston's charters are not the same as the students in the district: their families are less mobile, and they are more likely to be a better "fit" with the philosophies of the charters than the kids who left.

This is one reason I contend you can't make a comparison between the achievement of the two groups of students, even if you use a randomized controlled trial design: the kids who attrit from the charters are likely quite different from the kids who remain.*

Now, a reader who comments over at Diane Ravitch's blog pointed out that there might be another explanation for all this: maybe the charters are retaining students for an extra year to get them "caught up." If that's the case -- and, certainly, some Boston Commonwealth charters do advertise the fact they will retain students if they think it's warranted -- then the cohorts would appear to be shrinking over time, because there would be more freshman than upperclassmen, even if none of the students left.

It's an interesting theory; however, I think there's plenty of evidence to show retention does not explain all, or even most, of the cohort shrinkage found in Boston's charter schools. Let's start with this:


This is the "intake" rate for Boston's charters and the Boston Public Schools (BPS) in 2016. According to the MA Department of Elementary and Secondary Education: "The Intake Rate measures the number of students that enroll in the state, a district, or school after the beginning of the school year..." In other words, who is taking more of the students who are coming into schools mid-year?

The answer is clear: BPS and the Horace Mann charters have much greater intake rates than the Boston Commonwealth charters. Granted, this doesn't tell us much about the intake that occurs between school years; however, it does again provide some evidence that the students in BPS and the charters are not the same.

Next, let's look at how the cohorts shrink in BPS:


As each cohort progresses from freshman to senior year, it shrinks. Some of that is due to dropping out; some is due to student mobility; some is due to transfers out of the system and into other public or private systems; some may even be due to retention.

There's a good case to be made that BPS has made substantial improvements in its ability to retain students through high school:


The senior class of 2005 was only 65% of its size in its freshman year; that's improved to 85%. So, now that we have some context, let's look at the same data for some Boston charter schools. We'll start with the largest of the high school charters, City On a Hill:


If COAH was retaining students in their freshman year, we would certainly expect a drop in students from freshman to sophomore years. But then we'd expect to see the number of students level off between Grade 10 and Grade 12. Except that's not the case; COAH's cohorts continue to shrink in their upperclass years.

Let me show it in numbers; here are the cohorts and their enrollments for grades 9 to 12:


Yes, the cohorts shrink significantly between Grade 9 and Grade 10, but then they shrink again between Grade 10 and Grade 11. And most also shrink between Grade 11 and Grade 12.

I find it very unlikely that retention explains this. And even if it did: is that really a good thing? How many years does it take to graduate from COAH? Five? Six? More?

Here's the cohort shrinkage for Match Charter:


Again, in most cohorts, the attrition that starts from Grade 9 to Grade 10 continues in the upper grades. I've posted some more examples below, but let's step back a bit and assess all this before we get to them.

Yes, the cohorts shrink for BPS -- but they shrink much more for the high school charters. And they shrink through the four years of high school, suggesting that retention isn't the sole explanation; if it were, students are being retained more than one year, which is hardly ideal.

I would argue this data is yet another piece of evidence that throws into doubt the claim that Boston's charter high schools can easily scale up their "successes." Even if they are retaining students, these charters schools are seeing their cohorts shrink substantially. Clearly, after gaining admission, many of their students are "voting with their feet" in years other than their freshman year -- and they are not being replaced by the students who are, allegedly, on waiting lists.

There's one other thing worth noting here -- something you'll miss if you don't pay attention to the vertical scales of my graphs:


As I pointed out before, the Commonwealth charter schools are a tiny fraction of the total Boston high school population. What happens if the cap is lifted and they instead enroll 25 percent of Boston's students? What about 50 percent? 

Let's suppose we ignore the evidence above and concede a large part of the cohort shrinkage in charters is due to retention. Will the city be able to afford to have retention rates that high for so many students? In other words: what happens to the schools budget if even more students take five or six or more years to get through high school? 

In a way, it doesn't really matter if the high schools get their modest performance increases through attrition or retention: neither is an especially innovative way to boost student achievement, and neither requires charter school expansion. If Boston wants to invest in drawing out the high school careers of its students, why not do that within the framework of the existing schools? Especially since we know redundant school systems can have adverse effects on public school finances?

I'll have more to say about Question 2 in a bit. For now, here are some more cohort enrollment patterns for Boston charter high schools:






Again: it would be very hard to explain all of these solely through retention. But even if we could: that still isn't an argument for raising the charter school cap.

More to come -- stand by...  


* Nerds, we'll talk about instrumental variable estimation and attrition soon, I promise.

5 comments:

Harneen said...

Thanks for your posts about MA. We are in a huge fight here for the future of public schools. To add to your research...I served on the state Board of Elementary and Secondary Education for 11 years and in MA it is that Board who provides all charter accountability. Prior to 2010 charters did no backfilling. They weren't required to and didn't. In 2010 an Act Relative to the Achievement Gap was passed and one of its' provisions required schools to start backfilling. BUT and there is a big but...they only have to back fill in the first half of their grade span. So for a K-6 the charter must backfill in K1, K2, 1 and 2 grade. They are not required to backfill in 3, 4, 5, or 6 grades. In which of those grades is our state test MCAS PARC 2.0 administered? 3, 4, 5, 6. In addition they only have to backfill through February 15. Which months is our state test administered? March-April-May. High schools are required to backfill even less. They only are required to backfill through 9th, so no new students in 10, 11 or 12th grade.
You have done a great job unpacking the limited use of the Departments data on their webpage. We have also tracked cohort enrollment by year from the DESE webpage and show similar patterns of large numbers of families 'choosing' to unenroll every year. No comprehensive data exists on what happens to those families and why they are leaving. The DESE definition of attrition is how many kids are enrolled in June and return in September. Not how many begin with a cohort and are still there when the cohort graduates. These comments are really for you and your work. I wasn't sure how to contact you. Let me know if I can be of any help. I am a Boston Public School parent and have been on the debate circuit these last few weeks for NoOn2, the charter ballot initiative. Can't wait until the election is over.

zaidazee said...

These data are similar to the data posted by John Lerner on the Citizens for Public Schools website in which he compares the cohort attrition to rising MCAS scores.

http://www.citizensforpublicschools.org/charter-schools/as-charter-school-classes-shrink-mcas-scores-rise/

Stephen said...

(part 1)
If we wish to understand attrition rates at schools in Boston, rather than mingling members of multiple incoming entering classes in unknown ratios into a single starting "cohort", won't we achieve far more reliable results by examining Massachusetts Department of Elementary and Secondary Education (DESE) attrition data that shows for each school how many leave during the summer, plus its stability rate data for how many stay enrolled during the school year?

In my prior message I pointed to such DESE data for a couple of superb schools: MATCH Charter school and Nathan Hale (traditional public school).

As for a broad comparison, in respect to summer attrition, DESE has calculated school rates, weighted by enrollment, as:
Boston Public Schools: 14.2%
Boston Commonwealth Charter Schools: 9.3%
http://www.doe.mass.edu/research/reports/2016/02CharterReport.pdf (pg. 36)

While DESE doesn't provide a similar weighted average for school year stability, the state charter school association attempted the calculation and came up with:
Boston Commonwealth Charter Schools: 92.2%
Boston Public Schools: 86.5%

I haven't confirmed that, but have heard no complaint about its accuracy; it seems plausible to me after looking at the rates for a variety of individual schools.

Jazzman: "If COAH was retaining students in their freshman year, we would certainly expect a drop in students from freshman to sophomore years. But then we'd expect to see the number of students level off between Grade 10 and Grade 12. Except that's not the case; COAH's cohorts continue to shrink in their upperclass years."

I previously shared with you the most recently available COAH retention data:
9th: 27.8%
10th: 3.3%
11th: 3.2%
12th: 7.4%
http://www.doe.mass.edu/infoservices/reports/retention/

So, clearly, a large portion of the decline in enrollment between 9th and 12th grade is indeed caused by front-loaded retention. And, also clearly, that doesn't explain all of it. After some years with 100 newly enrolled 9th graders each year, if retention rates like those cited above were the only factor, I think we might at some point see enrollment along these lines:
9th: 138 students;
10th: 103
11th: 103
12th: 107

So, to achieve a fuller understanding of the enrollment decline, we would best move on to other DESE data, and learn:

COAH had a zero percent dropout rate:
http://profiles.doe.mass.edu/dropout/default.aspx?orgcode=04370505&orgtypecode=6&leftNavId=15627&

8.4% summer attrition rate:
http://profiles.doe.mass.edu/attrition/default.aspx?orgcode=04370505&fycode=2015&orgtypecode=6&

and
91.8% school year stability rate:
http://profiles.doe.mass.edu/mobility/default.aspx?orgcode=04370505&fycode=2015&orgtypecode=6&

So, it appears that throughout the entire summer and school year, COAH does better on every one of those measures than is typically the case for traditional public schools in the same community. COAH was the school you originally cited as a prime example of Commonwealth charter schools losing students. Rather than a poster child for attrition loss aimed at ridding schools of poor performers, COAH is perhaps instead a poster child for the practice of callously dumping high perfomers into college after four years, while keeping the academically less accomplished students close at hand, for at least one additional year.

Stephen said...

(part 2)
I do concur with a good portion of what you write most recently; it is lack of intake/backfill that explains why, as Commonwealth charter schools lose some students (at a relatively low rate), their enrollments decline in upper grades. The intake/backfill rate at COAH was just 1.7%.

If I understood you correctly, it may be an appropriate time to gracefully pivot from condemnation of Boston charter schools for excessive attrition to instead examining:
1) whether the shockingly low attrition rates of Boston Commonwealth charter schools suggests they are dealing with an easy, pliable, less mobile population
2) whether their startlingly low loss of students is intensively concentrated on those least likely to succeed academically thereby skewing test results favorably
3) whether some combination of losing the most troublesome students while not backfilling with new students could in additional ways be creating a climate fostering spectacular test results

Intuition alone irresistably inclines me to favor some of these hypotheses to some degree.

But the data I consult fails to provide much additional boost.

I have heard it asserted by others with pitbull persistence that the highest performing charters assuredly have the highest attrition... They must attain the highest performance by most sweepingly pushing out the least scholarly scholars. Though that's plausible, the Boston data runs contrary.

In Massachusetts, DESE constructs a numeric percentile score that aims to rank schools against others statewide that serve the same or very similar grade levels. My understanding is that it derives from a complex blend of test scores, growth, graduation rates, etc. that's tough to decipher, and that I might quarrel with if I fully understood it. But it's the best single score I can find for purposes of ranking schools as to how much better kids read, write and calculate as they progress through school.
http://profiles.doe.mass.edu/accountability/report/aboutdata.aspx

So here's that score for Boston's 2015 Commonwealth charter schools in the first column, and in the second column, DESE's attrition score for the same score. Would be pleased if you'd throw those in a scatter chart here, JJ.

for Boston's 2015 Commonwealth charter schools:
97 5.3
93 1.6
91 5.3
78 9.1
71 6.9
62 12
58 8.6
57 8
54 11
51 8
42 7.7
36 11.3
32 10
23 18.9

OK, no positive correlation between attrition and performance there. We could imagine that the impact is via school-year rather than summer loss, but with only 82 students total in all grades leaving all Commonwealth charter schools in the city during the school year, that's not a compelling putative explanation.

Stephen said...

(part 3)
So, what about the possibility that, what relatively little attrition there is, is hyperconcentrated among the lowest performing students... and that a peer mix more conducive to effective learning is thus enhancing academics among those who remain?

As you know, JJ, Angrist et al examined that possibility in "Stand and Deliver: Effects of Boston’s Charter High Schools on College Preparation, Entry, and Choice" and concluded:

“Charter schools are sometimes said to generate gains by the selective retention of higher-performing students — see, e.g., Skinner 2009. In this view, charter effectiveness is at least partly attributed to a tendency to eject trouble-makers and stragglers, leaving a student population that is easier to teach.”
[…]
“These results suggest that positive charter effects cannot be attributed to low-quality peers leaving charter schools. If anything, selective exit of low achievers is more pronounced at Boston’s traditional public schools.”
http://economics.mit.edu/files/9799

It seems consistent with available evidence that kids who are struggling academically at traditional schools are more likely to be retained and then drop out of school, in contrast to at charter schools being retained and then, thanks to strong and sustained intervention, hanging in there till graduation.

One could reasonably suppose that the circumstance is far different in elementary/middle school where dropping out isn't an option. But the notion that charter schools are aggressively 'counseling out' or otherwise removing the lowest performing students doesn't seem supported by what we find at the schools at the very top of the list of comparative performance I provided above.

Those would be the Brooke charter schools. And please see via the link below the graphs on pp 32-33 that illustrate this: “Furthermore, even the small numbers of students who do leave Brooke each year are just as likely to be performing well academically as their peers who persist, as demonstrated in the graphs below.”
http://www.doe.mass.edu/boe/docs/fy2016/2016-02/item3-tabA1-1.docx

Wish we could insert those graphs here and see how well your readers' friends could distinguish between "Stayed" and "Left" if those were covered.

The economics of whether it's preferable to anticipate short-term monetary savings via a school system where kids are more likely to drop out of school, or instead bear the burden of a school where kids are more likely to be retained for an extra year and graduate is one I'd be glad to take up with you, JJ, if I thought there'd be a controversy. But I anticipate we'd reach swift agreement.

Boston's Commonwealth Charter schools: Excellent principals. Effective teachers. Lots of time with the kids. Results that parents passionately appreciate. Can we find that in the Boston district schools also? Of course! Nathan Hale Elementary! New Mission High! I could go on. And on. But not as long as we might wish.