I will protect your pensions. Nothing about your pension is going to change when I am governor. - Chris Christie, "An Open Letter to the Teachers of NJ" October, 2009

Thursday, September 22, 2016

Massachusetts Charter Schools and Their Problems With "Attrition"

UPDATE: A commenter at Diane Ravitch's site pointed out that I had mistakenly included Phoenix Academy Charter in my group of Boston charter schools. He's quite correct; it's actually in the nearby suburbs. I've updated this post to reflect this, and apologize for the error.

Also: I left off the link to the MAPCSA post; I've added it.


The debate about lifting the Massachusetts charter school cap continues to rage on, in anticipation of November's vote on Question 2:
Question 2 on the November ballot will ask voters if they support giving Massachusetts the authority to lift the cap on charter schools. As it stands, no more than 120 charter schools are allowed to operate in the state; there are currently 78 active charters.
A "Yes" vote on Question 2 would give the Massachusetts Department of Elementary and Secondary Education the authority to lift the cap, allowing up to 12 new charter schools or expansions of existing charters each year.
Priority would be given to charters that open in lower-performing districts. New charters and charter expansions approved under this law would be exempt from existing limits on the number of charter schools, the number of students enrolled in them and the amount of local school districts' spending allocated to them.
Pro-charter researchers have been weighing in. I'll get to their arguments in due time, but for now, I want to concentrate on a key issue in the charter cap debate: attrition.

Determining whether attrition affects charter school results is central to the argument for (or against) their expansion in Massachusetts and elsewhere. If charter schools shed kids year after year -- especially if those kids are low-performing -- then their vaunted performance advantages are in question, particularly when compared to public district schools that aren't losing students.

The Massachusetts charter sector has been pushing back hard on this point. Here, for example, are the "facts" from the Massachusetts Charter Public* School Association:
ATTRITION RATES
The attrition rate in Boston and in Gateway City charters “has remained lower” than the attrition rates of district schools in those communities, according to data by (DESE) in Dec. 2015 (2014-2015 school year).
The attrition rate at Boston charters (9.3%) is significantly lower than in BPS (14.2%).
In Gateway Cities, charter attrition rates (6.2%) are lower than Gateway districts (11.4%).
From 2012-2014, an average of just 82 students left charters and returned to Boston Public Schools, according to BPS numbers – one-tenth of one percent of BPS total enrollment of 57,000.
Yeah, uh... no. Not really.

You see, "attrition," for the MA Department of Elementary and Secondary Education, has a very specific meaning:
This report provides the percentage of attrition by grade from the end of one school year to the beginning of the next for students enrolled in public schools, including charter schools, in the state. The information is as of October 1 of the school year selected. [emphasis mine]
In other words, "attrition" is the percentage of students who only leave a school's rolls during the summer. Which may be an interesting statistic, but does not include all of the students who leave during the school year.

Further, "attrition" as defined by MA-DESE only tells half of the story we need to hear if we're going to evaluate charter school performance. What we really need to consider is whether the students moving out of charter schools are being replaced at rates equal to the replacement rates for students moving out of public district schools.


If we only consider a school's attrition over time -- all of its attrition, not just the students leaving in the summer -- we don't consider its "backfill": the students coming in to replace the ones who left. Students in economic disadvantage are often more mobile than students who are not, which means that urban centers, where charters proliferate, are more likely to have student populations who move in and out of different school districts.

Here's an urban school that backfills as much as it attrits students.

Notice the school gains as many students as it loses. Let's assume this is just for one grade; if so, that "cohort" will remain the same size no matter how large its attrition rate is.

But what happens if the school doesn't backfill?

This school has exactly the same attrition rate as the first school. But because it doesn't backfill, its cohorts keep shrinking. Every year it loses students, but it doesn't take in enough in the same grade to replace them.

Again, this is a key issue in determining if charters can be scaled up to take a larger share of students. If charters are not backfilling, they are probably serving a less mobile student population -- and one that is likely in less economic disadvantage. They are relying on the public district schools to take the students that are coming into the district, which raises some profound questions about how, exactly, the "successful" charters get their gains.

So now that we've described the real issues in this debate, let's go to the data. We'll focus on Boston as the city is, by far, the largest district in Massachusetts and will likely see the greatest amount of charter expansion if Question 2 passes. 

In this analysis I focus on high schools, for several reasons. First, we don't have to concern ourselves with the differences in grade level served between charters -- who often serve elementary and middle grades along with high school grades -- and public district schools. Second, we're less likely to see the type of attrition that occurs between grades 8 and 9, when students move into high school and often attend a private school.

Let's start by seeing how Boston's high school students divide up between charters and public district schools:


A few things to note here. First, there are two flavors of charter school in Boston: independent charters, and "Horace Mann" charters, which are sanctioned by the Boston Public Schools and staffed (mostly) with unionized teachers. I've marked the independents in red, the Horace Mann charters in purple, and BPS in blue.

There is no question that the independent charter sector is still relatively small in Boston, at least as far as high schools are concerned. That alone ought to give supporters of Question 2 pause: how can they be so sure these schools can maintain their alleged "gains" (we'll talk about whether these "gains" actually exist in another post) if they expand? What if they can only function on a smaller scale?

This is why we have to look at the size of the cohorts as they pass through from Grade 9 to 12. What, for example, happened to the graduating Class of 2014 as they moved from freshman to senior year?



We lost a few schools because they are so new that they hadn't yet had a cohort pass through from freshman to senior year. I also took out Boston Day and Evening Academy Charter because it is an alternative high school that mixes grade levels.

Every independent charter school in Boston had a higher cohort attrition rate in 2014 than BPS as a whole.

In the case of City on a Hill and Phoenix, their 2011 freshman class shrank by more than half by the time they were seniors. That is a remarkable difference compared to BPS.

But is it part of a pattern?


In the last decade, Boston's charter sector has had substantially greater cohort attrition than the Boston Public Schools. In fact, even though the data is noisy, you could make a pretty good case the difference in cohort attrition rates has grown over the last five years.

Is this proof that the independent charters are doing a bad job? I wouldn't say so; I'm sure these schools are full of dedicated staff, working hard to serve their students. But there is little doubt that the public schools are doing a job that charters are not: they are educating the kids who don't stay in the charters, or who arrive too late to feel like enrolling in them is a good choice.

This is a serious issue, and the voters of Massachusetts should be made aware of it before they cast their votes. We know that charter schools have had detrimental effects on the finances of their host school systems in other states. Massachusetts' charter law has one of the more generous reimbursement policies for host schools, but these laws do little more than delay the inevitable: charter expansion, by definition, is inefficient because administrative functions are replicated. And that means less money in the classroom.

Is it really worth expanding charters and risking further injury to BPS when the charter sector appears, at least at the high school level, to rely so heavily on cohort attrition?

It doesn't matter what you call it -- you're still shrinking.



* Notice where the world "public" is put in the title of this group? They desperately want us all to believe charters are "public" schools, even when the courts and other public authorities have ruled repeatedly that they are not.

8 comments:

Peter Greene said...

Thanks for this.

Stephen said...

It looks to me, JJ, as if you may have some misunderstandings that may be worth correcting.

Without providing a link, you quote the Massachusetts Charter Public School Association.
I think this is the full document for those who may be interested:
http://www.masscharterschools.org/media/news/just-facts-public-charter-schools-massachusetts

The segment you quote from states:
"The attrition rate at Boston charters (9.3%) is significantly lower than in BPS (14.2%).
In Gateway Cities, charter attrition rates (6.2%) are lower than Gateway districts (11.4%).
"From 2012-2014, an average of just 82 students left charters and returned to Boston Public Schools, according to BPS numbers – one-tenth of one percent of BPS total enrollment of 57,000."

And you responds to that:
"Yeah, uh... no. Not really."

You claim, properly, that our department of education's "attrition" rate is inadequate to provide a full perspective insofar as it addresses students leaving during the summer. What you may not understand is that the sentence that starts "From 2012-2014" is supplementing the summer attrition figures by citing the number of students who leave charter schools to return to BPS schools not during the summer but rather during the school year. I think the Association's point there is that there's not much such movement during the year relative to summer attrition which is when most of the mobility between sectors occurs.

Our state department of education makes very easily available both summer attrition rates and also measures that show the rates at which students leave during the year. For example, to view the summer attrition and the school year stability rates of one superb charter school and one superb traditional public school, see
MATCH attrition: http://profiles.doe.mass.edu/attrition/default.aspx?orgcode=04690000&fycode=2015&orgtypecode=5&
MATCH stability: http://profiles.doe.mass.edu/mobility/default.aspx?orgcode=04690000&fycode=2015&orgtypecode=5&
Nathan Hale attrition: http://profiles.doe.mass.edu/attrition/default.aspx?orgcode=00350243&fycode=2015&orgtypecode=6&
Nathan Hale stability: http://profiles.doe.mass.edu/mobility/default.aspx?orgcode=00350243&fycode=2015&orgtypecode=6&

You write: "If charter schools shed kids year after year -- especially if those kids are low-performing -- then their vaunted performance advantages are in question, particularly when compared to public district schools that aren't losing students."

Yes, but in fact the evidence demonstrates that charter schools in this area shed considerably fewer kids than public district schools.

Stephen said...

I guess you recognize that may well be the case so move swiftly on to change the subject:
"What we really need to consider is whether the students moving out of charter schools are being replaced at rates equal to the replacement rates for students moving out of public district schools."

Your conclusion there is correct: there's far less backfill in the charter sector.

You write: "Every independent charter school in Boston had a higher cohort attrition rate in 2014 than BPS as a whole."

Is this a specifically New Jersey use of the phrase "cohort attrition rate"? I find it difficult to locate a definition in the academic research literature that combines what is normally considered to be attrition (loss of students) with backfill (gain of students) and calls the merged product "attrition". Though I do see it used that way by both you and Bruce Baker and in New Jersey.

In respect to what may constitutes a "cohort" in respect to individual schools in Boston, it appears that you blend members of multiple entering classes, and of multiple cities. For example, you write:

"In the case of City on a Hill and Phoenix, their 2011 freshman class shrank by more than half by the time they were seniors."

Phoenix doesn't have a school located in Boston. Perhaps you are thinking of the Phoenix charter school not too far away in the city of Chelsea. But given the nature of the school's specialization it makes it doubly hard to justify comparing it with a traditional public school in Boston:

"The school she founded, Phoenix Charter Academy, was conceived as an alternative school for students who had given up on traditional high school, or who felt that their schools had given up on them. As it celebrates its 10-year anniversary this week, it remains a haven for students who thought they might never finish high school."
https://www.bostonglobe.com/metro/2016/03/31/chelsea-charter-program-that-reached-struggling-kids-reaches-milestone/KYZKbHYHdsqCVD6XM8xxeI/story.html

Stephen said...


(had to break this up due to message size limit)
As for City on a Hill, you omit the implications of this, from its annual report: "There is no social promotion at City on a Hill. 100% of students promoted to the next level in each subject demonstrated mastery of the school’s common-core aligned college prep curriculum by earning 70% or above on written and oral proficiencies."
http://www.cityonahill.org/wp-content/uploads/2015/08/City-on-a-Hill-2014-15-Annual-Report-Final.pdf

Presumably as a consequence of that policy City on a Hill has extremely front-loaded grade level retention. The 9th grade class enrollments you examine merge students who are newly starting out and others who are repeating the grade. That distorts any attempt to compare 9th grade and 12th grade enrollment figures to measure attrition.

Massachusetts grade-level retention reports can be found here:
http://www.doe.mass.edu/infoservices/reports/retention/

Looking at Appendix B for 2015-2016, one finds for City on a hill the following grade-level retention rates:
9th 10th 11th 12th
27.8 3.3 3.2 7.4

In the past, in the several messages here, you and I discussed whether such a front-loaded grade-level retention pattern skews the results you and your NJ colleagues try to portray:
https://dianeravitch.net/2016/07/14/jersey-jazzman-takes-charter-cheerleaders-to-the-woodshed-for-destroying-public-schools/#comment-2582329

I continue to believe it does.

The one clearly valid point you make, not a novel one, is that there's less backfill at charter schools here in Massachusets charter schools than at traditional public schools. I'd be interested to see you debate with Michael Petrilli on whether or not that's helpful:
https://edexcellence.net/articles/backfilling-charter-seats-a-backhanded-way-to-kill-school-autonomy

You write: "Is it really worth expanding charters and risking further injury to BPS when the charter sector appears, at least at the high school level, to rely so heavily on cohort attrition?"

Further injury? Could you explain what you meant by that? This has what has been transpiring at BPS in parallel with the growth of charter schools:
http://www.bostonpublicschools.org/domain/238

ciedie aech said...

In our district it has been these non-filled classes which then get held up to the nation as being MUCH IMPROVED.

Unknown said...

I'm posting the following comment on behalf of my colleague Monty Neill, who was having trouble posting:

Good post, thank you. While MA has a ‘reimbursement’ law that is more generous than most states, the legislature for years has only partially funded it, leaving districts in an even bigger hole. While duplicative admin costs are part of the problem, the big one for public schools is that if a few students leave each of several grades, the school cannot lay off the teacher. But it has to cut, so it lays off the art or music teacher, or buys no new books or technology. Thus, Boston students know well the cuts they have been facing in their schools – and is a key reason thousands of them walked out to protest the budget last spring. Monty Neill

Unknown said...

JJ, still curious to read your thoughts on whether the methodologies used in the many studies showing superior student performance in MA charters controls for the "backfill" issue you claim exists. I still content that they do.

Regards,

Nat

urbansocrates said...

The attrition rates in urban schools have often been left out of the picture by DESE, which tends to focus on each school as a unit full of students marching through in cohorts with little attrition. But comparing attrition rates (real ones, that include attrition during the school year) is key to understanding the problems of urban education. High attrition skews standardized test scores downward and undermines their use as a measure of school performance, yet the DESE's reports overlook this. When I worked in a Worcester public school, the average attrition rate over the course of a year in a Worcester school was about 40%. The superintendent noted that the MCAS scores treated school populations as a unit, when in fact they were not.

In this instance, charter schools, by filtering out students with high mobility, have given themselves an edge in claiming superiority. That's why this complex issue is a poor one for voters to decide on -- it takes a lot of work to get the facts right and understand them, something that ballot question voters are unlikely to do.