I will protect your pensions. Nothing about your pension is going to change when I am governor. - Chris Christie, "An Open Letter to the Teachers of NJ" October, 2009

Monday, August 21, 2017

The Secrets to Their "Success"

"THE" Jose Vilson had a Twitter thread up this past week about an encounter with a parent whose child is enrolled in one of the Success Academy Charter Schools. I won't quote it here because I really want you to read the whole thing, then come back so we can talk...

Whenever I hear anecdotes about charter schools, my first reaction is to go to the data. Not because I don't think stories like this parent's are worthwhile -- to the contrary, they are very important and should be told. But I do believe data can help to confirm what we might already suspect.

And what do I suspect about Success Academy?

I've been teaching long enough to know that schools and teachers vary significantly in their effectiveness, and both can make a difference in the lives of children -- particularly children who are living in economic disadvantage.  But I also know that the reformy claims of "miracle" schools are almost always way overblown. Yes, some charter schools get better results than we would expect. Yes, some may engage in a few innovative practices that might be worth considering.

But schools like Success Academy almost always have structural advantages -- advantages that have nothing to do with their governance -- over the schools against which they compare themselves:
  1. Different student populations.
  2. Resource advantages.
  3. A less-experienced, less-expensive faculty.
  4. A longer school day/year and/or smaller class sizes and/or tutoring, made possible by #2 & #3 in combination with free-riding on the public district schools.
  5. Strict disciplinary codes which encourage students who do not thrive in a "no excuses" environment to leave.
In the minority of cases where "successful" charters out-perform expectations, I have seen no compelling evidence that freedom from teachers unions and public district school regulations, curricular innovations, or parental "choice" are what lead to "success." Instead, some combination of the five factors above almost always provide the most reasonable explanation for the difference in outcomes.

For years, I've been going to the data when confronted with a "miracle" charter school. For years, I've seen the same patterns emerge. Of course, there are many ways to approach this kind of analysis; let's apply one, which I first learned from Bruce Baker (I know, you're so surprised...), to Success Academy's oldest school and see if we can find any secrets of their "success."

The idea is simple: we'll look at Success Academy Harlem 1 and all the other schools that share its zip code. This way, we'll only be comparing schools situated in a similar geographical area.

To make an apples-to-apples comparison, we have to consider the grade levels that schools enroll. In the 2014 federal data, Success Academy Harlem 1 had a handful of Grade 9 students; most were in the lower grades. In another charter in the zip code, about half the students were in high school; two other NYCDOE schools in the area were mostly high schools. Keep this in mind as we continue.

Success Academy Harlem 1 enrolls a smaller percentage of Students With Disabilities (SWD) than its comparable neighboring public district schools. Compared with the other elementary schools in the neighborhood, SA enrolls far fewer special needs students proportionally; that's true for Harlem 1 and Harlem 4. It's worth noting that the school with the highest SWD percentage in the area, Opportunity CS, was threatened with closure last spring for poor performance (I can't find an update on its status for this fall).

Success Academy Harlem 1 enrolls a smaller percentage of Limited English Proficient (LEP) Students than its comparable neighboring public district schools. Again, compared with just the elementary schools, the difference is very substantial.

Success Academy Harlem 1 enrolls a smaller percentage of economically disadvantaged students than the neighboring public district schools serving comparable grades. Certainly, a 70 percent ED rate is high. But it's not as high as the elementary schools within the same zip code.

Let's look at the cohort attrition patterns. This is the size of the "class" of students in each school as those classes move from grade to grade. Here's the Class of 2018 (the year is when the cohort will be seniors in high school).

Success Academy loses many of its students as the cohort moves from grade to grade. Leo Casey of the Shanker Institute used a different data source and found the same patterns. Here's the Class of 2019:

So what does the data tell us about Success Academy Harlem 1 compared to its geographic neighbors run by NYCDOE?
  • Fewer students with disabilities.
  • Fewer Limited English Proficient students.
  • Fewer economically disadvantaged students.
  • Higher cohort attrition rates.
Something else we know about SA -- their philanthropic infrastructure gives them an enormous resource advantage:
As those of us who follow education policy and live in the greater-NYC area know, Moskowitz's Success Academy has benefitted enormously from philanthropic giving. $8.5 million from a hedge-funder in 2015. $9.3 million in one night later that year. $25 million from another in 2016. Plus another $10 million from some pikers...

This is all in addition to the monies SA gets from the government for the students it enrolls. Moskowitz's powerful friends have even made sure that she doesn't have to play by the rules that everyone else must follow. The result is a school system swimming in money -- a system that relies on funds that no one else gets to access.
Couple this with the widespread reporting about "got-to-go" lists and significant teacher turnover -- which undoubtedly keeps staffing costs lower -- and it's clear that there really is no secret to all the "success."

Now, I'm not about to say that this automatically shuts down any conversion about whether SA, or any other "high-performing" charter chain, should be allowed to exist, let alone expand. There is a serious debate to be had about segregation, class, race, and schooling. The sad truth is that reformers are correct in pointing out that we have highly segregated and underfunded public schools in our urban cores, and many parents perceive that a no-excuses charter is the best option they have for their child.

But claims that Success "succeeds" because it is free from union influence, or because it isn't directly accountable to a democratically elected school board, or because its teachers weren't trained in traditional, university-based teacher prep programs should be challenged. The data makes clear there are significant differences betweens SA's student population and those of its neighboring public schools. 

This matters. They can bring up test scores all you want, but unless and until SA's cheerleaders -- and the charter school backers across the nation -- start being honest about why a small group of them do better than predicted, support for charter schools will continue to plummet. I'm not saying Betsy DeVos and Donald Trump don't have something to do with that.

I am saying that it's well past time for the charter industry to start being honest with itself -- and the rest of us.

Happier times.


kenny said...

how do we account for the fact that, when compared to some of the wealthier school districts in the ny area, like jericho, scarsdale, chappaqua, it outperforms them? i would think those schools are flush with resources as well?

Stephen said...

"In the minority of cases where 'successful' charters out-perform expectations, I have seen no compelling evidence that freedom from teachers unions and public district school regulations, curricular innovations, or parental 'choice' are what lead to 'success.' "

In respect to impacts of teachers unions, I haven't seen anything more persuasive than this in Chapter 6, "Are Teachers Unions Good for Students?" by Dan Goldhaber from "Collective Bargaining in Education: Negotiating Change in Today's Schools":
"Eberts and Stone find the total (direct and indirect) effect of collecive bargaining agreements is to raise student achievement by about 3 percent over that of districts without such agreements. However, this increase in achievement comes at a cost, as the operating expenses in districts with such agreements are about 15 percent higher per pupil than in comparably achieving districts without such agreements.
"An interesting finding from this study is that the effects of bargaining are not the same for all students. The overall 3 percent increase in student achievement is driven mainly by a large 7 percent increase in the achievement of students whose test performance is close to the mean. Students who are either well below or well above the mean are marginally disadvantaged by the existence of collective bargaining agreements."

Not being expert in the subject, I'd be interested if you have any different sense of what's most valid in that respect. I've been especially troubled when I've seen teachers unions limit use of low-cost tutoring systems.

Stephen said...

Your focus on a charter school having fewer students with disabilities or fewer who are proficient in English is unpersuasive. You don't demonstrate, for example, that an ELL student would be expected to make less rapid progress than a non-ELL student. That would actually seem counter-intuitive to me. And you don't attend to the fact that a successful school may have a markedly diminishing number ELL students as they progress through school, because the kids become proficient in English, and fewer SWD, because students lose that classification once they start making good progress in a regular classroom.

Elizabeth Setren looked closely at such issues in respect to Boston charter schools:

"At the same time, critics of charter schools not only note that special needs students appear underrepresented in charters, but they also question whether charters serve these students well. Perhaps urban charters’ remarkable achievement gains are generated in part by a tendency to focus on non-special needs students.

"This paper reports new lottery-based evidence of charter effectiveness for special needs students. The results show that special education and ELL students experience large academic gains in charter schools: over 0.26 standard deviations in math and over 0.19 standard deviations on English on the state standardized exams. These gains are similar to those made by non-special needs students in charter schools. Charters also significantly increase the likelihood that special needs students meet a key high school graduation requirement, become eligible for a state merit scholarship, and take an AP exam. Special education students in charters score on average 115.7 points higher on the SAT than their traditional public school counterparts.

"Charters generate academic gains even for the most disadvantaged charter applicants. Special needs students who scored in the bottom third on their state exams in the year of the lottery experience large positive effects of over 0.22 standard deviations in math.English Language Learners with the lowest baseline English exam scores have the largest gains. Students with the most severe needs– special education students who spent the majority of their time in substantially separate classrooms and ELLs with beginning English proficiency at the time of the lottery–perform significantly better in charters than in traditional public schools.

"I also document striking differences in special needs classification practices in Boston charter and traditional public schools. Charter enrollment nearly doubles the likelihood that a student in special education at the time of the lottery loses this classification by the beginning of the following school year. Moreover, charters are three times as likely to remove an ELL classification. Charters are also three times more likely than traditional public schools to move special education students into general education classrooms. Classification practices are weakly correlated to charter gains, suggesting that special needs classification is not essential for special needs students to make progress."

Stephen said...

And, of course, JJ, your continuing resort to "cohort attrition" analysis as a substitute for actual attrition analysis is increasingly difficult to explain generously. As you very likely know, in respect to Success Academy, a close look at its actual attrition figures by Beth Fertig/WNYC showed those to be lower than average for NYC district schools, lower than average for NYC charter schools and only 57.4% as high as nearby district schools.

Duke said...

Stephen, you're welcome to leave comments if you wish.

But don't you have your own blog?

Stephen said...

JJ: "But don't you have your own blog?"

If I do, I'm not sure where it is or how to log into it.
I did open up one of those twitter account things and sent out a tweet, but like a leaf descending in the wilderness, it may have gone unnoticed.

Anonymous said...

I am curious about the claim that a less experienced faculty is an advantage for charter schools. Many threads on public education blogs (Dr. Ravitch's blog, for example), well qualified and experienced teachers are viewed as essential for students to be effectively educated.

If, as you say, having relatively inexperienced teachers is an advantage, the traditional public school contracts which reward teacher longevity and additional education, are structured incorrectly.