NJ Charter Report Does NOT Compare "Peer" Schools

The NJ charter school report, put out by CREDO and Stanford University, is not a piece of hack junk. It is, however, sloppy in its use of language. Take pages 5-6:

Compared to the educational gains that charter students would have had in a traditional public school (TPS), the analysis shows that students in New Jersey charter schools on average make larger learning gains in both reading and mathematics. At the school level, 30 percent of the charter schools have significantly more positive learning gains than their TPS counterparts in reading, while 11 percent of charter schools have significantly lower learning gains. In math, 40 percent of the charter schools studied outperform their TPS peers and 13 percent perform worse. These school-level results are notably more positive than the analogous pattern presented in the 2009 report. [emphasis mine]

I've reread this paragraph several times, and there is only one way to interpret it: that CREDO looked at charters and their "counterparts" or "peers" and concluded the charters do better.

Except that isn't what the study did at all.

What CREDO did was look at all of the kids going to charter schools and find "twins" for these kids in the public schools. For example, if Michelle and Diane are both in Fifth Grade, both "*not in poverty," both not Limited English Proficient (LEP), both girls, both got similar test scores in 4th and 3rd grade and both don't have a special education need, they are "virtual twins." CREDO matched up these "twins" across the district, then compared their progress on test scores.

What they didn't do - and this is absolutely critical to understanding the report - what they didn't do was compare schools that were "twins." That would be impossible, because the "successful" charter schools in Newark have no peers: they have far fewer students in poverty, who don't speak English at home, or who have a special education need.

In other words: if Michelle goes to a charter and Diane goes to a public school, Michelle is much more likely to be surrounded by her "twins" than Diane. Is there anyone out there who believes that won't make a difference?

This gets right to the heart of the matter: whether these "successful" charters are replicable. Because if the secret is to segregate the kids, that doesn't really augur well for charter expansion.

via Bruce Baker.

* "Poverty" in the CREDO report is not handled well: it conflates "Free Lunch" eligibility with "Free/Reduced Lunch" eligibility. They are not the same, and the difference does matter. More to come.