Bad Education Researcher Media Appearances

Apparently, according to NPR, the only people casting doubt on charter school research who matter are those who are normally charter cheerleaders:

There have been lots of skirmishes over charter school data over the years. But few have created as big a ruckus as the 26-state study of charter schools released recently by Stanford University's Center for Research on Education Outcomes, or CREDO. 

[...] 

None of these findings were in dispute. But when Jeanne Allen looked at the study, it upset her. 

"The way that CREDO has manipulated data and made conclusions about policy based on that data is absolutely 'un-credible,' " she says.

Uh-oh - reformy turf wars! What's the problem, Ms. Allen?

Allen heads the Center for Education Reform. She loves charter schools and would do anything to support them — short of endorsing a study that she says makes bogus comparisons between charter school kids and regular public school kids 

"They compared those students to students that don't even exist," Allen says.
In other words, she says, the CREDO study did not compare real kids to real kids. Instead, researchers took selected data and created a "composite" student to represent public school kids. 

Yes, they did: CREDO created "virtual" twins of charter school students in an attempt see whether the charter school had an effect on student learning. Contrary to Allen's assertion, this is a perfectly valid research method - as long as we acknowledge its many limitations.

Which continues to be the central problem with the CREDO study: it's not that the study uses a particularly bad methodology, but rather that ideologues (who would normally count Allen among their ranks) are taking huge, unjustified leaps in applying the study's conclusions to actual policy making.

Here in New Jersey, for example, Education Commissioner Chris Cerf used the report to justify his expansion of charters, patting himself on the back in the process for closing a few of the poorly performing schools. Cerf never seems to understand, however, that many of the "positive" outcomes of charters are most likely due - at least in part - to peer effect. As Bruce Baker says:

There just aren’t enough non-disabled, non-poor, fluent English speaking females in Newark to fully replicate district-wide the successes of the city’s highest flying charters.

Baker has taken the lead in showing that "successful" charters often engage in patterns of student attrition and cream-skimming: in other words, "successful" charters do not educate the same children as their neighboring public schools, which is why they aren't replicable on a large scale. Gary Miron, Ed Fuller, and Julian Vasquez Heilig have all done important work on this topic as well. For the record, I'd like to believe I've helped build this case, too: see here, here, here, here, and here for a few examples.

Add to all this evidence the fact that "successful" charters like KIPP tend to spend a lot more per student than neighboring public schools, and you've got the makings of a pretty good case that the "secret sauce" of charters just isn't very spicy. Recruit and retain the best kids, spend more, and counsel out the kids who don't easily meet the expectations - that's pretty much all you need to do to run a "successful" charter school.

Unfortunately, the very people who authored the CREDO study are overstepping their bounds and claiming that their study shows another reason for some charter schools' "successes":

Like previous studies, the one from CREDO concluded that kids in most charter schools are doing worse or no better than students in traditional public schools. About a third, though, are doing better. And that's a big jump from four years ago. 

The gains among blacks, Latinos and kids whose first language is not English have been impressive and surprising, says CREDO Director Margaret Raymond. 

"The fact that we can show that significantly disadvantaged groups of students are doing substantially better in charter school in reading and math, that's very exciting," she says. 

More and more charter school students are doing better, Raymond says, because they're getting anywhere from three to 10 extra weeks of instruction compared to their public school counterparts. 

"The average charter school student in the United States is benefiting from additional days of learning," she says, "compared to where they were four years ago and compared to traditional public schools they otherwise would've attended. [emphasis mine]

No. No, no, NO! Let's be very, very clear here: the CREDO study did not measure student instructional time; consequently, the study cannot show any correlation between instructional time and test-based student outcomes. I went over the entire report once again before I posted this, and I can state without hesitation that there is no data - none - that shows whether charters engage in more or less instructional time than public schools.

Why, then, would Raymond make such a claim? Unfortunately, it gets back to her insistence on using a flawed metric to demonstrate student "achievement" gains: "days of learning." I've posted about this before: Raymond and other researchers continue to illustrate the differences in charter school learning "gains" by translating the number of additional items a student scores correctly on a standardized test into "days of learning."

Any working teacher will tell you this illustration is absurd on its face: children do not learn in a simple, linear fashion, so the notion that a charter gains "x days of learning" has no practical analogue in the real world. But Raymond has taken this nonsense even further: she's now conflating actual time in school with her illustration of student learning gains.

What's particularly sad is that I don't even think she's aware of what she's doing. She probably thinks this NPR reporter - and his listeners - understand that she's not talking about actual weeks of additional schooling, but is merely trying to make the "gains" more understandable but translating them into "weeks of learning." But go read the entire report; listen to the audio clip. Any listener not closely familiar with the CREDO study is going to come away with the impression that the study shows charter schools get their "gains" by providing more instructional days, even though the study says nothing of the sort.

So why does this matter? Because, for far too long, charter cheerleaders have not been honest when touting the benefits of charter expansion. They keep making the case that the "flexibility" charters enjoy, or a "no excuses" discipline plan, or their longer school year, or the notion that they just care so gosh-darn much is the reason for their success.

The plain truth is that the evidence keeps piling up: there is no "secret sauce." Charters that "succeed" get their gains from segregation, student attrition, and unequal allocation of resources. The CREDO study does nothing to help us confirm or deny this hypothesis: it does not address the question at all, which is why its usefulness is quite limited.

One last point: Claudio Sanchez, the NPR reporter on this piece, could have spoken with any number of highly respected education researchers who could have given him real insight into the CREDO study; Baker, Miron, and Vasquez Heilig are only three of the credentialed, highly capable scholars who could have easily broken this issue down for his listeners.

That Sanchez ignored them and chose instead to interview two think-tanky types - Allen and Nina Rees of the National Alliance for Public Charter Schools, neither of whom is qualified to offer an expert opinion on CREDO's research methods - speaks volumes as to how broken our nation's reporting on education policy has become.

This bottle is empty.