I will protect your pensions. Nothing about your pension is going to change when I am governor. - Chris Christie, "An Open Letter to the Teachers of NJ" October, 2009

Wednesday, September 23, 2015

Don't Believe the Data In "The Prize"

I have a new policy brief out this week at the NJ Education Forum about the (mis-)use of data in Dale Russakoff's The Prize. Let me add a few words here:

The Prize, as I described in my initial review, is the story of how then-Mayor Cory Booker and Governor Chris Christie worked to cajole a $100 million donation from Facebook founder Mark Zuckerberg to support the "reform" of the Newark school system,

Understand, I actually liked much of The Prize. I think Russakoff does a good job of describing the struggles an urban pubic district school faces when educating students whose home lives are far from ideal. I think she tells the story of the behind-the-scenes machinations of the power players well, and I think it's important to tell that story.


The Prize is turning out to be an influential book, and Russakoff is making many media appearances where she gets to present her ideas about urban school "reform." She was quoted in the NY Times by Joe Nocera, who used her figures to make the case that the Newark Public Schools (and, by extension, other urban districts) has a bloated bureaucracy compared to its neighboring charters. Connor Williams cites her figures at The Seventy Four to make the case Newark overspends on non-instructional budget items. Terry Gross hosted Russakoff on NPR's Fresh Air where, once again, she laid out the case that NPS wastes money compared to the city's charter schools, and gets better results for it.

Folks, no one has looked at this more than I have (and I've had a lot of expert help). And I will tell you without the slightest hesitation: according to state data, none of this is true.

Russakoff relies far too much on figures from NPS's high level administrators and far, far too much on proprietary data points from TEAM Academy Charter School, the Newark branch of the KIPP charter network. In fact, Russakoff makes the fundamental mistake (which is really amazing to me, as she is a veteran reporter with years of experience) of extrapolating what she sees at one charter school (SPARK Academy, part of TEAM/KIPP) out to the entire Newark charter sector.

Obviously, you can't do this. Yes, it's perfectly fine to illustrate your points with an example, but you have to have the facts on your side if you're going to make the leap and say your example is typical for the population you are studying. And the facts just don't support what Russakoff is trying to argue:

- Newark's charter schools do not devote more money to instruction than NPS. 

NPS spends just about as much "in the classroom" as TEAM/KIPP.

- NPS spends far more on student support services -- social work, counseling, nursing, child study team, libraries, etc. -- than any Newark charter school.

No charter comes anywhere closing to spending as much on student support as NPS, probably because they serve many more special needs students than the charters.

- NPS spends far less on administration than the charters. This is largely because Newark's charter schools spend a lot on administrative salaries.

NPS has the lowest administration costs in the city. Why?

Because their administration salary costs are low. Take a look at TEAM/KIPP's while you're here.

- There is no evidence that this spending is wasted compared to the charters: NPS deploys as many teachers, and far more support staff, than the Newark charters.

NPS deploys more support staff per pupil than most charters.

Russakoff uses TEAM/KIPP as her exemplar, but they don't deploy nearly the support staff per pupil that NPS does.

- NPS's physical plant costs, where we would expect to find much budgetary "bloat," are not extraordinary compared to the charters.

Plant costs are right at the median compared to charters.

I'm not making this stuff up with a stash of super-secret data that only I can see; this is all based on publicly available figures from the NJDOE. All you need is Excel and a little patience (OK, I have some help on that front, too).

There's more in the brief. No, Newark's spending isn't an "extraordinary sum in national terms." It's high, but not ridiculously high. No, there's no reason to believe an NPS school could save $800 per pupil per year on janitorial services. No, the charters really don't do better on student outcomes when you properly account for factors such as resource differences, peer effects, and student cohort attrition. 

No, there's no evidence the quality of teaching in NPS schools is poor compared to the charter sector; in fact, there's some reason to believe it may be better due to racial alignment and experience. No, there's no reason to believe state superintendent Cami Anderson had targeted the lowest performing schools in the district for "renewal."

Let me be clear: not all of the outcome differences between some Newark charters and the district can be explained away by population differences, resource differences, and attrition. At some of these charters, there may be instructional practices worth emulating. Too bad our local media only ever seems to care about one charter chain in Newark, which apparently has a very good PR department.

Further, there may well be "bloat" in the NPS budget; certainly, we should be looking at the high salaries for the top administrators in the system. There may well be deficits in the teaching corps at NPS, and those need to be addressed properly.

And most importantly: even though the data shows NPS does not "waste" money relative to the charters, that doesn't mean NPS has the funds it needs to do its job. The district has been underfunded according to the state's own law for years now. Charter schools, which were "held harmless" in the last state budget, are a significant contributing factor to this serious problem, especially because they are not, as a sector, serving the same population as NPS.

So I am not saying everything is fine at the Newark Public Schools - far from it. What I am saying is that Russakoff's diagnosis of NPS's ills is not supported by the evidence. The "facts" that Dale Russakoff has been presenting about Newark's schools, both in The Prize and in her media appearances, are either completely out of appropriate context or not supported by state data sources.

Read The Prize for its political tales; it's well-written and important. But don't rely on Russakoff's data points or her analysis of education policy: they just don't hold up. The book that accurately describes the state of Newark's schools and what they have to teach us about urban education policy has yet to be written.


Giuseppe said...

I listened to the Russakoff interview on FRESH AIR and ended up screaming at the radio, but as the rage escalated I almost smashed the radio on the floor. Geez, the usual BS about the wonderful charter schools and the "FAILING" district schools. Oh, did I forget to mention, for the zillionth time, the district schools are ******FAILING******!!!!!!!!!!!!!!!! and the charter schools are a success and are so much better than the >>>>FAILING<<<< district schools, blah, blah, blah, ad nauseam. I like Terry Gross and most of her shows, this felt like a knife in the back by an old friend, a betrayal. But it's just the usual media love fest for charter schools and the demeaning of those supposedly failing district schools. Will it never end? I wish Terry G would allow JJ and Bob Braun to counter all the BS against Newark public schools. I'm guessing that Terry is not especially knowledgeable about the whole school deform movement and just takes the pro charter people at face value and assumes that all their data and figures are accurate. In other words, Gross is clueless and maybe Dale is too. Dale is not a total pro charter robot but she certainly seems to think that the Newark charters are wonderful beyond words. Very disappointing. Though Russakoff did admit that charter schools did not perform that well nationally.

Christine said...

Thanks for this. Was listening to the Fresh Air segment and felt it needed correction.

Dienne said...

In addition to what Giuseppe said, I'd mention that schools are FAILING!!! because of all the BAD TEACHERS who are TENURED and protected by their big, bad UNIONS.

As great as Russakoff's story about Christie, Booker, Cerf and Anderson is, we on the pro-public education side need to be really careful about Russakoff herself. She can teacher and union bash with the best of them. Nowhere in the book does she question the motives or methods of "reform", just the fact that the big players in Newark managed to botch the implementation.

Jasoneducator said...

Perhaps the clue to why Russakoff doesn't do but so deep a dive into the data is in her anecdotes in the book. She looked at her KIPP school, contrasted it with a school that had Cami's attention and good teachers, and saw with her own eyes that there was more support staff in the charter. Also, there is the reality of her rubber room equivalent that's described in the book. I'm not on the ground, and I'm only starting my journey in educational research so I trust your data more than I do Russakoff's narrative and thank you for your balancing voice.