"One Newark": A Market For "Lemons"

I've got a new brief over at the NJ Education Policy Forum out today about One Newark. As usual, I present my geekspeak-free (mostly) version of this brief here.

Here's the background: Cami Anderson, the state-appointed superintendent of Newark, has put out a plan to restructure the district's schools. Bruce Baker* and I have written a lot about the flaws in this plan, called One Newark: see here, here and here for our critques. The central problems are:

• Certain schools were targeted for closure, "renewal" (where all of the staff is fired), or takeover by charter management organizations (CMOs), but the sanctioning of these schools, which are supposedly "ineffective," seems to be arbitrary and capricious.
• Newark Public Schools clearly did not take student characteristics into account when rating the schools' effectiveness. In other words: they don't seem to care that a school with many kids in economic disadvantage, or who don't speak English at home, or have special needs, has a far tougher job than a school that doesn't serve as many of these kids.
• The sanctions under One Newark are applied unequally under race and class lines: more kids who are black and economically disadvantaged will see their schools sanctioned, and more teachers who are black and teach the poorest students will lose their jobs.
• To be frank: NPS does not appear to have a very good command of statistical methods. But even when judged by NPS's own criteria, the plans they offer make no sense.

This last one is the focus of the new brief, and it relates to how NPS is "selling" Newark's schools -- both charter and public -- to its families.

Under One Newark, every family fills out an application, listing their top eight choices. These choices include some, but not all, of the charter schools in the city. What's really surprising (and, to me, more than a little disturbing) is that NPS rates each school on the One Newark application itself.

So as a parent looks at the choices for her child, she will see that each school is listed as "Falling Behind," "On The Move," or "Great." Here's a part of the paper application (click to enlarge):

Think about this a minute: Anderson's NPS administration says that some of Newark's schools -- schools Anderson and her staff themselves are ultimately responsible for -- are "Falling Behind." It's a remarkable assertion, and one that has profound consequences: would you enroll your child into a school that, on the very application you must fill out, is listed as "Falling Behind"?

If NPS is going to make claims about these schools, they must have the data to back those claims up. And logic suggests that they must be taking student characteristics into account; otherwise, how could a school that serves poorer children and more children with special needs possibly "compete" with a school that does not?

What I've done in this brief is use a basic statistical tool -- linear regression -- to show that, when accounting for student characteristics, many of the schools NPS says are "Falling Behind" are actually doing quite well. Conversly, many of the schools that are "Great" are really "great" only because they serve far fewer special needs, Limited English Proficient (LEP), and free lunch-eligible (FL) students than most of the other NPS schools. 

Here's the breakdown:

I want to make an important point here: the prediction model I'm using is a model that NPS's analysts themselves used when answering our initial critique of One Newark. NPS themselves said that averaging scale scores is the preferred dependent variable (the brief points out the problems with this); NPS themselves said that a linear regression with these covariates is valid. I'm not substituting their dependent or independent variables**; this entire exercise is based on what NPS's analysts themselves say is valid.

Except it's clear it isn't. In the One Newark application above, Miller Street is "Falling Behind." But look at the graph: Miller Street is above prediction, based on NPS's own model! How can they justify labeling this school as "Falling Behind" -- right in the application itself -- when their own model says it isn't?!

I've got some other stuff in the brief about SGPs, another component of NPS's rating system (a system, by the way, that they have not, to my knowledge, ever explained in detail). But here's the real kicker:

If the rating system in the One Newark application isn't telling us about a school's effectiveness, what is it actually rating? Well, if you look at the student demographics of the schools in each classification, a pattern emerges:

Under One Newark's rating system, "Great" schools are actually the ones that have fewer black boys who are economically disadvantaged and/or have special eduction needs.

This is an ugly truth: "Great" schools, on average, have only 8 percent of their student population classified as needing special education services; "Falling Behind" schools more than double that percentage. "Great" schools have fewer children who qualify for free lunch, and significantly fewer black children. Even the difference in gender is significant.

So the rating system of One Newark is really about who goes to the school, rather than how effective the school is as measured by test scores (a dubious way to measure "effectiveness" to begin with).

The title of the brief comes from a classic economics paper written by George Akerlof. The paper asserts that markets are skewed by "asymmetrical information": when the seller knows something the buyer does not. Akerlof uses the used car market as an example: he contends that the effect of this asymmetry is not only more "lemons" on the road, but fewer "non-lemons" available to buyers. 

This is exactly what is happening in Newark. "Non-lemons" like Hawthorne Avenue are being turned over to private charter operators, taking away families' options to send their children to a relatively effective school. "Non-lemons" like Miller Street are tagged as "Falling Behind" when they really aren't. Meanwhile, only one Newark charter school is tagged as "Falling Behind," while several don't beat prediction (including some that interestingly declined to be part of the One Newark universal application).

I think "portfolio" districts are a bad idea for any number of reasons (more on this later this summer). However, if NPS is committed to this structuring of Newark's schools, they need to take the role of an impartial arbiter and an unbiased source of good "consumer information" for Newark's families -- kind of a Consumer Reports for parents. This brief shows they are anything but. The information they are providing to parents cannot be justified; it doesn't make sense, even by the district's own criteria. 

We have lemon laws to protect used car buyers; shouldn't we have at least the same protections for families that are forced to be in "choice" districts? 

One Newark: buyer beware!

* Bruce Baker is my advisor at Rutgers in PhD program.

** I am adding gender as a covariate, which is not only perfectly acceptable, but warranted in this case.