I will protect your pensions. Nothing about your pension is going to change when I am governor. - Chris Christie, "An Open Letter to the Teachers of NJ" October, 2009

Friday, May 23, 2014

"One Newark": A Market For "Lemons"

I've got a new brief over at the NJ Education Policy Forum out today about One Newark. As usual, I present my geekspeak-free (mostly) version of this brief here.

Here's the background: Cami Anderson, the state-appointed superintendent of Newark, has put out a plan to restructure the district's schools. Bruce Baker* and I have written a lot about the flaws in this plan, called One Newark: see here, here and here for our critques. The central problems are:

  • Certain schools were targeted for closure, "renewal" (where all of the staff is fired), or takeover by charter management organizations (CMOs), but the sanctioning of these schools, which are supposedly "ineffective," seems to be arbitrary and capricious.
  • Newark Public Schools clearly did not take student characteristics into account when rating the schools' effectiveness. In other words: they don't seem to care that a school with many kids in economic disadvantage, or who don't speak English at home, or have special needs, has a far tougher job than a school that doesn't serve as many of these kids.
  • The sanctions under One Newark are applied unequally under race and class lines: more kids who are black and economically disadvantaged will see their schools sanctioned, and more teachers who are black and teach the poorest students will lose their jobs.
  • To be frank: NPS does not appear to have a very good command of statistical methods. But even when judged by NPS's own criteria, the plans they offer make no sense.
This last one is the focus of the new brief, and it relates to how NPS is "selling" Newark's schools -- both charter and public -- to its families.

Under One Newark, every family fills out an application, listing their top eight choices. These choices include some, but not all, of the charter schools in the city. What's really surprising (and, to me, more than a little disturbing) is that NPS rates each school on the One Newark application itself.

So as a parent looks at the choices for her child, she will see that each school is listed as "Falling Behind," "On The Move," or "Great." Here's a part of the paper application (click to enlarge):

Think about this a minute: Anderson's NPS administration says that some of Newark's schools -- schools Anderson and her staff themselves are ultimately responsible for -- are "Falling Behind." It's a remarkable assertion, and one that has profound consequences: would you enroll your child into a school that, on the very application you must fill out, is listed as "Falling Behind"?

If NPS is going to make claims about these schools, they must have the data to back those claims up. And logic suggests that they must be taking student characteristics into account; otherwise, how could a school that serves poorer children and more children with special needs possibly "compete" with a school that does not?

What I've done in this brief is use a basic statistical tool -- linear regression -- to show that, when accounting for student characteristics, many of the schools NPS says are "Falling Behind" are actually doing quite well. Conversly, many of the schools that are "Great" are really "great" only because they serve far fewer special needs, Limited English Proficient (LEP), and free lunch-eligible (FL) students than most of the other NPS schools

Here's the breakdown:

I want to make an important point here: the prediction model I'm using is a model that NPS's analysts themselves used when answering our initial critique of One Newark. NPS themselves said that averaging scale scores is the preferred dependent variable (the brief points out the problems with this); NPS themselves said that a linear regression with these covariates is valid. I'm not substituting their dependent or independent variables**; this entire exercise is based on what NPS's analysts themselves say is valid.

Except it's clear it isn't. In the One Newark application above, Miller Street is "Falling Behind." But look at the graph: Miller Street is above prediction, based on NPS's own model! How can they justify labeling this school as "Falling Behind" -- right in the application itself -- when their own model says it isn't?!

I've got some other stuff in the brief about SGPs, another component of NPS's rating system (a system, by the way, that they have not, to my knowledge, ever explained in detail). But here's the real kicker:

If the rating system in the One Newark application isn't telling us about a school's effectiveness, what is it actually rating? Well, if you look at the student demographics of the schools in each classification, a pattern emerges:

Under One Newark's rating system, "Great" schools are actually the ones that have fewer black boys who are economically disadvantaged and/or have special eduction needs.

This is an ugly truth: "Great" schools, on average, have only 8 percent of their student population classified as needing special education services; "Falling Behind" schools more than double that percentage. "Great" schools have fewer children who qualify for free lunch, and significantly fewer black children. Even the difference in gender is significant.

So the rating system of One Newark is really about who goes to the school, rather than how effective the school is as measured by test scores (a dubious way to measure "effectiveness" to begin with).

The title of the brief comes from a classic economics paper written by George Akerlof. The paper asserts that markets are skewed by "asymmetrical information": when the seller knows something the buyer does not. Akerlof uses the used car market as an example: he contends that the effect of this asymmetry is not only more "lemons" on the road, but fewer "non-lemons" available to buyers. 

This is exactly what is happening in Newark. "Non-lemons" like Hawthorne Avenue are being turned over to private charter operators, taking away families' options to send their children to a relatively effective school. "Non-lemons" like Miller Street are tagged as "Falling Behind" when they really aren't. Meanwhile, only one Newark charter school is tagged as "Falling Behind," while several don't beat prediction (including some that interestingly declined to be part of the One Newark universal application).

I think "portfolio" districts are a bad idea for any number of reasons (more on this later this summer). However, if NPS is committed to this structuring of Newark's schools, they need to take the role of an impartial arbiter and an unbiased source of good "consumer information" for Newark's families -- kind of a Consumer Reports for parents. This brief shows they are anything but. The information they are providing to parents cannot be justified; it doesn't make sense, even by the district's own criteria. 

We have lemon laws to protect used car buyers; shouldn't we have at least the same protections for families that are forced to be in "choice" districts? 
One Newark: buyer beware!

* Bruce Baker is my advisor at Rutgers in PhD program.

** I am adding gender as a covariate, which is not only perfectly acceptable, but warranted in this case.


Giuseppe said...

It is really outrageous and despicable that some of those schools were labeled as falling behind. The teachers and staff in those schools must be so humiliated and demoralized by this bogus rating system. All their hard work and efforts are rewarded with stigmata; they have been crucified on the altar of school reform and school privatization.

Kelly said...

It is so contradictory that a school such as McKinley, that is labeled as "falling behind" and actually is underperforming based on the chart was not slated as a renew school, but a school like Luis Munoz Marin Elementary is labeled as "on the move" and is indeed performing slightly over the expectations on the chart is being made into a renew school. The entire staff fired, few rehired by way of "back door deals", and teacher morale at an all time low because most are scrambling to find jobs and support families.

To pour more salt in the wound, the supposed jobs posted on "Talent Match" are almost never really available. They're either filled within the school or through "word of mouth" before they're ever posted or, *get this*, if there are openings in an existing renew school, the Admin goes to TFA fairs to fill the positions with cheap, uncertified teacher-trainees from outside the district (they are encouraged to do that!). Meanwhile, many great teachers who've been in the district 8, 10, 15 years, are not being contacted for any positions they have applied for. Most of these loyal, highly qualified teachers will eventually be placed in the least desirable positions or end up in the EWP pool (along with the accompanying stigma).

The whole scam is unfair to students, families and teachers! And mark my words, there will be chaos in September when students will be scattered and lost across the city while staff will be left in the dark about who belongs where. Not looking forward to any of it.

edazare said...
This comment has been removed by the author.
edazare said...

hmmm. i wonder how many people choose the schools listed as "falling behind," unless they know more about it from friends/family.
I'm reminded of all the years that I received a mandatory letter stating that my child's urban school did not meet "adequate yearly progress" so we were allowed to switch schools. So disheartening. But since we were already in the school, and could see for ourselves that the teachers were amazing, and our child was learning well, we stayed.