Data Abuse in New Jersey

We've now had over a year of ACTING Commissioner Chris Cerf's reign at the NJ DOE. Looking back, one overarching theme emerges:

Data abuse.

Cerf likes to present himself as a wonk, despite the fact that he is a lawyer by training and holds no advanced degrees in education. He love charts, graphs, figures, and "factesque" data. The problem is that he seems to believe the role of research is to support his ideology, rather than guide him to the truth.

Unfortunately for him, highly qualified education researchers have been watching the NJDOE. And their examinations of Cerf's presentations are quite damning.

Let's begin with Cerf's latest focus: poverty and student achievement. Bruce Baker of Rutgers takes down a particularly brazen bit of Cerf's mendacity:

I just couldn’t pass this one up. This is a graph for the ages, and it comes from a presentation by the New Jersey Commissioner of Education given at the NJASA Commissioner’s Convocation in Jackson, NJ on Feb 29. State of NJ Schools presentation 2-29-2012

Please turn to Slide #24:

The title conveys the intended point of the graph – that if you look hard enough across New Jersey – you can find not only some, but MANY higher poverty schools that perform better than lower poverty schools.

This is a bizarre graph to say the least. It’s set up as a scatter plot of proficiency rates with respect to free/reduced lunch rates, but then it only includes those schools/dots that fall in these otherwise unlikely positions. At least put the others there faintly in the background, so we can see where these fit into the overall pattern. The suggestion here is that there is not pattern. [emphasis mine]

Cerf seems to think that just because there are some schools that "beat the odds," poverty is nothing more than a pesky excuse. It would be like my telling Mrs. Jazzman that the gallon of ice cream I eat every night isn't causing me to gain weight, because there are people in the world somewhere who eat ice cream and never gain weight; also, there are fat people who never eat ice cream. QED.

What's slick about this is that what Cerf puts out there may, in fact, be true - but it's so twisted and removed from context that it actually conveys a falsehood. Matt DiCarlo takes Cerf to task for doing just this when dissembling about the "achievement gap":

The existence of a gap only tells you that there are differences in outcomes (e.g., scores) between two groups, which are almost always defined in terms of income or race. States and districts could have a large achievement gap, but still contain a relatively high proportion of students reading at grade level. For instance, a relatively affluent district could see virtually all third graders reading at grade level, but still have a significant achievement gap – with one group (e.g., high-income students) performing much higher than the other. Conversely, one could imagine a struggling district with a much smaller gap, but where virtually all students still scored below grade level. 

If the NJDOE wants to make the (perfectly compelling) case that something should be done to help the 40 percent of third graders who it says cannot read at grade level, then those are the statistics they should cite. But it’s a little perplexing to use these figures, by themselves, as a justification for their focus on race- and income-based achievement gaps, which are a different type of measure. The better argument would have been to simply say that the state’s overall scores are quite high, but that this masks the fact that many students are still struggling. Differences in scores between subgroups are usually an underlying component of overall performance, but the latter might be obscured by the former, and vice-versa. It’s critical to examine both.* [emphasis mine]

But there's a reason Cerf focuses so much on the "gap": it allows him to casually dismiss the fact that New Jersey's students do exceedingly well. And, as DiCarlo points out, the state is moving in the right direction:

The simple table below compares the change (between 2005 and 2011) in average NAEP scale scores for NJ students who are eligible for free/reduced-price lunch (lower-income) versus those who are not eligible (higher-income). I want to quickly note that these data are cross-sectional, and might therefore conceal differences in the cohorts of students taking the test, even when broken down by subgroups.*** 

 

This table shows that, in three out of four NAEP tests, both low- and higher-income cohorts’ scores have increased substantially, at roughly similar rates. In fourth grade math, students eligible for free/reduced-price lunch scored six points higher in 2011 compared with 2005, the equivalent of roughly half a “year of learning,” compared with a similar, statistically discernible five point increase among non-eligible students. The results for eighth grade math and fourth grade reading are more noteworthy – on both tests, eligible students in NJ scored 12 points higher in 2011 than in 2005, while the 2011 cohorts of non-eligible students were higher by roughly similar margins. 

In other words, achievement gaps in NJ didn’t narrow during these years because both the eligible and non-eligible cohorts scored higher in 2011 versus 2005. Viewed in isolation, the persistence of the resulting gaps might seem like a policy failure. But, while nobody can be satisfied with these differences and addressing them must be a focus going forward, the stability of the gaps actually masks notable success among both groups of students (at least to the degree that these changes reflect “real” progress rather than compositional changes).

So, compared to similar children across the nation New Jersey has high-performing kids who are both poor and not-poor. And, as Howard Wainer points out, the same phenomenon occurs when looking at race:

But focusing on the difference blinds us to what has been a remarkable success in education over the past 20 years. Although the direction and size of student improvements are considered across many subject areas and many age groups, I will describe just one -- 4th grade mathematics. In the figure, the dots represent the average scores for all states that are available for NAEP's 4th grade mathematics test (with New Jersey's dot labeled for emphasis). These are shown broken down by race (black and white students) as well as by year (1992 and 2011). We can see that there have been steep gains for both racial groups over this period (somewhat steeper gains for blacks than for whites). Of course we can also see the all-too-familiar gap between the performance of black and white students, but here comes Achilles. New Jersey's black students performed as well in 2011 as New Jersey's white students did in 1992. Given the consequential differences in wealth between these two groups, which has always been inextricably connected with student performance, reaching this mark is an accomplishment worthy of applause, not criticism.

The last thing that we see is that the performance of New Jersey's students was among the very best of all states in both years and for both ethnic groups. [emphasis mine]

Bruce Baker goes into this further, and knocks down the "solution" proposed by the Christie administration to the "gap": take more money from poor schools and give it to not-poor ones (yeah, that makes perfect sense...).

It's worth noting that these are three distinguished scholars who are relying on standard statistical and research techniques, using commonly available data, that they have published publicly so as to invite scrutiny. This stands in stark contrast to the secretive, ideological "research" put out by Cerf's DOE, which is produced by highly-paid hacks and stands as an embarrassment to the state.

I know that it's easy for some to be deceived by Cerf's apparent love of data. He is a charismatic speaker and clearly an intelligent man. He has a grasp of the basic language of statistics. He talks about wanting to help disenfranchised children with great sincerity.

But there is little doubt that he is twisting the data for his own ends - probably knowingly. Everything coming out of his office these days needs to be examined very, very carefully.