How NOT To Evaluate Education Policy: A Newark Example

One thing I've learned from years of writing on education policy: the more convoluted the talking point, the more likely it's twisting the facts.

For example [all emphases below are mine]:

Today, African-American students in Newark are four times more likely to go to a quality school than they were in 2006. 

That's from a piece by Kyle Rosenkrans, who is launching something called the New Jersey Children's Foundation (because what we really need around here is another education "advocacy" group...).

I've seen variations of this talking point in other places:

“There is not a city in America that has experienced a greater expansion of educational opportunity than Newark over the last decade,” said Ryan Hill, CEO of KIPP New Jersey charter schools. He points to test score results showing African American kids in Newark are now four times as likely to enroll in a school where students outperform the state average than they were in 2006.

At least Hill is specific, unlike Rosenkrans's use of the watery term "quality school." But where did this data nugget come from? I can't trace the source of the "four times" claim, but the same methodology was used back in 2015, when Andrew Martin, who also worked for KIPP-NJ, wrote a piece in The 74:

The percentage of black Newark students attending a school that beat the state proficiency average has tripled in the past 10 years, and this increase can be attributed almost entirely to the growth of the charter sector.

Martin's method was updated in 2017 by Jesse Margolis in a report on Newark's schools:

Since 2014, the share of Black students in grades 3-8 in Newark attending a school that beat the state proficiency average has risen dramatically, reaching 27% by 2017. Black students in Newark are now three times more likely to attend a school that performs at or above the state average then they were in 2009, prior to the reforms.

So this talking point has been passed around and updated for four years... but, so far as I know, no one has stopped and asked whether this is a valid way to assess improvements in Newark's schools.

In my view, it isn't.

Let's set aside the very real problem of making any comparison between two groups on proficiency, simply noting for now that proficiency is a binary measure (you're either proficient or you're not) that can mask student growth or decline that does not cross the proficiency threshold. We'll also set aside the problem of comparing two groups -- here, black students in Newark and all other students in the state -- without making adjustments for socioeconomic status, special education needs, limited English proficiency, prior test score achievement, resources, and a raft of other things that affect test scores.

Instead, let's ask a simple question: Can a district improve the likelihood of students attending a "good" school without actually improving student achievement?

Yep:

Here's a hypothetical school district with 7 schools, each with 100 students (I am keeping the numbers simple to make this easy to follow). Let's say a school whose proficiency rate is 50 percent or higher is "successful." In our example, only one school meets this mark. The 100 students enrolled are in a "successful" school -- regardless of their own proficiency. 

The other schools are not "successful": only 20 students out of 100 are proficient. The probability of attending a "successful" school in this district is 14 percent.

Now a new school comes along and draws students from each "failing" school: 5 who are proficient, and 5 who aren't. Again, no individual students have changed their proficiency status. What happens?

60 more students are now in a "successful" school -- even though no individual student has moved from non-proficient to proficient. The district now has 160 students in "successful" school; the probability rises to 23 percent -- all without changing any individual student's proficiency!

But the district keeps hearing how great it's doing, so it keeps shuffling students around:

The new school draws another 10 students from each "failing" school: 5 proficient, 5 not. By now, the district realizes its schools are inefficiently small; so it closes one of the "failing" schools and distributes the students to the other "failures."

Again: no student has changed in their individual proficiency. But the probability of attending a "successful" school has shot up!

220 students are now in a "successful" school; the probability is now 31 percent, more than double.

Any district can easily raise the chance of attending a "successful" school simply by shuffling kids around.

This appears to be what happened in Newark. We know that charter school students tend to have higher prior achievement than NPS students; the Harvard report on Newark reforms said so. Simply clustering more "proficient" kids into charter schools will raise the probability of students being in a "successful" school.

So, no, this is not a valid way to judge whether Newark's schools have improved. Any measure of school effectiveness that can be increased without improving student outcomes isn't something we should be talking about.

So let's not.