I will protect your pensions. Nothing about your pension is going to change when I am governor. - Chris Christie, "An Open Letter to the Teachers of NJ" October, 2009

Sunday, June 30, 2019

The Facts About Newark's Schools: An Update

Thanks to, among other things, Cory Booker's presidential campaign, Newark's schools remain in the spotlight. Back in 2017, Bruce Baker and I released Newark’s Schools: The Facts in an attempt to give some context to the topic. The report is admittedly long, but the story of Newark's schools can't be told in a few talking points.

That said, if I had to boil down what we found, it would be the following:
  • Newark's school system gained significant resource advantages over comparable districts, through the Abbott rulings -- a series of court cases that directed more funds to a select group of disadvantaged New Jersey school districts -- and through the proliferation of charter schools, especially those run by large charter management organizations (CMOs) such as KIPP and Uncommon Schools.
  • Much of the vaunted "growth" in Newark's test outcomes can be explained by the transition from one form of the state test (NJASK) to another (PARCC). Other, similar districts close to Newark showed similar gains in student growth, suggesting Newark wasn't doing anything unique to realize its own modest gains.
  • While Newark's charter schools have resource advantages, they aren't particularly efficient producers of student gains.
  • Newark's high-profile charter schools enroll a fundamentally different type of student than the Newark Public Schools (NPS), the city's public school district. NPS enrolls more special needs students, especially those with costly disabilities. NPS enrolls far more Limited English Proficient students, and there are differences in socio-economic status, although this varies across the charter sector.
  • Newark’s high-profile charters show substantial cohort attrition: many students leave between grades 7 and 12 and are not replaced. As those students leave, the relative test scores of those schools rise. These schools also have very high suspension rates.
  • By their own admission, the high-profile charters focus intensely on passing the state tests. There is evidence they do not put as many resources into non-tested subjects as NPS. 
  • The charters have a unique resource model: they hire many teachers who are relatively inexperienced, yet are paid more relative to similar teachers. Those teachers, however, work longer days and years.
Now, these points stand in opposition to the conventional wisdom on Newark's schools, which says that superior leadership, superior instruction and curriculum, and a culture of "high expectations" has turned Newark's education system around. 

Let me be very clear on this: I'm not saying the city didn't make improvements in the way it runs its schools or educates its students. I am saying, however, that the influence of charter school instructional practices and school/community leadership has been largely oversold. In addition: the gains the city has made do not come anywhere close to overcoming the disadvantages of inadequate funding, intense segregation, economic inequality, and structural racism Newark's beautiful and deserving children must contend with every day.

The only way to understand this reality is to take the time to understand the context in which Newark's schools operate. A lot of folks would rather not do that; they'd rather believe a few well-crafted data points designed to uphold a preferred narrative. 

I find this point of view to be enormously frustrating. If we really care about improving the education of children in disadvantaged communities, we owe it them to take the time to get the facts right. Otherwise, we're going to learn the wrong lessons and implement the wrong policies.

To that end, I want to set down, for the record, a few more critical facts about Newark schools:

- There is nothing special about Newark's graduation rate increases. Back in 2016, I wrote a brief that tested the claim that Newark's (and Camden's) graduation rates were growing at an accelerated rate. The problem with the claim, I note, was that the city's graduation rate was being compared to the entire state's. But that comparison is invalid because many New Jersey high schools already have grad rates near 100 percent -- and you can't go higher than that!

And yet people who really should know better continue to make the claim. What they should be doing instead is comparing changes in Newark's graduation rate to changes for similar districts. Here's the latest data:


Yes, Newark's graduation rates have been climbing -- but so have the rates of similar districts. "DFG" stands for "District Factor Group"; DFG-A districts, like Newark and Camden, are the most disadvantaged districts in the state. Aside from an initial leap in 2012, there is little reason to believe Newark has seen outsized gains in graduation rates.

I am increasingly convinced these gains are due to a policy of "credit recovery," where students at risk of dropping out receive alternate instruction -- often on-line -- in core subjects. The quality of this instruction, however, may not be very good. This is an area of policy crying out for a meaningful investigation.

- The claim that more Newark students are enrolled in schools that "beat the state average" is largely pointless, because you can get gains in this measure simply by shuffling around students. I went into detail about this in a post earlier this year:


This is a great example of a talking point designed to sell an agenda, as opposed to illuminating an issue. It sounds great: more kids are in "better" schools! But the fact that the measure can be improved without changing any students' outcomes renders it useless.

- Newark's demographics have changed; the student population is fundamentally different compared to other DFG-A districts. Take, for example, child poverty:



There's some noise here, but in general Newark's childhood poverty rate, which used to be high compared to other DFG-A districts, is now matching those districts. And there have been other demographic changes.


Newark's Hispanic student population has ticked up slightly (again, we've got some noise in the data). But the proportion of Hispanic students in the other DFG-A districts has risen substantially. And that means...


Newark's proportion of Limited English Proficient (LEP) students has stayed relatively constant, while the other DFG-A districts have seen big increases.

You might wonder why I'm comparing Newark to other DFG-A districts, or why I'm going back to around 2006. The NJ Children's Foundation, a new education "reform" group, released a report that asserts Newark has made outsized gains in outcomes since 2006. What the authors don't do, however, is present a clear picture of how the student populations may have changed. 

The only attempt they make is to track free and reduced-price lunch eligibility; however, as I've pointed out before, this measure is increasingly unreliable as more districts, including Newark, have moved to universal enrollment for federally-subsidized meals.

There's little point in analyzing relative changes in student outcomes if student populations characteristics are shifting. Before anyone claims that Newark has seen outsized success, they should at least acknowledge how its demographic changes may impact test scores.

- Newark has seen gains in test scores relative to the rest of the state; however, those gains are nowhere near enough to indicate the district has closed the opportunity gap. One of the most interesting graphs from the NJCF report was this (p. 16):


As Bruce and I noted in our 2017 report: the change from the NJASK to the PARCC in 2014-15 explains a lot of the "growth" in Newark's test outcomes. When you have a leap that large in one year, there's little reason to believe it's because your district suddenly and dramatically improved its instruction; more likely, the gains are due to Newark aligning its curriculum better to the PARCC than other districts. That's not necessarily because Newark's students are now better readers and mathematicians; they may simply be better test takers.

In any case, while the district improved its standing relative to other DFG-A districts, it made very little change in its standing relative to the entire state. Again: while Newark may have seen its poverty rate decline more than other districts, it's still a disadvantaged district compared to all districts.

Getting gains on tests is better than not getting gains, or suffering losses. But the magnitude of those gains is important. The "percentile rank"* of Newark compared to all of New Jersey has not shifted so significantly that the disadvantages from which the city's children suffer are being substantially mitigated.

- Newark's high profile charters schools continue to have high rates of cohort attrition and/or student suspensions. Bruce Baker and I have made so many versions of this graph I've lost count:


North Star Academy Charter School, the Newark branch of Uncommon Schools, sees its cohorts -- the students in the "Class of 'xx" --  shrink substantially between Grades 5 and 12. The Class of 2019 had 254 students in Grade 5; that was down to 149 in Grade 12.


TEAM Academy Charter School shows similar patterns. One of the arguments I hear is that the attrition occurs because students place into competitive magnet or private high schools. But the attrition occurs between all grade levels, not just Grades 8 and 9. Something else is going on.


This is a new metric from the NJDOE: total days of school missed due to out-of-school suspensions (OSS). I divided this by the number of students (divided by 100) to get how many days were missed, on average, per 100 students. The rate is higher at TEAM than at NPS, but far higher at North Star. There's good reason to believe the suspension rate contributes to the shrinking cohorts at North Star -- and that likely affects test scores.

- Newark's charter schools employ strategies to gain resource advantages that can't be scaled up. I included this table in my last report in NJ charter schools:

Let's break this down: the teaching staffs in Newark's charter schools are far less experienced than those at NPS. This, among other factors, explains why the costs per pupil are lower at Newark's charters. But, as we pointed out in 2017, the high-profile Newark charter schools tend to pay teachers more than NPS schools given the same level of experience. For this extra pay, the charter teachers work a longer school day and year.

In the absence of other learning opportunities, it's good for kids to have more time in school; unquestionably, the extra time helps boost test scores. But it's difficult to imagine a city the size of Newark could recruit and maintain an entire teaching staff with this little experience. If extra time in school is good for students, the practical solution is not to keep churning the teaching staff; instead, the district should pay teachers more to work longer hours and years.

As I've noted before, there's also a real question as to whether the charters are "free-riding" on public school district teacher wages. In other words: would these charters still be able to recruit enough inexperienced teachers if those teachers didn't see a transfer to a public school -- with fewer hours, better work conditions, and eventually better pay -- in their future?


Look, I am all for Newark talking up its educational successes. The students, families, educators, and community leaders should be proud of their schools -- and, yes, that includes the charter schools. But when we enter the realm of state and national education policy, we've got to be more judicious. We need to see the larger picture, with all of its subtleties.

Pre-digested taking points and incomplete analyses don't help us get where we need to be. We have to move past preferred narratives and look at all of the relevant facts. I'll keep putting them out there.

I hope some of you will challenge yourselves to listen.




* One of these days, we're going to have to have a chat about whether "percentile rank" is a good measure of relative standing. I would argue it spreads out what may be very small differences in outcomes. More to come...

4 comments:

Stephen said...

JJ: "If we really care about improving the education of children in disadvantaged communities, we owe it them to take the time to get the facts right."

Sounds good. In your NEPC report regarding Newark: https://nepc.colorado.edu/sites/default/files/reviews/TTR%20Weber-Baker_2.pdf
you and Bruce Baker state:

"As we have noted in the past, the CREDO reports employ a virtual matching methodology that is only as good as the variables used to create matches to counterfactual students. This places a significant limitation on the interpretation of the effect sizes found, as the variables used to match students are crude, binary measures of student characteristics such as socio-economic status, special education status, and limited English proficient (LEP) status."
"This is highly problematic when studying a school district like Newark, which enrolls a high percentage of economically disadvantaged students, LEP students, and special education students.”

Your implication there seems to be that while the charter school student and VCR control may both be limited in English proficiency (LEP) and economically disadvantaged, the control records may represent students in the district schools who are more severely LEP and who are more dramatically economically disadvantaged.

One might find that persuasive… if one didn’t know, and as you omit to mention, that the charter students and controls in the CREDO research were actually also matched on a crucial non-binary characteristic: test scores.

In respect to attrition you define cohorts as the students in the "Class of 'xx'". In the past, you've silently assigned some unstated number of individual students first to one Class and then additionally also to another... although ultimately each of them will only be permitted to graduate once. Can we be confident that you haven't done any of that here? Starting your analysis with 5th grade would seem likely to helpfully reduce that. Any possibility of obtaining the actual numbers of those who depart schools and comparing those among Newark schools. Or the NJDOE still not make that possible?

JJ: "We have to move past preferred narratives and look at all of the relevant facts. I'll keep putting them out there."

As you do that... you cite charter schools as having up to 1/2 day per year of days lost per student due to suspension... might you wish to calculate how many day-equivalents they gain due to extended school days and years? Perhaps compare charter schools with TPS in regard to days lost from unexcused absences and dropout rates? You examine the percentage of LEP students... why not look at status when newly enrolled and FLNE status? Have Newark schools perhaps become increasingly adept at increasing English proficiency, termininating students' LEP status? Can you get a sense of that from the available data?

Best wishes.

Noah said...

Brilliant work. Thank you!

Duke said...

Stephen:

- Previous test scores are not going to capture variation that is explained by other variables. If LEP and FL/RPL were not relevant covariates, I doubt CREDO would include them. Our point is that the data used in these studies is limited; therefore, caution should be used when interpreting the effects. Further, when researchers have more refined data, as in the case of the CREDO study of Newark and FRPL, they should use it.

- I have no idea what you're saying I did "silently" with my attrition analysis. These are the fall enrollments from NJDOE, and they show shrinking cohorts as students pass grade-to-grade. And I did start with Grade 5.

- The issue of suspensions and charter enrollments has more to do, in my opinion, with pressures to leave a charter than with the amount of time spent in schools.

- You ask several questions I can't answer because the data is limited. I have repeatedly & publicly called for NJDOE to study these issues at the student level and report the results. I've told NJDOE officials in person they need to collect and disseminate better data on charters. Not sure what else I can do.

Stephen said...

I am not at all suggesting that "LEP and FL/RPL were not relevant covariates".

By only alluding to "crude, binary measures" and stating, correctly, that students vary in relation to each other within those measures you seemed to be implying as others have done, that the VCR control records might be less academically capable than the matched charter school student... The charter school may have ELL students but matched district ELL student may actually tend to be much less advanced in learning English...

Orfield and Luce made the same kind of argument:
"However, there are reasons to worry about the CREDO results. The studies match charter
students with traditional school students based on race/ethnicity, gender, English proficiency,
free/reduced price lunch status, special education status and grade level. The method is designed to
control for selection bias by creating a control group like those used in randomized experiments, but
the list of matching variables does not include anything that reliably captures parental engagement, a
primary source of selection bias in charter studies (Maul, 2013; Miron & Applegate, 2009).6 This
compromises the findings that are positive for charters – they may simply be the result of selection
bias."
http://dx.doi.org/10.14507/epaa.24.2203

But Orfield and Luce there also omitted mentioning that CREDO methodology includes matching baseline test scores, and acknowledging that would have pulled the rug out from their argument.

JJ: "I have no idea what you're saying I did 'silently' with my attrition analysis. These are the fall enrollments from NJDOE, and they show shrinking cohorts as students pass grade-to-grade. And I did start with Grade 5."

Standard practice among "cohort attrition" enthusiasts seems to be to take, for example, a 9th grade class of 100 students label each and every one them, say, the class of 2023. Then, if 3 or 13 or 23 of them repeat 9th grade, label each of those, additionally, as class of 2024. So the same student silently appears initially, in undisclosed fashion, in the lovely graphs as both class of '23 and '24... while only actually having an opportunity to graduate once... and being mourned as a member of the class of '23 who has supposedly been squeezed out/kicked out or who abandoned the school miffed at having lost a half day of its excellent services due to suspension. In this particular instances, where you start with 5th grade, that would seem to greatly reduce the incidence of front-loaded grade-level retention, which is good.

But I can't help but suppose that the whole "cohort attrition" riff could be abandoned if NJ education advocates put more effort into attaining actual attrition statistics. But I respect the fact that you yourself have "told NJDOE officials in person they need to collect and disseminate better data." Perhaps they're more obdurate than I imagine.