I will protect your pensions. Nothing about your pension is going to change when I am governor. - Chris Christie, "An Open Letter to the Teachers of NJ" October, 2009

Wednesday, January 9, 2019

Kids In Disadvantaged Schools Don't Need Tests To Tell Them They're Being Cheated

There was a very raw, very discouraging story in The Trentonian this past week that is worth your time, whether you live in Jersey or not. Here's an excerpt:
Over the past year, this newspaper spoke with high school students educated in the Trenton Public Schools (TPS) district. The interviews took place in the presence of an adult and the teens were granted anonymity to speak freely and honestly. Each interview started with vague questions, such as “What is it like to live in Trenton?” While some students also spoke about nice and community-oriented neighbors, each of the conversations began with a discussion about violence. 
[...] 
“The school smells like weed,” a highschooler said. “They smoke in the hallways and stairwells almost everyday.” 
While some students said school guards “try to stop bad behavior” and convince kids to stay out of trouble, others described guards as “too young,” with “not enough care” for what happens. 
“This guy told the security guards what was going to happen to him, but they didn't care enough to do anything about it, so he got jumped,” a teen said. “They don't take their job seriously.” 
And as for teachers: “I feel like it depends on whether they know the student wants to change,” a teen said. 
Students said some teachers will remain persistent in trying to convince a kid to stay out of trouble. But if they realize their advice is not improving behaviors, “they just give up.” 
“I think that’s why a lot of people say teachers don't care either,” a student said.
When asked to estimate the percentage who don’t seem to care about the students in their school, the majority of the teens said approximately 70-75 percent of teachers seem like they don't want to be there.
 
One teen suggested the teachers have cause for not caring: “They have to teach in Trenton and dealing with kids’ attitudes is just overwhelming for them after a little while. It's not getting better, it's getting worse.” 
Teens estimated 60-70 percent of students seem to not take school seriously. One teen described negativity as their greatest challenge living in the capital city. 
“Negativity seems to be everywhere in Trenton; you can't run from it,” she said.
With a toxic environment as described by teens interviewed for this report, it’s no wonder that the TPS district high school graduation rate for the class of 2017 was only 70 percent, according to state department of education data. That low figure is due in large part to Daylight/Twilight’s graduation rate of 34 percent. Both Trenton Central High School campuses graduated more than 80 percent of its 2017 class.
 
The statewide graduation rate in 2017 was 90.5 percent. The graduation rates both statewide and in TPS have gradually increased over recent years.
Again, this is a tough piece of writing. But I thought it was well worth it, because the whole story raises some very uncomfortable questions as states head into their budgeting seasons:

A few days ago, a New Jersey appellate court threw out regulations that made the PARCC Algebra I and English Language Arts (ELA) Grade 10 tests a requirement for high school graduation. One of the most prevalent arguments for the PARCC -- which is based on the Common Core standards -- is that if we didn't have high standards and tough tests to match them, we are "lying" to students.

This argument is aimed particularly at schools like those in Trenton: "failing" schools, as reformers are so eager to call them. Apparently, many of us have deceived ourselves into thinking everything is fine in places like Trenton. Worse, students and parents, according former SecEd Arne Duncan, believed the lie. Only the hard, cold reality of testing could free us all from our delusions.

Now, I've been around a lot of testing skeptics, and I can assure you I've never once met one who was convinced that schooling in disadvantaged neighborhoods was generally acceptable. I've never met a union official who believed schools in impoverished cities didn't need improving. I never met anyone who works in a school or advocates for public education who was fine with the opportunity gap that plagues so many children in this country.

But I'll set that aside and instead make this point: stories like the Trentonian's give us clear evidence that kids who are in these schools themselves know full well what is going on. They are saying, with unmistakable clarity, that their instruction is unacceptably poor. They are telling us many of their peers have given up and have no interest in school.

What are multiple administrations of standardized tests going to tell us that these kids aren't already telling us themselves?

As I said last time: I am all for reasonable accountability testing. Test outcomes have been used by researchers and advocates to show indisputable truths, such as:

1) When it comes to school, money matters. A lot.
2) Children who come from disadvantage need more resources to equalize their educational opportunities.

Some make the argument that we have to have tough tests to motivate schools to improve. Most of these people don't know the first thing about how these tests are constructed or what they actually measure, but even if they did: What, exactly, do they want to do with the information they get from these tests? 

Do they really want to deny students, who went to their classes and played by the rules, their diplomas? What will that accomplish? Will it make the students and teachers "do more with less"? Will it improve the lives of students who don't pass these much more challenging PARCC and upcoming PARCC-like tests?

The idea of requiring students to pass tests with current passing rates of 46 percent strikes me as both cruel and capricious. Cruel because denying graduates a chance to participate in the workforce or join the military or pursue higher education is unnecessarily harsh when these students did exactly what they were told to do. Capricious because we changed the rules on these students quickly and with little forethought, and we didn't even stop to ask if their schools have what they need to help students pass these tests.

Now, if these same people who are demanding more and harder tests were also the same ones demanding full, adequate, and equitable funding for schools, I'd be more amenable to their arguments. Unfortunately, however, these folks -- like Duncan -- don't ever put school funding at the top of their lists of policy preferences.

I've heard a lot of crowing from certain quarters about how great it is that Trenton is getting a new high school in 2019. Certainly, the kids deserve it... but why did it take so long when high school kids in Trenton have been attending a school that is dirty and dangerous for years?

The infamous "Waterfall Staircase" at Trenton Central.

And what about all the K-8 schools in Trenton that are in need of repair and renovation? Where is any urgency to address this?

A couple of years ago I gave a presentation to the Trenton Education Association. Here's one of the slides:



Why are plant costs so much higher in Trenton? Because it costs much more to maintain old buildings that weren't properly cared for over the years. TPS is playing a constant game of catch up. Why?



Because the district has been underfunded millions less than what the state's own law says it needs.

Again: if the folks demanding harder tests were proposing that Trenton, and all other districts, got the funding they needed to at least generate average test outcomes, I'd at least say they were being honest about what it takes to close the opportunity gap. But no -- what we get from them instead is school "choice" and finger-pointing at unions and test-based teacher evaluation and a lot of other stuff that has never been shown to work and/or been brought up to scale... and in some cases actually drains more money away from the schools serving the most disadvantaged students.

This kids in Trenton are telling us the their education is not equivalent to the schooling kids in the Windsors and Princeton receive. We don't need more tests to tell us that -- the kids themselves are saying it.

The only question left is: what are we going to do about it?

Tuesday, January 1, 2019

NJ Court Strikes Down Graduation Test; An Opportunity to Re-Think Testing?

Miss me?

Yesterday, a New Jersey appellate court struck down regulations that required students to pass two PARCC tests -- the statewide tests implemented a few years ago under the Christie administration -- as a requirement to graduate. Sarah Blaine has an excellent legal analysis you should read about the ruling. Let me add a few thoughts to it, coming less from a legal perspective and more from an educational one:

SOSNJ has posted a copy of the ruling on its Facebook page. The ruling notes that back in 1979 the NJ Legislature called for "'a statewide assessment test in reading, writing, and computational skills . . . .' N.J.S.A. 18A:7C-1. The test must 'measure those basic skills all students must possess to function politically, economically and socially in a democratic society.' N.J.S.A. 18A:7C-6.1."

As Sarah points out, state statutes are usually vague when it comes to things like setting standards; functioning "politically, economically and socially" will mean different things to different people. It's also not clear why the Legislature thought a test was necessary for graduation. Was it to uphold the value of a New Jersey diploma by making sure all holders could meet a certain standard? Was it to hold districts accountable for their programs? Was it to make sure the state, as a whole, was providing the resources needed for students to have educational success?

Or was it possible the Legislature never really thought about the purpose of the test? I'm not sure; I do, however, know that in the time since the passage of the law, psyshometricians have been giving long and hard thought to something called validity: in the most basic terms, whether a test measures what we want it to measure, and whether it should be used for the purposes in which it is used.

The passage of the federal No Child Left Behind in 2002 act gave us gobs of test outcome data, which has been used by policymakers and researchers to evaluate and justify educational policies. Unfortunately, many of these folks have failed to ask the most basic question of the tests they rely on: are they valid measures of what we want to measure?

This might seem picky: a test is a test, right? Why can't we just say a kid needs to pass a test to graduate? It sounds simple at first... until you get to the problem of what to test, and whether you should use a test for purposes other than those for which it was originally designed, and even what outcome should be considered "passing."

As I've noted before, there's been a lot of fuzzy thinking about this over the last few years. "College and career ready," for example, has been held up as a standard all children should meet. But it's a meaningless phrase, artificially equating admission to institutions that, by design, only accept a certain percentage of the population to being able to participate in the workforce. "College and career ready" essentially means that everyone should be above average, a logical impossibility.

Clearly, the NJ Legislature didn't mean to set such a standard. On its face, the law is calling for a test that shows whether or not a student has reached a level of education that allows them to participate in society. Which brings us to the PARCC tests in question: the Algebra 1 and English Language Arts (ELA) Grade 10 tests.

Forget for a moment whether these tests "measure those basic skills all students must possess to function politically, economically and socially in a democratic society." Because it's a hard enough lift just showing that these tests measure a student's abilities in language and algebra in a way that is valid and reliable. I'm going to be way too simplistic here, but...

By valid, we mean the extent to which the test is actually measuring what it purports to measure. This is much trickier than many people are willing to admit: take, for example, word problems on an algebra test. Do they measure the ability to apply mathematical concepts to real-world situations... or do they measure a student's proficiency in the English language?

For a test to be valid, we have to present some evidence that our interpretation of the test's score can be used for the purposes we have set out. If, for example, a mathematically adept student can't pass an algebra test because they aren't able to read the questions in English, we have a potential validity problem -- we may not have a meaningful measure of what we want to measure. Maybe we want to use the test to place the student in the correct math class. It could be our test outcome isn't giving us the feedback we need to make the right decision -- we have to give some reason to believe it is.

By reliable, we mean that the test can consistently gauge a student's abilities. Do test scores vary, for example, based on whether they are taken on a computer or on paper? Do they vary based on the weather? All sorts of unobserved factors can influence a test outcome, and some tests vary more on these factors than others.

Validity and reliability are actually closely intertwined. What we should remember, however, is that test outcomes are always measured with error, and will vary due to differences in things we don't want to measure. This applies to the PARCC tests -- especially when we use those tests to determine whether a student should receive a diploma, a task for which they were not designed.

In yesterday's ruling, the court points to two problems with using the PARCC tests to fulfill the mandates of the state's law:
We hold N.J.A.C. 6A:8-5.1(a)(6), -5.1(f) and -5.1(g) are contrary to the
express provisions of the Act because they require administration of more than one graduate proficiency test to students other than those in the eleventh grade, and because the regulations on their face do not permit retesting with the same standardized test to students through the 2020 graduating class. As a result, the regulations as enacted are stricken.
It's worth noting that the court said there may be other problems with the regulations calling for the use of PARCC,  but that these two problems -- the tests aren't equivalent to an eleventh grade test, and there are no provisions to allow retesting -- are enough to overturn the regulations.

What the court is basically doing here is calling into question the validity and the reliability of the PARCC for the purpose of granting a diploma. The validity problem comes from the fact that neither the Algebra 1 nor the ELA-10 exam is measuring what the law says it's supposed to measure: whether an 11th grader is able to "function politically, economically and socially in a democratic society." How could they? They're not 11th grade tests!

The court rightfully restrained itself from going any further than this basic flaw in the regulations, but I'm under no such restriction, so let me take this a step further. Where has anyone ever made the case that passing the Algebra 1 exam is a valid measure of the "computational" skills needed to be a fully capable citizen? Keep in mind that NJDOE's own guide to the exam says that, among other tasks, Algebra 1 students should be able to:
Identify zeros of quadratic and cubic polynomials in which linear and quadratic factors are available, and use the zeros to construct a rough graph of the function defined by the polynomial.

Graph the solutions to a linear inequality in two variables as a half-plane (excluding the boundary in the case of a strict inequality), and graph the solution set to a system of linear inequalities in two variables as the intersection of the corresponding half-planes.
 
Given a verbal description of a linear or quadratic functional dependence, write an expression for the function and demonstrate various knowledge and skills articulated in the Functions category in relation to this function.
The passing rate last year for the Algebra 1 test was only 46 percent. I know this is a cliche, but it rings true: given that rate, how many members of the NJ Legislature or the State Board of Education could pass the PARCC Algebra 1 test right now? If they can't, does that mean they don't have anything to contribute to our state?

One of the reasons the senior leaders of the NJDOE under the previous administration pushed so hard for a switch to the PARCC was that the previous math tests (the NJASK) had what are called "ceiling effects" -- basically, too many kids were getting perfect (or close to perfect) scores on the test. PARCC cheerleaders told us this was a huge problem; we needed to be able to sort the kids at the top of the distribution because... uh... reasons?

The PARCC Algebra 1 is not, therefore, measuring whether a student meets a basic level of achievement in math. It's a test that is attempting to gauge algebra ability, and it includes plenty of items that have low passing rates so as to tease out who is at the top of the score distribution. On its face, therefore, the test is not suitable for the purposes set out in law -- a point both Sarah and Stan Karp of the Education Law Center have been making for years.

The reliability problem in the regulations comes from the lack of retesting opportunities for students who fail the PARCC tests on their first try. Again: all tests outcomes are measured with error. A student who fails one administration of a test may have a "true test score" that is much higher; however, due to circumstances having nothing to do with their academic abilities, they may get a score lower than their "true" score.

When the stakes are as high are they are in graduation test, there must be a chance for students to take the test again. But this is difficult when the test, like the PARCC, has to limit its administrations due to security concerns. I can't say for sure that the HSPA, New Jersey's old test, addressed this problem as well as it should. There were, however, other alternative tests available to students if they didn't pass the HSPA.

I can only guess as to what comes next, but it's highly unlikely, given the rhetoric during the campaign, that the Murphy administration will challenge this ruling before the state Supreme Court. Which means the state's students are in a real bind: the law says they have to pass an 11th grade test, but the state doesn't have one ready to go. It takes a good bit of time to develop a valid, reliable test.  Maybe there's something available off-the-shelf -- but it would still be unfair to students to spring a brand new test on them without giving their schools the chance to actually teach the content on which those tests are based.

In the short term, the Legislature should work quickly and amend the law so today's high school students can get their diplomas without having to pass tests that are invalid for the purpose of "measur[ing] those basic skills all students must possess to function politically, economically and socially in a democratic society." 

I know that some legislators have invested a great deal of their reputations into the PARCC, but they need to step up and do the right thing here. A lot of kids have been working hard and playing by the rules, and they shouldn't feel their diplomas are at risk simply because the state acted rashly. No student should miss out on graduating due to this ruling.

In the long term: we are well past the time for this state to have a serious conversation, informed by expertise, about what exactly we are trying to achieve in our schools and how testing can help us get there. I have no doubt the usual suspects will claim anyone taking this position is setting low standards and in the pocket of the teachers union and doesn't really care about kids and blah blah blah...

Those, however, are the same folks who pushed hard for PARCC without engaging in a meaningful debate with skeptics about the purposes of testing and the consequences of implementing the current regime. They never bothered to address the carefully laid out, serious concerns of folks like Sarah and Stan and many others. Many of them made wildly ambitious claims about the benefits of PARCC, suggesting the tests could be used for all sorts of purposes for which they were never designed.

They are also (mostly) the same folks who have continually downplayed the role of school funding in educational outcomes; they insisted on higher standards without pausing for a second and asking whether schools have what they need to meet those standards. Remember: our current funding formula, which isn't even being followed, came years before we moved to the Common Core and PARCC. If we've adjusted our standards upward, isn't it sensible to think we'd have to adjust the resources needed upward as well?

Let me be clear: I am all for accountability testing. I think there is a real and serious danger of short-changing schools and exacerbating inequality if we don't use some universal measure to assess how students are performing. But we've got to have an understanding of what tests can and can't do if we're going to use them to evaluate policies. And we've got to be extremely cautious when we attach high-stakes decisions, such as graduation, to test outcomes.

Let's view this ruling as an opportunity: a chance to make smart, well-informed decisions about testing. New Jersey happens to be the home to some of this country's most highly-regarded experts in testing, learning, and education policy. Let's bring them in and have them help fix this mess. Our kids deserve no less.