NJ Teacher Evaluation: SOMEONE'S Got To Lose!

Tim Matheney is the current Director of the Office of Evaluation at the New Jersey Department of Education. He is touring around the state these days, trying to sell the new, top-down, test-based teacher evaluation system, AchieveNJ. I've blogged extensively about my problems with this system, but let's give Matheney a chance to respond to its critics:

Pamela Keuett, a Neptune Township middle school science teacher, voiced concern over whether the new rating system will create competition as opposed to collaboration between educators.

Matheney said the new paradigm of measuring students’ progress will be weighed against other students from across the state that test within the same percentile, thereby creating a more collaborative environment.

“An individual student is compared to his or her academic peers in terms of how much growth they make over the course of a year,” Matheney said. “Teachers aren’t being compared to other teachers per se.” [emphasis mine]

Sorry, but no. No way.

Let's go back to my previous explanation of Student Growth Percentiles: the fact is that they cannot account for teacher effect, even though the students within a "peer group" start with the same test scores.

Further, an analysis of SGP data from New York State makes clear that teachers of students who are in economic distress will pay a penalty under an SGP-based system. But let's put all that aside and look at Matheney's contention: teachers will not be competing against each other because students will only be judged against their "academic peers."

If that's true, then it must also be true, by logical extension, that every teacher in an SGP-based system has a chance to get a good SGP rating. If the "good" ratings are limited, however, then teachers will, by definition, be competing against each other for those best ratings. And since teacher ratings are dependent on the median SGP (mSGP) of their students, it follows logically that limiting the number of "good" student SGP ratings will mean limiting the number of "good" teacher mSGP ratings.

In other words: if only some students will get "good" SGP ratings, then only some teachers will get "good" test-based evaluations.

Well, guess what: that's exactly what's going to happen. AchieveNJ is designed to limit the number of "good" SGPs, which means teachers will be competing with each other to get their best possible mSGP ratings.

Let's look at an example. As the NJDOE itself explains in this video, test score "gains" on the NJASK will be assigned an SGP score from 1 to 99 in what statisticians call a "normal distribution." As NJDOE's technical manual on SGPs explains, the raw scores that students get are "forced" into what is commonly known as a bell curve. Here's a diagram from the video:

What's important to remember is that the SGP is not an "absolute" number: it only has meaning in relationship to the students' other "academic peers." Let's suppose that Jenny, for example, gained 40 points between her 3rd Grade and 4th Grade NJASK tests.

Jenny may have gained 40 points, but her SGP is 75, because her gain was as good or better than 75% of her "academic peers" - the children who got the same score as Jenny in 3rd Grade. Let's say one of her peers is Susie.

Susie got exactly the same score on the NJASK in 4th Grade that she got in 3rd. But that doesn't mean she is "average" compared to her peers: they averaged a 20 point gain. Susie, therefore, has an SGP of 25.

What would it look like if Jenny's and Susie's "peers" averaged the same score in 3rd Grade as they did in 4th?

The entire cohort did "worse," but the SGPs don't change. All of the "academic peers" are being judged against each other, and not on their "absolute" growth.

What's more, it doesn't matter how the scores of the "peers" are distributed:

The "growth" this time is distributed unevenly, but that doesn't matter: all that the SGPs show is how the students scored in comparison to one another. Notice also that even though Susie is now showing "growth," her SGP hasn't changed. It doesn't matter if she "grew" or not; all that matters is whether she "grew" more or less than Jenny and her other "peers."

Now let's imagine we have another administration of the test, based on the same score distribution. This time, Susie scored 29 points higher; she's now at the 50th percentile for growth. That's great... but someone has to be at the "slot" for an SGP of 25. 

Because we are judging students relative to each other, every space must be occupied by the correct number of students. It is impossible to have everyone be "above average"* - someone has to take Susie's place. In this case, Jenny didn't do as well the second time around.

Do you see the problem now? A certain number of children MUST occupy every place in the distribution - no exceptions. It may not be Jenny that moves down, but someone has to; someone must be in every slot, or else the distribution will not be correct.

When the NJDOE says that SGPs will create a collaborative environment they are dead wrong: SGPs are predicated on competition, because some children must be "above average" and some must be "below." AchieveNJ is, therefore, a competition between teachers to get the best mSGP: it moves the incentives away from collaboration.

By the way: Tim Matheney was a principal for many years; apparently, a good one. He should know better. Why he is signing off on this without acknowledging the flaws inherent in AchieveNJ is beyond me; he's certainly not doing his former colleagues any favors.


* I'm using the term "average" colloquially here. I could say "above the median," although the SGP process is, in a way, converting a median to a mean. "Above the 50th percentile in SGP" is maybe the best way to express it... but that's not very good prose, is it?