I will protect your pensions. Nothing about your pension is going to change when I am governor. - Chris Christie, "An Open Letter to the Teachers of NJ" October, 2009

Sunday, May 24, 2015

Standardized Tests: Symptoms, Not Causes

Let's talk about tests for a bit:

All standardized tests, by design, yield "normal" or bell curve distributions:


You sometimes hear quanty folks say: "God loves the normal distribution." In this case, however, man is creating God in his own image, because the tests are designed to yield this distribution of scores.


In a normal distribution, most students get average scores: that's why the highest point in the curve is in the middle. As we move to the right from the middle, the test scores go up, but fewer and fewer students get those higher scores. Likewise, as we move to the left from the center, fewer students get the lowest scores.

How does this happen? Well, to start, and as testing expert Daniel Koretz has explained, we only test students on what we reasonably expect them to be able to do. We don't test fourth graders on trigonometry because we know most of them can't solve those kind of problems and wouldn't be able to even if we drilled them on it repeatedly. Similarly, we don't test high school juniors on adding one-digit numbers because we know that would be too easy.

Are there fourth graders that know the difference between a sine and a cosine? Are there juniors who can't add 5 and 4? Of course there are, but we know they are far away from the norm; we don't set our standards based on these outliers. Because we like to see ourselves as normally distributed, we create standards and tests that give us normal distributions. 

This can't be stressed enough in the testing debates: we design tests not based on objective criteria, but on socially constructed frameworks that assume some of us are above average, some of us are below, and most of us are in the middle.

Now, sometimes we design a test that is "too hard" or "too easy." What does that mean?



Here's a test where many students got high grades; it's skewed negative, meaning many students got grades close to the top of the test's scale. In fact, the "right tail" of the curve is cut off: a good number of students got the highest grade possible. This is a "ceiling effect," and many people, including the NJDOE and other authorities in education, do not like this distribution. They think the test is "too easy," because they assume that students must be normally distributed.

Hence the shift to tests like the PARCC, which reformy folk say is testing "higher order thinking" and "real world problem solving" and so on. Frankly, if you can find a real world situation where people use phrases like "constant of proportionality," I'd be surprised. What's really going on is that the questions are more difficult so the test can lose its skewness and return a normal distribution. 

The irony here is that the promoters of standardized testing are using an argument against inequity to insist on the continued use of these tests (if not their expansion), and to shame those who are opting their children out of these tests as perpetuators of race and class injustices. 

We'll leave aside the point for the moment that these same people also refuse to actually provide the funds the law -- based on a large and growing body of empirical research -- says are needed to equalize the test-based goals they've set out. 

Instead, let's look at the logic of their argument. Somehow, everyone has to be "college and career ready" (as if that is an objective criterion), and we're going to insist that everyone perform at the highest possible levels. But we're going to use a test that forces a normal distribution; and if that test actually shows that many students are meeting a high standard, we'll declare the test "too easy," and redesign it so we get back the bell curve we crave.

Does everyone see the problem here? We're insisting that all children demonstrate high performance on a test that, by design, only allows a few children to demonstrate high performance.

This is where "proficiency rates" enter the conversation:* 



All a proficiency rate does is set the cut point along the normal distribution. Where the rate is set is entirely up to whomever has the power to set it; it can be as low or as high as they like.

But what usually happens is that proficiency is determined by another test that -- surprise! -- yields a normal distribution. And that test -- say, the SAT -- is tied to some other normalized standard, like college freshman GPA. Why do I say that's normalized? Because even the reformiest of the reformy admit not everyone can or should go to college, and college standards are determined, like those in K-12 schools, by what we reasonably expect the average college student do be able to do. It's all normalized.

This is a fundamental contradiction inherent in the arguments for standardized testing as necessary prerequisites for addressing inequitable outcomes in education. Standardized tests, by design, give us bell curves, and reformy types insist we change them if too many students are getting high scores. But then they moan that not enough students are above average!

Further: they fail to see what the tests are really measuring:




The correlation between socio-economic status and test scores is absolutely iron-clad. Does anyone think eliminating the ceiling effect is going to change this? Granted, there is likely a ceiling on how income effects test scores: a kid in a family making $300K a year probably isn't at much, if any, disadvantage compared to a kid in a family making $500K.

But the wealthy have always enjoyed an advantage in our false meritocracy. The biases in the tests themselves, coupled with the inequitable distribution of resources available for schools, all but guarantee the majority of the variation in test scores will be explained by class.

The neo-liberal view appears to be that this is inevitable and just, so long as we decouple these inequities from race. If we can get some more students of color into elite schools, and create a few more black and brown millionaires and billionaires, everything will be "fair." The owners of the country can then sleep soundly at night, content that they may be classists, but they aren't racists.

I'm all for social mobility, but increasing it isn't the same as decreasing inequity. There are millions of people in this country doing difficult, necessary jobs. It's wrong to consign people of color to these jobs through a system of social reproduction in our schools. But it's also wrong to pretend that we have a system where everybody can be above average, and in doing so can make a better life for themselves.

So long as we keep making bell curves, somebody has to be on the left side. Somebody has to do the work that needs to get done. But there's no reason those decent, hardworking people shouldn't have good wages and good medical care and good housing and disposable income and workplace rights and time to spend raising their children.

Yes, their children should have the same opportunities to move to the right side of the bell curve. But if they do, somebody is going to have to take their place. Maybe if the consequences for being on the left side of the bell curve weren't so dire, affluent people wouldn't be as obsessed with maintaining the advantages they enjoy in keeping their children on the right side. Maybe they would stop pushing their children to the breaking point just to stay ahead of the pack:



Maybe we'd allow children to become themselves and realize their full potentials, free of the fear that their "failure" will inevitably banish them to a life of toil and misery. Maybe we'd start to see schooling not as preparation for a life of stepping on our fellow citizens, and instead as a process by which we become a people who balance our own self-interest with caring for our fellow citizens.

And then maybe we wouldn't feel the need to make these bell curves at all.


* UPDATE: Hey, I got it to loop!

** UPDATE 2: Sometimes you can look at something a hundred times and you never really see it. Thanks to MFortun in the comments for pointing out my axis titles were backwards!

15 comments:

Suzanne Libourel said...

Years back, when I was in school, most of the classroom tests we took were criterion based. We were graded against a particular standard and our scores were generally 'skewed right' indicating that most of us scored C and above.

When high-stakes standardized tests were introduced, supposedly to address inequities in education, norm-referenced tests became the vogue. Even in the classroom today, tests teachers create have become more 'difficult' for all students, forcing more and more students' scores lower and reducing the number of students who score higher grades.

Educators, just like the general public, have been duped into believing only a few students 'deserve' As and Bs, and even Cs are no longer 'average'. Were we to go back to more criterion based testing, more students would succeed and the 'reformy' crowd would have no platform upon which to perpetuate the concept of the 'failing school'.

If only more decision makers understood this ......

Peter Greene said...

Man. Standing ovation, sir.

Duke said...

Thanks, Peter - high praise coming from you.

And thanks, Suzanne, as always. There are people who would say ALL tests are normalized, again because we choose what to teach students based on what the average student can and can't do.

But I agree with your point: I see the top of the case being pushed and pushed harder and harder... and for what? When I was in high school, only the most elite students took calculus in their senior year. We've now made junior year calculus the standard for elite college-bound students in high-performing districts.

Is that really necessary? Or did we shift because "too many" seniors were taking calculus, so we couldn't cream off the top as easily?

P. Grunther said...

Reading JJ's posts brings a small amount of sanity into my life - thanks. As a teacher I KNOW when my kids are learning and both they and I realize that a standardized test bears no relation to what is going on in the classroom. People are being conditioned not to trust a classroom teacher's assessment of their own students. Is it any wonder that the war on public school teachers and teacher unions is being led by the very people who stand to profit from the privatization of public schools? The push for standardized testing is nothing more than a tool for devaluing public education, making it vulnerable to privatization. And, as JJ so aptly points out, it is our children, our students who are paying the price. A wise friend of mine says, "Nobody should ever make a profit from education or health care - these are basic, democratic rights and should never be in the hands of those who stand to make a profit." If only...

Duke said...

"People are being conditioned..."

Yep. That's why teaching true critical thinking is so dangerous.

MFortun said...

Great post, but aren't your axes switched? Shouldn't the X-axis be test scores, and y the number of test takers?

Dienne said...

Excellent, except for this: "Further: they fail to see what the tests are really measuring...."

Oh, they see it alright. That's why they're playing this game. They know perfectly well that only half of all kids can be above average and they're never going to let go of a system that ensures that their children remain on the right side of the curve.

NY Teacher said...

Why wasn't the 100% proficiency clause of NCLB challenged on constitutional grounds?
It was entrapment, pure and simple. A law that even the best intentioned and hardest working school districts had to break.

No different than if all cars were required to travel on one road, yet were fined if they exceeded the zero mile per hour speed limit.

Duke said...

MF: AAARRRGGGGHHHH!!!! Good catch - stand by...

Linda Myrick said...

Excellent points all. I'd also like to comment on the underlying assumption that those on the left side of the curve are "less than." The academic skills supposedly measured by these tests are determined, in self-fulfilling fashion, by those with these traits. Questions designed, written in academic language spoken and understood by those "in the club" for students with the background and breeding to decipher what they are looking for. The very nature of the questions skews many brilliant children to the left regions of the curve. These tests miss their brilliance. WE are vulnerable to missing their brilliance as we continue to participate in a system that places children on bell curves using questions that are culturally biased and based on elitist assumptions.

Robert D. Skeels * rdsathene said...

Duke:

Your scholarship and analysis are a national treasure. Thank you for always providing irrefutable evidence to counter the neoliberal corporate edreform's cacophony of lies and propaganda.

Michael Fiorillo said...

Not just very smart, but very wise.

Thanks for this and everything you do.

laMissy said...

Nifty deconstruction of our testing "meritocracy". Now if we could only get it on the front page of every major newspaper and website in the country!

Duke said...

Thx, everyone. - JJ

Old Teacher said...

Once again pointing out the argument I have made with my reformy administrators; A test is a reflection of the assumptions made by those designing and writing the test. They create a bell curve and order us to straighten it out. I know how Sisyphus felt...From an old teacher and on old profiler, nice explanation of the intricacies of test design. Even a lay person can see what's wrong here. May I send this on to my ill informed bosses? I know they are just yes men, but I like tweaking their noses once in a while.