Rhee's Phony Math

So the edu-blogosphere has lit into the StudentsFirst "report card" - and rightly so. Aside from the fact that SF doesn't rate states by their students' achievement level - and isn't that the entire point of Rhee-form anyway? - the categories and rubrics Michelle Rhee's outfit came up with are absurd. G.F. Brandenburg, who knows Rhee as well as anyone, takes a good look at how subjective and inane SF's criteria are.

But even if SF had rational and relevant benchmarks for ranking states, the "report card" would still be utterly phony, because it's based on faulty math. Here's p.66 of the report, which explains SF's methodology:

No. No, no, no. Sweet lord, no.

Folks, it is a basic precept of mathematics that you must round averages to the correct significant figure. You are not allowed to express an average in any more digits than the original data, because you run the risk of compounding measurement errors. Further, this is what is referred to as an "ordinal" scale: it does not describe the degree of difference between the items measured. You can't average ordinal measures; it's simply not allowed.

Let me give an example I've used before:

Two cities claim to average the most snowfall: Ravitchton and Rheeville. Snowfall in both is measured in feet:

Day	Ravitchton	Rheeville
Mon	4 ft.	3 ft.
Tue	4 ft.	3 ft.
Wed	4 ft.	4 ft.
Thurs	3 ft.	3 ft.
Fri	4 ft.	3 ft.
AVERAGE SNOWFALL	3.8 ft.	3.2 ft.

Looking at this, you'd think Ravitchton had 0.6 ft. more snow per day, on average, than Rheeville. But let's suppose these are the more accurate measures, in tenths of a foot:

Day	Ravitchton	Rheeville
Mon	3.9 ft.	3.1 ft.
Tue	3.6 ft.	3.2 ft.
Wed	3.8 ft.	4.1 ft.
Thurs	2.9 ft.	3.1 ft.
Fri	3.8 ft.	3.0 ft.
AVERAGE SNOWFALL	3.6 ft.	3.3 ft.

All of a sudden, the difference between the two towns shrinks to 0.3 ft. When we rounded above, we overstated the difference between the two towns.

But let's take this even further, and treat snowfall the way SF treats its rubrics. In a rubric like the one used by StudentsFirst, the rounding really only goes down. Either you make the cut-off point, or you drop down to the next lowest level. If you're not a "4" on the rubric, you're a "3," even if you're really, really close to the cut point, and better than all the other "3's"; this is why it's an ordinal scale.

So what happens when we round everything down?

Day	Ravitchton	Rheeville
Mon	3 ft.	3 ft.
Tue	3 ft.	3 ft.
Wed	3 ft.	4 ft.
Thurs	2 ft.	3 ft.
Fri	3 ft.	3 ft.
AVERAGE SNOWFALL	2.8 ft.	3.2 ft.

We've now completely flipped the measure: the town that got more snow is now the town that got less.

It's a silly little example, but it illustrates the point: you can't average these measures without compounding the errors inherent in them. This is one of the most basic principals of statistics, mathematics, and social science - and StudentsFirst violated it.

So even if you grant StudentsFirst all of their criteria - which I don't - their rankings are completely ridden with errors. There is no reason to believe in their state rankings, even by their own standards.

I ask this seriously: is there anyone on the StudentsFirst staff who understands what a blunder this is? Is there anyone on the staff who has enough of an understanding of statistics that they are capable of being embarrassed by this error?

The mathematics underlying the entire StudentsFirst report is completely phony. Do they really not understand that?

Well, um...

ADDING: This isn't Rhee's first time screwing up math; when she was chancellor in Washington, she put in place an entire teacher evaluation system based on this phony precision. Why the teachers union wasn't jumping up and down about this then, I have no idea.