Dare To Compare

A big part of the reformy agenda is predicated on the notion that even affluent American kids suck at learning; this must mean our schools stink and we need to ___ (fill in your particular reformy "solution" here).

Thus, we have pieces by Amanda Ripley bemoaning our awful, awful schools:

The most respected international tests of teenagers around the world (PISA) has consistently shown that our most-affluent kids do not perform as well as the most-affluent kids in the highest-performing countries around the world (even though our rich kids are richer than their rich kids). PISA measures students’ economic, social and cultural status to get a sense of their socio-economic background. In reading, American kids’ best subject, our most affluent students still rank behind the most affluent kids in six other countries. (Even though we spend far more money per student than all of those countries.)

Rich Kids Ranking (PISA Reading 2009)

1. New Zealand

2. Korea

3. Belgium

4. Finland

5. Canada

6.  Australia

Now, I've already dealt with the many flaws in Ripley's presentation here, here, and here. But let's put aside the fact that she mischaracterized and/or misunderstood Diane Ravitch's claims, or presented the 75th percentile in socio-economic status as "rich kids," or missed the curvilinearity of the relationship between SES and test scores, or her lack of accounting for differences between countries even when using the same SES measurement tool, or her attempt to throw out Ravitch's simple point - found in a standard source - based on nit-picking about how to define poverty, or the dubiousness of her claim that we spend so more more on schools than the rest of the world...

Let's instead focus on the list above, which comes from Table II.3.1 on p. 165 of PISA 2009 Results: Overcoming Social Background: Equity in Learning Opportunities and Outcomes (Volume II). These are not "rich kids"; they are the top 75th percentile. And there's no excuse to characterize them that way when you can go right to Table II.1.1 on p.152 of the same report and get the scores for kids in the 95th percentile of SES - the real "rich kids." Let's look at them instead.

Wow, it's even worse for the US: we're #8!!! Man the lifeboats!

1. New Zealand 
2. Australia 
3. Japan 
4. Finland 
5. Canada 
6. Korea 
7. Belgium 
8. United States 

I mean, how could we let little Belgium beat us! And those damn Koreans again! Those who defend the status quo stand by idly while these other countries thrash us so badly!

Oh, wait - a ranking list doesn't tell us how badly, does it? No, and Ripley doesn't include the raw scores on her list either. Well, let's see just how badly the Belgians are rubbing our noses in their waffles:

Belgium's "rich kids" beat our Ritchie Riches by one point. The Koreans - two points. Should we rush to emulate the Koreans based on this?

But let me show you another neat trick; this is exactly the same data:

All of a sudden, the US doesn't look so bad compared to the folks who are "beating" us. Why? All I did was change the y-axis of the graph. Is this more accurate? I have no idea.

And that is precisely the point: if we are so concerned about the Belgians beating us by one point, or the Finns beating us by ten, we need to have some clue as to what one point means. Think of it this way: if you lose by four in basketball, it was a close game; if you lose by four in baseball, you were soundly defeated; if you lose by four in soccer, you got trashed. The point differentials depend entirely on the context; ranking countries accounts for none of this.

I am not expert on the PISA, and I'm not prepared to dive into the meaning of the score differences without compensation or graduate credits (I much prefer the first over the second). I will say this: for all OEDC students, kids in the 95th percentile performed 138 points higher than kids in the 50th percentile. So should we worry too much about a differential of 2 points? 5 points? 10?

I'll also point out the 95th percentile kids in the US did better than every other country's kids at the 90th percentile.

To sum up: there is no reason, based on these scores, to believe that America's "rich kids" are getting trashed by the rest of the world. Unless and until someone goes into the weeds and really looks at what these point differentials mean, the fact that Finland's richest kids beat our richest kids by 10 points is hardly cause for sounding the alarm.

I'll have one more post on this topic tomorrow.