I have to admit it: Amanda Ripley really got under my skin. Twice. So much so that I went back to her blog to look at her two recent posts - here and here - about how much American schools suck and how people who tell you that poverty is the overriding cause are selling you magic beans.
Lucky for you, Ripley has "...been to Finland, Korea and Poland working on this book, and I have the luxury of spending hours reading PISA results." Apparently that gives her license to mock Diane Ravitch, David Sirota, and liberals in general. It also seems to give her license to make the following mistakes:
- Failing to understand that Ravitch's comparison of low-poverty US school districts to low-poverty countries allows for poor kids to be included in both sets.
- Wonky alert! Failing to account for the "linearity" of the relationship between socio-economic status and test performance; in other words, your scores rise even faster the wealthier you get (up to a limit, obviously).
- Failing to understand that the group she repeatedly calls "rich kids" are really the top 25% of socioeconomic status - not 10%, or 5% - and that countries could differ greatly in the homogeneousness of that group.
I've been thinking about this last one some more: what is the difference between the scores of the average kid in the 75th percentile for SES compared to the kid at the 95th percentile? Well, I couldn't find the data desegregated this way for the 2003 PISA, which Ripley uses to bash the performance of "rich kids" in math and science (without any sources or references to how she got the data, by the way). But I did find a clue in the 2009 PISA, which does disaggregate the data enough to possibly gain a little insight (p.152):
What this tells us is that the US has the second largest difference between its top 5% wealthiest kids and its top 25% wealthiest kids. Which implies that the kids in the 75th to maybe 90th (maybe 95th) percentile are "pulling down" the wealthier kids more than in other countries.
Maybe. Because who knows how much this really matters in determining overall scores for the top quarter of kids? And because after I looked at this for a bit, it occurred to me that I was making an assumption about something Ripley wrote that I probably shouldn't:
See, I read that, and I thought, "Oh, the PISA must use income to compare socio-economic status." Silly me...The free/reduced-price lunch figure measures the number of kids from families making 185% of federal poverty line, right? So that means a family of 4 needs to make less than about $40,000 to qualify. Under this measure, roughly 40% of American kids qualify as “poor.”OK. Then the other measure is the measure usually used in international comparisons of poverty. That is the percentage of kids from families earning less than 50% of the median income in that country. (In the US, this comes out to about 22%. NOT 40%.)In other words, Ravitch is comparing the test scores of kids from families that earn more than $40,000 in the U.S. to the scores of all kids in Finland (where the median household income is about $40,000).I don’t have tenure, but even I know you can’t mix and match data like this. Unless you are really, really desperate to find a certain answer, that is. [Edit: After this post went up, an alert reader informed me that Ravitch does not have tenure either; NYU confirms that she is a nontenured “research scientist.”]Conversely, PISA’s own measure of socio-economic background, the one you can find detailed in Table II.3.1 in PISA Volume II, offers a more valid comparison. And yet Ravitch does not cite it—because it does not show what she wants it to show. [emphasis mine]
The PISA determines a student's SES by asking students questions about his or her parent's job, education, and possessions in the home (p. 29):
Socio-economic background refers to a combination of characteristics of a student’s family that describes its social, economic and cultural status. Socio-economic background is measured by the PISA index of economic, social and cultural status (ESCS). This index captures a range of aspects of a student’s family and home background that combines information on parents’ education and occupations and home possessions. The index was derived from the following variables: the international socio-economic index of occupational status of the father or mother, whichever is higher; the level of education of the father or mother, whichever is higher, converted into years of schooling; and the index of home possessions, obtained by asking students whether they had a desk at which they studied at home, a room of their own, a quiet place to study, educational software, a link to the Internet, their own calculator, classic literature, books of poetry, works of art (e.g. paintings), books to help them with their school work, a dictionary, a dishwasher, a DVD player or VCR, three other country-specific items and the number of cellular phones, televisions, computers, cars and books at home. The rationale for choosing these variables is that socio-economic background is usually seen as being determined by occupational status, education and wealth. As no direct measure of parental income or wealth was available from PISA (except for those countries that undertook the PISA Parent Questionnaire), access to relevant household items was used as a proxy. [emphasis mine]They didn't have the parents' incomes, so they used questionnaires completed by the kids to make comparisons. I'm no sociologist, so I'm not qualified in the slightest to discuss the validity of all this. But for a woman who is so worried about defining poverty, it seems like quite a stretch to claim so assuredly that this is a "more valid" comparison than Ravitch's. At the very least, Ripley needs to explain the methods of the ECSC and her rationale for why she thinks it's superior.
Has she bothered to think about this at all?
Which is actually my point: this stuff is really, really knotty. International comparisons of academic achievement are hard enough; comparisons of relative wealth and socio-economic status are damn near impossible.
I'm not saying the PISA should be ignored; it's probably as good as we can get at the moment. We should continue to look at other countries and compare their methods, their outcomes, and the characteristics of their students.
But Ripley goes much, much further:
To prove her thesis, Ripley believes she has to show that "rich kids" in the US suck at learning (she's not worried that even if she showed the schools for "rich kids" suck, it wouldn't tell us a thing about schools for "poor kids" anyway). So she dismisses US poverty measures and claims international measures of SES are superior, without any exploration of why that may be. She confidently asserts that their "rich kids" do so much better than our "rich kids," without stopping to think calmly for 5 minutes what the definition of "rich kids" really is. She ranks countries without showing the differences in their mean scores. She presents no caveats about the tests and their alleged universality.Interestingly, this is not the kind of talk you hear in places with higher-performing education systems. In those countries, the very same countries that Ravitch says should be models for US schools, educators also think poverty is a big problem. But they think it istheir problem. They think it is a problem so intractable that our schools must be outstanding in order to help overcome it. See the difference?Of course, if we think about it calmly for more than 5 minutes, we can probably agree that poverty interacts with schools, like a chemistry experiment. Bad schools make poverty worse, and great schools make it possible to overcome poverty. In fact, great schools are among the most effective anti-poverty measures known to humanity. Neither schools nor poverty work in isolation.And yet this debate rages on, with a stunning lack of sophistication.
But she does tell us she's been to Finland, Korea, and Poland. Swell.
Folks, I wish I could tell you this stuff is atypical for our country's education dialogue; sadly, it's not. We are awash in reforminess: people who confidently aver that their particular prescription for American education's ills will cure all of our problems. But when you really start to pick apart their arguments, you find they have neither the tools, the training, nor the talent to back up just about anything they snottily claim to be the truth.
Lord, save us from these supremely self-confident "sophisticates."
ADDING: Paul Thomas has much more.