I have to admit it: Amanda Ripley really got under my skin. Twice. So much so that I went back to her blog to look at her two recent posts - here and here - about how much American schools suck and how people who tell you that poverty is the overriding cause are selling you magic beans.
Lucky for you, Ripley has "...been to Finland, Korea and Poland working on this book, and I have the luxury of spending hours reading PISA results." Apparently that gives her license to mock Diane Ravitch, David Sirota, and liberals in general. It also seems to give her license to make the following mistakes:
- Failing to understand that Ravitch's comparison of low-poverty US school districts to low-poverty countries allows for poor kids to be included in both sets.
- Wonky alert! Failing to account for the "linearity" of the relationship between socio-economic status and test performance; in other words, your scores rise even faster the wealthier you get (up to a limit, obviously).
- Failing to understand that the group she repeatedly calls "rich kids" are really the top 25% of socioeconomic status - not 10%, or 5% - and that countries could differ greatly in the homogeneousness of that group.
I've been thinking about this last one some more: what is the difference between the scores of the average kid in the 75th percentile for SES compared to the kid at the 95th percentile? Well, I couldn't find the data desegregated this way for the 2003 PISA, which Ripley uses to bash the performance of "rich kids" in math and science (without any sources or references to how she got the data, by the way). But I did find a clue in the 2009 PISA, which does disaggregate the data enough to possibly gain a little insight (p.152):
What this tells us is that the US has the second largest difference between its top 5% wealthiest kids and its top 25% wealthiest kids. Which implies that the kids in the 75th to maybe 90th (maybe 95th) percentile are "pulling down" the wealthier kids more than in other countries.
Maybe. Because who knows how much this really matters in determining overall scores for the top quarter of kids? And because after I looked at this for a bit, it occurred to me that I was making an assumption about something Ripley wrote that I probably shouldn't:
See, I read that, and I thought, "Oh, the PISA must use income to compare socio-economic status." Silly me...The free/reduced-price lunch figure measures the number of kids from families making 185% of federal poverty line, right? So that means a family of 4 needs to make less than about $40,000 to qualify. Under this measure, roughly 40% of American kids qualify as “poor.”OK. Then the other measure is the measure usually used in international comparisons of poverty. That is the percentage of kids from families earning less than 50% of the median income in that country. (In the US, this comes out to about 22%. NOT 40%.)In other words, Ravitch is comparing the test scores of kids from families that earn more than $40,000 in the U.S. to the scores of all kids in Finland (where the median household income is about $40,000).I don’t have tenure, but even I know you can’t mix and match data like this. Unless you are really, really desperate to find a certain answer, that is. [Edit: After this post went up, an alert reader informed me that Ravitch does not have tenure either; NYU confirms that she is a nontenured “research scientist.”]Conversely, PISA’s own measure of socio-economic background, the one you can find detailed in Table II.3.1 in PISA Volume II, offers a more valid comparison. And yet Ravitch does not cite it—because it does not show what she wants it to show. [emphasis mine]
The PISA determines a student's SES by asking students questions about his or her parent's job, education, and possessions in the home (p. 29):
Socio-economic background refers to a combination of characteristics of a student’s family that describes its social, economic and cultural status. Socio-economic background is measured by the PISA index of economic, social and cultural status (ESCS). This index captures a range of aspects of a student’s family and home background that combines information on parents’ education and occupations and home possessions. The index was derived from the following variables: the international socio-economic index of occupational status of the father or mother, whichever is higher; the level of education of the father or mother, whichever is higher, converted into years of schooling; and the index of home possessions, obtained by asking students whether they had a desk at which they studied at home, a room of their own, a quiet place to study, educational software, a link to the Internet, their own calculator, classic literature, books of poetry, works of art (e.g. paintings), books to help them with their school work, a dictionary, a dishwasher, a DVD player or VCR, three other country-specific items and the number of cellular phones, televisions, computers, cars and books at home. The rationale for choosing these variables is that socio-economic background is usually seen as being determined by occupational status, education and wealth. As no direct measure of parental income or wealth was available from PISA (except for those countries that undertook the PISA Parent Questionnaire), access to relevant household items was used as a proxy. [emphasis mine]They didn't have the parents' incomes, so they used questionnaires completed by the kids to make comparisons. I'm no sociologist, so I'm not qualified in the slightest to discuss the validity of all this. But for a woman who is so worried about defining poverty, it seems like quite a stretch to claim so assuredly that this is a "more valid" comparison than Ravitch's. At the very least, Ripley needs to explain the methods of the ECSC and her rationale for why she thinks it's superior.
Has she bothered to think about this at all?
Which is actually my point: this stuff is really, really knotty. International comparisons of academic achievement are hard enough; comparisons of relative wealth and socio-economic status are damn near impossible.
I'm not saying the PISA should be ignored; it's probably as good as we can get at the moment. We should continue to look at other countries and compare their methods, their outcomes, and the characteristics of their students.
But Ripley goes much, much further:
To prove her thesis, Ripley believes she has to show that "rich kids" in the US suck at learning (she's not worried that even if she showed the schools for "rich kids" suck, it wouldn't tell us a thing about schools for "poor kids" anyway). So she dismisses US poverty measures and claims international measures of SES are superior, without any exploration of why that may be. She confidently asserts that their "rich kids" do so much better than our "rich kids," without stopping to think calmly for 5 minutes what the definition of "rich kids" really is. She ranks countries without showing the differences in their mean scores. She presents no caveats about the tests and their alleged universality.Interestingly, this is not the kind of talk you hear in places with higher-performing education systems. In those countries, the very same countries that Ravitch says should be models for US schools, educators also think poverty is a big problem. But they think it istheir problem. They think it is a problem so intractable that our schools must be outstanding in order to help overcome it. See the difference?Of course, if we think about it calmly for more than 5 minutes, we can probably agree that poverty interacts with schools, like a chemistry experiment. Bad schools make poverty worse, and great schools make it possible to overcome poverty. In fact, great schools are among the most effective anti-poverty measures known to humanity. Neither schools nor poverty work in isolation.And yet this debate rages on, with a stunning lack of sophistication.
But she does tell us she's been to Finland, Korea, and Poland. Swell.
Folks, I wish I could tell you this stuff is atypical for our country's education dialogue; sadly, it's not. We are awash in reforminess: people who confidently aver that their particular prescription for American education's ills will cure all of our problems. But when you really start to pick apart their arguments, you find they have neither the tools, the training, nor the talent to back up just about anything they snottily claim to be the truth.
Lord, save us from these supremely self-confident "sophisticates."
ADDING: Paul Thomas has much more.
5 comments:
"But for a woman who is so worried about defining poverty, it seems like quite a stretch to claim so assuredly that this is a "more valid" comparison than Ravitch's."
The reason it's more valid is that even if it's a flawed way to measure poverty, at least different countries aren't having poverty measured in completely different ways. If we could use FRL status for all the countries in the world, that would be great.
But the one thing we can't do is what Ravitch does: compare the supposed performance of "low poverty" kids in different countries when poverty is being defined in completely different ways.
First of all, Stuart, it wouldn't be great to use FRPL across countries. As I've said, any metric has problems: $40K may go a lot further in one country than another. Again, there is no accounting for all the free stuff other OEDC countries give their people - health care, college education, housing subsidies - that Americans must either pay for or do without. Let alone differences in taxation.
But let's put all that aside and follow your argument - and Ripley's - to its conclusion:
Ravitch points to a government study that says that kids in districts with less than 10% poverty do better than countries with less than 10% poverty.
Ripley says, "Oh, no, you can't do that! The US has a different measure of poverty than the rest of the world!"
By that logic, the US is so very generous in its estimation of poverty that it classifies loads of children as being poor that other countries would say were not poor. So that 21%, or 15%, or whatever poverty rate in the US is WAY inflated.
On international comparisons, these other countries are under an unfair burden: they have to include a lot of kids we would consider "poor" in their "not poor" group of students.
And the difference in the way the US defines poverty is so great, it not only makes up for the difference in mean scores: it REVERSES the trend! When you exclude the same kids the US excludes, the rest of the world trashes us!
And the difference in the definition of poverty is so great that the US may overstate poverty 6 TIMES more than Finland does!
That's how I read your argument. And I find it absurd on its face. We overstate poverty to such a large extent it not only closes, but REVERSES the gap in achievement? Come on, man.
More soon on "rich kids."
I'm not sure what "trend" or "gap in achievement" you're talking about. The point, which you partially seem to understand, is that other countries count a lot of kids as "not poor" merely because their family income is over half of what the median income is in that country, whereas those same children would be considered "poor" here.
To take one striking example, Slovakia. Median income: $9,071. (See http://en.wikipedia.org/wiki/Median_household_income) Poverty rate by the measure that Ravitch is using: 2.1%, lower than Finland. See http://www.nationmaster.com/graph/eco_pop_bel_med_inc-economy-population-below-median-income
So Slovakia is beating us on poverty, right? Well, 50% of their population is below $9,071 in income, because that's the median, so it's not really all that comforting that only 2.2% are below $4,536 (or half the median).
The point is this: if it turned out that Slovakia was beating the US on PISA scores (it's not, but this is just an example), it would be completely absurd to pull Ravitch's trick of saying, "Look, their poverty rate is only 2.1%, and if we look only at US districts with really low poverty rates (using a completely different US standard), then we're doing just as good as Slovakia!!!!"
Stuart, I think you know full well I understand Amanda's point completely. Here's the argument so far:
Ravitch contends that poverty is primarily responsible for the gap between the mean scores of the US and other countries. She cites a government study that shows low-poverty school districts do better than low-poverty countries on PISA scores.
Ripley says: "You can't do that! We don't measure poverty the same way!"
I say: "Are you telling me that we so overestimate poverty in this country that if we counted poverty the same as other countries, our low-poverty districts would be getting beat by these other countries? And beat as badly as Ripley says?"
You: "You only partially understand."
No, I understand perfectly. I am simply taking your argument to its logical conclusion:
You - or, more correctly, Ripley - contend the US includes so many "affluent" kids in its measures of poverty it skews the data so much that the US is actually way BEHIND other countries in educating its most affluent.
Again: I find that absurd on its face. It reminds me of the nonsense I hear on Fox News when wingnuts say "Well, the poor in the US have it so much better than the poor in other countries!" Really? The poor in Finland - which Ripley uses as an example - have it so much worse than our poor? Even with the strong social safety net there? Yet they suck it up and deal with it much better than we do? And that accounts for the gap between Ripley's contention that the US sucks at educating all kids and Ravitch's contention that poverty accounts for most of our sucking?
Seriously?
You cite the low income in Sovakia. From p.152 of the PISA Vol II report, here are the mean indices of ESCS for three countries:
Finland - 0.37
US - 0.17
Slovakia - -0.09
So the difference in mean socio-economic status is nearly as great between Finland and the US as between the US and Slovakia. Well, this is the measure Ripley prefers. If we're going assert the poverty difference between Finland and the US is over-exaggerated, shouldn't we say the same about the US and Slovakia? Why are they whining?
My point is that the income differences can't tell the entire story. $9,000 may go a lot further in Slovakia than it does in the US. I don't know, but neither does Ripley.
And, as I will show tomorrow, if you want to compare truly "affluent" students - using Ripley's preferred measure - the US does not come up short.
Again, we're in an Occam's razor situation. You and Ripley keep trying to convince us that the relatively high performance of students in our low-poverty districts is due to the fact that the US has so fewer poor kids in those districts than low-poverty countries have in their entire nations (Ripley misstates this in her first post to assert that the US has NO poor kids in these districts, which isn't true).
But we know child poverty and income inequity are a major problem in this country. Isn't the simpler explanation that we haven't dealt with these as well as the nations we know have a strong safety net?
Post a Comment