I will protect your pensions. Nothing about your pension is going to change when I am governor. - Chris Christie, "An Open Letter to the Teachers of NJ" October, 2009

Sunday, August 21, 2011

School Ranking Fail

Every year, the Washington Post's Jay Matthews ranks America's high schools. His system makes things easy for him: he simply takes the number of AP or IB tests a school gives each year and divides it by the numbers of seniors. He doesn't bother with the actual results of the tests; too complicated, I guess.

Since I'm a Jersey guy, I thought I'd look up the Jersey schools. #1 is Dr. Ronald E. McNair in Jersey City, which is a magnet school and, I understand, outstanding. I thought Matthew's rules would have excluded it, because competitive schools like Stuyvesant in New York City with high concentrations of the best and the brightest are supposed to be left out of this ranking; I guess McNair made it. Good for them.

Jerseyites, guess who came in at #2? Millburn? Ridge? Princeton? Chatham?

Nope. Elizabeth.

Yeah, me too. Let's take a look:

2009-10 SAT Math Scores:
State Average: 520
Elizabeth HS Average: 445

2009-10 SAT English Scores:

State Average: 496
Elizabeth HS Average: 429

2009-10 SAT Essay Scores:
State Average: 499
Elizabeth HS Average: 432

Does this seem like a school that ranks highly on, and I quote, a "measure of how effectively a school prepares its students for college"?

Don't get me wrong: I think Elizabeth is actually doing a pretty good job (especially for a district that seems to feel it's important to put pictures of the Board of Education on the front page of its website - a little tacky, folks). 54% of the students speak Spanish as their first language at home, so it's not like Elizabeth doesn't have a real challenge in getting kids ready for a career or college.

It's especially hard since the faculty has the specter of a corrupt BOE hanging over their heads all the time.

But is it the #2 school in the entire state for college preparation? I really don't think so.

Hey, look who's #11: North Star Academy, a charter school! How are they doing with getting kids ready for college?

2009-10 SAT Math Scores:
State Average: 520
North Star Academy Average: 516

2009-10 SAT English Scores:
State Average: 496
North Star Academy Average: 474

2009-10 SAT Essay Scores:
State Average: 499
North Star Academy Average: 477

Let me just add some words about North Star from Bruce Baker:
My point here is not that these are bad schools, or that they are necessarily engaging in any particular immoral or unethical activity. But rather, that a significant portion of the apparent success of schools like North Star is a) attributable to the demographically different population they serve to begin with and b) attributable to the patterns of student attrition that occur within cohorts over time.
Click through to see Baker's entire analysis. Still, for a school that has 55% of it's students qualify for free lunch, getting this close to the state average in the SAT English and essay tests - and beating the math average - is no small feat.

But Matthews says this is the #11 school in state for college preparation. Come on.

Readers from other states - take a look at your results. I'd be curious to know what you think.

What's the point here? Well, lets start with this: stupid analytical methods lead to stupid results. It may well be that Mathews is dealing with a corrupt data set here (the NJ report card for Elizabeth HS shows 236% of the eligible students took the SAT; pretty neat trick. Do the juniors and some of the sophomores all take the SAT too?). But I think the big problem here is that Matthews is trying to take the easy way out, and he's getting results that don't make a lot of sense.

Another thought - Matthews says:
While not a measure of the overall quality of the school, the rating can reveal the level of a high school’s commitment to preparing average students for college.
Well, if it's not a measure of overall quality, then why are you ranking the schools? Aren't these rankings going to be inevitably interpreted this way? Why do this exercise in the first place?

Data is a lot like a high-powered automatic gun: nobody should use it in public unless they have been properly trained and understand its potential for disaster. Putting this stuff out in the open without really refining the methods is just asking for trouble; just look at the LA Times, which is opening a huge can of worms with their ill-advised and hugely inaccurate teacher evaluation scheme.

Of course, like the LAT, the Post is making money off of the deal. Every time I've signed on tonight, I see a B4K banner ad on top (now we know where some of that hedge fund dough is going). This is the sort of easy, cheap stuff a newspaper in dire financial straights can pump out quickly, no matter the inaccuracies.

So goes our media's coverage of education...

3 comments:

Jason V. said...

I think this obsession with numbers (numerical rankings, standardized test scores, etc.) is a modern manifestation in American society of anti-intellectualism. Trying to distill everything into a super-simple number that will tell us all we need to know, where 'higher = better,' overlooks all the nuances and variations that are inherent in all human activities, in favor of a quick, 'now-I-know-it-all' answer.

Call it "The Cult of the Data Oracle".

Duke said...

Jason, I've not thought of it in this way, but I think your make a very perceptive point.

Rankings ALLOW us not to make serious evaluations.

Personally, I do that all the time. Want to go to a nice restaurant? What's the Zagat ranking?

But our thinking should be much more rigorous with something as important as schools.

Thx for commenting.

Lisa said...

The entire premise of this ranking is patently absurd--and actually harmful. It's an ill-advised strategy for kids trying to get into their choice of college, and damaging to self-esteem, which is crucial for academic and personal success.

I know many honor students who graduated in the top 2%-5% of of their class and attended top universities who CHOSE not to even take AP and IB courses because they and their parents (understandably) believed that the chance of getting a 95 in an AP course, when they were guaranteed a 100 in an honors course, wasn't worth the risk of lowering their cum. I realize this is anecdotal, but after getting two kids into college, I saw this often in my own family and peers. College entrance and scholarship packages are particularly competitive right now. A difference of a few 100ths of a point in a student's cum can drop his class ranking two or more places. I know of many kids who chose not to take an AP course or two and risk the cum and class rank they worked so hard to attain. My own son made this choice in the subject area that wasn't his best, and he graduated HS with 20 college credits (his best friend with 48) from AP courses. It's also not cheap to take AP exams. Each exam fee is now $87, and taking three or four exams (esp. with two or more kids) is a nice chunk of change, so some kids take the course but don't take the exam if they feel they're not guaranteed a 4 or 5, which is usually needed for credit in selective universities. This variable alone renders the statistics flawed, if not useless.

Aside from the strategies used to get into one's choice colleges, Mathews states, "AP, IB and AICE are important because they give average students a chance to experience the trauma of heavy college reading lists and long, analytical college examinations." The key word in that statement is "trauma," and an interesting and accurate choice of words it is. He is advocating trauma for average kids. To what end...and benefit?

In order take all the AP courses available, students need to start taking them in their second year of high school. AP courses are college courses. I question the psychological benefit of counseling a 14 or 15 year old adolescent who is a solid, top student in average, above average, or even honor classes enjoying success, to submit their self-esteem, self confidence, and motivation to the battering it takes to struggle in the bottom of a college course. How does this benefit them, their academics, and their growth and development? I would like to see the research that Mathews cited that supports his claim that "low-performing students who got only a 2 on an AP test did significantly better in college than similar students who did not take AP," and a follow-up study on how many are on anti-depressants.

As a parent I find this ranking's premise appalling, and as an educator I find it sophomoric and reckless.