I will protect your pensions. Nothing about your pension is going to change when I am governor. - Chris Christie, "An Open Letter to the Teachers of NJ" October, 2009

Thursday, November 7, 2013

No One Manipulates the Press Like TFA

In general, I like Leslie Brody of the Record: she's smart and fair and usually does a good job. Which is why I found this puff piece on Teach For America in New Jersey so frustrating:
Teach for America got a major public relations boost in September when a federally funded study found that on average, its “corps members” slightly outperformed teachers who entered the profession after traditional teacher preparation, which can involve years of course work and supervised practice. The eight-state study by Mathematica Policy Research found that on average, the students of Teach for America corps members made more gains in math in one year – amounting to an average of 2.6 months of additional learning time. It was unclear whether the program’s selectivity in recruiting or other factors accounted for its positive results. 
The report came at a time of intense debate over how to attract quality candidates into teaching, especially in high-poverty areas, and how to help them succeed in the job. [emphasis mine]
Sigh. Look, I know this stuff is knotty. And kudos to Brody for talking to Aaron Pallas, one of the smartest guys on the planet when it comes to this stuff. But nobody should be taking TFA's word on their "success" without really looking at what the research says. And, as I wrote in September, the Mathematica TFA study is hardly a ringing endorsement:
- Effect sizes matter. The study shows a difference of .13 standard deviations in high school math and .06 standard deviations in middle school math. That is equivalent to moving a student from the 27th percentile to the 30th. The study uses the "x months of learning" conversion to say that's 2.6 months of learning; I find that to be misleading at best. Having four TFA teachers in a row isn't going to mean that a student will take calculus in their senior year instead of pre-calculus: these effects aren't necessarily cumulative.

What really should be reported is how many more questions the TFA students got right on the final tests. And it can't be many when an effect size is that small. Statistically significant? Sure. But practically significant? Come on. And I'm not even going to deal with the scaling problems here...

In addition, TFAers come from elite schools, which means they generally have good test-taking abilities. Are they able to impart those abilities on to their students? And is that "real" learning, or simply learning how to beat the system?

In any case, the results here seem like a great candidate for a treatment from the Mountain-Out-Of-A-Molehill-INATOR: there's just not that much here to get excited about.

Some randomness isn't full randomness. OK, so the kids are randomly assigned -- but the teachers weren't. The TFAers aren't going up against the entire population of "regular" math teachers; they are matched only with those teachers who:
  • Teach the same subject,
  • In the same school,
  • To which only TFAers are assigned.
That is a very, very limited control group from which to draw broad conclusions about the effectiveness of TFA. There is plenty of reason to suspect that many districts do not distribute teachers across their schools equally; and that's not even addressing the distribution between districts. 

So the students may be randomly assigned, but the teachers in the control group most certainly are not. That is a big hit to the generalizability of this study. And speaking of those teachers...

Heterogeneity of the control group. The study tries to account for some differences between the control group teachers, but it is a very limited description. That's not a criticism; it's simply pointing out the limitations of the study. Colleges are reported dichotomously as "selective" or "not selective," as if there isn't a whole world of difference between programs in "not selective" colleges (for example: state schools with scholars as faculty vs. crappy, on-line, for-profit "universities"). Interestingly, TFA teachers were more likely to have degrees in secondary math education than comparison teachers (18.8% vs. 15.9%).

I also found this telling: 70 percent of both the TFAers and the comparison teachers had 20 or fewer days of student teaching in math. That says as much, to me, about the quality of training of the comparison teachers as it does about the TFA group.
And so on. Again: I know this is tough stuff, and Brody did present an opposing view to the usual TFA propaganda. Maybe I'm just annoyed at how damn good TFA is at getting their brand out there with very little challenge to the hegemony they've created for themselves, both in the media and political spheres.

But TFA isn't a "solution" to any "problem" with urban education. As Diane Ravitch has said: we're not going to staff our schools with Ivy League graduates, because the raw numbers needed are just too great. Let's focus instead on making the profession more attractive to college graduates of both private and state colleges and universities. Let's start improving teacher working conditions, which are student learning conditions.

Relying instead on the kindness of strangers who don't stay very long is not a very well-though out strategy.


alm said...

propaganda? yikes - It's a Mathematica policy evaluation - one of the most rigorous and independent in the business.
An effect size of 0.13 for a 1-year treatment isn't enormous, but it's certainly nothing to sniff at. If you google black-white achievement gap you'll see estimates that range from 0.8 SD to 1.0+, depending on grade level -- you'd need 6-8 years of a 'treatment' of that size to close the black/white achievement gap, provided that the impact was additive. Another benchmark - that's roughly the same effect size as reductions in class size in the Robinson '90 meta-analysis of class size research. (If someone has a more current statistic there, please post.)

The TFA literature on closing the achievement gap could stand to be a little more sober on the size of these impacts, but it's certainly directionally in TFA's favor, and reasonably educationally meaningful.

re: TFA as a solution - are there enough Ivy graduates to staff every position in high poverty schools? certainly not. Are there enough high achieving college students generally (top 10% of their class or above?) well, getting closer. Are there enough of those students to 'shift the quality curve'? (replace lowest performing 10% over time, either through natural attrition or more rigorous eval programs?) yes, certainly.

chunga said...

@alm - the effect size is small enough that the other questions/limitations of the study make the claims about it clearly overblown, including those of Mathematica (in spite of their reputation). This is especially so when much of the prior research doesn't match these results (refer to http://www.greatlakescenter.org/docs/Policy_Briefs/Heilig_TeachForAmerica.pdf or http://uaedpolicy.weebly.com/uploads/6/1/7/1/6171842/tfa-2_1.pdf or http://reconsideringtfa.wordpress.com/research/).

Mrs. King's music students said...

I'm confused. Why aren’t we asking why an Ivy Leaguer would go into teaching in the first place? And why would anyone think that the Ivy League fools who do pick up the flag (hi Larry) aren’t dogged mercilessly by incompetent/lowest common denominator ed. admins and union leaders who don’t want to be shown up by them?

In districts like Camden, the real problem for Christie is not the training of quality teachers that’s holding back achievement. It’s the refusal of top-notch people to work or take the fall for him.

Mrs. King's music students said...

PS - To separate actual teachers from the profiteers in TFA - their blogs are very complimentary toward trad route teachers and school leaders who end up mentoring them on the job.

Leyo said...

I have been impressed with your message. Great view! your content was very pleasant and interesting to readers. Thank you very much for posting this.The claims pages