I will protect your pensions. Nothing about your pension is going to change when I am governor. - Chris Christie, "An Open Letter to the Teachers of NJ" October, 2009

Thursday, January 5, 2012

Stupid Teacher Evaluation Tricks

In our brave, new, reformy world, teachers need to be evaluated through multiple measures. So we're going to use test scores to rate their effectiveness, but we're also going to use observations from their principals. Because it wouldn't be right to just use one or the other: reformy types all agree on this.

But how we will make both work together? Lucky for you, America, the great state of Tennessee is leading the way on teacher evaluation:
In Murfreesboro City Schools, principals rated nearly half the teachers a five — the best score possible on the state’s new evaluation. 
But in Fayette County Schools in far West Tennessee, only 1 percent garnered that rating. 
The first glimpse of how educators fared under the system, which ultimately will affect whether they earn and keep tenure, demonstrated how subjective the process can be. The Tennessee Department of Education released principal observation data in December after The Tennessean and Williamson County Schools Director Mike Looney filed separate open records requests for it. 
Looney said he wanted the data for comparison after Williamson principals rated 97 percent of teachers a three or higher, and state education officials questioned those ratings. He said his county has a high level of teacher talent plus motivated students, and state officials shouldn’t pressure districts to align scores with projections. 
“To come to some conclusion that our scores are too high ... is preposterous,” Looney said. “We are not going to feel compelled or pushed into making our teachers fit some bell curve.”
Oh, Mr. Looney, you poor, deluded man. Don't you understand? Everyone knows that there must be a certain percentage of teachers who suck. Rich people say it's so; who are you to argue? We already do it with the kids; read Todd Farley. He tells stories of adjusting standardized test grades to make them fit pre-determined distributions drawn up by psychometricians. If we're doing it for the kids, why shouldn't the teachers also be subject to the statisticians' whims?

Of course, this logic requires that we all accept certain... contortions:
Tennessee and 16 other states redesigned teacher evaluation models in the past two years, tying ratings to student test scores, according to the National Council on Teacher Quality. Tennessee, further along than most, both designed and piloted its new system in the 2010-11 school year and put it into effect for all districts this school year. 
Under the new system, 35 percent of the final score is on student learning gains and 15 percent on data the school chooses, such as ACT scores. Principals use a long list of measures for success to do their observations, which count for the other half. 
The state predicted that districts would rate 3-5 percent of teachers as ones; 10-25 percent as twos; 40-50 percent as threes; 10-25 percent as fours and 5-10 percent as fives. No district that submitted data hit all those ranges. 
The projections were based on value-added scores — which measure how much students learned in a year — how other districts using the same observation form distributed scores, and research from the National Institute for Effective Teaching, said Emily Barton, the state’s assistant commissioner for curriculum and instruction. [emphasis mine]
Oh dear - you mean the principal observations didn't match up with the value-added scores (the scores based on standardized test results)?

Wait - I thought the entire point of the principal evaluations was that they would mitigate against unreliable value-added scores. We all know the tests are insanely unreliable as measures of teacher effectiveness, but I thought we teachers didn't need to worry about that. These (usually irrelevant) test scores were only supposed to be a part of our evaluations: only 50%, or 30%, or even 20% of your final rating.

So it should be a good thing that the test scores don't match up with the principal evaluations. We should be happy about this - it means the system is working!

Right?
At the end of the school year, districts with observation scores that don’t closely align with value-added growth scores could penalize principals by taking 10 percent off their own evaluations, Barton said. For now, the state focus is that educators continue to get used to the new evaluations and that teachers get constructive feedback.
Let's recap:
  • Test scores are admittedly unreliable measures of teacher quality.
  • So we need principal observations to mitigate against their unreliability.
  • But we're going to check principal observations against test scores.
  • And if there is a discrepancy, we'll punish the principal.
I need a drink...

9 comments:

Anonymous said...

Duke, do you really believe that no New Jersey teachers "suck" in your words?

There are no mid-life alcoholics? No child abusers? No test cheaters?

Your vitriolic flapping makes you sometimes fall off the perch of sanity.

Duke said...

Anon, go through this entire blog and find me one - just one - instance where I say that no teachers suck.

Of course there are bad teachers - even ones who earn tenure. I get that; everyone who works in schools gets that. That's not the issue.

The issue is whether we are going to use methods to make high-stakes decisions that are contradictory, capricious, and - in this case - just plain old stupid.

Further: why aren't teachers themselves the ones driving this? Why aren't we using high-quality research, instead of this think-tanky crap? Why do we listen to Michelle Rhee and Derrell Bradford and Chris Cerf over true experts in the field?

You posit a false choice, my friend: either we accept a clearly idiotic scheme like the one here, or bad teachers roam free. There is no reason we can't find a fair, effective way to remove bad teachers.

Also: we really don't know how many "bad" teachers there are. There is certainly plenty of evidence that the "problem" of bad teachers is way overblown, especially compared to poverty.

Finally: spare me the concern trolling. I've been doing this long enough that you know what you get with me. I back up what I say with plenty of high-quality sources. If my snark makes you uncomfortable, sorry, but it's not changing any time soon.

jcg said...

Hi Jazzman,
Do you know how difficult it is to walk into a classroom of 25 enthusiastic pre-service teachers and try to explain to them that sociopaths designed and adopted the TN TEAM evaluation system and they will be forced to endure it if they stay in the state to teach?

TN's Teach for America cultists in DoE insist that a 1 -5 Likert scale and an incomplete checklist of teacher behaviors is ALWAYS reliable and valid, not a principal's lying eyes, or, gasp, a teacher's professional judgement. The Milken Foundation TEAM evaluation trainer told us that if scores don't align into a normal distribution that "principals are gaming the system". That's the logic driving TN DoE and 29 of Nashville's 1%ers.

In the end, it's going to be the 29 usual suspects who make the decisions about TEAM- the wealthy special interests for whom our state officials work:
(h/t http://enclave-nashville.blogspot.com/search?updated-max=2011-12-01T21:28:00-06:00&max-results=20)

"After the state agreed to ease up on overworked educators and streamline the unrealistic teacher evaluation process, the Music City mogul class flushed apoplectic:

Keel Hunt, a public relations executive on the steering committee of Nashville’s Agenda, said education reform has emerged time and again as a focus for the group and Nashville. He is one of 29 people who put their names to the letter — others were Metro Nashville Director of Schools Jesse Register and Orrin Ingram, president and CEO of Ingram Industries.

Their letter contends that changing the evaluations could jeopardize the Race to the Top grant.

“I hope (the letter) helps underscore the importance that these people feel this issue has,” Hunt said. “It’s very important to stay the course. ... This is very important work, and there is broader interest in school success that goes beyond what one or two people feel.”


To the 28 billionaires(and 1 Broad trained superintendent) who signed this letter, disagreement with TEAM is simply whining from a couple of overwrought women.

Anonymous said...

I don't have to search your site, (what a wade through a bilious swamp that would be) you said this above:

"Everyone knows that there must be a certain percentage of teachers who suck. Rich people say it's so; who are you to argue?"

That seemed to me to be in your usual sarcasm font that meant you felt their WEREN'T a certain percentage of teachers who "suck". If there aren't a certain percentage that suck then, logically "no teachers suck", in your opinion. Or I read it that way, seems awfully reasonable as I look at it again.

So, besides proposing a thousand year think tank for a debate that has gone on for decades, what is your formula for measuring teacher performance, identifying poor performers, attempting to improve them and removing the ones that don't imporve?

Not platitudes -- specifics. God knows that you have preached enough about how wrong the current plan is that you MUST have a specifc plan (or admit to being a hypocrite).

Duke said...

Anon, I write in English. Yeah, I dangle my prepositions way too often, and split my infinitives, and commit a host of other sins against the language...

But I'm pretty clear in what I say. If you misread it, that's on you, not me. I never said there weren't bad teachers, and your parsing of my words to arrive at that conclusion is worthy of Fox News.

As to my proposals: they are all over this blog. If you don't care to find them, that's also your problem. But I'll give you one for free: the NJEA tenure plan is a serious proposal and would do a lot of good while saving money. Have you read it?

By the way: I'll write about what I want to write about, thank you very much. I don't respond to demands from trolls to outline things the way you want them. I've been very clear about what needs to be done - far more than the reformy types who speak in platitudes far more than I do.

Meg in Mich said...

If there are Teachers that suck - its like anywhere else - a good HR manager (I guess that is the principal) would evaluate them and fire them. In Michigan you have 4 years to figure this out before allowing the state to award tenure. And then if you do have tenure - 3 bad evaluations and a teacher can be fired. Pretty simple - in what psycho world do we create another layer of bureaucracy to make this more complicated? Perhaps one where either HR managers just arent doing their jobs and blaming the unions. LAZY - I am a teacher and in my whole 18 years barely a soul has even asked me what I do. Therein lies our problem. This is an artificial crisis designed to bust unions for corporate gain.

Duke said...

Tell you what, Anon: I'll save you the horror of looking through the cesspool that is my archives:

http://jerseyjazzman.blogspot.com/2012/01/repost-ok-smart-guy-what-would-you-do.html

Specific enough for you?

Duke said...

Meg, well said.

jcg, totally stealing your post for tomorrow!

jcg said...

Jazz, no sense in arguing with AnonTroll- 'it' has NO idea how to interpret test statistics.

The fact that Milkin's TEAM teacher eval completely distorts the meaning of reliable and valid measurement is lost on 'it'.

A real test of validity and reliability of TEAM would require an aggregate of the principals' scores (with strict controls for external threats)and several rewrites and retests of the instrument across a broad population of teachers. THAT is how quality assessments are developed and it takes years and money to do this type of research.

But the socipoaths and money addicts in the corporate edu-business can't wait for real research- they need a reliable source of income from our tax dollars now, before the next bubble crashes their risky investments.

Further, TN corporate edu-deformers and test-profiteers begin with an unfounded, unscientific, faulty assumption: that in every school, city, district, and state 25% of the teachers are ineffective. Everyone who understands test construction knows that when an assessment is designed based on faulty assumptions, that the assessment is invalid.

Thus, we have a teacher evaluation model designed on distortions of assessment reliability and validity and an instrument developed from a false assumption.

This model was developed by the same group of thieves who are running the financial industry and who are fighting accountability to taxpayers who bailed them out with millions of dollars in lobbying.

Take a little personal satisfaction that Anon is either a profiteer, a dupe, or a persona. He's got nothing over on you and our professional ethics.