I will protect your pensions. Nothing about your pension is going to change when I am governor. - Chris Christie, "An Open Letter to the Teachers of NJ" October, 2009

Friday, May 3, 2013

Computerized Testing Disaster Preview

It's going to be so freakin' awesome when we finally get all the kids taking all those Common Core and PARCC and state-level standardized tests on computers! Because - considering they must all take them at the same time - what could possibly go wrong?
School testing came to a halt statewide early Monday because the CTB/McGraw-Hill testing company servers in New Jersey crashed around 9 a.m., state education officials said.

In the last week of Oklahoma's April testing window, the outage has raised rescheduling concerns among school officials across the state.

"We're working with (McGraw) very closely" to resolve the problems, said Sherry Fair, spokeswoman with the state Department of Education.

School districts were advised to cancel online testing because of the server crash, although some students in a few schools somehow were able to complete tests.

The outage affected Oklahoma Core Curriculum tests for grades three to eight and end-of-instruction tests for students up to grade 12, officials said. [emphasis mine]
Golly, a computer problem caused by thousands of users accessing servers at the same time. Who could have predicted?
A few online tests have alternates available. But a state Education Department official told schools most incomplete tests will be invalidated and require pencil-paper testing.

Educators have little faith that McGraw could get paper tests to schools by Friday, particularly since the company was late in providing them before testing in the first place.

"I think it goes without saying that a company bidding on high-stakes testing should have the technology infrastructure to avoid outages such as this," said Joe Slitzker, information technology director at Sapulpa Public Schools. 
Read the whole thing. McGraw-Hill sounds like a clown show operation. And what happens when the problems are at the client end - when the schools' computer network is the problem?

Here in New Jersey, the DOE is pushing on-line testing hard. They seem to think it will work just fine; but how would they know? And where is the backup plan when something like what happened in Oklahoma inevitably happens here? Because there's no doubt what happened in Tulsa is not an isolated incident:
School districts across several states are rescheduling high-stakes tests that judge student proficiency and even determine teachers' pay because of technical problems involving the test administrators' computer systems.
Thousands of students in Indiana, Kentucky, Minnesota andOklahoma have been kicked offline while taking tests in recent weeks, postponing the testing schools planned for months and raising concerns about whether the glitches will affect scores.
"There's been pep rallies and spirit weeks all getting ready for this. It's like showing up for the big game and then the basketball is deflated," said Jason Zook, a fifth-grade teacher at Brown Intermediate Center in South Bend, Ind.
Many frustrated students have been reduced to tears and administrators are boiling over, calling the problems "disastrous" and "unacceptable" at a time when test results count so heavily toward schools' ratings under the federal No Child Left Behind law. In places such as Indiana, where former Gov. Mitch Daniels approved changes tying teachers' merit pay to student test scores, the pressure is even greater.
Nice job, Mitch: enjoying your new office, you massive, wimpy hypocrite? Of course, it was just so absolutely critical that teachers' pay get tied to test scores, even if the incompetent Daniels and his minion, Tony Bennett, couldn't set up a system that works. Now Bennett's gone and Glenda Ritz has to clean up his mess.

She won't be alone for very long, though: these thoughtless decisions will plague our schools for years - and that plague has already spread to states all over the country. The reformies' blind faith in technology isn't going to save their agenda; it's actually going to make things far, far worse.

I'm sorry, Dave, but I'm afraid I can't administer these tests...


Unknown said...

Not to mention that the schools themselves don't have the technology to implement PARCC, as explained by Madison Superintendent Michael A. Rossi, and described as a "train wreck" in his open letter to the NJ State BOE. PARCC is only one small part of the problems districts are having with all the rushed initiatives being pushed on them.

Excerpted below is his report on the PARCC initiative... keep in mind that Madison is a pretty wealthy district. I recommend reading the rest of his letter as well.


"PARCC: I cannot say this any other way but to describe it as a train wreck right now. The power point on this mentions no less than 12 tests for our current 6th graders when they reach high school. Those would be on top of Common Core assessments, benchmark assessments in non-tested areas, SATs, ACTs, APS, and then classroom teacher tests. Functionally, at this point, Madison cannot implement PARCC because the design is to be on computers that do not use XP as an operating system, which our entire district has. Even with the resources here we cannot turn that many computers over by 2014-2015. Combine all this with the most recent announcement that PARCC will not work with I-PADs and other BYOD initiatives suffice it to say we will grind all computer assisted teaching to a halt to do the PARCC testing."

Seems to me that "train wreck" was being generous... and optimistic.


Mrs. Droz said...

13 years into the 21st century, and we are still stuck in 20th century mindsets regarding education. The disconnect with these "reformers" regarding education is staggeringly mind boggling. They are wasting millions of dollars on reforms which do nothing but push the same methods of teaching (drill and test) that were used in the 20th century. We need to teach our students competitive skills to help them not just get jobs, but create jobs that haven't been developed yet.

Districts who get it by using BYOD, and other means to get into the 21st century, are held back by these "reformers" who are ironically woefully behind the times with their ideas on reform. They need to look at people like Ian Jukes who cites studies done on students that the visual stimulation given to children is rewiring their brains so much that they remember visual information over time tremendously better than auditory information. This means, as educators, we have take this into account when we think of pedagogy. But, when assessments are being steam rolled in, how can we.

alm said...

When you take classes on assessment design/psychometrics, you learn about something called the 'test information function.'

A TIF is a way to quantify how much new information a test can provide about a learner's performance on a subject.

Imagine the readers of your blog were talking a physics assessment, but all of the questions were taken from 400 level/PhD coursework, and we all score a zero. For the vast majority of the population, the TIF of the test I just described would be zero, or close to zero -- anyone with less than a graduate degree in physics would not be expected to answer any of the questions correctly. The test would confirm that we are below PhD-level proficiency in physics, but wouldn't help determine who was ready for Physics 101 vs., say, those who needed some math remediation.

What's my point? Well, a very low performing (or high performing) student taking a fixed, pen-and-pencil version of a state assessment can experience something along the lines of what I described above -- the test instrument can be so out of pace with their ability that the process provides very little information about what instruction the student needs next.

A computer adaptive test can avoid this problem by tailoring the presentation of questions based on the student's responses. This means:

--better information about *all* students -- not just the kids at the center of the distribution
--better supports for students with special needs
--potentially less time testing

There are definitely potential issues with any transition. We need good people looking at this and testing these systems. But we have the technology - Netflix reliably delivers streaming video to millions of households a day. It can be done. Writing performance guarantees or penalties into the contracts is one possible avenue.

The overwhelming negativity and deep cynicism of evident in this post is probably what Alexander Russo meant when he called your blog "overheated".

Unknown said...

Re: alm
I see you haven't watched an eight-year-old with ADHD take a computerized test in a computer lab. #potentialissue #uselessdata


alm said...

Indeed, I have :).

It is a potential issue -- the potential richness of a computerized test offers the chance to exactly that sort of useless data. See for instance http://www.kingsburycenter.org/sites/default/files/Student%20test-taking%20effort%20and%20teacher%20effectiveness.pdf

The issues that you raise are reasons that we should use caution moving forward. But caution is about nuance -- and nuance requires everyone to dial back the rhetoric just a bit.