I will protect your pensions. Nothing about your pension is going to change when I am governor. - Chris Christie, "An Open Letter to the Teachers of NJ" October, 2009

Monday, March 2, 2015

@NJSBA and PARCC: Going Along to Get Along?

The resistance to the PARCC -- the new, standardized, computerized tests being administered in New Jersey beginning this month -- continues to grow. Parents, teachers, and students are rightly concerned that these tests are taking too much time, are unnecessarily complex and confusing, disadvantage students with less access to technology, and narrow the curriculum.

In response to the grassroots movement to opt students out of the PARCC, a coalition has formed, consisting of various education stakeholder groups across the state. We Raise New Jersey includes the NJPTA, the Principals and Supervisors Association, the New Jersey Chamber of Commerce, the New Jersey Council of County Colleges, JerseyCAN, and the New Jersey School Boards Association (NJSBA).

I find this list to be very interesting. As my blogging bud Darcie Cimarusti points out, the national PTA has received millions of dollars in funding over the years from the Gates Foundation, the principal driver of the Common Core standards to which the PARCC is aligned. JerseyCan also gets funds from Gates, among other reformy groups.

But it's the NJSBA that really caught my attention. This is a group that is supposed to serve school boards across the state. And yet the PARCC is clearly an unfunded mandate from the state on local school districts, draining resources away from operations and toward the administration of tests. This is especially onerous for local school districts who have had to upgrade their computer networks, yet received no additional funding from the state.

I would think the NJSBA would have lobbied against the PARCC; at least, they should have insisted that their member school boards receive funds from the state to help defray the costs of this mandate. Instead, they're supporting a testing regime that will hurt the bottom lines of their member school boards' budgets. Why are they going along with this?

Perhaps because the NJSBA itself is an unfunded mandate. Let's go back to 1997, when the funding of the NJSBA was a source of contention:
For more than 80 years, state law has required every school district in the state to be a dues-paying member of the New Jersey School Boards Association. But that could change with the passage of a bill pending in the state legislature that proposes making membership in the statewide organization voluntary.
The suggested change has stirred some controversy over which setup would better serve the state's 1.2 million public-school students. And some school board members are accusing lawmakers of trying to weaken the school boards association as payback for the organization's vocal opposition earlier this year to new education funding laws.
The school boards association provision is only one of several dozen state mandates that would be abolished under the legislation, which is scheduled to face the Assembly and Senate appropriations committees today. The school boards association and its supporters are calling for legislators to amend the bill to keep association membership mandatory.
Districts pay an average of $11,700 in annual dues to fund the association's $8.9 million budget. Association advocates say it would cost far more for districts to replicate the services their dues provide. Belluscio said association lobbying efforts have led to legislative changes that have saved school districts hundreds of millions of dollars.
But legislators who support the change say that lifting mandatory membership would force the association to be more efficient and more accountable to its constituents.
They also point to neighboring Pennsylvania, where 500 of the state's 501 school districts voluntarily participate in the school boards association. In fact, New Jersey and Washington are the only states that mandate district membership in school boards associations.
If membership were made voluntary, Belluscio said, the school boards association would have to shift resources away from direct services to marketing itself.
But observers say that's not necessarily a bad thing. For the Pennsylvania association, that has meant tailoring services to constituents.
``We've had virtually 100 percent membership over the years,'' said spokesman Thomas Gentzel. ``We like to think it's because we provide a good array of services for our members and we have to be responsive to them.''
And then there's the issue of the possible political motivation behind New Jersey's proposed legislation.
``There has been some talk that this is kind of a payback move,'' said Lindenwold school board member Jim Dougherty. ``There is some discussion that the legislature and the governor are more than a little ticked off because of the position the association took in the funding situation.'' [emphasis mine]
The NJSBA, it seems, has a history of walking on thin ice: piss off the wrong people, and its unusual source of mandatory funding could be put in jeopardy.

Fast forward to 2010, when the NJSBA once again found its mandatory dues questioned:
The publicly financed lobby for New Jersey’s school boards is spending millions to renovate its headquarters, even as local districts face massive state aid cuts, defeated budgets and construction proposals, and pending teacher layoffs. 
The New Jersey School Boards Association collects more than $7 million a year from 588 member districts, which are legally required to join. It has socked away so much in dues and conference fees — $12.3 million, an amount greater than the group’s annual operating budget — that it is paying cash for the improvements. 
It also paid $1.6 million in cash for 10 suburban acres where it had hoped to build an $18 million conference center. But the board abandoned that plan and put the land back on the market. 
The most recent projected cost for the headquarter’s renovations was $6.3 million. But that figure could grow an additional $600,000 to $1 million, as the contractor decides whether to fix or replace the building’s walls of glass windows, officials said. In the meantime, its 70 employees — including five lobbyists paid to influence legislation — are working in leased office space. [emphasis mine]
Remember, this was back in 2010, when local school budgets were being decimated. In those lean times, the practices of NJSBA were not siting well with taxpayers or legislators:
The School Boards Association has come under some criticism in recent months, after The Record reported that its staff is enrolled in the state-run health and pension systems, even though they are not government employees. Workers at two other Trenton lobbying groups — the Association of Counties and the League of Municipalities — also are in the programs, as a result of 1950s legislation that declared they were acting in the public interest.  
In all, New Jerseyans will fund retirement payouts and lifetime health benefits for 107 non-government employees with combined pensionable salaries of $7 million. Right now, taxpayers are giving $1.3 million a year to 62 retirees of the groups. Gov. Chris Christie has said the benefits arrangement must end.
Given all this, the NJSBA found, once again, that its funding was under scrutiny:
Last year, New Jersey districts paid $7.6 million in dues — a 73 percent increase from the $4.4 million paid in 1999, according to the association’s financial reports. In 2009, it also had revenue of $2.7 million from conference fees, ad sales and services. 
This year, dues will be 5 percent lower, said Frank Belluscio, a spokesman for the association. Each district will receive a $2,000 credit to apply toward services. As a result of the law Corzine signed, the group also has replaced its annual three-day Atlantic City workshop with a shorter program in Central Jersey, which shortens the drive for most participants and eliminates the need to stay overnight at school board expense.  
Belluscio also pointed out that many districts get back their dues and then some because of their affiliation with the association’s energy cooperative.  
The give-backs to the districts pleased Assemblyman John Burzichelli (D-Gloucester), who in December had proposed making association membership voluntary rather than compulsory. Last week, he said he is withdrawing the bill. 
"It seems to me that the service they’re providing is very helpful, especially as we’re going through this transition between potential consolidations, upheavals in school districts and things of that nature," Burzichelli said. "I applaud them for taking the steps they’re taking."
See how it works? Don't make waves, keep legislators happy... and nobody messes with your source of revenue. Win-win. Want another example?

Plug "NJSBA" and "SFRA" into Google. SFRA is the School Funding Reform Act, the state's school aid formula, which has not been fully funded since Chris Christie came into office and is now $6 billion behind what the law dictates. You would think the group that represents local school boards would be apoplectic over the state's repeated refusal to come up with the money its member districts are due.

But the best I could find from NJSBA in the last few years on school underfunding -- a period where inequality between high and low-spending districts has increased rapidly -- was this, from 2011:
“The New Jersey School Boards Association believes in fair and equitable distribution of state aid," said Raymond R. Wiss, president of the New Jersey School Boards Association via release.  "In 2008, NJSBA supported the principles of the School Funding Reform Act, based on the act’s recognition that at-risk students attend schools in communities throughout New Jersey, not just in 31 communities.  The 2008 funding law also attempted to help those middle- and moderate-income communities, which suffer from high property tax burdens and still have been unable to fund their education programs at levels considered adequate by the state.
“Today’s court decision does not resolve these matters,” Wiss concluded.
That is some weak, lukewarm tea -- probably because it's several years old. In fact, some of NJSBA publications on the underfunding of SFRA read, to me, more like apologies for the Christie administration than indictments. And if they've had anything to say lately about the Bacon lawsuit, which seeks to remedy the underfunding of rural New Jersey school districts, I haven't been able to find it.

Now, the NJSBA might make the case that lobbying for the full funding of SFRA isn't part of their mandate. OK... then why dive into the debate about PARCC? Why actively lobby and dispense pro-PARCC propaganda with groups like the NJPTA and JerseyCAN? Why take a stand on this issue at all if NJSBA won't even take a strong stand on getting their member districts funding the state itself says they are due under SFRA?

It looks like NJSBA has learned its lesson: if you want to get along, go along. They should embroider it on the pillows in their new offices...

The NJSBA slogan.

ADDING: The New York State School Boards Association is a plaintiff against the state in a lawsuit for equitable funding. I guess the NJSBA is too busy redecorating to get involved in that fight around here...

Sunday, March 1, 2015

Pension "Reform" = Screwing Public Employees

Leave it to the Star-Ledger's Editorial Board to reliably present the plutocratic argument for screwing over public employees:
Gov. Chris Christie deserves the scorn that has been heaped on him for breaking his 2011 promise on pension reform. It was a cynical move that created lasting distrust, making it much harder to find common ground this time around. 
But it's time to move on. Because the hard reality is that without a second round of reform, the state is doomed to a future of spending cuts, tax increases, and economic decline. The math is beyond argument. [emphasis mine]
You see, New Jersey public employees, it doesn't matter that you are under-compensated by 5.88% compared to similarly educated and credentialed employees in the private sector. It doesn't matter that your pension payments are compensation for work you have already done. It doesn't matter that pensions actually save taxpayers money in the long run. It doesn't matter that reformy folks like the S-L Editorial Board think having great teachers is a top priority, and labor market economics dictate that attracting the best and the brightest into teaching means paying teachers competitive wages.

No, all that matters is that we "move on." Which means, of course, that you, New Jersey public employees, need to get screwed yet again:
For a hair-raising analysis of this, consider the most sobering fact presented last week by the governor's bipartisan panel on pension and health reform: It found that without a course correction, these costs will absorb 23 percent of the state budget within two years, roughly double the portion today. 
Whether you are conservative or liberal, that is a potential disaster. It amounts to just over $8 billion a year. 
For perspective, the millionaire's tax favored by Democrats would raise $600 milliona year, barely enough to make a dent. There is no feasible way to raise taxes high enough to cover this cost. 
The same is true on the spending side. You could fire every state employee tomorrow, and you still wouldn't raise enough money to fill this hole.
And here we see the true function of the S-L editorial page: creating a "truth," based on assumed "facts," that takes off the table anything but a modest tax increase on the state's wealthiest citizens.

Let's look, once again, at the state and local tax rates by income for the Garden State:

New Jersey's wealthiest citizens are paying a much lower tax rate than the state's poorest taxpayers. The millionaires tax the S-L is willing to accede to should, at the very least, force the state's top 1 percent to pay the same rate as its bottom 20 percent.

Will that be enough? Almost certainly not, which is why we need to start hacking away at the billions of dollars in tax expenditures this state gives away to special interests. We also need to reign in the outrageous increases in health benefits premiums the state allows every year. Having a conversation about legalizing marijuana is also in order, even though I have serious reservations about going down that road.

Also, as David Sirota has repeatedly pointed out, we need to stop letting Christie's political patrons manage the pensions: they're doing a lousy job and they cost a fortune.

The point I'm trying to make here is that we have more options than just reinstalling a small millionaires tax while taking a hatchet to public employee compensation. But the Star-Ledger doesn't want to consider any of these other courses; as usual, giving public employees the shaft is always the paper's favored plan:
The core conclusion of the governor's commission is that benefits for public workers should be roughly equal to benefits in the private sector. That would result in big savings. The commission estimated that savings on health care alone would amount to $1.7 billion a year, money it would recycle into the pension funds. 
That's a reasonable place to start. The unions hate this idea, saying the answer to this imbalance is to lift benefits in the private sector, rather than set off a "race to the bottom." That's an appealing idea, but not a realistic one. In practice, it amounts to an unreasonable demand that middle-class families should pay higher taxes to cover the costs of benefits they don't receive themselves.
Again: even when accounting for benefits, New Jersey public employees are under-compensated compared to the private sector. So if the S-L thinks it's "reasonable" for public and private employees to have comparable benefits, isn't it also "reasonable" that they have comparable wages? Or do we not care about the quality of our public work force?

You can't keep slashing the compensation of public employees while continuing to expect high-quality candidates to enter public service.

How many capable young people are going to enter law enforcement if they know the benefits their predecessors earned will no longer be there, and their wages won't be any higher to cover the difference? How many bright college graduates are going to enter teaching when pensions and health benefits are removed, but salary remains the same? Why would anyone in their 20s with good prospects be drawn to public service when the private sector pays better and offers the same level of health insurance and retirement benefits?

Back in 1985, Tom Kean, Sr. implemented a plan to raise the minimum salary for teachers. Kean understood you need to pay competitive wages to attract good people into the profession. A few years later, the demagogic Jim Florio, caught up in a recession, wagged his finger at "the education establishment" for paying teachers decently. He then proceeded to short change the pensions, leading to the current mess.

History, then, teaches us what pension "reform" is really all about: cutting the compensation of public employees under the premise of a fiscal crisis. And it's the job of media outlets like the S-L's op-ed page to proclaim that the situation is so intractable that we have no choice but to break promises and allow Chris Christie's bandit friends on Wall Street to keep feeding at the pension trough. And all while folks like the Newhouse brothers, owners of the Ledger, continue to pay historically low tax rates while gobbling up historically high shares of income.

The last thing public employees should do in this situation is back down. The pensions are our money. We earned them. The state owes us this money for work we have already done. The courts have now declared multiple times that the state must meet its obligations.

Don't believe the Star-Ledger when they tell you that it isn't fair to demand that the state make its payments; it's only unfair not to make them -- especially when the Ledger's owners and their friends aren't paying their fair share.

Tax expenditures? Never heard of them...

Monday, February 23, 2015

Chris Christie's Bizarro World

You mean a state can't pass a bill and then proceed to ignore it?! Wow, who knew:
New Jersey lawmakers warned of potential "draconian" budget cuts to come up with $1.57 billion if the state is forced to make a full pension payment this year. 
A state Superior Court judge ruled today that unions are entitled to that payment as part of a 2011 pension overhaul law passed by the Democratic-led Legislature and signed by Republican Gov. Chris Christie that forced the state to increase its payments into the system.

"The impact on programs at the end of the year would be devastating," state Assembly Majority Leader Lou Greenwald (D-Camden) said.

But Greenwald said there could be a solution on the horizon. He said public workers unions had "come to the table with real suggestion and real reform to try to create a pension payment and longevity that is sustainable and reliable and predictable."

"The reality is we have to either make draconian cuts and make the payment," Greenwald said. "Or we have to be doing what we have been doing over the course of the last number of months, which is work hand-in-hand with the people that are most dramatically impacted by the pension, which is the public workers. It's their retirement."
I have no idea what that last part means. But notice the construction: either huge cuts, or a further change to the pension. Not even the Democratic Leader, it appears, can dare to suggest that maybe we need to collect more revenue.

It is a testament to how far out of touch this state is with reality that we won't even mention raising taxes when it's become increasingly clear we have to do just that. Judge Mary Jacobson is merely stating the obvious: a state can't just walk away from its debts when it is perfectly capable of raising more money.

But according to Christie, meeting your contractual obligations somehow makes you the equivalent of Che Guevara:
"Once again liberal judicial activism rears its head with the court trying to replace its own judgment for the judgment of the people who were elected to make these decisions," Christie spokesman Michael Drewniak said in a statement. 
"This budget was passed by the legislature and signed by the governor with a pension payment," he said. "The governor will continue to work on a practical solution to New Jersey's pension and health benefits problems while he appeals this decision to a higher court where we are confident the judgment of New Jersey's elected officials will be vindicated." [emphasis mine]
Yeah, you know what else was passed by the legislature and signed by the governor? The law that requires the state to make its pension payments!

Since when did ignoring the law become a conservative value? Since when was it considered liberal to do what you said you were going to do? Since when is it "judicial activism" for a court to insist that the executive and legislative beaches of the New Jersey government follow the laws they themselves wrote and passed?

Jacobson herself says it best:
"The Governor now takes the unusual position in this court of claiming that this legislative contractual guarantee, which embodied significant reforms for which he took substantial credit with great national fanfare, violates the New Jersey Constitution." 
"Despite the inherently limiting consequences of creating contractual rights through statutory language, the New Jersey Legislature deliberately and unequivocally created contractual rights for public employees for seven years of increasing state payments to support the actuarial soundness of the pension funds and to remedy decades of underfunding."  
"The court rejects defendants' position that finding Chapter 78 unconstitutional somehow does not result in the inevitable abandonment of the state's obligation to its employees and holds instead that the New Jersey Constitution does not bar enforcement of Chapter 78's contractual obligation." 
"Unsurprisingly, the governor does not make any promises that the budget for Fiscal Year 2016 will include a full payment for the pension systems. The court is unwilling to rely on what has now become a succession of empty promises.
"In short, the aim of the legislation is not being met. The goal of Chapter 78 was to reduce the unfunded liability to put the State pension system on sounder financial footing. The legislation defined what was reasonable, and the current underfunding falls far short of that goal. The state's failure to pay $1.57 billion is a substantial impairment both in terms of the absolute magnitude of the failure and as a percentage of the total payment that was expected under the statute." [emphasis mine]
That is exactly right: Christie and the Legislature passed Ben-Pen to great fanfare, and Christie ran around the state -- and then the country -- bragging about how was a mover and a shaker who knew how to get things done. Any time a public worker complained, Christie dressed them down, saying we should be grateful he "saved" our pensions.

But only the unions bothered to point out that the state needed to come up with an additional $5 billion a year in pension payments. The press certainly didn't seem to care that Christie didn't have a plan: editorial boards across the state lined up to endorse his re-election, but never bothered to ask how he would meet his pension obligations.

In a way, I can't blame Christie for lashing out at this inevitable ruling. Until recently, he's been living in Bizzaro World, a place where passing a pension bill that requires huge infusions of cash without any plan to raise revenues earns you the endorsement of the state's largest newspaper. The governor must be wondering why he's getting so much grief now; after all, he got a free pass back when it mattered...

I really don't know where any of this is going, but it's clear that any solution is going to have to involve raising more revenues. It's also clear that, so long as Chris Christie suffers from the delusion that one day he will hold national office, our governor will never agree to raising taxes.

Which means that this state is caught in a death spiral, trapped by the ambitions of a man who thinks keeping your word is a vice.

I'm sick of you liberal activist judges forcing me to obey my own laws!

Sunday, February 22, 2015

Sunday Night Music: Snarky Puppy

So here's something only a proud dad/jazzman would say: my son hipped me to this.

Tight, tight, tight. Nice to see some young guns bringing it.

Enjoy the Oscars.

Saturday, February 21, 2015

If You Don't Want Civil Disobedience, Stop Politicizing Our Schools

We had yet another interesting juxtaposition of events in education "reform" this week. As most of you know by now, the Newark Students Union staged another protest, this time occupying State Superintendent Cami Anderson's office. Bob Braun, as usual, had the best coverage and analysis of the event.

Meanwhile, on the other side of Essex County, a different superintendent made waves with an op-ed in NJ Spotlight (I write regularly for them as well: here's my latest piece). Jim Crisfield, departing superintendent of Millburn's schools, objects to the grassroots movement to opt out of the PARCC:
I do understand the concerns people have with the PARCC tests, and I in fact share some of them. I feel the PARCC tests as currently configured take too much time to administer, and I strongly object to how they are used to compare districts (or schools) to one another. And worse yet, very few educators, anywhere, will agree with the notion that standardized test results are either a valid or a reliable way to evaluate teachers. 
Having said that, assessment is a natural and necessary component of the education process. Great teachers deploy assessment techniques all the time to help shed light on both their students' needs and the efficacy of their teaching. PARCC results, we are told and we hope, will provide us with valuable insights into our students' needs and how we can meet them, so I am willing to give PARCC the benefit of the doubt to see if that promise will be fulfilled. After all, it's not like we haven't had standardized testing for, well, decades (if by a different name -- Iowa, Early Warning Test, NJASK, and the like).
What distinguishes PARCC from these prior versions, among other things, is the highly charged political climate of 2015. It seems as if everything now needs to be viewed (and acted upon) through a political lens. PARCC is linked to the Common Core, which in turn elicits angry, visceral reactions from several different quarters. And we then start down the road of letting politics interfere with the educational process. Politics, especially the partisan variety, has no place in the classroom and can in fact be quite distracting.
Coming out of all this political hysteria is a fast-brewing notion that it is a right to opt out of things happening at school with which one doesn't agree. [emphasis mine]
Let's stop here for a second to get a few things out of the way:

First, I know Jim Crisfield; in fact, he hired me for my current job when our family moved back to New Jersey. I didn't always agree with Jim (does anyone ever agree with their boss all the time?), but I respected and liked him. To be clear, the headline of this post has nothing to do with Jim: he's a real educator, and Pennsylvania is lucky to have him.

Second: yes, Crisfield is leaving New Jersey. Why? Because the Christie administration's truly stupid superintendent pay cap would have required him to take a huge cut in pay this year, and Crisfield wasn't willing to subject his family to that.

Yes, reformy New Jersey, this is what your beloved, teacher-bashing governor has wrought: you're losing school leaders who actually agree with you on particular education policies. And now you're stuck with superintendents who spend their days checking to see if teachers have rivets on their pants pockets (I swear I'm not making that up). Nice work...

Third: as Sarah Blaine points out in her eloquent response to Crisfield, his use of the word "hysterical" is very unfortunate. I can confirm Blaine's suspicions: I'm certain Jim didn't mean to give insult to the mothers who object to the PARCC and are opting their children out. I know this from having watched him work for years: I know he respects teachers, and I know he respects women. But that doesn't excuse his unintentional slight; I'd like to see him apologize for that.

Now on to Crisfield's argument:
Herein lies the danger. True, there is precedent for telling the school that your child will not participate in things ("family life" and "sex education" classes are the most salient example, and probably the old fashioned way of dissecting things in biology class can be included as well, and of course there is also the vaccination requirement that has been an opt-out candidate for years). 
Those topics (which often center on religious objections, by the way) notwithstanding, very few public school things have been candidates over the years for opting out. If a parent didn't like the way the local public school was approaching a given topic, they could find another way to educate his or her child (private, parochial, or even home-schooling options).
But opting out of things with such broad brush strokes is different, and taken to its extreme, this new version of opting out will destroy public education as we know it today. If we don’t stop facilitating and/or encouraging all this "opting out" or "refusing" (or whatever it's called), we might as well set up a la carte public schools. Opting out of Common Core? There go all of the child's language arts and math-class activities. Every. Single. One. Everything we do in language arts and math is aligned to the common core!
Further, what's to stop a parent of a high school student in 2015 from opting out of a bunch of other things that school does, too. What's the difference? Why not opt out of having one's child take that nasty calculus exam that she didn't study for because she was out of town over the weekend? Why not opt out of her having to go outside for PE during first period because she doesn't like the cold, and then opt her out of having first lunch, because she is way too cranky in the afternoon if she eats lunch at 10:30 a.m. [emphasis mine]
As Blaine points out in her post, this is a slippery slope argument and, as such, quite weak. Not one person I've heard or read who supports opting out of the PARCC has ever suggested parents should be able to pull their children out of any exam at any time.

Further, I disagree with Crisfield's analysis that the anti-PARCC movement is "letting politics interfere with the educational process." And this is where the protests against Anderson in Newark and the opt-out movement intersect.

It would be lovely if politics was banned from our schools, and policy was created solely through evidence-based processes that rationally balanced the interests of all stakeholders. But we live in America, and that's not how anything works, let alone education.

The imposition of the PARCC on this state is a political act. Neo-liberals like Andrew Cuomo and Barack Obama have found common cause with conservatives like Chris Christie to push the idea that American schools are failing. This allows them the space to claim they are really interested in addressing economic inequality without having to promote any policies that actually redistribute wealth. Nearly the entirety of their program to address chronic poverty and the erosion of the middle- and working-classes is to be found in education "reform."

Standardized tests are a critical component to making this political case: when you can show that schools in impoverished communities are "failing," you can flip cause and effect and declare: "If only test scores would go up, everyone would be 'ready' to move into a high-paying job!" It never dawns on reformy folks that this country needs millions of workers to do hard, often dirty, often dangerous jobs that do not require college "readiness," and that the people who do these jobs deserve living wages and lives of dignity.

Opting out of the PARCC is an act that calls b.s. on the claim that standardized testing is necessary to inform student instruction; this testing is, in fact, almost completely useless for that purpose. If anything, standardized tests are only useful as accountability measures. But if that's true, it's a waste of time and money to test every child in nearly every grade in multiple subjects; we'd be much better off using appropriate sampling methodologies to get the data we need to hold schools and systems accountable.

No, the current testing regime is really in place to create a narrative*: American schools suck, and fixing them will fix poverty (and, presumably, racism). Same with the loss of democratic, local control of urban schools: these districts are "broken," and only the subversion of democracy can save them.

As I've noted time and again, the districts in New Jersey that do not allow their citizens any say over their schools are the districts full of people of color and people in economic disadvantage.

Again, this has nothing to do with educational outcomes: it is a political reality. No one has ever shown that districts that lose local control achieve better learning outcomes for their students. Certainly, two decades of state control in Newark hasn't resulted in schools that get test scores equivalent to those in Millburn.

Cami Anderson is a political appointee. She was installed by a governor who overwhelmingly lost in Newark, but garnered enough votes in the white suburbs to win re-election. Does anyone think it's a coincidence, then, that suburban school districts whose leaders have engaged in corruption at least as bad -- arguably far worse -- as that found in Newark 20 years ago retain local control, while Newark and Paterson and Camden and Jersey City do not?

Again: it would be lovely to have a world where students don't have to resort to actions like occupying offices in order to gain agency for themselves, their families, and their neighbors. But state control is a political act; it requires a political response.

Now, some of you are probably thinking that I'm being unfair. After all, don't we all just want what's best for our students? Can't people have legitimate differences on testing and state control? Isn't it poisonous -- and unnecessarily political -- to suggest that this debate isn't taking place in good faith?

Certainly, I'm not going to tar everyone who sees value in the PARCC or isn't for the immediate return of local control of schools in Newark with the same brush. Again, I think Jim Crisfield, for example, is sincere in his willingness to give PARCC a try. I can tell this because he's willing to acknowledge the arguments of the anti-PARCC side and express his own reservations about the test.

But there are others far more vocal than Crisfield, and far less willing to acknowledge any doubts about things like PARCC and state control. Their certainty annoys and troubles me; in addition, I'd be far more willing to accept that these people's intentions were sincere were it not for two troubling facts:

1) There is no evidence reformy policies work. Again, where is any evidence state control makes urban schools better? Where is any evidence the PARCC is a "better" test than the NJASK, or any other test? Where is any evidence "college and career readiness" is an appropriate goal, or that it is the sole responsibility of the K-12 school system?

For that matter, where is the evidence in favor of vouchers and charters and test-based teacher evaluations and all the other reformy stuff we are told will fix our schools? The burden of proof is on the reformies, but they haven't shown what they want will actually work.

2) Why aren't the reformies also advocating for things like school funding reform and reduced class sizes, which have tons of evidence to support them? The fine, reformy folks keep telling us that these new tests are so wonderful -- but where are they when it comes time to stand up for adequate school funding? They tell us that state control is necessary in Newark -- but where are they when it comes time to demand that class sizes in the city should be reduced?

It's hard to accept the sincerity of the arguments in favor of PARCC and state control when the same folks making those arguments are silent (and sometimes openly hostile) on the issues of school funding and class size reduction. Could it possibly be that reformy folks only want to advocate for "reforms" that are cheap, while dumping the problems of income inequity, racism, and a lack of social mobility almost entirely on our nation's schools?

Until the reformies are willing to have this debate, I really don't want to hear from anyone about how we need to keep politics out of our schools. Our schools are already politicized -- the reformies made sure of that. If they don't like the civil disobedience that has sprung up in resistance to their political acts, they should stop engaging in them.

This blog remains a proud supporter of the Newark Students Union.

*It's also in place to make some companies money -- a lot of money. Anyone who denies this is being willingly naive.

Thursday, February 19, 2015

Are College Remediation Rates SOLELY K-12 Schools' Responsibility?

It seems to me, from the outside looking in, that the reformy argument for the Common Core and its aligned tests, such as the PARCC, is largely based on the "problem" of college remediation rates.

JerseyCAN says the Common Core will help remediation rates. Community college presidents say Common Core and the PARCC will help close the "preparation gap." NJDOE officials say the Common Core and PARCC will "remedy" the problem of college remediation. The PARCC people themselves say aligning instruction with the test will help bring remediation rates down.

This is as good an example as I have seen of how the reformy mind works:
  • College remediation rates are too high, because...
  • High schools are graduating too many students who aren't really ready for college, therefore...
  • We need to design K-12 instruction with the goal of "college readiness," which means...
  • We must have tests like the PARCC which predict that readiness.
It's all so logical, isn't it? One point flows to the next...

But neither the premises nor the logical connections between them ever get challenged. No one ever stops to think:
  • Are college remediation rates really unacceptably high?
  • If they are, is it really the fault of K-12 schools that so many students aren't "college ready"?
  • Is it really better for society if we withhold high school diplomas from those who aren't "college ready"?
  • Are students really not aware of their own "college readiness"?
Carol Burris wrote a couple of posts this year for the Washington Post -- see here and here -- that shed light on the question of whether college remediation rates are all that bad. I want to explore her ideas and some others in future posts, but for right now, let's focus on this:

Can the blame for a lack of "college readiness" be put entirely on our K-12 system?

Burris starts her second post with an interesting anecdote:
I was standing on line at a local sandwich shop, when the young man behind me got my attention. He had graduated a few years ago from my high school, and he was anxious to share good news.
He beamed as he told me how well he was doing at Hofstra University. He was studying what he loved, and doing very well. He did not start out at Hofstra, however. Tom began at a local community college and then transferred to the four-year university.
I remember him as a great kid who struggled in math due to his learning disability. He was able to get through geometry, but math beyond that was too tough. After he told me about his success at Hofstra, I asked him how he did at the local community college where he began.
The young man said that after taking a test with, as he described it, “math I had not seen in years”, he was placed in two remedial courses. He got through them, and his family luckily did not mind spending tuition on classes for which he would get no credit. In English, he went right into a college level course.
I asked him if he felt more prepared for college math after having remediation. He laughed and described the credit0bearing course as “really not math at all…it was easier than high school.” Why this school insisted that he take two, non-credit bearing courses, only to then have him take a course he could have successfully completed without remediation, is something that neither he nor I understand. [emphasis mine]
This story highlights a real problem in making the claim about a lack of college readiness: there really isn't an objective definition of what "college readiness" actually is. I know there is a movement afoot to tie this to the SAT, but as Burris notes, colleges aren't even in agreement on the cut score that should constitute "readiness."

But let's set that aside and look at the highlighted quote: "math I had not seen in years." The implication here is that there was a time of at least a couple of years between this young man's last math course and his community college placement test. New York State requires 3 units (years) of math to earn a diploma. So it's reasonable here to think he had been out of high school at least a year before enrolling in community college.

Is this typical? Well...

This comes from the American Association of Community Colleges. 63 percent of community college students enrolled for credit are age 22 or older -- which means it is reasonable to assume that many of those students haven't had a high school math course for years.

I'm not a cognition and learning expert, so I won't claim to know the literature. There is at least some evidence, however, that memory of learning falls off quickly for those who don't achieve a high level of proficiency. But even if you do, you'll experience a sharp drop in memory early, followed by a period of stabilization. Of course, the issue is highly complex and outcomes depend on many factors.

As a music teacher, I know that it doesn't take long for facility at playing an instrument to drop off quickly, especially if you haven't achieved high levels of playing before (although there are other benefits to music training that have long-term benefits). This is almost certainly true in other domains. Learning is contextual, so if you lose the context, you almost certainly lose at least some of the learning. Given all this...

Is it at all reasonable to hold K-12 education responsible for all deficits in learning for those students who haven't been in high school for years?

You can't hold people or systems responsible for things that you can't attribute to them. Yes, I'll admit the K-12 school system can't shrug off it's responsibilities by claiming: "Hey, you won't remember any of this stuff anyway!"

But it is unreasonable to attribute a college student's math proficiency entirely to his K-12 school system if he hasn't been in that system for a long time. In other words: it's not necessarily your high school geometry teacher's fault if you need a refresher course in geometry years after you were in her classroom.

My point here is that even if we set aside highly subjective nature of college remediation rates, using them as "evidence" of K-12 failure is extremely problematic. We need much better research to show that insufficient instruction in high school and before is leading to the current community college remediation rate.

Once again, this is the problem with so much "reformy" thinking -- it is a victim of Yogi's Fallacy:

Before rushing off to "fix" our schools, we ought to think a lot harder about why outcomes are the way that they are, and whether our "fixes" will actually make things worse. More in a bit...

ADDING: Ajay in the comments points us to this post from Judith Scott-Clayton:
While remediation rates have risen slightly over time — to 22 percent of all first-time first-year students in 2003-4 from 18 percent in 1995-96, according to Department of Education statistics— the increases have been striking for students with strong high school grades. 
For students with high school grade-point averages between 3.5 and 4.0, remediation rates have more than doubled (see chart below). This is not a result of high school grade inflation – the percentage of students with G.P.A.’s in this range has not changed – but is consistent with increasingly ubiquitous placement testing.
Department of Education, National Center for Education Statistics, Beginning Postsecondary Students database; computation by N.C.E.S. QuickStats.
Screening seemingly prepared students for remediation is questionable for at least two reasons. First, the benefits of remediation are far from obvious: remediation has been referred to as the Bermuda Triangle of postsecondary education, because the majority of those who enter never make it out
Across several rigorous, quasi-experimental studies of the causal impact of remediation, only one found positive effects on college outcomes, while others found null to negative effects. 
Second, the tests commonly used to screen for college readiness are only weakly related to college outcomes, as two recent studies by the Community College Research Center show. (Disclosure: I am a senior research associate at the center and the author of one of these studies.) Some students manage to pass the tests even though they are not ready for college-level work, while even more who are ready for college-level work are kept out. 
My own research, using data from a large urban community college system with particularly high remediation rates, estimates that one in four students assigned to math remediation could have passed a college-level math course with a grade of B or better and one in three students assigned to English remediation could have passed freshman composition with a B or better.
Off hand, I think the GPA argument is a bit more complex: if we're getting more high-GPA students from lower-performing schools entering colleges, that could explain some of the increases in remediation.

But Scott-Clayton's larger point is solid: remediation is a questionable policy. And, again: how much of the "problem" of remediation is attributable to the K-12 system?

Big, knotty topic -- the sort of stuff we love here...

Saturday, February 14, 2015

PARCC Cheerleaders: Snide, Dismissive, & Wrong

The PARCC tests -- new, computerized, Common Core-aligned standardized tests -- are coming to New Jersey, and many people aren't happy. NJEA's recent poll clearly shows parents and other stakeholders hold standardized testing in low regard, and the burgeoning "Opt Out" movement continues to draw attention.

As I wrote yesterday, the confident claim of Common Core supporters that PARCC is a "better" test has little evidence to back it up. But no matter; these folks continue to make broad, unsubstantiated assertions about the new tests, based merely on their own beliefs.

Worse, the PARCC cheerleaders have developed a nasty habit of snidely dismissing anyone who doesn't share their enthusiasm for the Common Core State Standards and the tests aligned with them. Here's a case in point, from the reliably reformy Atlantic Monthly and author Laura McKenna:
The PARCC test for its part doesn’t require much more time than previous assessments. In the past, all public-school students in New Jersey, for example, took a state-designed standardized math and reading test. Fifth-grade students had 316 minutes to fill in the bubbles on an answer sheet. The PARCC’s fifth-grade test, meanwhile, will take 405 minutes. That might seem like a big difference for a 10-year-old, but the 89-minute difference doesn’t have much impact on the 180-day school year. That’s about a quarter of the time that my teenage boys like to spend playing Super Mario Brothers on any given Saturday.
First of all, there are plenty of people -- myself included -- who thought the NJASK took too much time to begin with. As Sarah Blaine points out, a New Jersey 4th Grader will spend more time in one year taking the PARCC the PARCC and other standardized tests* than a law school graduate will spend taking the bar exam.

There's no need for this: we could be using appropriate sampling methodologies to get all the information we need to direct education policy. The mere administration of the PARCC requires significant time and resources from schools; this is certainly a legitimate concern.

In addition, the time giving the tests is only part of the time devoted to the tests. And there is no question that tests like the PARCC redirect instructional hours toward test prep. Here's Bob Shepherd:
4. The tests have enormous incurred costs and opportunity costs.
First, they steal away valuable instructional time. Administrators at many schools now report that they spend as much as a third of the school year preparing students to take these tests. That time includes the actual time spent taking the tests, the time spent taking pretests and benchmark tests and other practice tests, the time spent on test prep materials, the time spent doing exercises and activities in textbooks and online materials that have been modeled on the test questions in order to prepare kids to answer questions of those kinds, and the time spent on reporting, data analysis, data chats, proctoring, and other test housekeeping. [emphasis mine]
I'm guessing McKenna wouldn't allow her own kids that much time playing Super Mario Brothers. I'm also guessing she wouldn't like it if all this test prep leads to a narrowing of the curriculum at her children's school. Sadly, she appears to be unaware that this is what is happening in many school districts:
The reality of the Common Core model is much more boring. America’s schools could be better, no doubt. They could be more equal. They could be more effective in preparing kids for the new, global economy and the ever-growing rigors of higher education. But there is no evidence that one set of standards, that a single standardized test, will alter the basic school experience of children. They will probably still have to do book reports on Abraham Lincoln and To Kill a Mockingbird. They almost certainly will still have time to joke around on the playground with their buddies. They will be evaluated by teachers’ exams and rubrics and probably won’t be penalized by the Common Core tests. [emphasis mine]
As I've noted before, the notion that these tests do not have high stakes for children is just flat out wrong: even if you don't buy into the notion of these tests influencing a child's self-image, the use of these tests for placement into advanced courses by itself makes them high-stakes.

Further, McKenna's attempt to downplay the influence of standardized testing on schools is contradicted by a boatload of evidence. ASCD has an excellent summary of just how powerful the influence of high-stakes, standardized testing has been:
In contrast, since the advent of No Child Left Behind (NCLB), with its high stakes for schools, the traditional pattern of time allocation across subjects in elementary schools has changed markedly. Five years into NCLB, researchers found that 62 percent of a nationally representative sample of all districts in the United States—and 75 percent of districts with at least one school identified as needing improvement—increased the amount of time spent on language arts and math in elementary schools. These increases were substantial: a 47 percent increase in language arts and a 37 percent increase in math. Correspondingly, these districts decreased time allotted to other subjects and activities, including science, social studies, art, music, physical education, and recess (McMurrer, 2007).
Other studies from the NCLB era conclude that the higher the stakes are for educators, the more curriculum and instruction reflect what's on the test—particularly in low-performing schools where the threat of sanctions is strongest. A study of a large urban district from 2001 to 2005 (Valli & Buese, 2007) found that as worries about adequate yearly progress increased, teachers matched the content and format of what they taught to the state test. These researchers concluded that the content of the tests had effectively become the learning goals for students.
Au's 2007 synthesis of 49 recent studies found a strong relationship between high-stakes testing and changes in curriculum and pedagogy. More than 80 percent of the studies in the review found changes in curriculum content and increases in teacher-centered instruction. Similarly, a study of California, Georgia, and Pennsylvania school districts found that teachers narrowed their curriculum and instruction to focus on tested topics and also increased their use of test-like problem styles and formats (Hamilton et al., 2007).
Now, it may be that McKenna's school district hasn't felt this pressure acutely yet; that doesn't mean it's not happening in other places. And it doesn't mean that suburban districts, fearing the crashing of test scores like we saw recently in New York, aren't also now feeling the heat. In fact, the reformy types behind the PARCC in New Jersey have been explicitly clear in stating that they are looking to set up a narrative of failure for suburban public schools.

Given all this, I'd say suburban parents in New Jersey and elsewhere are more than justified in their apprehension about the PARCC. I'd say they have every right to question why local high school end-of-course assessments should be replaced by the PARCC. I'd say that after having seen the narrowing of the curriculum in New Jersey's urban schools, suburban parents should be asking if the same thing is going to happen to their schools post-PARCC.

McKenna and her fellow cheerleaders, however, see such concern as the fretting of a bunch of ill-informed, naive ninnies:
A typical suburban parent, like all parents, has an intense, natural instinct to protect his or her kids. We parents are hard-wired to protect our babies from the unknown—and for the most part, this is a good thing. After all, protection of offspring and suspicion of outsiders have kept the human species alive for millions of years. But this instinct sometimes takes parents in the wrong direction. Just look at the anti-vaccination movement: Though the instincts of anti-vaccination parent activists are pure, their actions have resulted in what’s arguably a public-health crisis in the country.
Isn't that lovely? According to Laura McKenna, if you have any doubts about the PARCC, you are the equivalent of Jenny McCarthy!

Am I the only who sees the irony in a group of people, claiming that the PARCC measures critical thinking, making such poorly reasoned arguments? We have reams of research showing the relative benefits and risks of vaccination; where is any equivalent evidence regarding standardized testing, let alone the PARCC? Where is any research confirming that the PARCC is a "better" test than what came before? Where is the data that shows PARCC has greater external validity, or accurately assesses the quality of instruction? Where's the proof?

This condescension is completely unwarranted and, frankly, obnoxious. People have legitimate questions about the role of standardized testing in America's schools. They have every right to ask if the influence of the education-industrial complex, the profiteers behind the testing movement, has grown too great. They have more than enough reasons to be leery of the PARCC or any other standardized testing regime.

I'll give McKenna this: at least she seem to understand that these tests are not about informing instruction:
Parents need to understand why a new universal set of standards is important, particularly parents in good school districts where schools are working well. They need to know how their kids will benefit from this program—and if their kids won’t benefit, parents need to know why these test results serve the larger public good, that they can help shape policies that will help others. Parents need to know that their kids will continue to be graded based on their teachers’ assessments and that the tests really serve to provide data for administrators and political leaders who can set policies based on students’ overall performance. Parents need to know how the Common Core differs from previous state curricula and how it will affect their kids on a daily basis. Simple facts—that the Common Core does not prescribe certain textbooks, for example—would go a long way in dispelling confusion. [emphasis mine]
McKenna agrees: PARCC tests aren't designed to inform student instruction (of course, even NJDOE officials admit this when pressed). However, if these tests are really tools for making policy, there is no reason to test every child twice in every school year in most grades. We could get all the data we need for far less cost and with far less intrusion.

McKenna and her fellow PARCC cheerleaders shouldn't snidely dismiss their fellow parents who continue to point this out.

A lack of external validity.

* I try to get this stuff correct, so to be clear: the science test is what puts Grade 4 over the top, after the students take math and language arts tests. But science is only tested in Grades 4 and 8 (there's also the biology test in high school). The science test is not PARCC.

Friday, February 13, 2015

What We DON'T Know About the PARCC

One of the more annoying aspects of the current debate about PARCC -- the new statewide, standardized tests coming to New Jersey and a dozen other states -- is how the test's advocates project such certainty in their claim that the PARCC is a superior test.

They will tell you the PARCC will help ensure that students are "college-ready." They will tell you the PARCC will "provide parents with important information." They will tell you the PARCC is "generations better" than previous standardized tests.

People are certainly entitled to their opinions, but let's be clear: at this point, there is very little evidence to back up any assertions of the PARCC's superiority. In truth, there is a great deal we don't know about the PARCC:

We don't know if the PARCC is more reliable or valid than the NJASK, or any other statewide standardized test.

Those who claim that the PARCC sample items that have been released are "better" than the questions on the old NJASK have the rest of us at a disadvantage: we never got to see the NJASK. In fact, any claims of the PARCC's superiority over the old tests fail if only because the NJASK was never properly studied; we don't really know how "good" or "bad" the NJASK actually was.

There are two major considerations for any test: validity and reliability. Validity speaks to whether the test measures what you want to measure; reliability deals with the consistency of a test's results. I've been looking, and, so far, I've found no evidence the PARCC is more reliable than any other standardized statewide test.

And we have very little information as to the external validity of the PARCC, if only because it is so new. We don't know if better results on the PARCC correlate more tightly to better outcomes in college or career. How could we? We haven't even administered the test yet!

When anyone asserts that the PARCC is somehow "better" than another test, they are offering an opinion based on personal preference. That's perfectly fine (and it's worth noting that some people's preferences are better-informed than others). But claims about PARCC's superiority over what came before it are not currently backed up by objective evidence, and PARCC's cheerleaders ought to be far more circumspect in making their claims.

We don't know if the PARCC has better predictive validity for "college and career readiness" than other standardized tests.

I'm going to make a bet right now: $50 (hey, I make a teacher's salary...) says scale scores on the PARCC and scale scores on the NJASK for individual students are highly correlated. Of course, no one with access to this data is going to take me up on this bet, because they know that a student who scores well on one standardized test will almost certainly score well on a different one.

The primary task of standardized tests is to rank and order students. If you doubt me, look at how the NJDOE is going to report the results: it's all based on how students do compared to other students.

The notion of "college and career readiness" (which I think is utterly phony anyway) isn't supposed to be tied to the ranking of students. It's supposed to be about whether students have acquired the knowledge needed to be successful adults. But ranking students is what the PARCC is designed to do; setting the proficiency levels comes later (see below).

I can guarantee you that the ranking and ordering of New Jersey's students on the PARCC will barely differ from their ranking on the NJASK.* If that's the case, what could possibly make the PARCC any "better"?

We don't know the extent the "rigor" which the PARCC is allegedly measuring is developmentally appropriate.

As parents and other stakeholders take a closer look at the sample items that have been released, they grow increasingly concerned that the PARCC is not developmentally appropriate. Russ Walsh has produced evidence that some PARCC sample tests overreach in the difficulty level of their reading passages.

There's no point in setting high standards for students if they can't reasonably achieve them. And I haven't seen any evidence that the standards the PARCC demands can be achieved by the large majority of our students.

What I do know is that tests like the PARCC must have items with various degrees of difficulty in order to create a normalized or "bell-curve" distribution of scores. This summer, a committee consisting of lord-knows-who will "benchmark" the test and set the performance levels -- after the test has been administered.

New York went through this process last year, leading to the crashing of proficiency levels and the wailing and gnashing of teeth by reformies like Governor Andrew Cuomo. He promptly decided to dump all the blame on teachers and ignore his own failure to provide adequate funding for New York's schools. This, of course, appears to be the real purpose of standardized tests: ammunition for politicians to get what they want.

It's perfectly fine to benchmark an exam after the fact. But doing so highlights the normative nature of setting proficiency levels. The criteria for setting "cut scores" -- the levels needed to show various levels of proficiency -- isn't based on some objective idea of learning; it's based on how all of the students did on the test. Which is why the cut scores are going to be set after the PARCC is administered, when the benchmarkers can see the results for each test item and determine how difficult it was.

I know this is knotty stuff: lord knows I've struggled with writing about it before. But the critical point is this: given the variation in the abilities of our students and the amount of resources we are willing to devote to public schools, it is reasonable to question whether all students can achieve the levels of "rigor" the PARCC is calling for.

That isn't a statement excusing low expectations for children: it's a statement informed by the knowledge that this test is going to rank and order students. And, logically, not everyone can be above average.

We don't know how large the bias resulting from the computerized format of the PARCC will be.

I was just at a "Take the PARCC" event last night (more on that later). Even the parents and teachers I spoke with who didn't have a problem with the content of the PARCC admitted that children who are more computer literate are going to have an advantage on this exam.

I'll leave it to others to point out the design flaws in the user interface of the PARCC (scroll windows within scroll windows?). And I'll certainly acknowledge paper tests can and do have design flaws.

But there's no question in my mind that a child with regular access to a modern computer with high-speed internet access at home will feel far more comfortable in the PARCC testing environment than a child without that access. At the very least, we ought to study the extent of this bias before we make high-stakes decisions based on PARCC results.

We don't know if the PARCC is sensitive to changes in instruction.

I know regular readers have seen this a billion times, but once again...

It is impossible to deny the correlation between socio-economic status and standardized test scores. And yet we're using these scores to make high-stakes decisions about schools and teachers and even students (yes, we are) without appropriately acknowledging this bias.

Worse: even the PARCC people admit they don't know how this test does at measuring the quality and alignment of instruction. We are attributing all sorts of causes for the variations in PARCC test scores without even knowing the extent of the relationship between school and teacher effectiveness and those scores.

Do I even have to point out how insane this is?

Look, I'm not going to defend the NJASK or any other pre-PARCC test. As I've said many times: we barely knew anything about that test, or many of the other statewide tests that were administered in the wake of No Child Left Behind. 

I'll also risk alienating some of you by stating, once again, that I believe there is an appropriate and reasonable use for standardized testing, especially in the formulating of policy. I think tests results can help inform decisions, even if using them to compel decisions is totally unwarranted and, frankly, ignorant.

Lord knows my job as a researcher and blogging smart-ass would be far more difficult if I didn't have test scores to work with. Much of my work in advocating for teacher workplace rights and fair/adequate school funding and reasonable charter school policies relies on standardized test results.

But when I and others use this data, we use it appropriately, with full acknowledgment of its limitations and flaws. And we certainly don't make unsubstantiated claims about how the tests themselves are going to radically improve instruction and outcomes for students.

It's time for the PARCC cheerleaders to take a step back and think more clearly about their claims. It's time for them to start showing a little more humility and a little more healthy skepticism. It's time for them to stop holding on to arguments that have little evidence to back them up.

We know way less about the PARCC than many would have us believe. We have very little evidence that it is "better" than what came before. Let's at least wait until we've studied it before we claim otherwise.

A lack of external validity.

* One caveat: we might see the ceiling go up a bit, especially in math. More on this later.