I will protect your pensions. Nothing about your pension is going to change when I am governor. - Chris Christie, "An Open Letter to the Teachers of NJ" October, 2009

Thursday, August 24, 2017

The Worst Argument For Charter Schools

Eva Moskowitz, founder of Success Academy Charter Schools, has a book coming out. The timing is lousy, as she apparently praises the chairman of her board of directors, Dan Loeb, who is in a lot of hot water for saying something incredibly offensive and stupid.

But let me skip that and get to something Moskowitz reportedly writes in the upcoming book:
In the book, Moskowitz is less apologetic. Accusing the media of “sandbagging” her, Moskowitz devotes two full chapters to the journalists who she feels have been unfair to Success, including former Daily News columnist Juan Gonzalez, whose reporting on Success she calls “a sad waste of his talents,” and the Times’ Kate Taylor, whose investigative reports have revealed the harsh discipline meted out at some Success schools.

"Now I’m going to share with you some facts about Taylor and her editors that I fear may come across as an ad hominem attack but I hope you’ll ultimately conclude isn’t," Moskowitz writes.

She then reveals that Taylor attended a private high school, and that her editors either attended private schools or grew up in the suburbs, suggesting the Times’ journalists are incapable of understanding the black and Latino children who disproportionately attend Success schools. (Moskowitz attended Stuyvesant High School, the University of Pennsylvania, and received a Ph.D. from Johns Hopkins.) 
Moskowitz reprints some of her email exchanges with Times editors defending Taylor and her reporting — and she ends one chapter ominously, by saying, “alas, Kate Taylor wasn’t done with us yet.”
A little background: Taylor was the reporter who broke the story about a teacher at Success Academy yelling at a first grade girl and ripping up her worksheet in front of her class. According to Taylor's story, the teacher's behavior is typical for SA. Gary Rubinstein, a veteran NYC teacher, reviewed many of SA's videos -- which have since been scrubbed -- and found that while nothing approached the video Taylor and The NY Times published, they did highlight a teaching style that is "very robotic and cold."

Taylor also broke the "got-to-go-list" story, which confirmed the culling of "difficult" students of which SA had long been accused. But it's not like Taylor's stories were an unusual pieces of bad press for SA and Moskowitz: in 2015, The PBS News Hour ran a story by John Merrow that documented SA's harsh disciplinary practices for students as young as five years old (Moskowitz's response was to release a 10-year-old's disciplinary record; no, I'm not kidding).

The plain fact is that Moskowitz has had many critics, and SA has been under scrutiny for a long time. And why wouldn't it be? The organization engages in self-promotion at a level that would make Donald Trump blush.* And Moskowitz is truculence personified: she lives to do battle with the teachers union, the mayor, the press, and anyone else who gets in her way. Did she really think she could wage continuous war against her perceived enemies and no one would take a look inside her schools?

So now that Moskowitz and SA are under the microscope, what does she do? Engage in an ad hominem assault, hoping we don't call it what it is. Moskowitz wants us to dismiss all of Taylor's reporting because her parents enrolled her in a private school, and because her editors didn't grow up in the city; you can't get more ad hominem than that.

It's a line of attack I've seen charter cheerleaders use before: Chris Cerf, New Jersey's top charter booster, used it just the other day when he went after Save Our Schools New Jersey.

But it's a terrible argument in favor of charter schools, for at least two reasons:

First, and most obviously: what does Taylor's alma mater have to do with anything? Does it change the fact that one of Moskowitz's teachers was caught on camera yelling at a six-year-old? Does it change the fact that an SA principal drew up a "got-to-go" list? Does it change any of Taylor's subsequent reporting that suggests these were not, in fact, isolated incidents?

I obviously haven't read the book, but if the best Moskowitz can do to answer the charges against SA is to recount the education of the journalists who report on her schools... well, that pretty much speaks for itself, doesn't it?

Next: when Moskowitz accuses the Times staff of the crime of growing up outside of the city, she extends a comparison that she has made repeatedly: Success Academy vs. suburban schools. Just yesterday, when the New York State test scores were released, SA sent out a press release that explicitly compared its students' results to students in public schools on Long Island and in Westchester County.

To be clear: this comparison is wholly invalid (I'll try to get into detail about that later). But if Moskowitz is going to play this game, she ought to follow it through to its logical conclusion...

Would any parent sending their child to any of these five suburban districts put up with "got-to-go" lists? Would they be fine with SA's high suspension rates and high attrition rates without backfilling? Would they put up with a young teacher humiliating a first grader in front of her peers? Would they stand for their school district publishing the disciplinary record of a student? Would they be fine with special needs children being housed in schools that have clear resource disadvantages? Would they want their district to train their teachers to be "cold and robotic"? Would they be happy with a curriculum hyper-focused on test preparation?

Every time Eva Moskowitz brings up the suburbs, she's asking the rest of us to compare her version of schooling with the education offered in suburban schools. She thinks the test scores are proof that what she's offering is somehow equivalent to what affluent parents provide for their children. But it's transparently clear that Success Academy is in no way similar to the schools in Scarsdale. Yes, the test outcomes may be similar, or even better -- but the experience of schooling for children at SA would never, ever be tolerated in the leafy 'burbs of New York.

One interesting thing about the press release: note how it compares the per pupil spending of the Jericho school district to SA (I don't know what the source for Jericho is, but it doesn't match up with my data). I have no idea where SA got its $14,027 figure, but as I have pointed out repeatedly, that figure undoubtedly does not include the many structural financial advantages SA has over NYCDOE schools (and many other NYC charters, for that matter).

In addition to the very large sums SA pulls in from its philanthropic sponsors, and the expenses it saves by enrolling a student population less in need of extensive resources compared to its neighboring public schools, the chain free rides on higher public school salaries -- likely including the suburban schools to which SA compares itself.

But let's set all that aside to make a larger point:

New York State spends more money per pupil than any other state in the nation, even if we make appropriate adjustments for regional and other variations. But NY is also one of America's most inequitably funded statewide school systems. There are many reasons for this, including a host of "stealth inequities" baked into the state's funding formula. But there's also no question that the state's repeated refusal to fund its own law regarding school aid has left many districts, including New York City, without the funds they need to provide students with an adequate education.

Success Academy is always bragging on their allegedly large waitlists. But would their lists be nearly as long if every school in New York City was adequately and equitably funded? Would families still be clamoring to get their kids into SA if all NYC schools had small class sizes and healthy facilities?

There's no question that education is, to at least some extent, a positional good.** So if parents perceive a school like SA gives their kid a leg up on other children, many will gravitate toward it. But if SA's job is to sell themselves as the best option, they are certainly helped by Albany when the state refuses to follow its own law and provide the NYC public schools with what they need to do their job.

Which is, in the end, why comparisons to suburban schools will always be the worst argument urban charter schools can make -- because they only serve to point out that school "choice" will never be a substitute for equity.

I've taken some heat from those on "my side" before when I've said this, but I still believe it: there may well be a place for charter schools. But self-serving self-promoters like Moskowitz do us no favors when they try to convince us (and themselves) that what they are offering is somehow a cure for structural inequity.

It isn't.

ADDING: One rebuttal we often hear from charter supporters to those, like me, who think we need to address funding inequity immediately goes like this: "We can't wait! We need to do something about 'bad' urban schools right now!"

Why, then, do "successful" charter chains like Success Academy and KIPP take so long to grow their student populations once they've been approved? SA doesn't immediately fill its new schools with students of all grades; it starts with seats only in K and Grade 1 and then grows them one grade level at a time. In Camden, KIPP is growing its Lanning Square site -- a site that was supposed to be a public district school -- one grade at a time.

So where's the urgency?

It seems to me that, if we had the will, we could raise taxes on the wealthy immediately and put that money into existing public schools much more quickly than trying to create multiple, redundant, new networks of self-governing charters. We could fully fund the state aid laws in New York and New Jersey and everywhere else much faster than we could create "sector agnostic" school districts.

So why don't we do that?

It's funny -- You'd almost think that the reason the people who would have to pony up to fully fund our public schools love charter schools so much is that charter boosterism gets them off of the hook for having to pay more in taxes.

Not "ha-ha" funny, but... funny...

ADDING MORE: Oh, my... (I posted this kind of glibly earlier, but you should really read this: it's a Twitter thread about Success Academy's ties to the Trump administration put together by Leo Casey. It's a real eye-opener; I'm very curious to hear what the "liberal" members of SA's board will have to say about it...).

AND MORE: Jack Covey in the comments below reminds of this Politico story from 2016:
Success Academy CEO Eva Moskowitz did not approve of the finding — made by an “ethnographer” she hired to study her rapidly expanding charter school network — that some teachers at the high-performing network might be responding to the enormous pressure placed on them by cheating. 
So Moskowitz, Success's combative founder, deployed senior managers to inform the staffer, Roy Germano, that he was banned from visiting schools for the remainder of the year. Moskowitz disparaged Germano to other employees, according to a memo written by Germano in July 2015 and obtained by POLITICO New York, and he was told to halt his research projects immediately.

Germano was fired last August, approximately a month after the report was completed, and is now a research scholar at New York University.

Germano’s reports and memo, along with a trove of other documents obtained by POLITICO — a separately commissioned internal draft risk assessment report, a compilation of exit interviews, and internal Success staffing records, among other documents — paint a picture of a growing enterprise facing serious institutional strain in the form of low staff morale, unusually high turnover, and the kind of stress that could drive teachers to exaggerate their students’ progress.
“It seems possible if not likely that some teacher cheating is occurring at Success on both internal assessments and state exams,” reads the July report by Germano, which was titled “Research Proposal: An Investigation into Possible Teacher Cheating.” [emphasis mine]
The response from SA is funny:
A spokesman for Success questioned Germano's findings, and said that the network uses an outside auditor to review its state exams. 
"As to the allegations raised in the title of Mr. Germano’s memo, though he interviewed just 13 teachers out of 1,400 to justify that title, we conducted a thorough investigation and found no evidence to substantiate his speculation," Stefan Friedman, the spokesman, said in a statement. "Instead, we found teachers to be appropriately coaching students on do-nows, exit tickets and number story problems, an encouraged practice that helps students understand and master the material. Any suggestion that we utilized these methods -- or anything untoward -- on state standardized exams is categorically false and not supported by a scintilla of fact."
So why did Germano only interview 13 teachers?
“I am told Eva Moskowitz made disparaging comments about me in reaction to the report,” Germano wrote in the July memo, which was a follow-up to an initial report he wrote about cheating completed in May. “I was told to write a follow-up report that would essentially downplay my findings and told by [recently departed Success vice president] Keri Hoyt not to use the word ‘cheating’ in any future reports. Finally, I was told that I was banned from visiting schools for the remaining 4 weeks of the school year, and that I could only visit schools next year if accompanied by ‘a chaperone.’” (Hoyt could not be reached for comment for this story.)
Kinda hard to interview teachers when you're not allowed on campus...

 * OK, I'm wrong: nothing could make Donald Trump blush.

** Sorry, can't find a free version. But I do want to talk more about this idea later...

Monday, August 21, 2017

The Secrets to Their "Success"

"THE" Jose Vilson had a Twitter thread up this past week about an encounter with a parent whose child is enrolled in one of the Success Academy Charter Schools. I won't quote it here because I really want you to read the whole thing, then come back so we can talk...

Whenever I hear anecdotes about charter schools, my first reaction is to go to the data. Not because I don't think stories like this parent's are worthwhile -- to the contrary, they are very important and should be told. But I do believe data can help to confirm what we might already suspect.

And what do I suspect about Success Academy?

I've been teaching long enough to know that schools and teachers vary significantly in their effectiveness, and both can make a difference in the lives of children -- particularly children who are living in economic disadvantage.  But I also know that the reformy claims of "miracle" schools are almost always way overblown. Yes, some charter schools get better results than we would expect. Yes, some may engage in a few innovative practices that might be worth considering.

But schools like Success Academy almost always have structural advantages -- advantages that have nothing to do with their governance -- over the schools against which they compare themselves:
  1. Different student populations.
  2. Resource advantages.
  3. A less-experienced, less-expensive faculty.
  4. A longer school day/year and/or smaller class sizes and/or tutoring, made possible by #2 & #3 in combination with free-riding on the public district schools.
  5. Strict disciplinary codes which encourage students who do not thrive in a "no excuses" environment to leave.
In the minority of cases where "successful" charters out-perform expectations, I have seen no compelling evidence that freedom from teachers unions and public district school regulations, curricular innovations, or parental "choice" are what lead to "success." Instead, some combination of the five factors above almost always provide the most reasonable explanation for the difference in outcomes.

For years, I've been going to the data when confronted with a "miracle" charter school. For years, I've seen the same patterns emerge. Of course, there are many ways to approach this kind of analysis; let's apply one, which I first learned from Bruce Baker (I know, you're so surprised...), to Success Academy's oldest school and see if we can find any secrets of their "success."

The idea is simple: we'll look at Success Academy Harlem 1 and all the other schools that share its zip code. This way, we'll only be comparing schools situated in a similar geographical area.

To make an apples-to-apples comparison, we have to consider the grade levels that schools enroll. In the 2014 federal data, Success Academy Harlem 1 had a handful of Grade 9 students; most were in the lower grades. In another charter in the zip code, about half the students were in high school; two other NYCDOE schools in the area were mostly high schools. Keep this in mind as we continue.

Success Academy Harlem 1 enrolls a smaller percentage of Students With Disabilities (SWD) than its comparable neighboring public district schools. Compared with the other elementary schools in the neighborhood, SA enrolls far fewer special needs students proportionally; that's true for Harlem 1 and Harlem 4. It's worth noting that the school with the highest SWD percentage in the area, Opportunity CS, was threatened with closure last spring for poor performance (I can't find an update on its status for this fall).

Success Academy Harlem 1 enrolls a smaller percentage of Limited English Proficient (LEP) Students than its comparable neighboring public district schools. Again, compared with just the elementary schools, the difference is very substantial.

Success Academy Harlem 1 enrolls a smaller percentage of economically disadvantaged students than the neighboring public district schools serving comparable grades. Certainly, a 70 percent ED rate is high. But it's not as high as the elementary schools within the same zip code.

Let's look at the cohort attrition patterns. This is the size of the "class" of students in each school as those classes move from grade to grade. Here's the Class of 2018 (the year is when the cohort will be seniors in high school).

Success Academy loses many of its students as the cohort moves from grade to grade. Leo Casey of the Shanker Institute used a different data source and found the same patterns. Here's the Class of 2019:

So what does the data tell us about Success Academy Harlem 1 compared to its geographic neighbors run by NYCDOE?
  • Fewer students with disabilities.
  • Fewer Limited English Proficient students.
  • Fewer economically disadvantaged students.
  • Higher cohort attrition rates.
Something else we know about SA -- their philanthropic infrastructure gives them an enormous resource advantage:
As those of us who follow education policy and live in the greater-NYC area know, Moskowitz's Success Academy has benefitted enormously from philanthropic giving. $8.5 million from a hedge-funder in 2015. $9.3 million in one night later that year. $25 million from another in 2016. Plus another $10 million from some pikers...

This is all in addition to the monies SA gets from the government for the students it enrolls. Moskowitz's powerful friends have even made sure that she doesn't have to play by the rules that everyone else must follow. The result is a school system swimming in money -- a system that relies on funds that no one else gets to access.
Couple this with the widespread reporting about "got-to-go" lists and significant teacher turnover -- which undoubtedly keeps staffing costs lower -- and it's clear that there really is no secret to all the "success."

Now, I'm not about to say that this automatically shuts down any conversion about whether SA, or any other "high-performing" charter chain, should be allowed to exist, let alone expand. There is a serious debate to be had about segregation, class, race, and schooling. The sad truth is that reformers are correct in pointing out that we have highly segregated and underfunded public schools in our urban cores, and many parents perceive that a no-excuses charter is the best option they have for their child.

But claims that Success "succeeds" because it is free from union influence, or because it isn't directly accountable to a democratically elected school board, or because its teachers weren't trained in traditional, university-based teacher prep programs should be challenged. The data makes clear there are significant differences betweens SA's student population and those of its neighboring public schools. 

This matters. They can bring up test scores all you want, but unless and until SA's cheerleaders -- and the charter school backers across the nation -- start being honest about why a small group of them do better than predicted, support for charter schools will continue to plummet. I'm not saying Betsy DeVos and Donald Trump don't have something to do with that.

I am saying that it's well past time for the charter industry to start being honest with itself -- and the rest of us.

Happier times.

Tuesday, August 15, 2017

Chris Cerf Is Better Than You -- Just Ask Him

Chris Cerf is, of course, the State Superintendent of the Newark Public Schools. He holds this position not because the citizens of Newark chose him, nor because he was appointed by a democratically elected school board, nor because he was chosen by the democratically elected mayor, nor because he had an outstanding track record as a chief officer in another public school district (he never ran a district before Newark*).

No, Cerf only holds his office because Chris Christie -- the man for whom Cerf worked for three years as Education Commissioner -- put him there.

And why did Christie choose Cerf? We can only guess, but as this interview from the past Sunday makes plain, Cerf does share his boss's love of taking cheap shots at those who disagree with him. Before we get to that, however, let's see what spin Cerf is serving these days about charters:
Q. What do you think of the idea that charters siphon off resources from public schools?

A. It presumes charters are not public schools. But they are. They are publicly funded, open to all, tuition-free, accountable to democratically-elected authorities, and in New Jersey, not for-profit.
What nonsense. Yes, charter schools are publicly funded. But Newark Prep -- which was approved by Cerf when he was Commissioner but closed down last year due to poor performance -- was managed for years by K12 Inc., a for-profit corporation. And Camden Community Charter School -- also approved by Cerf, also closed due to poor performance this past year -- was also, according to the Star-Ledger, managed by a for-profit corporation.

But what's really galling here is Cerf's insistence that charters are overseen by "democratically-elected authorities." Because not one charter school in Newark was required to gain approval from the local school board, or the mayor, or the city council. All they needed was a nod from the Education Commissioner, who serves at the pleasure of the governor.

Keep in mind that Chris Christie lost in Newark by a landslide in the 2013 election: 6,443 to 29,039. The notion, then, that Newark's charter sector is "accountable to democratically-elected authorities" is a sick joke. But when it comes to charters, Cerf lives in an alternate universe:
The only difference is that a charter is managed by a board of citizens, as opposed to a local school board. Secondly, charters receive only 90 percent of the per-pupil operating cost of a traditional public school, which means that for every student who leaves for a charter, the district retains 10 percent to cover its fixed expenses.

So it's only a negative if the district cannot manage its own expenses to keep its fixed and legacy costs to a number equal to or less than 10 percent. [emphasis mine]
Cerf's hypocrisy here would be funny if the consequences weren't so serious. Because it wasn't even two years ago that Chris Cerf was complaining to the State BOE that charters had an unfair fiscal advantage, and that advantage was hurting his district:
Much of the budget pressure has come from payments the district is required to make to the city’s charter schools. Cerf, a cheerleader for charter schools as state commissioner, yesterday acknowledged that some funding stop-gap is needed to help the district.
He said the mandatory funding for charter schools year to year is “disproportionately hurting the district schools,” adding, “We can’t just turn the other way and let that happen.” [emphasis mine]
Oh, come on, Chris -- can't you keep your fixed and legacy costs in line?

As Cerf undoubtedly knows, New Jersey charter schools have been "held harmless" in their revenues for three years running. This means charters haven't had to take the financial hit that district schools, like Newark's, have had to take because Christie has refused to fully fund the School Funding Reform Act -- the state's own law that ensures districts get adequate and equitable funding.

For every year Christie has been in office -- including the three years Cerf was his Education Commissioner -- Christie has refused to follow the state's own law and provide New Jersey's schools with the funding they need to fulfill the mandates of the state constitution. In Newark, for FY2017 alone, this meant Cerf's district was $181 million below its adequacy target, or $3,615 per pupil.

Cumulatively, NPS is $611 million in the hole since the beginning of Christie's reign. It could have been worse: back in 2011, the state's Supreme Court ordered Christie to restore half-a-billion dollars in state aid to the neediest districts. But that didn't stop Cerf, working on Christie's behalf, from running up and down the state, making the case that New Jersey's schools didn't need the funding the state's own law said they needed.

I contend that there is no one individual over the last eight years -- and yes, I do include Christie -- who has done more to undermine the state's own law on school funding than Chris Cerf. It was Cerf, after all, who authored the 2012 Education Funding Report, an incoherent, rambling mess that tried to make the case that while money matters in education, the schools serving the most disadvantaged students were clearly suffering because they had too much.

As Bruce Baker noted in real time, once you got past the reformy pablum served up in the report, the actual policy of Christie and Cerf was to cut aid to districts serving larger proportions of at-risk students, while simultaneously giving more aid to districts enrolling proportionally fewer at-risk students.

Cerf's report made the case the "weights" for at-risk students were too high in SFRA, even as the empirical evidence showed they were, in fact, very likely too low. He tried to argue that eliminating tenure was absolutely critical for student success, even though there is no evidence whatsoever that tenure impedes student achievement (in fact, there's some evidence strong collective bargaining improves staff effectiveness). Cerf also argued for changing the way students are counted in the aid formula, a clear attempt to decrease funding going to districts enrolling more at-risk students.

But the Funding Report was only the beginning of Chris Cerf's assault on equitable and adequate school funding. In 2013, Cerf stood in front of the NJEA and presented a brazenly deceptive graph in an attempt to downplay the effects of poverty on student outcomes.

Here is the full set of data that Cerf's slide omitted:

Still amazes me every time I see it.

There was also Cerf's twisting of national test scores to make New Jersey's schools appear to be failing their students; Matt DiCarlo did a great job at the time deconstructing Cerf's arguments. And there was the December, 2012 report that bizarrely tried to rationalize the Christie aid cuts; again, Bruce Baker pointed out how absurd its arguments were. Plus there was the proposed policy of underfunding the districts Cerf's DOE classified as "Priority" more than ones classified as "Reward."

All of this went on while New Jersey, under Chris Christie, retreated from funding equity -- even as the record clearly showed the state had made significant gains during the period of meaningful funding reform.

Again: it's not hard to make the case that Chris Cerf was as responsible as anyone for giving cover to the underfunding of New Jersey's schools under Chris Christie -- including, notably, the Newark Public Schools. And yet, despite this track record, Cerf has the stones to make the case that he cares more for the wee children than anyone who dares to disagree with his reformy views:
I look with astonishment at groups like Save Our Schools, highly represented by white wealthy suburbanites that have made it their mission to undermine the opportunity of poor African-American students to have access to quality education. 
Many don't honor their own principles by sending their children to private schools or living in leafy green suburbs. I ask myself whether the strength of their argument would be affected if the focus of charters did not include suburbs like Princeton. 
Mostly, I'm aghast at seemingly progressive individuals who so deeply misunderstand the profound injury their position will cause families of vastly more limited means. [emphasis mine]
Let's start with the low-hanging fruit: Chris, when you were Education Commissioner, did you ever walk into the governor's office and fault him for sending his children to highly-resourced private schools?

Yes, there are people in Save Our Schools who live in the suburbs. Yes, there are people who question charter school proliferation who send their kids to private schools. The difference, Chris, is that none of them, so far as I know, have ever tried to undermine the state's own law regarding full and fair funding of public schools.

For three years, Chris, you did everything within your power to keep schools serving large numbers of at-risk students from getting the funds the state's own law says they should get -- all while your boss gave gobs of tax breaks to wealthy, connected special interests. You went out of your way to justify Christie's underfunding of SFRA with some of the most fallacious nonsense imaginable, costing Newark hundreds of millions of dollars for their schools.

And still, despite your own past, you publicly question others' motives:
I stay up at night wondering how otherwise good-hearted people could say they want to impose a moratorium or somehow stop the addition of charter schools, when so many children are choosing them and being successfully launched into adulthood on the basis of that choice. 
Unfortunately, I think the answer is a raw political one. By statute, traditional public schools are unionized and charter public schools could be unionized, but are not required to be. 
The interest group in the state that spends by far the most money in Trenton, the teacher's union, would be economically hurt by the increased number of charter schools because it would lose dues-paying members. I think this is a purely economic argument. They do not want to lose members. All their talk about creaming and hedge funds is just so much propaganda that they have focus grouped and determined works in the public debate.

Is it true? Is the charge that Newark's charters don't enroll the same sorts of students as NPS "just so much propaganda"?

Let's go to the data, straight from Cerf's former office, the NJDOE (and including the three years when he ran it):

Year after year, NPS enrolls proportionally more special education students than the Newark charter sector.

The special education students the charters do take tend to have lower-cost disabilities compared to the classified students enrolled at NPS. (The lower-cost disabilities are speech impairments (SPL) and specific learning disabilities (SLD); see here for more information.)

Newark's charter sector enrolls very few students who are Limited English Proficient. 

Yes, the free and reduced-price lunch eligibility rates between the district and the charters look the same. But keep in mind that NPS has been expanding its free lunch program** for some time, which means the incentive for families to fill out applications has waned. The charters, on the other hand, have a great incentive for their families to fill out the application, as their funding increases based on the number of FRPL students they enroll.

Whether there is, in fact, a difference in the socio-economic status of charter students compared to district students is a serious question. Cerf says there's "zero factual evidence" to support a charge of creaming. I say: What have you done, Chris, to explore the question, either as Commissioner or State Superintendent?

Who did you contract with to look into why charters don't enroll as many LEP students as their host districts? Who did you assign from your staff to write a report on the disparity in special education rates between the two sectors?

As you know, Chris, Julia Sass Rubin and I wrote the report using your department's data that explored these questions. But why did we have to do it? Why did I have to be the one to use your department's data and point out charter schools spend far more on administrative costs and far less on support services than the New Jersey charter sector?

Why did I have to be the one who pointed out that charter staffs have far less experience than district staffs, and consequently make less money? The charters you laud are basically free-riding on district salaries negotiated by the unions you casually denigrate. Did this ever concern you? Did you ever even think about it? Did you ever hire people for your senior staff who had the capacity to study this stuff?

Chris Cerf has a nasty habit of questioning other people's motives. You would have thought maybe a few years actually running a school district would give him a little bit of humility, and a little bit of space to reflect on his own culpability in underfunding New Jersey schools.

Sadly, Cerf appears set in his ways. He'd rather take his pot shots than own up to the terrible legacy he left as Commissioner. Luckily for Newark, he won't be around much longer. Hopefully, he'll soon return to the private sector, where his record speaks for itself.

Accountability begins at home.

ADDING: Remember this? Good times...

ADDING MORE: As Darcie Cimarusti reminds us, Atlantic City Community Charter is also managed by a for-profit CMO. And, according to the Star-Ledger, Central Jersey Arts Charter School contracts with a for-profit manager.

Pesky facts...

* Maybe now that Newark is regaining control of its schools, it can look around for a superintendent who has actually run a district - unlike the last two appointed by Chris Christie.

** This is a good thing. It's one of the few things I give Cerf and Cami Anderson credit for.

Saturday, August 5, 2017

CREDO Charter School Studies' "X Days Of Learning" Conversion: Still Unvalidated

This past week, the Center for Research on Education Outcomes (CREDO) at Stanford University released yet another report in a series on the effects of charter schools on test scores -- this time focusing on Texas.

Almost immediately, members of the the local media trumpeted the results as "proof" that charter schools are realizing meaningful gains in student outcomes:
For the first time, Texas charter schools have outperformed traditional public schools in reading and closed the gap in math, researchers at Stanford University have found.
Students at Texas charter schools, on average, received the equivalent of 17 additional days of learning per year in reading and virtually the same level of education in math when compared to traditional public schools, according to a study released Wednesday by the Center for Research on Education Outcomes, or CREDO.
Rather than looking at raw standardized test scores, CREDO researchers quantify the impact of a school by looking at student improvement on the tests relative to other schools. The researchers then translate those results into an equivalent number of "days of learning" gained or lost in a 180-day school year.
The center's staff produced similar analyses in 2013 and 2015, finding Texas charter schools had a negative impact on reading and math performance.
"The most recent results are positive in two ways. Not only do they show a positive shift over time, but the values themselves are both positive for the first time," the researchers wrote.
CREDO's studies of charter school performance are widely respected in education circles. The center compares students from charter and traditional public schools by matching them based on demographic characters -- race, economic status, geography and English proficiency, among others -- and comparing their growth on standardized tests. Scores from 2011-12 to 2014-15 were analyzed for the most recent report. [emphasis mine]
That's from the Houston Chronicle, which published just one paragraph suggesting the CREDO studies might have credible critics:
Skeptics of CREDO's study typically offer three main criticisms of the research: it focuses exclusively on standardized test results, incentivizing schools that "teach to the test"; it ignores other advantages of traditional public schools, such as better access to extracurricular activities; and it doesn't account for the fact that charter school students are more likely to have strong, positive parental influence on their education.
Sorry, but that's, at best, an incomplete description of the serious limitations of these studies, which include:

Here is how the CREDO Texas study reports its findings:

Stanley Pogrow published a paper earlier this year that didn't get much attention, and that's too bad. Because he quite rightly points out that it's much more credible to describe results like the ones reported here as "small" than as substantial. 0.03 standard deviations is tiny: plug it in here and you'll see it translates into moving from the 50th to the 51st percentile (the most generous possible interpretation when converting to percentiles).

I have been working on something more formal than a blog post to delve into this issue. I've decided to publish an excerpt now because, frankly, I am tired of seeing "days of learning" conversions reported in the press and in research -- both peer-reviewed and not -- as if there was no debate about their validity. 

The fact is that many people who know what they are talking about have a problem with how CREDO and others use "days of learning," and it's well past time that the researchers who make this conversion justify it. 

The excerpt below refers to what the eminent psychometrician Michael T. Kane coined a "validity argument." To quote Kane: "Public claims require public justification." I sincerely hope I can spark a meaningful conversation here and get the CREDO team to adequately and publicly justify their use of "days of learning." As of now, their validity argument is cursory at best -- and that's just not good enough.

I have added some bolding to the excerpt below to highlight key points.

* * *

Avoiding the Validity Argument: A Case Study

As an illustration of the problem of avoiding the validity argument in education policy, I turn to an ongoing series of influential studies of charter school effects. Produced by The Center for Research on Education Outcomes at Stanford University, the so-called CREDO reports have driven a great deal of discussion about the efficacy of charter school proliferation. The studies have been cited often in the media, where the effects they find are reported as “days of learning.”[1] Both the National Charter School Study (Raymond et al., 2013) and the Urban Charter School Study Report on 41 Regions (CREDO, 2015) include tables that translate the effect sizes found in the study into “days of learning.” Since an effect size of 0.25 SD is translated into 180 days, the clear implication is that an effect of this size moves a student ahead a grade level (a typical school year being 180 days long). Yet neither study explains the rationale behind the tables; instead, they cite two different sources, each authored by economist Eric Hanushek, as the source for the translations.

The 2015 study (p. 5) cites a paper published in Education Next (Hanushek, Peterson & Woessmann, 2012) that asserts: “On most measures of student performance, student growth is typically about 1 full std. dev. on standardized tests between 4th and 8th grade, or about 25 percent of a std. dev. from one grade to the next.” (p. 3-4) No citation, however, is given to back up this claim: it is simply stated as a received truth. 

The 2013 study (p. 13) cites a chapter by Hanushek in the Handbook of the Economics of Education (Hanushek & Rivkin, 2006), in which the author cites his own earlier work:
“Hanushek (1992) shows that teachers near the top of the quality distribution can get an entire year’s worth of additional learning out of their students compared to those near the bottom. That is, a good teacher will get a gain of 1.5 grade level equivalents while a bad teacher will get 0.5 year for a single academic year.” (p. 1068)
No other references are made within the chapter as to how student gains could be presented as years or fractions of a year’s worth of learning.

The 1992 citation is to an investigation by Hanushek of the correlation between birth order and student achievement, and between family size and student achievement. The test scores used to measure achievement come from the “Iowa Reading Comprehension and Vocabulary Tests.” (p. 88) The Iowa Assessments: Forms E and F Research and Development Guide (2015), which traces the development of the Iowa Assessments back to the 1970s, states:
“To describe the developmental continuum or learning progression in a particular achievement domain, students in several different grade levels must answer the same questions in that domain. Because of the range of item difficulty in the scaling tests, special Directions for Administration were prepared to explain to students that they would be answering some very easy questions and other very difficult questions.” (p. 55-56)
In other words: to have test scores reported in a way that allows for comparisons across grade levels (or, by extension, fractions of a grade level), the Iowa Assessments deliberately place the same questions across those grade levels. There is no indication, however, that all, or any, of the statewide tests used in the CREDO studies have this property.[2]

Harris (2007) describes the process of creating a common score scale for different levels of an assessment as vertical scaling. She notes: “Different decisions can lead to different vertical scales, which in turn can lead to different reported scores and different decisions.” (p. 233) In her discussion of data collection, Harris emphasizes that several designs can be used to facilitate a vertical scale, such as a scaling test, common items, or single group to scaling test. (p. 241) 

In all of these methods, however, there must be some form of overlapping: at least some students in concurrent grades must have at least some common items on their tests. And yet students in different grades still take tests that differ in form and content; Patz (2007) describes the process of merging their results into a common scale as linking (p. 6). He notes, however, that there is a price to be paid for linking: “In particular, since vertical links provide for weaker comparability than equating, the strength of the validity of interpretations that rest on the vertical links between test forms is weaker.” (p. 16)

So even if the CREDO studies used assessments that were vertically scaled, the authors would have to acknowledge that the validity of their effect sizes was at least somewhat compromised compared to effect sizes derived from other assessments. In this case, however, the point is moot: it appears that many of the assessments used by CREDO are not vertically scaled[3], which is a minimal requirement for making the case that effect sizes can be translated into fractions of a year’s worth of learning. The authors are, therefore, presenting their results in a metric that has not been validated and could be misleading.

I use this small but important example to illustrate a larger point: when influential education policy research neglects to validate the use of assessments, it may lead stakeholders to conclusions that cannot be justified. In the case of the CREDO reports, avoiding a validity argument for presenting effect sizes in “days of learning” has led to media reports on the effects of charter schools and policy decisions regarding charter proliferation that are based on conclusions that have not been validated. That is not to say these decisions are necessarily harmful; rather, that they are based on a reporting of the effects of charter schools that avoided having to make an argument for the validity of using test scores.

[2] Nor is there any indication the national and international tests Hanushek cites in his 2006 paper, such as the National Assessment of Educational Progress, share questions across grade levels. In fact, Patz (2007) notes: “NAEP attempted to vertically link forms for grades 4, 8, and 12, but abandoned the approach as the comparisons of students separated by 4 or 8 years were found to be ‘largely meaningless’ (Haertel, 1991).” (p.17)

[3] Some statewide assessments are vertically scaled, including the Smarter Balanced assessments; see: https://portal.smarterbalanced.org/library/en/2014-15-technical-report.pdf


Center for Research on Education Outcomes (CREDO) (2015). Urban Charter School Study Report on 41 Regions. Palo Alto, CA: Center for Research on Education Outcomes (CREDO), Stanford University. Retrieved from: http://urbancharters.stanford.edu/summary.php

Dunbar, S.& Welch, C. (2015). Iowa Assessments: Forms E and F Research and Development Guide. Iowa City, IA: University of Iowa. Retrieved from: https://itp.education.uiowa.edu/ia/documents/Research-Guide-Form-E-F.pdf

Hanushek, E. A. (1992). The trade-off between child quantity and quality. Journal of political economy100(1), 84-117.

Hanushek, E. A., & Rivkin, S. G. (2006). Teacher quality. Handbook of the Economics of Education2, 1051-1078.

Hanushek, E. A., Peterson, P. E., & Woessmann, L. (2012). Achievement Growth: International and US State Trends in Student Performance. PEPG Report No.: 12-03. Program on Education Policy and Governance, Harvard University.

Harris, D. J. (2007). Practical issues in vertical scaling. In Linking and aligning scores and scales (233-251). New York: Springer.

Kane, M. (2013). Validating the interpretations and uses of test scores. Journal of Educational Measurement 50(1), 1–73.

Raymond, M. E., Woodworth, J. L., Cremata, E., Davis, D., Dickey, K., Lawyer, K., & Negassi, Y. (2013). National Charter School Study 2013. Palo Alto, CA: Center for Research on Education Outcomes (CREDO), Stanford University. Retrieved from: http://credo.stanford.edu/research-reports.html