I will protect your pensions. Nothing about your pension is going to change when I am governor. - Chris Christie, "An Open Letter to the Teachers of NJ" October, 2009

Saturday, September 2, 2017

What Really Makes a "No Excuses" Charter School?

Once again, Matt Barnum points us to a study showing positive effects from a "no excuses" charter school. The paper, by Matthew Davis and Blake Heller, takes advantage of the fact that, for three years, the Noble charter school network on Chicago had more applicants than it had available seats, and therefore held a lottery. The research compared students who were and weren't offered a seat a Noble, and found those who were offered charter admission were somewhat more likely to attend and persist in college.

I've written quite a bit about the limitations of these charter "lottery" studies before: see here, here, and here. And one of the biggest issues I've consistently found is that the definition of the treatment -- here defined as acceptance into a "no excuses" charter school -- is quite fuzzy.

What exactly does "no excuses" actually mean? Davis and Heller cite five factors identified by Dobbie & Fryer in a 2013 paper: "frequent teacher feedback, data driven instruction, high-dosage tutoring, increased instructional time, and a relentless focus on academic achievement."* But Dobbie & Fryer never describe how these factors vary when compared to public district schools; they only studied how they vary within a group of NYC charter schools.

So we really don't know if these are the behaviors that lead to the effects found in the lottery studies when charters are compared to counterfactual public district schools. This is serious limitation, because it may well be that the five factors aren't the differences that really lead to these charter school outcome gains.

In other words: what if "no excuses" really means something other than what this research assumes it means?

Davis and Heller do cite another source to help define "no excuses" -- a source that Dobbie & Fryer, tellingly, also cite. A source that shows up time and again in the charter "lottery" studies to help explain what a "no excuses" charter school is: see here, here, here, here, here, and here, among others.

No Excuses, by Abigail & Stephan Thernstrom, is the touchstone for these researchers' descriptions of the charter schools they study. Unfortunately -- and I'm going to be charitable here -- the text is simply not worthy of citation in a serious journal article. 

I'd say the book's description of its qualitative methods was inadequate, but that would be incorrect: it doesn't describe its methods at all. For example: aside from trafficking in some highly questionable racial profiling, the book features two chapters where it describes "superb" "high-poverty public schools." (p. 43)

But we have no idea why the Thernstroms picked these schools or what methods they used to study them; the only clue is on page 43: "We chose these particular examples of fabulous education only because they came to our attention and we visited them."** We further have no idea how they came to the conclusion these schools differ in behaviors or practices from public district schools.

And yet this is a primary source many of the lottery studies use to define their treatment. When Davis and Heller assert the Noble schools got a 10 percentage point gain in college persistence because of the chain's "no excuses" pedagogy,*** they are echoing the Thernstroms' assertions about the value of increased instructional time and high expectations...

Except that's not all the Thernstroms have to say about what makes a "no excuses" school. In fact, No Excuses is quite blunt about how "fabulous" schools get the gains that they get. What follows is an extended excerpt from the book that cuts right to the heart of how the Thernstroms believe "no excuses" charter schools operate. Certainly, this passage isn't the entirety of the Thernstroms' argument; however, they are remarkably candid about the role of self-selection in these "fabulous" schools.

If quantitative researchers are willing to cite No Excuses, they should at the very least acknowledge this part of the Thernstroms' thesis.

[All emphases are mine.]

* * *

(From No Excuses by Abigail & Stephan Thernstrom. Simon & Schuster (2003), pp. 47-49.)

Schools That "Cream"?

Not all minority students in the inner city are alike, of course. Some are more academically promising than others. Some have parents who place a particularly high value on education. Do the schools we so admire cream? Are the schools getting unrepresentative academic results because their students are not representative of the neighborhoods from which they draw?

With one exception, none of the schools selects students on the basis of test scores, grades, or recommendations. It is true that, in theory, Rafe Asquith's kids are part of a "gifted" program, although the school's definition of "gifted" means he has, in fact, a mixture of academically talented and struggling youngsters. There are about ten such "gifted" classes at Hobart, and the others do not yield remotely comparable results. Some of these schools, like the Amistead Academy in New Haven and the KIPP academies, simply select their students by a lottery.

At almost all the schools we celebrate, though, there is an application process that tends to -- and is intended to -- discourage families unlikely to cooperate with the school. Indeed, one of the five pillars upon which the KIPP schools rest is "choice and commitment." Students, their parents, and the faculty of a KIPP academy "have made a choice to be at the school," the literature reads. "No one is assigned or forced to attend these schools." Choice entails commitment -- a commitment that not every parent and student is willing to make. And thus KIPP and other schools of choice are, in this minimal sense, self-selective. Families interested in KIPP must be willing to come in for initial interviews, and to sign on to a rigorous program that includes long days, a long year, and summer school, as well as a strict behavioral code.

Parents attracted to North Star come to an open house, where Norman Atkins, the cofounder, gives them a "very negative sell," as he describes it. The school is not for them, he says, if they want to take the family on vacation in July, if they don't like school uniforms, if two hours of homework a night seems excessive. If they still find North Star appealing, they watch a simulated school day on a Saturday and can pick up an application. Students are picked randomly from the applicant pool. Among the schools we refer to in the pages below, only the Fredrick Douglass Academy in Harlem looks at test scores and other indices of academic and personal strength in admitting students.

The commitment that KIPP demands is spelled out in contracts that set expectations and are signed by teachers, parents, and students. Students must agree, for instance, to "always work, think, and behave in the best way [they] know how and... do whatever it takes for [them] and [their] fellow students to learn." They must promise to attend school faithfully, do their homework, and "behave so as to protect the safety, interests, and rights of all individuals in the classroom." And they must understand that that are "responsible for [their] own behavior." The contract parents sign is similar, but includes provisions on helping their "child in the best way we know how" and making themselves available to their children and the school. They, too, must agree that the ultimate responsibility for the "behavior and actions" of their child lies with the family.

North Star parents are urged to sign a voluntary "Covenant" that includes such items as: "We will make certain that our child attends school every day, except in cases of illness or another legitimate reason," and "we will check our child's homework each night and provide quiet time and space for the work to be completed." At South Boston Harbor Academy (SBHA) and most of the other successful schools we visited, there were similar agreements, although many parents inevitably fail to keep their end of the bargain. In running SBHA, the biggest surprise, the principal told us, was the lack of parental support for the high standards; parents often took their children out of school for an appointment or family vacation or made excuses for inexcusable absences.

And yet, with almost no exceptions, the children who violate the terms, or whose parents do, are not expelled. Norman Atkins stated the problem well: "There is always," he said, "in good schools serving inner-city populations, a tension between rigor and rescue." North Star can stick to the letter of its law and toss a recalcitrant student out, but where will he or she go?

Nevertheless, the fact that these are schools of choice is not incidental to their success. "Most of what we do here, if it's not reinforced at home, becomes incredibly challenging," one North Star staff member said to a doctoral candidate writing a dissertation on the school. Others at North Star, KIPP, and elsewhere have echoed the sentiment. Families that choose a school have at least, on paper, committed themselves to supporting the academic program. And schools of choice gain leverage from the simple fact that students who come willingly can be told, you have another option: Return to a school where fights are common, kids talk in class and curse teachers, and no one does any homework. But, remember, since the students at these schools learn little, their future is dim.

"This school is all about choice," James Verrilli, cofounder of North Star, noted in his opening remarks to students on their first day a few years back. "See that back door? See any locks on it? Is this a prison? Am I forcing you to be here? ... If you cannot live by our rules, if you cannot adapt to this place, I can show you the back door." Norman Atkins, that same day, delivered much the same message: "We have 479 students, 479 students, who want to be sitting where you're sitting, who wish deep down that they had your seat. And you got it. So if you're not making the most of your experience here -- if you're not working to your absolute top potential, if you're not going home, running home to do your homework every night, if you're not respecting your teachers and each other with all your energy -- some of the other kids would like your spot."

That is precisely the message, of course, that no regular public school delivers. An urban principal whose school we visited told us that she was sending her own children to parochial schools. She wanted a school that could impose limits, with consequences for those who refused to abide by them. Why couldn't she do that, herself, in the school she runs? we asked. "I'm powerless," she answered. The students are stuck with the school and the school is stuck with them.

* * *

While I think this passage pretty much speaks for itself, let me add a few quick thoughts:

- One response to the "creaming" argument I often read from the economists who conduct these lottery studies is that the random assignment to treatment controls for unobserved variables. In other words: because the lotteries are random, we can assume that, on average, things like parental motivation are the same in both the group of students who are accepted into the charter schools, and the group of students who are not.

There are two problems with -- or perhaps more precisely, limitations on -- this argument. First, at least some of the students who "lost" the lottery do not enroll in the counterfactual public district schools. They might go to a parochial or private school, or go to a public school out of state, or be home schooled. It's reasonable to think at least some parents who are highly motivated and have greater means will choose these options for their children; to the extent that they do, there is a difference between the control and treatment groups. We can try to measure that difference with crude binary variables or error-prone test scores, but those are ultimately proxies for what we really care about.

[ADDING: To be clear, it appears this isn't an issue with Davis and Heller's study, because the students are tracked to college regardless of where they eventually went to high school. This is an issue in other lottery studies, however, especially if the outcomes measured are statewide accountability tests. Private school students don't take these tests, so they can't be included in the control group.]

The greater limitation, however, is one that is often lost in the discussion of these studies, and is directly related to the passage above: The estimated effects of charter schools in these studies can only be generalized to students who enter the lottery.

We really have no idea whether or not expanding charter schools will yield the same effects if the pool of applicants expands as well. The passage above suggests that the students and families who enter the lottery perceive themselves as quite different from the families who don't enter the lottery. The charter school officials and the Thernstroms seem to agree. Is the charter effect really little more than a peer effect? That would greatly limit the effects of charter school expansion.

- The Thernstroms make a big deal about the relatively low expulsion rate for "no excuses" charters. But we've got enough data now to know that's not really how the charter attrition game is played.

Charter schools don't have to expel students; they just have to hassle them with high suspension rates or constant disciplinary actions for minor infractions. Eventually, the kids will voluntarily transfer out.

- One rebuttal I often hear from charter supporters is that things are changing so quickly within the sector that the last piece of research which might call into question charter "success" is outdated. I will concede that KIPP, for example, seems to have had something of a change of heart in the last couple of years, and is backing away somewhat from strict disciplinary measures and high attrition rates.

But part of that is they've built a brand, and everyone now knows what it is. Parents send their kids to a KIPP school knowing what they'll get. It's not like a decade ago when most people weren't sure what a "no excuses" charter school was; parents in charter-heavy districts are much more likely to know what they're in for, which means the self-selection increasingly can take place before students enter the lottery.

In any case, there's at least some evidence that as KIPP has built up, it's effects have cooled off. Charter effects are not guaranteed to continue as populations expand.

- Notice how much Atkins relies on being able to claim he has a waitlist for students to get into North Star. It's worth remembering that the size of these lists is largely a myth. And as I've said before: if the waitlist is so long, why doesn't North Star backfill? And why spend money on advertising and marketing if you can't even enroll everyone who allegedly wants your product?

Waitlists are marketing tools. When you can claim a large waitlist, you can make families believe your school is special. And if schooling is, to at least some extent, a positional good, a waitlist sends the message that a school is offering something lots of people want, but only some people get. That's a big part of the appeal of charter schools -- don't let anyone tell you otherwise.

* * *

I've said it so many times I've run out of ways to say it: I'm not against charter schools per se. But it's well past time to start being honest about why certain charters get the gains that they do.

Noble students are somewhat more likely to go to college and persist.**** That's great -- but Noble has some other features that are important to consider before we declare it a model fit for replication. Its cohorts, for example, shed large numbers of students -- including those in Davis & Heller's study:

Noble has also engaged in some questionable disciplinary practices (to their credit, Davis and Heller note this), some questionable recruitment practices, and some questionable student retention practices. The leaders of Noble appear to believe they cannot maintain their schools with a unionized faculty. Why? What else is going on at Noble that has nothing to do with "frequent teacher feedback, data driven instruction, high-dosage tutoring, increased instructional time, and a relentless focus on academic achievement"?

Yet another charter lottery study isn't going to answer this critical question. We need to get past facile explanations of the "success" of a small number of charter schools and really start digging into why they get the (largely marginal) gains that they get. And if we are to believe the sources cited by research showing positive charter school effects, one of the factors leading to charter gains is student self-selection.

Easy to say; hard to define...

ADDING: I have to note that No Excuses has perhaps the most hacktastic chapter ever written on school funding. An excerpt:
Consider the recent historical record. Per pupil spending on America's elementary and secondary schools (in constant dollars) nearly doubled between 1970 and 2000. Our educational system is not performing twice as well as a result. In fact, on the basis of the evidence NAEP has collected about what American students know, it seems safe to say that the nation's schools, overall, are no better than they were three decades ago. Black students have come a considerable distance since 1970, but the rise of a large black middle class and the end of segregated schools in the South may account entirely for that change. (p. 153)
Hey, Bruce, got anything to say about this?

* We also don't know how charters are able to offer more instruction and individual tutoring. Are they collecting more resources through philanthropy? Are they free-riding on public school salaries? If so, it's unlikely they can continue to use those strategies if the sector expands.

** That quote is followed by an endnote citing Samuel Casey Carter's No Excuses, a 2000 book published by the right-wing Heritage Foundation. It's essentially a catalog of 21 schools chosen ostensibly because at least three-quarters of their students qualify for free lunch yet score above the 65th percentile on a national standardized test. But no sampling methodologies are documented, no qualitative validity methods are described, and the tests are not consistent across all schools.

Again: this is not a source that should be cited in a serious, peer-reviewed journal article.

*** First: I'm going to skip a discussion of whether this is a practically significant gain or not. I'll only suggest this may be a point open for debate.

Second: yes, Davis and Heller do, in the last paragraph of the the paper, acknowledge that the exact mechanics of how "no excuses" style teaching leads to gains is still unknown. But the idea that "no excuses" might be student self-selection, or peer effects, or increased resources, or any number of other factors isn't a part of their argument:
Viewed in concert with work by Dobbie and Fryer (2015) and Angrist et al. (2016), we see an increasingly clear picture of a rapidly growing style of school management that boosts the college enrollment of poor urban high schoolers. Unfortunately, the similarity of the educational approaches considered in each study means that we can say very little about what mechanisms might be driving our results (at most, one might infer that the neighborhood-level interventions of the Harlem Children's Zone are not essential for increasing college enrollment, a conclusion shared by Dobbie and Fryer (2015)). Accordingly, understanding which "No Excuses" strategies are most important for generating long-term gains is an important topic for future research. Nevertheless, we are confident that this educational model has proved to be – and will continue to be – an effective part of the fight to increase low-income students' human capital.
They are "confident" that "no excuses" works, even if they don't know how it works. But what if "no excuses" isn't anything that Dobbie & Fryer lay out? What if it's something else, like self-selection & peer effects? Shouldn't we address this possibility before suggesting things like "a relentless focus on academic achievement" -- whatever that is -- makes all the difference?

**** It's worth noting that there are several large assumptions Davis and Heller make in their study. They use last names as a proxy for race, they've got a spotty dataset for tracking college attendance, and they make assumptions about where the control students attended school.

That's not to say this is a bad study -- it's not. But it is, like all social science research, limited. And before we jump to conclusions and use research like this to guide policy, we should stop and carefully consider its limitations.

Thursday, August 24, 2017

The Worst Argument For Charter Schools

Eva Moskowitz, founder of Success Academy Charter Schools, has a book coming out. The timing is lousy, as she apparently praises the chairman of her board of directors, Dan Loeb, who is in a lot of hot water for saying something incredibly offensive and stupid.

But let me skip that and get to something Moskowitz reportedly writes in the upcoming book:
In the book, Moskowitz is less apologetic. Accusing the media of “sandbagging” her, Moskowitz devotes two full chapters to the journalists who she feels have been unfair to Success, including former Daily News columnist Juan Gonzalez, whose reporting on Success she calls “a sad waste of his talents,” and the Times’ Kate Taylor, whose investigative reports have revealed the harsh discipline meted out at some Success schools.

"Now I’m going to share with you some facts about Taylor and her editors that I fear may come across as an ad hominem attack but I hope you’ll ultimately conclude isn’t," Moskowitz writes.

She then reveals that Taylor attended a private high school, and that her editors either attended private schools or grew up in the suburbs, suggesting the Times’ journalists are incapable of understanding the black and Latino children who disproportionately attend Success schools. (Moskowitz attended Stuyvesant High School, the University of Pennsylvania, and received a Ph.D. from Johns Hopkins.) 
Moskowitz reprints some of her email exchanges with Times editors defending Taylor and her reporting — and she ends one chapter ominously, by saying, “alas, Kate Taylor wasn’t done with us yet.”
A little background: Taylor was the reporter who broke the story about a teacher at Success Academy yelling at a first grade girl and ripping up her worksheet in front of her class. According to Taylor's story, the teacher's behavior is typical for SA. Gary Rubinstein, a veteran NYC teacher, reviewed many of SA's videos -- which have since been scrubbed -- and found that while nothing approached the video Taylor and The NY Times published, they did highlight a teaching style that is "very robotic and cold."

Taylor also broke the "got-to-go-list" story, which confirmed the culling of "difficult" students of which SA had long been accused. But it's not like Taylor's stories were an unusual pieces of bad press for SA and Moskowitz: in 2015, The PBS News Hour ran a story by John Merrow that documented SA's harsh disciplinary practices for students as young as five years old (Moskowitz's response was to release a 10-year-old's disciplinary record; no, I'm not kidding).

The plain fact is that Moskowitz has had many critics, and SA has been under scrutiny for a long time. And why wouldn't it be? The organization engages in self-promotion at a level that would make Donald Trump blush.* And Moskowitz is truculence personified: she lives to do battle with the teachers union, the mayor, the press, and anyone else who gets in her way. Did she really think she could wage continuous war against her perceived enemies and no one would take a look inside her schools?

So now that Moskowitz and SA are under the microscope, what does she do? Engage in an ad hominem assault, hoping we don't call it what it is. Moskowitz wants us to dismiss all of Taylor's reporting because her parents enrolled her in a private school, and because her editors didn't grow up in the city; you can't get more ad hominem than that.

It's a line of attack I've seen charter cheerleaders use before: Chris Cerf, New Jersey's top charter booster, used it just the other day when he went after Save Our Schools New Jersey.

But it's a terrible argument in favor of charter schools, for at least two reasons:

First, and most obviously: what does Taylor's alma mater have to do with anything? Does it change the fact that one of Moskowitz's teachers was caught on camera yelling at a six-year-old? Does it change the fact that an SA principal drew up a "got-to-go" list? Does it change any of Taylor's subsequent reporting that suggests these were not, in fact, isolated incidents?

I obviously haven't read the book, but if the best Moskowitz can do to answer the charges against SA is to recount the education of the journalists who report on her schools... well, that pretty much speaks for itself, doesn't it?

Next: when Moskowitz accuses the Times staff of the crime of growing up outside of the city, she extends a comparison that she has made repeatedly: Success Academy vs. suburban schools. Just yesterday, when the New York State test scores were released, SA sent out a press release that explicitly compared its students' results to students in public schools on Long Island and in Westchester County.

To be clear: this comparison is wholly invalid (I'll try to get into detail about that later). But if Moskowitz is going to play this game, she ought to follow it through to its logical conclusion...

Would any parent sending their child to any of these five suburban districts put up with "got-to-go" lists? Would they be fine with SA's high suspension rates and high attrition rates without backfilling? Would they put up with a young teacher humiliating a first grader in front of her peers? Would they stand for their school district publishing the disciplinary record of a student? Would they be fine with special needs children being housed in schools that have clear resource disadvantages? Would they want their district to train their teachers to be "cold and robotic"? Would they be happy with a curriculum hyper-focused on test preparation?

Every time Eva Moskowitz brings up the suburbs, she's asking the rest of us to compare her version of schooling with the education offered in suburban schools. She thinks the test scores are proof that what she's offering is somehow equivalent to what affluent parents provide for their children. But it's transparently clear that Success Academy is in no way similar to the schools in Scarsdale. Yes, the test outcomes may be similar, or even better -- but the experience of schooling for children at SA would never, ever be tolerated in the leafy 'burbs of New York.

One interesting thing about the press release: note how it compares the per pupil spending of the Jericho school district to SA (I don't know what the source for Jericho is, but it doesn't match up with my data). I have no idea where SA got its $14,027 figure, but as I have pointed out repeatedly, that figure undoubtedly does not include the many structural financial advantages SA has over NYCDOE schools (and many other NYC charters, for that matter).

In addition to the very large sums SA pulls in from its philanthropic sponsors, and the expenses it saves by enrolling a student population less in need of extensive resources compared to its neighboring public schools, the chain free rides on higher public school salaries -- likely including the suburban schools to which SA compares itself.

But let's set all that aside to make a larger point:

New York State spends more money per pupil than any other state in the nation, even if we make appropriate adjustments for regional and other variations. But NY is also one of America's most inequitably funded statewide school systems. There are many reasons for this, including a host of "stealth inequities" baked into the state's funding formula. But there's also no question that the state's repeated refusal to fund its own law regarding school aid has left many districts, including New York City, without the funds they need to provide students with an adequate education.

Success Academy is always bragging on their allegedly large waitlists. But would their lists be nearly as long if every school in New York City was adequately and equitably funded? Would families still be clamoring to get their kids into SA if all NYC schools had small class sizes and healthy facilities?

There's no question that education is, to at least some extent, a positional good.** So if parents perceive a school like SA gives their kid a leg up on other children, many will gravitate toward it. But if SA's job is to sell themselves as the best option, they are certainly helped by Albany when the state refuses to follow its own law and provide the NYC public schools with what they need to do their job.

Which is, in the end, why comparisons to suburban schools will always be the worst argument urban charter schools can make -- because they only serve to point out that school "choice" will never be a substitute for equity.

I've taken some heat from those on "my side" before when I've said this, but I still believe it: there may well be a place for charter schools. But self-serving self-promoters like Moskowitz do us no favors when they try to convince us (and themselves) that what they are offering is somehow a cure for structural inequity.

It isn't.

ADDING: One rebuttal we often hear from charter supporters to those, like me, who think we need to address funding inequity immediately goes like this: "We can't wait! We need to do something about 'bad' urban schools right now!"

Why, then, do "successful" charter chains like Success Academy and KIPP take so long to grow their student populations once they've been approved? SA doesn't immediately fill its new schools with students of all grades; it starts with seats only in K and Grade 1 and then grows them one grade level at a time. In Camden, KIPP is growing its Lanning Square site -- a site that was supposed to be a public district school -- one grade at a time.

So where's the urgency?

It seems to me that, if we had the will, we could raise taxes on the wealthy immediately and put that money into existing public schools much more quickly than trying to create multiple, redundant, new networks of self-governing charters. We could fully fund the state aid laws in New York and New Jersey and everywhere else much faster than we could create "sector agnostic" school districts.

So why don't we do that?

It's funny -- You'd almost think that the reason the people who would have to pony up to fully fund our public schools love charter schools so much is that charter boosterism gets them off of the hook for having to pay more in taxes.

Not "ha-ha" funny, but... funny...

ADDING MORE: Oh, my... (I posted this kind of glibly earlier, but you should really read this: it's a Twitter thread about Success Academy's ties to the Trump administration put together by Leo Casey. It's a real eye-opener; I'm very curious to hear what the "liberal" members of SA's board will have to say about it...).

AND MORE: Jack Covey in the comments below reminds of this Politico story from 2016:
Success Academy CEO Eva Moskowitz did not approve of the finding — made by an “ethnographer” she hired to study her rapidly expanding charter school network — that some teachers at the high-performing network might be responding to the enormous pressure placed on them by cheating. 
So Moskowitz, Success's combative founder, deployed senior managers to inform the staffer, Roy Germano, that he was banned from visiting schools for the remainder of the year. Moskowitz disparaged Germano to other employees, according to a memo written by Germano in July 2015 and obtained by POLITICO New York, and he was told to halt his research projects immediately.

Germano was fired last August, approximately a month after the report was completed, and is now a research scholar at New York University.

Germano’s reports and memo, along with a trove of other documents obtained by POLITICO — a separately commissioned internal draft risk assessment report, a compilation of exit interviews, and internal Success staffing records, among other documents — paint a picture of a growing enterprise facing serious institutional strain in the form of low staff morale, unusually high turnover, and the kind of stress that could drive teachers to exaggerate their students’ progress.
“It seems possible if not likely that some teacher cheating is occurring at Success on both internal assessments and state exams,” reads the July report by Germano, which was titled “Research Proposal: An Investigation into Possible Teacher Cheating.” [emphasis mine]
The response from SA is funny:
A spokesman for Success questioned Germano's findings, and said that the network uses an outside auditor to review its state exams. 
"As to the allegations raised in the title of Mr. Germano’s memo, though he interviewed just 13 teachers out of 1,400 to justify that title, we conducted a thorough investigation and found no evidence to substantiate his speculation," Stefan Friedman, the spokesman, said in a statement. "Instead, we found teachers to be appropriately coaching students on do-nows, exit tickets and number story problems, an encouraged practice that helps students understand and master the material. Any suggestion that we utilized these methods -- or anything untoward -- on state standardized exams is categorically false and not supported by a scintilla of fact."
So why did Germano only interview 13 teachers?
“I am told Eva Moskowitz made disparaging comments about me in reaction to the report,” Germano wrote in the July memo, which was a follow-up to an initial report he wrote about cheating completed in May. “I was told to write a follow-up report that would essentially downplay my findings and told by [recently departed Success vice president] Keri Hoyt not to use the word ‘cheating’ in any future reports. Finally, I was told that I was banned from visiting schools for the remaining 4 weeks of the school year, and that I could only visit schools next year if accompanied by ‘a chaperone.’” (Hoyt could not be reached for comment for this story.)
Kinda hard to interview teachers when you're not allowed on campus...

 * OK, I'm wrong: nothing could make Donald Trump blush.

** Sorry, can't find a free version. But I do want to talk more about this idea later...

Monday, August 21, 2017

The Secrets to Their "Success"

"THE" Jose Vilson had a Twitter thread up this past week about an encounter with a parent whose child is enrolled in one of the Success Academy Charter Schools. I won't quote it here because I really want you to read the whole thing, then come back so we can talk...

Whenever I hear anecdotes about charter schools, my first reaction is to go to the data. Not because I don't think stories like this parent's are worthwhile -- to the contrary, they are very important and should be told. But I do believe data can help to confirm what we might already suspect.

And what do I suspect about Success Academy?

I've been teaching long enough to know that schools and teachers vary significantly in their effectiveness, and both can make a difference in the lives of children -- particularly children who are living in economic disadvantage.  But I also know that the reformy claims of "miracle" schools are almost always way overblown. Yes, some charter schools get better results than we would expect. Yes, some may engage in a few innovative practices that might be worth considering.

But schools like Success Academy almost always have structural advantages -- advantages that have nothing to do with their governance -- over the schools against which they compare themselves:
  1. Different student populations.
  2. Resource advantages.
  3. A less-experienced, less-expensive faculty.
  4. A longer school day/year and/or smaller class sizes and/or tutoring, made possible by #2 & #3 in combination with free-riding on the public district schools.
  5. Strict disciplinary codes which encourage students who do not thrive in a "no excuses" environment to leave.
In the minority of cases where "successful" charters out-perform expectations, I have seen no compelling evidence that freedom from teachers unions and public district school regulations, curricular innovations, or parental "choice" are what lead to "success." Instead, some combination of the five factors above almost always provide the most reasonable explanation for the difference in outcomes.

For years, I've been going to the data when confronted with a "miracle" charter school. For years, I've seen the same patterns emerge. Of course, there are many ways to approach this kind of analysis; let's apply one, which I first learned from Bruce Baker (I know, you're so surprised...), to Success Academy's oldest school and see if we can find any secrets of their "success."

The idea is simple: we'll look at Success Academy Harlem 1 and all the other schools that share its zip code. This way, we'll only be comparing schools situated in a similar geographical area.

To make an apples-to-apples comparison, we have to consider the grade levels that schools enroll. In the 2014 federal data, Success Academy Harlem 1 had a handful of Grade 9 students; most were in the lower grades. In another charter in the zip code, about half the students were in high school; two other NYCDOE schools in the area were mostly high schools. Keep this in mind as we continue.

Success Academy Harlem 1 enrolls a smaller percentage of Students With Disabilities (SWD) than its comparable neighboring public district schools. Compared with the other elementary schools in the neighborhood, SA enrolls far fewer special needs students proportionally; that's true for Harlem 1 and Harlem 4. It's worth noting that the school with the highest SWD percentage in the area, Opportunity CS, was threatened with closure last spring for poor performance (I can't find an update on its status for this fall).

Success Academy Harlem 1 enrolls a smaller percentage of Limited English Proficient (LEP) Students than its comparable neighboring public district schools. Again, compared with just the elementary schools, the difference is very substantial.

Success Academy Harlem 1 enrolls a smaller percentage of economically disadvantaged students than the neighboring public district schools serving comparable grades. Certainly, a 70 percent ED rate is high. But it's not as high as the elementary schools within the same zip code.

Let's look at the cohort attrition patterns. This is the size of the "class" of students in each school as those classes move from grade to grade. Here's the Class of 2018 (the year is when the cohort will be seniors in high school).

Success Academy loses many of its students as the cohort moves from grade to grade. Leo Casey of the Shanker Institute used a different data source and found the same patterns. Here's the Class of 2019:

So what does the data tell us about Success Academy Harlem 1 compared to its geographic neighbors run by NYCDOE?
  • Fewer students with disabilities.
  • Fewer Limited English Proficient students.
  • Fewer economically disadvantaged students.
  • Higher cohort attrition rates.
Something else we know about SA -- their philanthropic infrastructure gives them an enormous resource advantage:
As those of us who follow education policy and live in the greater-NYC area know, Moskowitz's Success Academy has benefitted enormously from philanthropic giving. $8.5 million from a hedge-funder in 2015. $9.3 million in one night later that year. $25 million from another in 2016. Plus another $10 million from some pikers...

This is all in addition to the monies SA gets from the government for the students it enrolls. Moskowitz's powerful friends have even made sure that she doesn't have to play by the rules that everyone else must follow. The result is a school system swimming in money -- a system that relies on funds that no one else gets to access.
Couple this with the widespread reporting about "got-to-go" lists and significant teacher turnover -- which undoubtedly keeps staffing costs lower -- and it's clear that there really is no secret to all the "success."

Now, I'm not about to say that this automatically shuts down any conversion about whether SA, or any other "high-performing" charter chain, should be allowed to exist, let alone expand. There is a serious debate to be had about segregation, class, race, and schooling. The sad truth is that reformers are correct in pointing out that we have highly segregated and underfunded public schools in our urban cores, and many parents perceive that a no-excuses charter is the best option they have for their child.

But claims that Success "succeeds" because it is free from union influence, or because it isn't directly accountable to a democratically elected school board, or because its teachers weren't trained in traditional, university-based teacher prep programs should be challenged. The data makes clear there are significant differences betweens SA's student population and those of its neighboring public schools. 

This matters. They can bring up test scores all you want, but unless and until SA's cheerleaders -- and the charter school backers across the nation -- start being honest about why a small group of them do better than predicted, support for charter schools will continue to plummet. I'm not saying Betsy DeVos and Donald Trump don't have something to do with that.

I am saying that it's well past time for the charter industry to start being honest with itself -- and the rest of us.

Happier times.

Tuesday, August 15, 2017

Chris Cerf Is Better Than You -- Just Ask Him

Chris Cerf is, of course, the State Superintendent of the Newark Public Schools. He holds this position not because the citizens of Newark chose him, nor because he was appointed by a democratically elected school board, nor because he was chosen by the democratically elected mayor, nor because he had an outstanding track record as a chief officer in another public school district (he never ran a district before Newark*).

No, Cerf only holds his office because Chris Christie -- the man for whom Cerf worked for three years as Education Commissioner -- put him there.

And why did Christie choose Cerf? We can only guess, but as this interview from the past Sunday makes plain, Cerf does share his boss's love of taking cheap shots at those who disagree with him. Before we get to that, however, let's see what spin Cerf is serving these days about charters:
Q. What do you think of the idea that charters siphon off resources from public schools?

A. It presumes charters are not public schools. But they are. They are publicly funded, open to all, tuition-free, accountable to democratically-elected authorities, and in New Jersey, not for-profit.
What nonsense. Yes, charter schools are publicly funded. But Newark Prep -- which was approved by Cerf when he was Commissioner but closed down last year due to poor performance -- was managed for years by K12 Inc., a for-profit corporation. And Camden Community Charter School -- also approved by Cerf, also closed due to poor performance this past year -- was also, according to the Star-Ledger, managed by a for-profit corporation.

But what's really galling here is Cerf's insistence that charters are overseen by "democratically-elected authorities." Because not one charter school in Newark was required to gain approval from the local school board, or the mayor, or the city council. All they needed was a nod from the Education Commissioner, who serves at the pleasure of the governor.

Keep in mind that Chris Christie lost in Newark by a landslide in the 2013 election: 6,443 to 29,039. The notion, then, that Newark's charter sector is "accountable to democratically-elected authorities" is a sick joke. But when it comes to charters, Cerf lives in an alternate universe:
The only difference is that a charter is managed by a board of citizens, as opposed to a local school board. Secondly, charters receive only 90 percent of the per-pupil operating cost of a traditional public school, which means that for every student who leaves for a charter, the district retains 10 percent to cover its fixed expenses.

So it's only a negative if the district cannot manage its own expenses to keep its fixed and legacy costs to a number equal to or less than 10 percent. [emphasis mine]
Cerf's hypocrisy here would be funny if the consequences weren't so serious. Because it wasn't even two years ago that Chris Cerf was complaining to the State BOE that charters had an unfair fiscal advantage, and that advantage was hurting his district:
Much of the budget pressure has come from payments the district is required to make to the city’s charter schools. Cerf, a cheerleader for charter schools as state commissioner, yesterday acknowledged that some funding stop-gap is needed to help the district.
He said the mandatory funding for charter schools year to year is “disproportionately hurting the district schools,” adding, “We can’t just turn the other way and let that happen.” [emphasis mine]
Oh, come on, Chris -- can't you keep your fixed and legacy costs in line?

As Cerf undoubtedly knows, New Jersey charter schools have been "held harmless" in their revenues for three years running. This means charters haven't had to take the financial hit that district schools, like Newark's, have had to take because Christie has refused to fully fund the School Funding Reform Act -- the state's own law that ensures districts get adequate and equitable funding.

For every year Christie has been in office -- including the three years Cerf was his Education Commissioner -- Christie has refused to follow the state's own law and provide New Jersey's schools with the funding they need to fulfill the mandates of the state constitution. In Newark, for FY2017 alone, this meant Cerf's district was $181 million below its adequacy target, or $3,615 per pupil.

Cumulatively, NPS is $611 million in the hole since the beginning of Christie's reign. It could have been worse: back in 2011, the state's Supreme Court ordered Christie to restore half-a-billion dollars in state aid to the neediest districts. But that didn't stop Cerf, working on Christie's behalf, from running up and down the state, making the case that New Jersey's schools didn't need the funding the state's own law said they needed.

I contend that there is no one individual over the last eight years -- and yes, I do include Christie -- who has done more to undermine the state's own law on school funding than Chris Cerf. It was Cerf, after all, who authored the 2012 Education Funding Report, an incoherent, rambling mess that tried to make the case that while money matters in education, the schools serving the most disadvantaged students were clearly suffering because they had too much.

As Bruce Baker noted in real time, once you got past the reformy pablum served up in the report, the actual policy of Christie and Cerf was to cut aid to districts serving larger proportions of at-risk students, while simultaneously giving more aid to districts enrolling proportionally fewer at-risk students.

Cerf's report made the case the "weights" for at-risk students were too high in SFRA, even as the empirical evidence showed they were, in fact, very likely too low. He tried to argue that eliminating tenure was absolutely critical for student success, even though there is no evidence whatsoever that tenure impedes student achievement (in fact, there's some evidence strong collective bargaining improves staff effectiveness). Cerf also argued for changing the way students are counted in the aid formula, a clear attempt to decrease funding going to districts enrolling more at-risk students.

But the Funding Report was only the beginning of Chris Cerf's assault on equitable and adequate school funding. In 2013, Cerf stood in front of the NJEA and presented a brazenly deceptive graph in an attempt to downplay the effects of poverty on student outcomes.

Here is the full set of data that Cerf's slide omitted:

Still amazes me every time I see it.

There was also Cerf's twisting of national test scores to make New Jersey's schools appear to be failing their students; Matt DiCarlo did a great job at the time deconstructing Cerf's arguments. And there was the December, 2012 report that bizarrely tried to rationalize the Christie aid cuts; again, Bruce Baker pointed out how absurd its arguments were. Plus there was the proposed policy of underfunding the districts Cerf's DOE classified as "Priority" more than ones classified as "Reward."

All of this went on while New Jersey, under Chris Christie, retreated from funding equity -- even as the record clearly showed the state had made significant gains during the period of meaningful funding reform.

Again: it's not hard to make the case that Chris Cerf was as responsible as anyone for giving cover to the underfunding of New Jersey's schools under Chris Christie -- including, notably, the Newark Public Schools. And yet, despite this track record, Cerf has the stones to make the case that he cares more for the wee children than anyone who dares to disagree with his reformy views:
I look with astonishment at groups like Save Our Schools, highly represented by white wealthy suburbanites that have made it their mission to undermine the opportunity of poor African-American students to have access to quality education. 
Many don't honor their own principles by sending their children to private schools or living in leafy green suburbs. I ask myself whether the strength of their argument would be affected if the focus of charters did not include suburbs like Princeton. 
Mostly, I'm aghast at seemingly progressive individuals who so deeply misunderstand the profound injury their position will cause families of vastly more limited means. [emphasis mine]
Let's start with the low-hanging fruit: Chris, when you were Education Commissioner, did you ever walk into the governor's office and fault him for sending his children to highly-resourced private schools?

Yes, there are people in Save Our Schools who live in the suburbs. Yes, there are people who question charter school proliferation who send their kids to private schools. The difference, Chris, is that none of them, so far as I know, have ever tried to undermine the state's own law regarding full and fair funding of public schools.

For three years, Chris, you did everything within your power to keep schools serving large numbers of at-risk students from getting the funds the state's own law says they should get -- all while your boss gave gobs of tax breaks to wealthy, connected special interests. You went out of your way to justify Christie's underfunding of SFRA with some of the most fallacious nonsense imaginable, costing Newark hundreds of millions of dollars for their schools.

And still, despite your own past, you publicly question others' motives:
I stay up at night wondering how otherwise good-hearted people could say they want to impose a moratorium or somehow stop the addition of charter schools, when so many children are choosing them and being successfully launched into adulthood on the basis of that choice. 
Unfortunately, I think the answer is a raw political one. By statute, traditional public schools are unionized and charter public schools could be unionized, but are not required to be. 
The interest group in the state that spends by far the most money in Trenton, the teacher's union, would be economically hurt by the increased number of charter schools because it would lose dues-paying members. I think this is a purely economic argument. They do not want to lose members. All their talk about creaming and hedge funds is just so much propaganda that they have focus grouped and determined works in the public debate.

Is it true? Is the charge that Newark's charters don't enroll the same sorts of students as NPS "just so much propaganda"?

Let's go to the data, straight from Cerf's former office, the NJDOE (and including the three years when he ran it):

Year after year, NPS enrolls proportionally more special education students than the Newark charter sector.

The special education students the charters do take tend to have lower-cost disabilities compared to the classified students enrolled at NPS. (The lower-cost disabilities are speech impairments (SPL) and specific learning disabilities (SLD); see here for more information.)

Newark's charter sector enrolls very few students who are Limited English Proficient. 

Yes, the free and reduced-price lunch eligibility rates between the district and the charters look the same. But keep in mind that NPS has been expanding its free lunch program** for some time, which means the incentive for families to fill out applications has waned. The charters, on the other hand, have a great incentive for their families to fill out the application, as their funding increases based on the number of FRPL students they enroll.

Whether there is, in fact, a difference in the socio-economic status of charter students compared to district students is a serious question. Cerf says there's "zero factual evidence" to support a charge of creaming. I say: What have you done, Chris, to explore the question, either as Commissioner or State Superintendent?

Who did you contract with to look into why charters don't enroll as many LEP students as their host districts? Who did you assign from your staff to write a report on the disparity in special education rates between the two sectors?

As you know, Chris, Julia Sass Rubin and I wrote the report using your department's data that explored these questions. But why did we have to do it? Why did I have to be the one to use your department's data and point out charter schools spend far more on administrative costs and far less on support services than the New Jersey charter sector?

Why did I have to be the one who pointed out that charter staffs have far less experience than district staffs, and consequently make less money? The charters you laud are basically free-riding on district salaries negotiated by the unions you casually denigrate. Did this ever concern you? Did you ever even think about it? Did you ever hire people for your senior staff who had the capacity to study this stuff?

Chris Cerf has a nasty habit of questioning other people's motives. You would have thought maybe a few years actually running a school district would give him a little bit of humility, and a little bit of space to reflect on his own culpability in underfunding New Jersey schools.

Sadly, Cerf appears set in his ways. He'd rather take his pot shots than own up to the terrible legacy he left as Commissioner. Luckily for Newark, he won't be around much longer. Hopefully, he'll soon return to the private sector, where his record speaks for itself.

Accountability begins at home.

ADDING: Remember this? Good times...

ADDING MORE: As Darcie Cimarusti reminds us, Atlantic City Community Charter is also managed by a for-profit CMO. And, according to the Star-Ledger, Central Jersey Arts Charter School contracts with a for-profit manager.

Pesky facts...

* Maybe now that Newark is regaining control of its schools, it can look around for a superintendent who has actually run a district - unlike the last two appointed by Chris Christie.

** This is a good thing. It's one of the few things I give Cerf and Cami Anderson credit for.

Saturday, August 5, 2017

CREDO Charter School Studies' "X Days Of Learning" Conversion: Still Unvalidated

This past week, the Center for Research on Education Outcomes (CREDO) at Stanford University released yet another report in a series on the effects of charter schools on test scores -- this time focusing on Texas.

Almost immediately, members of the the local media trumpeted the results as "proof" that charter schools are realizing meaningful gains in student outcomes:
For the first time, Texas charter schools have outperformed traditional public schools in reading and closed the gap in math, researchers at Stanford University have found.
Students at Texas charter schools, on average, received the equivalent of 17 additional days of learning per year in reading and virtually the same level of education in math when compared to traditional public schools, according to a study released Wednesday by the Center for Research on Education Outcomes, or CREDO.
Rather than looking at raw standardized test scores, CREDO researchers quantify the impact of a school by looking at student improvement on the tests relative to other schools. The researchers then translate those results into an equivalent number of "days of learning" gained or lost in a 180-day school year.
The center's staff produced similar analyses in 2013 and 2015, finding Texas charter schools had a negative impact on reading and math performance.
"The most recent results are positive in two ways. Not only do they show a positive shift over time, but the values themselves are both positive for the first time," the researchers wrote.
CREDO's studies of charter school performance are widely respected in education circles. The center compares students from charter and traditional public schools by matching them based on demographic characters -- race, economic status, geography and English proficiency, among others -- and comparing their growth on standardized tests. Scores from 2011-12 to 2014-15 were analyzed for the most recent report. [emphasis mine]
That's from the Houston Chronicle, which published just one paragraph suggesting the CREDO studies might have credible critics:
Skeptics of CREDO's study typically offer three main criticisms of the research: it focuses exclusively on standardized test results, incentivizing schools that "teach to the test"; it ignores other advantages of traditional public schools, such as better access to extracurricular activities; and it doesn't account for the fact that charter school students are more likely to have strong, positive parental influence on their education.
Sorry, but that's, at best, an incomplete description of the serious limitations of these studies, which include:

Here is how the CREDO Texas study reports its findings:

Stanley Pogrow published a paper earlier this year that didn't get much attention, and that's too bad. Because he quite rightly points out that it's much more credible to describe results like the ones reported here as "small" than as substantial. 0.03 standard deviations is tiny: plug it in here and you'll see it translates into moving from the 50th to the 51st percentile (the most generous possible interpretation when converting to percentiles).

I have been working on something more formal than a blog post to delve into this issue. I've decided to publish an excerpt now because, frankly, I am tired of seeing "days of learning" conversions reported in the press and in research -- both peer-reviewed and not -- as if there was no debate about their validity. 

The fact is that many people who know what they are talking about have a problem with how CREDO and others use "days of learning," and it's well past time that the researchers who make this conversion justify it. 

The excerpt below refers to what the eminent psychometrician Michael T. Kane coined a "validity argument." To quote Kane: "Public claims require public justification." I sincerely hope I can spark a meaningful conversation here and get the CREDO team to adequately and publicly justify their use of "days of learning." As of now, their validity argument is cursory at best -- and that's just not good enough.

I have added some bolding to the excerpt below to highlight key points.

* * *

Avoiding the Validity Argument: A Case Study

As an illustration of the problem of avoiding the validity argument in education policy, I turn to an ongoing series of influential studies of charter school effects. Produced by The Center for Research on Education Outcomes at Stanford University, the so-called CREDO reports have driven a great deal of discussion about the efficacy of charter school proliferation. The studies have been cited often in the media, where the effects they find are reported as “days of learning.”[1] Both the National Charter School Study (Raymond et al., 2013) and the Urban Charter School Study Report on 41 Regions (CREDO, 2015) include tables that translate the effect sizes found in the study into “days of learning.” Since an effect size of 0.25 SD is translated into 180 days, the clear implication is that an effect of this size moves a student ahead a grade level (a typical school year being 180 days long). Yet neither study explains the rationale behind the tables; instead, they cite two different sources, each authored by economist Eric Hanushek, as the source for the translations.

The 2015 study (p. 5) cites a paper published in Education Next (Hanushek, Peterson & Woessmann, 2012) that asserts: “On most measures of student performance, student growth is typically about 1 full std. dev. on standardized tests between 4th and 8th grade, or about 25 percent of a std. dev. from one grade to the next.” (p. 3-4) No citation, however, is given to back up this claim: it is simply stated as a received truth. 

The 2013 study (p. 13) cites a chapter by Hanushek in the Handbook of the Economics of Education (Hanushek & Rivkin, 2006), in which the author cites his own earlier work:
“Hanushek (1992) shows that teachers near the top of the quality distribution can get an entire year’s worth of additional learning out of their students compared to those near the bottom. That is, a good teacher will get a gain of 1.5 grade level equivalents while a bad teacher will get 0.5 year for a single academic year.” (p. 1068)
No other references are made within the chapter as to how student gains could be presented as years or fractions of a year’s worth of learning.

The 1992 citation is to an investigation by Hanushek of the correlation between birth order and student achievement, and between family size and student achievement. The test scores used to measure achievement come from the “Iowa Reading Comprehension and Vocabulary Tests.” (p. 88) The Iowa Assessments: Forms E and F Research and Development Guide (2015), which traces the development of the Iowa Assessments back to the 1970s, states:
“To describe the developmental continuum or learning progression in a particular achievement domain, students in several different grade levels must answer the same questions in that domain. Because of the range of item difficulty in the scaling tests, special Directions for Administration were prepared to explain to students that they would be answering some very easy questions and other very difficult questions.” (p. 55-56)
In other words: to have test scores reported in a way that allows for comparisons across grade levels (or, by extension, fractions of a grade level), the Iowa Assessments deliberately place the same questions across those grade levels. There is no indication, however, that all, or any, of the statewide tests used in the CREDO studies have this property.[2]

Harris (2007) describes the process of creating a common score scale for different levels of an assessment as vertical scaling. She notes: “Different decisions can lead to different vertical scales, which in turn can lead to different reported scores and different decisions.” (p. 233) In her discussion of data collection, Harris emphasizes that several designs can be used to facilitate a vertical scale, such as a scaling test, common items, or single group to scaling test. (p. 241) 

In all of these methods, however, there must be some form of overlapping: at least some students in concurrent grades must have at least some common items on their tests. And yet students in different grades still take tests that differ in form and content; Patz (2007) describes the process of merging their results into a common scale as linking (p. 6). He notes, however, that there is a price to be paid for linking: “In particular, since vertical links provide for weaker comparability than equating, the strength of the validity of interpretations that rest on the vertical links between test forms is weaker.” (p. 16)

So even if the CREDO studies used assessments that were vertically scaled, the authors would have to acknowledge that the validity of their effect sizes was at least somewhat compromised compared to effect sizes derived from other assessments. In this case, however, the point is moot: it appears that many of the assessments used by CREDO are not vertically scaled[3], which is a minimal requirement for making the case that effect sizes can be translated into fractions of a year’s worth of learning. The authors are, therefore, presenting their results in a metric that has not been validated and could be misleading.

I use this small but important example to illustrate a larger point: when influential education policy research neglects to validate the use of assessments, it may lead stakeholders to conclusions that cannot be justified. In the case of the CREDO reports, avoiding a validity argument for presenting effect sizes in “days of learning” has led to media reports on the effects of charter schools and policy decisions regarding charter proliferation that are based on conclusions that have not been validated. That is not to say these decisions are necessarily harmful; rather, that they are based on a reporting of the effects of charter schools that avoided having to make an argument for the validity of using test scores.

[2] Nor is there any indication the national and international tests Hanushek cites in his 2006 paper, such as the National Assessment of Educational Progress, share questions across grade levels. In fact, Patz (2007) notes: “NAEP attempted to vertically link forms for grades 4, 8, and 12, but abandoned the approach as the comparisons of students separated by 4 or 8 years were found to be ‘largely meaningless’ (Haertel, 1991).” (p.17)

[3] Some statewide assessments are vertically scaled, including the Smarter Balanced assessments; see: https://portal.smarterbalanced.org/library/en/2014-15-technical-report.pdf


Center for Research on Education Outcomes (CREDO) (2015). Urban Charter School Study Report on 41 Regions. Palo Alto, CA: Center for Research on Education Outcomes (CREDO), Stanford University. Retrieved from: http://urbancharters.stanford.edu/summary.php

Dunbar, S.& Welch, C. (2015). Iowa Assessments: Forms E and F Research and Development Guide. Iowa City, IA: University of Iowa. Retrieved from: https://itp.education.uiowa.edu/ia/documents/Research-Guide-Form-E-F.pdf

Hanushek, E. A. (1992). The trade-off between child quantity and quality. Journal of political economy100(1), 84-117.

Hanushek, E. A., & Rivkin, S. G. (2006). Teacher quality. Handbook of the Economics of Education2, 1051-1078.

Hanushek, E. A., Peterson, P. E., & Woessmann, L. (2012). Achievement Growth: International and US State Trends in Student Performance. PEPG Report No.: 12-03. Program on Education Policy and Governance, Harvard University.

Harris, D. J. (2007). Practical issues in vertical scaling. In Linking and aligning scores and scales (233-251). New York: Springer.

Kane, M. (2013). Validating the interpretations and uses of test scores. Journal of Educational Measurement 50(1), 1–73.

Raymond, M. E., Woodworth, J. L., Cremata, E., Davis, D., Dickey, K., Lawyer, K., & Negassi, Y. (2013). National Charter School Study 2013. Palo Alto, CA: Center for Research on Education Outcomes (CREDO), Stanford University. Retrieved from: http://credo.stanford.edu/research-reports.html