Test Scores Gains Are Not Necessarily a Sign of Better Instruction: A Cautionary Tale From Newark

This post is part of a series on recent research into Newark schools and education "reform."

Here's Part I.

Here's Part II.

* * *

In this series, I've been breaking down recent research about Newark, NJ's schools. Reformy types have been attempting to make the case that "reforms" in Newark over the past several years -- including charter school expansion, merit pay, Common Core alignment, school closures, and universal enrollment -- have led to gains in student learning. These "reforms" are purportedly the result of Facebook CEO Mark Zuckerberg's high-profile, $100 million grant to the city's schools back in 2010.

Zuckerberg recently funded a study, published by the Center for Education Policy Research at Harvard University this past fall, that shows a gain in "value-added" on tests for Newark compared to the rest of the state. (A technical paper, published by the National Bureau of Economic Research, is found here.)

Bruce Baker and I looked carefully at this study, and added our own analysis of statewide data, to produce a review of this research. One of our most important findings is that most of the "gains" -- which are, in our opinion, educationally small anyway (more on this later) -- can be tied to a switch New Jersey made in 2015 from the NJASK statewide exams to the newer PARCC exams.

As I noted in the last post, even the CEPR researchers suggest this is the most likely explanation for the "gains."

Assuming both tests have similar levels of measurement error, this implies that the PARCC and NJASK were assessing different sets of skills and the districts that excelled in preparing students for PARCC were not necessarily the same as the districts that excelled at preparing students for NJASK. Thus, what appears to be a single-year gain in performance may have been present before 2015, but was simply undetected by earlier NJASK tests. (p. 22, NBER, emphasis mine)

As I pointed out last time, there has never been, to my knowledge, any analysis of whether the PARCC does a better job measuring things we care about compared to the NJASK. So, while the PARCC has plenty of supporters, we really don't know if it's any better than the old test at detecting "good" instructional practices, assuming we can hold things like student characteristics constant.

But even if we did have reason to believe the PARCC was a "better" test, I still would find the sentence above that I bolded to be highly problematic. Let's look again at the change in "value-added" that the CEPR researchers found (p. 35 of the NBER report, with my annotations):

"Value-added" -- ostensibly, the measure of how much the Newark schools contributed to student achievement gains -- was trending downward prior to 2014 in English language arts. It then trended upward after the change to the new test in 2015. But the CEPR authors say that the previous years may have actually been a time when Newark students would have been doing better, if they had been taking the PARCC instead of the NJASK.

The first problem with this line of thinking is that there's no way to prove it's true. But the more serious problem is that the researchers assume, on the basis of nothing, that the bump upwards in value-added represents real gains, as opposed to variations in test scores which have nothing to do with student learning.

To further explore this, let me reprint an extended quote we used in our review from a recent book by Daniel Koretz, an expert on testing and assessment at Harvard's Graduate School of Education. The Testing Charade should be required reading for anyone opining about education policy these days. Koretz does an excellent job explaining what tests are, how they are limited in what they can do, and how they've been abused by education policy makers over the years.

I was reading Koretz's book when Bruce and I started working on our review. I thought it was important to include his perspective, especially because he explicitly takes on the writings of Paul Bambrick-Santoyo and Doug Lemon, who both hold just happen to hold leadership positions at Uncommon Schools, which manages North Star Academy, one of Newark's largest charter chains.

Here's Koretz:

One of the rationales given to new teachers for focusing on score gains is that high-stakes tests serve a gatekeeping function, and therefore training kids to do well on tests opens doors for them. For example, in Teaching as Leadership^[i] – a book distributed to many Teach for America trainees – Steven Farr argues that teaching kids to be successful on a high-stakes test “allows teachers to connect big goals to pathways of opportunity in their students’ future.” This theme is echoed by Paul Bambrick-Santoyo in Leverage Leadership and by Doug Lemov in Teach Like a Champion, both of which are widely read by new teachers. For example, in explaining why he used scores on state assessments to identify successful teachers, Lemov argued that student success as measured by state assessments is predictive not just of [students’] success in getting into college but of their succeeding there. 

Let’s use Lemov’s specific example to unpack this. 

To start, Lemov has his facts wrong: test scores predict success in college only modestly, and they have very little predictive power after one takes high school grades into account. Decades of studies have shown this to be true of college admissions tests, and a few more recent studies have shown that scores on states’ high-stakes tests don’t predict any better. 

However, the critical issue isn’t Lemov’s factual error; it’s his fundamental misunderstanding of the link between better test scores and later success of any sort (other than simply taking another similar test). Whether raising test scores will improve students’ later success – in contrast to their probability of admission – depends on how one raises scores. Raising scores by teaching well can increase students’ later success. Having them memorize a couple of Pythagorian triples or the rule that b is the intercept in a linear equation[ii] will increase their scores but won’t help them a whit later. 

[...] 

Some of today’s educators, however, make a virtue of this mistake. The[y] often tell new teachers that tests, rather than standards or a curriculum, should define what they teach. For example, Lemov argued that “if it’s ‘on the test,’ it’s also probably part of the school’s curriculum or perhaps your state standards… It’s just possible that the (also smart) people who put it there had a good rationale for putting it there.” (Probably? Perhaps? Possible? Shouldn’t they look?) Bambrick-Santoyo was more direct: “Standards are meaningless until you define how to assess them.” And “instead of standards defining the sort of assessments used, the assessments used define the standard that will be reached.” And again: “Assessments are not the end of the teaching and learning process; they’re the starting point.” 

They are advising new teachers to put the cart before the horse.”[iii] [emphasis mine; the notes below are from our review]

Let's put this into the Newark context:

• One of the most prominent "reforms" in Newark has been the closing of local public district schools while moving more students into charter schools like North Star.
• By their own admission, these schools focus heavily on raising test scores.
• The district also claims it has focused on aligning its curriculum with the PARCC (as I point out in our review, however, there is little evidence presented to back up the claim).
• None of these "reforms," however, are necessarily indicators of improved instruction.

How did Newark get its small gains in value-added, most of which were concentrated in the year the state changed its tests? The question answers itself: the students were taught with the goal of improving their test scores on the PARCC. But those test score gains are not necessarily indicative of better instruction. 

As Koretz notes in other sections of his book, "teaching to the test" can take various forms. One of those is curricular narrowing: focusing on tested subjects at the expense of instruction in other domains of learning that aren't tested. Did this happen in Newark?

More to come...

---

^[i] Farr, S. (2010). Teaching as leadership; The highly effective teacher’s guide to closing the achievement gap. San Francisco: Josey-Bass. We note here that Russakoff reports that Teach for America received $1 million of the Zuckerberg donation “to train teachers for positions in Newark district and charter schools.” (Russakoff, D. (2016). The Prize; Who’s in charter of America’s schools? New York, NY: Houghton, Mifflin, Harcourt. p. 224)
[ii] A “Pythagorean Triple” is a memorized ratio that conforms to the Pythagorean theorem regarding the ratio of the sides of a right triangle. Koretz critiques the linear intercept rule, noting that b is often taught as the intercept of an equation in high school, but is usually the coefficient of an equation in college courses. In both cases, Kortez contends test prep strategies keep students from gaining a full understanding of the concepts being taught. See: Koretz, D. (2017) The testing charade; Pretending to make schools better. Chicago, IL: University of Chicago Press. pp. 104-108.
[iii] Koretz, D. (2017) The testing charade; Pretending to make schools better. Chicago, IL: University of Chicago Press. p. 114-115.

What Are Tests Really Measuring? A Tale of Education "Reform" in Newark

This post is part of a series on recent research into Newark schools and education "reform."

Here's Part I.

"What is a test, and what does it really measure?"

I often get the sense that more than a few stakeholders and policy makers in education don't take a lot of time to think carefully about this question.

There aren't many people who would claim that a test score, by itself, is the ultimate product of education. And yet test scores dominate discussions of education policy: if your beloved program can show a gain in a test outcome, you're sure to cite that gain as evidence in favor of it.

That's what's been happening in Newark, New Jersey these days. As I said in my last post, new research was published by the Center for Education Policy Research at Harvard University this past fall that purportedly showed a gain in "value-added" on tests for Newark compared to the rest of the state. The researchers have attempted to make the case that a series of reforms, initiated by a $100 million grant from Mark Zuckerberg, prompted those gains. (A more technical study of their research, published by the National Bureau of Economic Research, is found here.)

To make their case, the CEPR researchers do what many others have done: take test scores from students, input them into a sophisticated statistical model, and compare the gains for various groups. To be clear, I do think using test scores this way is fine -- to a point.

Test outcomes can and often do contain useful information that, when properly used, tell us important things. But we always have to remember that a test is a sample of knowledge or ability at a particular point in time. Like all samples, test outcomes are subject to error. Give a child who ate a good breakfast and got enough sleep a test in a quiet room with the heat set properly, and you'll get one score. Give that same child the same test but on an empty stomach in a freezing cold room, and you'll almost certainly get something else.

The variation in outcomes here illustrates a critical point: Often the scores on a test vary because of factors that have nothing to do with what the test is trying to measure. Psychometricians will often talk about construct validity: the extent to which a test is measuring what it is supposed to be measuring. Making a valid test requires not only creating test items that vary based on what we're trying to measure; it also requires defining what we're trying to measure.

Take, for example, New Jersey's statewide assessments in Grades 3 through 8 -- assessments required by federal law. For a number of years, the state administrated the NJASK: the New Jersey Assessment of Skills and Knowledge. It was a paper-and-pencil test that assessed students in two domains: math and English language arts (ELA).

Those are very big domains. What, exactly, comes under ELA? Reading and writing, sure... but reading what? Fiction? Informational texts? Toward what end? Comprehension, sure... but what does that mean? How does anyone demonstrate they comprehend something? By summarizing the text, or by responding to it in an original way? Is there a fool-proof way to show comprehension? And at what level?

These questions aren't merely a philosophical exercise -- they matter when building a test. What goes into the construct we are trying to measure? And, importantly, do the tests we give vary based on what we intend to use the tests to measure?

In the case of the recent Newark research, the economists who conducted the study made an assumption: they believed the test scores they used vary based on the actions of school systems, which implement programs and policies of various kinds. They assumed that after applying their models -- models that attempt to strip away differences in student characteristics and abilities to learn --  the variation in outcomes can be attributed to things the Newark publicly-financed schools, including the charter schools, do that differ from schools in other parts of the state.

It's a big assumption. It requires showing that the policies and programs implemented can be documented and, if appropriate, measured. It requires showing that those policies and programs only took place in Newark. And it requires making the argument that the variation found in test outcomes came only from those policies and programs -- what social scientists would call the treatment.

Further, this assumption requires making yet another assumption:

In 2015, New Jersey switched its statewide exam from the NJASK to the PARCC: the Partnership for Assessment of Readiness for College and Careers. PARCC is (mostly) a computerized exam. Its supporters often claim it's a "better" exam, because, they say, it measures things that matter more. I'm not going to get into that debate now, but I will note that, so far as I know, no one ever conducted a validity study of the PARCC compared to the NJASK. In other words: we're not sure how the two tests differ in what they measure.

What I can say is that everyone agrees the two exams are different. From what I've seen and heard from others, the PARCC math exam relies more on language skills than the NJASK math exam did, requiring students to do more verbal problem solving (which would put non-native English speakers at a disadvantage). The PARCC ELA exam seems to put more emphasis on writing than the NJASK, although how that writing is graded remains problematic.

Keeping this in mind, let's look at this graph from the CEPR research (p.35):

Until 2014, Newark's test score "growth" is pretty much the same as the other Abbott districts in the state. The Abbotts are a group of low-income districts that brought the famous Abbott v. Burke lawsuit, which forced the state toward more equitable school funding. They stand as a comparison group for Newark, because they have similar students and got similar test outcomes...

Until 2015. The Abbotts, as a group, saw gains compared to the rest of the state -- but Newark saw greater gains. Whether the size of those gains is educationally significant is something we'll talk about later; for right now, let's acknowledge they are statistically significant.

But why did they occur? Let me annotate this graph:

Newark's gains in "growth," relative to other, similar New Jersey districts, occurred in the same year the state switched exams.

And it's not just the CEPR research that shows this. As Bruce Baker and I showed in our review of that research, the state's own measure of growth, called Student Growth Percentiles (SGPs), also show a leap in achievement gains for Newark in the same year.

Again, the red line is the dividing point between the NJASK and the PARCC. In this case, however, we break down the districts into the Newark Public Schools, Newark's charter schools, and only those Abbotts in the same county as Newark. The districts close to Newark with similar demographics had similar gains in achievement "growth."

Let's step back and remember what the CEPR study was trying to understand: how a series of policies, initiated by Zuckerberg's donation, affected test score growth in Newark. What would we have to assume, based on this evidence, to believe that's true?

• That the Newark reforms, which began in 2011, didn't kick in until 2015, when they suddenly started affecting test scores.
• That the gains in the other Essex County Abbott districts (Irvington, Orange, and East Orange) were caused by some other factor completely separate from anything affecting Newark.
• That the switch from the NJASK to the PARCC didn't create any gains in growth that were unrelated to the construct the tests are purportedly measuring.

Test makers will sometimes refer to the concept of construct-irrelevant variation: that test outcomes will vary because things we do not want them to measure still affect the scores. If two children with equal mathematical ability take a computerized test, but one has greater facility in using a computer, their test scores will differ. The problem is that we don't want their scores to differ, because we're trying to measure math ability, not familiarity with computers.

Did Newark's students -- and Orange's and East Orange's and Irvington's -- do better on the PARCC simply because they felt more at ease with the new PARCC test than students around the rest of the state? Did these districts engage in test prep activities specific to the PARCC that brought scores up, but didn't necessarily reflect better instruction?

The CEPR study admits this is likely:

Assuming both tests have similar levels of measurement error, this implies that the PARCC and NJASK were assessing different sets of skills and the districts that excelled in preparing students for PARCC were not necessarily the same as the districts that excelled at preparing students for NJASK. Thus, what appears to be a single-year gain in performance may have been present before 2015, but was simply undetected by earlier NJASK tests. (p. 22, NBER, emphasis mine)

I'll get into that last sentence more in a future post. For now, it's enough to note this: Even the CEPR team acknowledges that the most likely explanation for Newark's gains is the state's switch from the NJASK to the PARCC. But aligning instruction with one test more than another is not the same as providing better instruction. 

Gains like these are not necessarily an indication of curricular or instructional improvements. They are not necessarily brought about by moving students into "better" schools. They could very easily be the result of the tests measuring different things that we don't really want them to measure.

We'll talk more about this -- and get the views of a Harvard education expert -- next.

Education "Reform" in Newark: The Facts - Prelude

Miss me?

Yesterday, the National Education Policy Center published a lengthy report written by Dr. Bruce Baker and myself that looks closely at school "reform" in Newark. I wrote a short piece about our report at NJ Spotlight that gives summarizes our findings. We've also got a deep dive into the data for our report at the NJ Education Policy website.

You might be wondering why anyone outside of New Jersey, let alone Newark, should care about what we found. Let me give you a little background before I try to answer that question...

In 2010, Mark Zuckerberg, the CEO and founder of Facebook, went on the The Oprah Winfrey Show and announced that he was giving $100 million in a challenge grant toward the improvement of Newark's schools. Within the next couple of years, Newark had a new superintendent, Cami Anderson. Anderson attempted to implement a series of "reforms" that were supposed to improve student achievement within the city's entire publicly-financed school system.

In the time following the Zuckerberg donation, Newark has often been cited by "reformers" as a proof point. It has a large and growing charter school sector, it implemented a teacher contract with merit pay, it has a universal enrollment system, it "renewed" public district schools by churning school leadership, it implemented Common Core early (allegedly), and so on.

So when research was released this fall that purported to show that students had made "educationally meaningful improvements" in student outcomes, "reformers" both in and out of New Jersey saw it as a vindication. Charter schools are not only good -- they don't harm public schools, because they "do more with less." Disruption in urban schools is good, because the intractable bureaucracies in these districts needs to be shredded. Teachers unions are impeding student learning because we don't reward the best teachers and get rid of the worst...

And so on. If Newark's student outcomes have improved, it has to be because these and other received truths of the "reformers" must be true.

But what if the data -- including the research recently cited by Newark's "reformers" -- doesn't show Newark has improved? What if other factors account for charter school "successes"? What if the test score gains in the district, relative to other, similar districts, isn't unique, or educationally meaningful? What if all the "reforms" supposedly implemented in Newark weren't actually put into place? What if the chaos and strife that has dogged Newark's schools during this "reform" period hasn't been worth it?

What if Newark, NJ isn't an example of "reform" leading to success, but is instead a cautionary tale?

These are the questions we set out to tackle. And in the next series of posts here, I am going to lay out, in great detail, exactly what we found, and explain what the Newark "reform" experiment is actually telling us about the future of American education.

One other thing: as I have said before, the "reformers" often appear to misunderstand how research should be used to inform public policy. Often, you will hear them say some variation of this: "We have to do something to improve our schools -- and you can't prove my preferred reform won't make schools better!"

In fact, the burden of proof is on those who contend that charter school expansion, school closures/"renewals," merit pay, Common Core, universal enrollment, and so on will improve schools. If I or others present evidence that calls into question any of these policies, it is up to the promoters of those policies to show why that evidence is not germane.

For example: there is no question charter schools -- particularly the Newark charters run by big charter management organizations (CMOs) like TEAM/KIPP and North Star -- enroll very few Limited English Proficient (LEP) students. This puts the burden of educating these children on the public district schools. It explains why charters can spend less on instruction and student support services compared to NPS.

It is up to charter defenders to show why this doesn't matter when asserting charter claims of relatively better productivity. The burden of proof is on them, not on those who question their claims.

Much more to come...

More to come...

Education "Reform" Is a Right-Wing Movement

One of the perils of "success" is that it opens you up to increased scrutiny. Just ask Eva Moskowitz, the founder of the "successful" Success Academies chain of charter schools. Her constant self-promotion was nothing less than an open invitation to the press and others to take a look at how her schools achieved their "success."

Sure enough, a hard look at SA reveals disturbing disciplinary practices, a highly questionable curriculum, huge resource advantages (gained by appealing to wealthy donors and by wage "free riding" on public district schools), a distinctly different student population compared to neighboring public district schools, and patterns of significant student cohort attrition.

In other words: the "success" of Success Academies is largely attributable to the chain's ability to game the system. There's just no evidence Moskowitz and her staff have found any innovative, let alone scalable, methods to improve schooling for urban students.

But more recent scrutiny of SA has revealed something else: Moskowitz's financial patrons are much more closely aligned with the far political right than the current "reform" narrative would like us to believe.

The problems for Moskowitz on this front started last August, when the chairman of SA's board, Dan Loeb, compared an African-American New York state senator to the KKK:

The hedge fund manager Daniel S. Loeb, a prominent supporter of charter schools and a major financial backer of Gov. Andrew M. Cuomo and congressional Republicans, accused the African-American woman who leads the Democrats in the New York State Senate of having done “more damage to people of color than anyone who has ever donned a hood.” [emphasis mine]

Somewhat lost in the outrage, however, was the fact that this wasn't Loeb's first time getting caught saying something offensive and stupid. As Chalkbeat notes:

This isn’t the first offensive comment he’s made. Far from it, in fact. Loeb is fast-fingered on Facebook and frequently uses derogatory language to lash out at people who have made him unhappy. Here are a few of the examples that have been reported previously:

• Another time Loeb compared the unions and their supporters to the KKK: Loeb posted the following on his Facebook page in 2016, first reported by Dealbreaker: “If you truly believe that education is the dividing line (and I concurr) then you must recognizer and take up the fight against the teachers union, the biggest single force standing in the way of quality education and an organization that has done more to perpetuate poverty and discrimination against people of color than the KKK.”
• Using a derogatory term for people of color: Loeb once got into a fight with Fairfax Financial, a Canadian insurance company, which resulted in a lawsuit. Reported Reuters in 2011: “Fairfax’s filing quotes Loeb as saying he found the situation somewhat ironic because “the odds are much greater of being strung up by a Canadian Jew than a Canadian schwarze.” Loeb, who is Jewish, used “schwarze,” a derogatory Yiddish word for a black person, to describe Watsa, who is of Indian ancestry.”
• Making light of domestic violence by comparing Obama to an abuser: In a 2010 letter to hedge fund managers who had supported Obama, Loeb wrote, according to CNBC: “I am sure, if we are really nice and stay quiet, everything will be alright and the President will become more centrist and that all his tough talk is just words; I mean he really loves us and when he beats us, he doesn’t mean it; he just gets a little angry.”
• Making a xenophobic, homophobic attack against a rival: A damning 2013 Vanity Fair profile dredged up an anecdote from 1999, when Loeb was feuding with John Liviakis, a San Francisco public-relations executive. In an “imaginary monologue” in the voice of Liviakis, Loeb wrote under a pseudonym: “Then I will laugh at you fools for buying my shares and I will celebrate with a bottle of grappa, some fresh feta, and a nice young boy-just like in the old country.” Liviakis sued him for libel.

Apparently, Moskowitz thinks this is exactly the sort of person who should be leading the board of a school system that primarily enrolls students of color. And Loeb's support isn't limited to SA:

Loeb’s allies say his mean-spirited comments don’t necessarily reflect deep-seated beliefs. “I have known Dan to be a champion for underserved children who has worked tirelessly for years on their behalf,” said Jenny Sedlis, the head of StudentsFirstNY and a former deputy to Moskowitz, last week. “I know from first-hand experience the post he made does not reflect his true beliefs or the person he is.”

He has championed progressive causes in the past. Most notably, Loeb helped get gay marriage on the books in New York by throwing his influence into winning over Senate Republicans. This position put him in line with most Democrats and with Moskowitz, who has had wide support in New York City’s gay community for nearly 20 years. It also suggests that some of his internet posts, which have included seemingly homophobic comments, do not necessarily reflect the entirety of his beliefs.

Look, I will be the first to admit that people -- especially me -- occasionally say and do stupid things. And people can and do grow. But there are at least two problems with Sedlis's defense of Loeb. First, he said what he said just a month ago, and he has said similar things in the past. So you'll forgive me, Ms. Sedlis, if I think the words coming out of Loeb's mouth are a better indicator of his world view than your "first-hand experience," whatever that might be.

Second: It's clear that Dan Loeb is not the only player in the education "reform" movement whose mindset aligns with the political right.

Last month, for example, Leo Casey posted a long Twitter thread that looked at the connections SA's patrons have to the conservative movement. Kenneth Campbell of AlterNet also picked up the story:

Success Academy, New York City’s largest charter school network, has been the subject of a steady stream of unflattering press reports since Board chair and billionaire hedge funder Dan Loeb accused New York Legislature's highest-ranking black woman of inflicting more damage to people of color than the KKK. But Loeb isn’t the only Success leader who traffics in incendiary racial commentary. Board member Charles Strauch has had a blog for years that specializes in right-wing race baiting and recycled conspiracy theories from the dregs of the Internet, many with a racial tinge. 

Strauch’s blog, Wealth Creates Good, was taken down on September 5th, not long after I began Tweeting excerpts of his posts to Success, asking for a response. (An archive of some of Starch’s post can still be viewed here.) [emphasis mine]

As a long-time blogger, I can tell you there are probably several things I've posted over the years that I don't agree with now. But I don't take down my posts because I believe anyone who puts their thoughts out into the public should either have to defend or disavow them. The fact that Strauch took down his blog speaks volumes.

Campbell continues:

Strauch isn’t even the highest profile right-winger on the Board. That honor goes to Suzie Kovner, who, with her husband, has contributed more than $3 million to Republican causes, including charter school expansion and school choice. Once called “George Soros’s right-wing twin” and “the patron saint of the neoconservatives” Kovner’s husband was formerly the chairman of the American Enterprise Institute, where he led a conservative PR effort to undermine unions and halt the spread of a plot to spread socialism throughout America, AKA the living wage. Meanwhile, Suzie Kovner has a seat on Success Academy’s Board, overseeing a network where more than three-quarters of students’ families are classified as low-income, and getting poorer, even as the wealthiest New Yorkers, the Loebs and the Kovners among them, have grown richer still.

Success Academy’s love affair with the reactionary right wing doesn’t stop at the boardroom. The controversial Mercer family, owners of the alt-right outlet Breitbart News, known for its crusades against immigrant students and warnings about how the LGBT agenda is hijacking America’s youth, has contributed more than $1 million to Success Academy through their family foundation. And of course there is Success Academy CEO Eva Moskowitz’s own support for school choice zealot Betsy DeVos. Despite concerns from faculty at her schools, Moskowitz has been reluctant to distance herself from President Trump and his Secretary of Education. After all, the network’s political influence has been key to helping Moskowitz realize her vision of growing a charter school empire in New York.

It's pretty clear at this point that any notion of Moskowitz being a liberal is a pretense. However, some might ask: "So what?" Does it really matter if SA takes the money of right-wingers in support of urban education?

I'd argue it matters a lot.

First, consider what the Success Academy formula for "success" is: privately-controlled schools, fueled by public money with private donations mixed in as an additive. Combine this with the clear differences in student populations and the wage free-riding and you've got a big problem with resource inequities between SA and the NYC public district schools.

New York State has been systemically underfunding its own law regarding state aid to schools for years. It's the very districts where charter schools are proliferating that have paid the largest price. I've seen the same pattern here in New Jersey, made worse by the "hold harmless" funding policy of the Christie administration.

Across the nation, spotty charter "success" stories, like SA, have been the justification for this chronic under-resourcing of schools. Bruce Baker has started digging into this and the preliminary results are disturbing: "choice" is being offered in place of adequate school funding.

Support for charter proliferation goes hand-in-hand with a lack of support for adequate and equitable public school funding. No wonder the political right, which has set the accumulation of wealth for a small elite as its highest priority, loves the charter movement.

Second, the ties between SA and the political right highlight a clear reality: The charter school movement is, at its core, an anti-teachers union movement. Unions have been the backbone of the Democratic Party for years -- especially public sector unions. And the teachers unions have been pretty much the last vestige of professional unionism.

I know the unions are trying to organize charter teachers; it's telling that the charter managers have fought so hard to stop them.

Third, charter schools have proliferated largely because agency has been stripped away from urban communities. One of the charter supporters' favorite chants is that charters allow families to "vote with their feet." But that's not the same as voting with your vote.

Charters in New Jersey sprang up in Newark, Jersey City, Paterson, and Camden: all cities where schools have been under state control. The state also controls the schools in Philadelphia, a major charter haven; same in Detroit. New York City and Chicago have "strong mayor" control of schools; the charter sector has grown in both cities.

When the voters of Boston were given the chance to decide on whether they wanted to see charters expand, they overwhelmingly said no (especially in communities of color). Now we find out the entire campaign to expand charters in Massachusetts was illegally funded by extremely wealthy plutocrats; in other words, the notion that there is a urban, grassroots movement for charter school proliferation is a conceit.

The charter school movement has thrived in a climate where communities of color have been denied agency over the governance of their schools. If that isn't an alignment with right-wing values, I don't know what is.

Part of the marketing of school "choice" has been the selling of a myth that charter schools and vouchers are disputed issues within the liberal circles -- that there are many on the left who embrace "choice" because it aligns with their overall liberal worldview. But we must always remember that the reformy types know they will always have conservatives on their side; there's no need to convince Republicans of the virtues of privatizing any civic institution.

No, the real work for reformies is in getting the left on board. When failed hedge fund manager Whitney Tilson started Democrats for Education Reform, he all but admitted he was infiltrating the left so he could sell them what was always an inherently right-wing plan:

“The real problem, politically, was not the Republican party, it was the Democratic party. So it dawned on us, over the course of six months or a year, that it had to be an inside job. The main obstacle to education reform was moving the Democratic party, and it had to be Democrats who did it, it had to be an inside job. So that was the thesis behind the organization. And the name – and the name was critical – we get a lot of flack for the name. You know, “Why are you Democrats for education reform? That’s very exclusionary. I mean, certainly there are Republicans in favor of education reform.” And we said, “We agree.” In fact, our natural allies, in many cases, are Republicans on this crusade, but the problem is not Republicans. We don’t need to convert the Republican party to our point of view…” [emphasis mine]

I really don't know how much more clear this could be:

- The education "reform" movement provides a pretext for underfunding public schools, which aligns with right-wing values.

- The education "reform" movement is inherently anti-union, which aligns with right-wing values.

- The education "reform" movement thrives when communities of color lose agency over their schools, which aligns with right-wing values.

- The education "reform" movement is financed by wealthy people who openly profess conservative values.

Can we please, then, stop this nonsense about charter schools and vouchers being a policy embraced by the left? Yes, there are some Democrats and other folks who are otherwise liberals who support "choice." But their embrace of "reform" -- whether out of ignorance or hypocrisy or, yes, even genuine belief -- is inconsistent with the liberalism they espouse in other policy areas.

Education "reform" is a right-wing movement. There is nothing remotely liberal about privatizing schools, demonizing unions, and making excuses for underfunding education. If you support charter schools and vouchers and call yourself a liberal, that is, of course, your right. But it's really no different than being a pro-assault weapon liberal, or a pro-life* liberal: you're holding a position on at least one issue (and likely others) that is philosophically aligned with the right.

More school choice "liberals."

* I hate -- hate -- that label; the proper term is "anti-choice." But I've been using "choice" here to mean school choice, and I don't want to confuse that with a woman's right to choose.

Help Elect Linda Weber to Congress

Dear Readers of "Jersey Jazzman,"

If you're a teacher like me, this past month you got your first paycheck since last spring.

I'd like you to consider investing some of it in the future of public education -- by donating to my wife Linda's campaign for Congress.

Click here to donate.

There will never be a better friend of public education in the House of Representatives than Linda Weber -- that's why the Network for Public Education Action gave her its early endorsement. Linda will fight for fair and equitable funding for schools, oppose federal funds for school vouchers, and demand that charter schools return to their original mission as laboratories of innovation and never exist to make a profit.

Linda is a strong advocate for a woman's right to choose, for the environment, for civil rights, for racial equality, for pay equity, and for LGBTQ rights. We are a proud union family, and Linda will work hard to protect the right of workers to collectively bargain.

Her opponent, Leonard Lance, is a Republican enabler of Donald Trump. Yet Hillary Clinton won this district in last year's presidential race. If we have any hope of flipping the House, we need to win New Jersey's 7th District. Linda Weber -- a smart, accomplished, progressive woman -- is our best shot for putting another Democrat in the House and finally holding Trump accountable.

Linda is up against a September 30 fundraising deadline. Any amount you can give would help us flip this seat, take back Congress, and save public education.

Click here to donate.

Many thanks,

Mark "Jersey Jazzman" Weber

To learn more about Linda, visit her campaign website.

What Really Makes a "No Excuses" Charter School?

Once again, Matt Barnum points us to a study showing positive effects from a "no excuses" charter school. The paper, by Matthew Davis and Blake Heller, takes advantage of the fact that, for three years, the Noble charter school network on Chicago had more applicants than it had available seats, and therefore held a lottery. The research compared students who were and weren't offered a seat a Noble, and found those who were offered charter admission were somewhat more likely to attend and persist in college.

I've written quite a bit about the limitations of these charter "lottery" studies before: see here, here, and here. And one of the biggest issues I've consistently found is that the definition of the treatment -- here defined as acceptance into a "no excuses" charter school -- is quite fuzzy.

What exactly does "no excuses" actually mean? Davis and Heller cite five factors identified by Dobbie & Fryer in a 2013 paper: "frequent teacher feedback, data driven instruction, high-dosage tutoring, increased instructional time, and a relentless focus on academic achievement."* But Dobbie & Fryer never describe how these factors vary when compared to public district schools; they only studied how they vary within a group of NYC charter schools.

So we really don't know if these are the behaviors that lead to the effects found in the lottery studies when charters are compared to counterfactual public district schools. This is serious limitation, because it may well be that the five factors aren't the differences that really lead to these charter school outcome gains.

In other words: what if "no excuses" really means something other than what this research assumes it means?

Davis and Heller do cite another source to help define "no excuses" -- a source that Dobbie & Fryer, tellingly, also cite. A source that shows up time and again in the charter "lottery" studies to help explain what a "no excuses" charter school is: see here, here, here, here, here, and here, among others.

No Excuses, by Abigail & Stephan Thernstrom, is the touchstone for these researchers' descriptions of the charter schools they study. Unfortunately -- and I'm going to be charitable here -- the text is simply not worthy of citation in a serious journal article. 

I'd say the book's description of its qualitative methods was inadequate, but that would be incorrect: it doesn't describe its methods at all. For example: aside from trafficking in some highly questionable racial profiling, the book features two chapters where it describes "superb" "high-poverty public schools." (p. 43)

But we have no idea why the Thernstroms picked these schools or what methods they used to study them; the only clue is on page 43: "We chose these particular examples of fabulous education only because they came to our attention and we visited them."** We further have no idea how they came to the conclusion these schools differ in behaviors or practices from public district schools.

And yet this is a primary source many of the lottery studies use to define their treatment. When Davis and Heller assert the Noble schools got a 10 percentage point gain in college persistence because of the chain's "no excuses" pedagogy,*** they are echoing the Thernstroms' assertions about the value of increased instructional time and high expectations...

Except that's not all the Thernstroms have to say about what makes a "no excuses" school. In fact, No Excuses is quite blunt about how "fabulous" schools get the gains that they get. What follows is an extended excerpt from the book that cuts right to the heart of how the Thernstroms believe "no excuses" charter schools operate. Certainly, this passage isn't the entirety of the Thernstroms' argument; however, they are remarkably candid about the role of self-selection in these "fabulous" schools.

If quantitative researchers are willing to cite No Excuses, they should at the very least acknowledge this part of the Thernstroms' thesis.

[All emphases are mine.]

* * *

(From No Excuses by Abigail & Stephan Thernstrom. Simon & Schuster (2003), pp. 47-49.)

Schools That "Cream"?

Not all minority students in the inner city are alike, of course. Some are more academically promising than others. Some have parents who place a particularly high value on education. Do the schools we so admire cream? Are the schools getting unrepresentative academic results because their students are not representative of the neighborhoods from which they draw?

With one exception, none of the schools selects students on the basis of test scores, grades, or recommendations. It is true that, in theory, Rafe Asquith's kids are part of a "gifted" program, although the school's definition of "gifted" means he has, in fact, a mixture of academically talented and struggling youngsters. There are about ten such "gifted" classes at Hobart, and the others do not yield remotely comparable results. Some of these schools, like the Amistead Academy in New Haven and the KIPP academies, simply select their students by a lottery.

At almost all the schools we celebrate, though, there is an application process that tends to -- and is intended to -- discourage families unlikely to cooperate with the school. Indeed, one of the five pillars upon which the KIPP schools rest is "choice and commitment." Students, their parents, and the faculty of a KIPP academy "have made a choice to be at the school," the literature reads. "No one is assigned or forced to attend these schools." Choice entails commitment -- a commitment that not every parent and student is willing to make. And thus KIPP and other schools of choice are, in this minimal sense, self-selective. Families interested in KIPP must be willing to come in for initial interviews, and to sign on to a rigorous program that includes long days, a long year, and summer school, as well as a strict behavioral code.

Parents attracted to North Star come to an open house, where Norman Atkins, the cofounder, gives them a "very negative sell," as he describes it. The school is not for them, he says, if they want to take the family on vacation in July, if they don't like school uniforms, if two hours of homework a night seems excessive. If they still find North Star appealing, they watch a simulated school day on a Saturday and can pick up an application. Students are picked randomly from the applicant pool. Among the schools we refer to in the pages below, only the Fredrick Douglass Academy in Harlem looks at test scores and other indices of academic and personal strength in admitting students.

The commitment that KIPP demands is spelled out in contracts that set expectations and are signed by teachers, parents, and students. Students must agree, for instance, to "always work, think, and behave in the best way [they] know how and... do whatever it takes for [them] and [their] fellow students to learn." They must promise to attend school faithfully, do their homework, and "behave so as to protect the safety, interests, and rights of all individuals in the classroom." And they must understand that that are "responsible for [their] own behavior." The contract parents sign is similar, but includes provisions on helping their "child in the best way we know how" and making themselves available to their children and the school. They, too, must agree that the ultimate responsibility for the "behavior and actions" of their child lies with the family.

North Star parents are urged to sign a voluntary "Covenant" that includes such items as: "We will make certain that our child attends school every day, except in cases of illness or another legitimate reason," and "we will check our child's homework each night and provide quiet time and space for the work to be completed." At South Boston Harbor Academy (SBHA) and most of the other successful schools we visited, there were similar agreements, although many parents inevitably fail to keep their end of the bargain. In running SBHA, the biggest surprise, the principal told us, was the lack of parental support for the high standards; parents often took their children out of school for an appointment or family vacation or made excuses for inexcusable absences.

And yet, with almost no exceptions, the children who violate the terms, or whose parents do, are not expelled. Norman Atkins stated the problem well: "There is always," he said, "in good schools serving inner-city populations, a tension between rigor and rescue." North Star can stick to the letter of its law and toss a recalcitrant student out, but where will he or she go?

Nevertheless, the fact that these are schools of choice is not incidental to their success. "Most of what we do here, if it's not reinforced at home, becomes incredibly challenging," one North Star staff member said to a doctoral candidate writing a dissertation on the school. Others at North Star, KIPP, and elsewhere have echoed the sentiment. Families that choose a school have at least, on paper, committed themselves to supporting the academic program. And schools of choice gain leverage from the simple fact that students who come willingly can be told, you have another option: Return to a school where fights are common, kids talk in class and curse teachers, and no one does any homework. But, remember, since the students at these schools learn little, their future is dim.

"This school is all about choice," James Verrilli, cofounder of North Star, noted in his opening remarks to students on their first day a few years back. "See that back door? See any locks on it? Is this a prison? Am I forcing you to be here? ... If you cannot live by our rules, if you cannot adapt to this place, I can show you the back door." Norman Atkins, that same day, delivered much the same message: "We have 479 students, 479 students, who want to be sitting where you're sitting, who wish deep down that they had your seat. And you got it. So if you're not making the most of your experience here -- if you're not working to your absolute top potential, if you're not going home, running home to do your homework every night, if you're not respecting your teachers and each other with all your energy -- some of the other kids would like your spot."

That is precisely the message, of course, that no regular public school delivers. An urban principal whose school we visited told us that she was sending her own children to parochial schools. She wanted a school that could impose limits, with consequences for those who refused to abide by them. Why couldn't she do that, herself, in the school she runs? we asked. "I'm powerless," she answered. The students are stuck with the school and the school is stuck with them.

* * *

While I think this passage pretty much speaks for itself, let me add a few quick thoughts:

- One response to the "creaming" argument I often read from the economists who conduct these lottery studies is that the random assignment to treatment controls for unobserved variables. In other words: because the lotteries are random, we can assume that, on average, things like parental motivation are the same in both the group of students who are accepted into the charter schools, and the group of students who are not.

There are two problems with -- or perhaps more precisely, limitations on -- this argument. First, at least some of the students who "lost" the lottery do not enroll in the counterfactual public district schools. They might go to a parochial or private school, or go to a public school out of state, or be home schooled. It's reasonable to think at least some parents who are highly motivated and have greater means will choose these options for their children; to the extent that they do, there is a difference between the control and treatment groups. We can try to measure that difference with crude binary variables or error-prone test scores, but those are ultimately proxies for what we really care about.

[ADDING: To be clear, it appears this isn't an issue with Davis and Heller's study, because the students are tracked to college regardless of where they eventually went to high school. This is an issue in other lottery studies, however, especially if the outcomes measured are statewide accountability tests. Private school students don't take these tests, so they can't be included in the control group.]

The greater limitation, however, is one that is often lost in the discussion of these studies, and is directly related to the passage above: The estimated effects of charter schools in these studies can only be generalized to students who enter the lottery.

We really have no idea whether or not expanding charter schools will yield the same effects if the pool of applicants expands as well. The passage above suggests that the students and families who enter the lottery perceive themselves as quite different from the families who don't enter the lottery. The charter school officials and the Thernstroms seem to agree. Is the charter effect really little more than a peer effect? That would greatly limit the effects of charter school expansion.

- The Thernstroms make a big deal about the relatively low expulsion rate for "no excuses" charters. But we've got enough data now to know that's not really how the charter attrition game is played.

Charter schools don't have to expel students; they just have to hassle them with high suspension rates or constant disciplinary actions for minor infractions. Eventually, the kids will voluntarily transfer out.

- One rebuttal I often hear from charter supporters is that things are changing so quickly within the sector that the last piece of research which might call into question charter "success" is outdated. I will concede that KIPP, for example, seems to have had something of a change of heart in the last couple of years, and is backing away somewhat from strict disciplinary measures and high attrition rates.

But part of that is they've built a brand, and everyone now knows what it is. Parents send their kids to a KIPP school knowing what they'll get. It's not like a decade ago when most people weren't sure what a "no excuses" charter school was; parents in charter-heavy districts are much more likely to know what they're in for, which means the self-selection increasingly can take place before students enter the lottery.

In any case, there's at least some evidence that as KIPP has built up, it's effects have cooled off. Charter effects are not guaranteed to continue as populations expand.

- Notice how much Atkins relies on being able to claim he has a waitlist for students to get into North Star. It's worth remembering that the size of these lists is largely a myth. And as I've said before: if the waitlist is so long, why doesn't North Star backfill? And why spend money on advertising and marketing if you can't even enroll everyone who allegedly wants your product?

Waitlists are marketing tools. When you can claim a large waitlist, you can make families believe your school is special. And if schooling is, to at least some extent, a positional good, a waitlist sends the message that a school is offering something lots of people want, but only some people get. That's a big part of the appeal of charter schools -- don't let anyone tell you otherwise.

* * *

I've said it so many times I've run out of ways to say it: I'm not against charter schools per se. But it's well past time to start being honest about why certain charters get the gains that they do.

Noble students are somewhat more likely to go to college and persist.**** That's great -- but Noble has some other features that are important to consider before we declare it a model fit for replication. Its cohorts, for example, shed large numbers of students -- including those in Davis & Heller's study:

Noble has also engaged in some questionable disciplinary practices (to their credit, Davis and Heller note this), some questionable recruitment practices, and some questionable student retention practices. The leaders of Noble appear to believe they cannot maintain their schools with a unionized faculty. Why? What else is going on at Noble that has nothing to do with "frequent teacher feedback, data driven instruction, high-dosage tutoring, increased instructional time, and a relentless focus on academic achievement"?

Yet another charter lottery study isn't going to answer this critical question. We need to get past facile explanations of the "success" of a small number of charter schools and really start digging into why they get the (largely marginal) gains that they get. And if we are to believe the sources cited by research showing positive charter school effects, one of the factors leading to charter gains is student self-selection.

Easy to say; hard to define...

ADDING: I have to note that No Excuses has perhaps the most hacktastic chapter ever written on school funding. An excerpt:

Consider the recent historical record. Per pupil spending on America's elementary and secondary schools (in constant dollars) nearly doubled between 1970 and 2000. Our educational system is not performing twice as well as a result. In fact, on the basis of the evidence NAEP has collected about what American students know, it seems safe to say that the nation's schools, overall, are no better than they were three decades ago. Black students have come a considerable distance since 1970, but the rise of a large black middle class and the end of segregated schools in the South may account entirely for that change. (p. 153)

Hey, Bruce, got anything to say about this?


* We also don't know how charters are able to offer more instruction and individual tutoring. Are they collecting more resources through philanthropy? Are they free-riding on public school salaries? If so, it's unlikely they can continue to use those strategies if the sector expands.

** That quote is followed by an endnote citing Samuel Casey Carter's No Excuses, a 2000 book published by the right-wing Heritage Foundation. It's essentially a catalog of 21 schools chosen ostensibly because at least three-quarters of their students qualify for free lunch yet score above the 65th percentile on a national standardized test. But no sampling methodologies are documented, no qualitative validity methods are described, and the tests are not consistent across all schools.

Again: this is not a source that should be cited in a serious, peer-reviewed journal article.

*** First: I'm going to skip a discussion of whether this is a practically significant gain or not. I'll only suggest this may be a point open for debate.

Second: yes, Davis and Heller do, in the last paragraph of the the paper, acknowledge that the exact mechanics of how "no excuses" style teaching leads to gains is still unknown. But the idea that "no excuses" might be student self-selection, or peer effects, or increased resources, or any number of other factors isn't a part of their argument:

Viewed in concert with work by Dobbie and Fryer (2015) and Angrist et al. (2016), we see an increasingly clear picture of a rapidly growing style of school management that boosts the college enrollment of poor urban high schoolers. Unfortunately, the similarity of the educational approaches considered in each study means that we can say very little about what mechanisms might be driving our results (at most, one might infer that the neighborhood-level interventions of the Harlem Children's Zone are not essential for increasing college enrollment, a conclusion shared by Dobbie and Fryer (2015)). Accordingly, understanding which "No Excuses" strategies are most important for generating long-term gains is an important topic for future research. Nevertheless, we are confident that this educational model has proved to be – and will continue to be – an effective part of the fight to increase low-income students' human capital.

They are "confident" that "no excuses" works, even if they don't know how it works. But what if "no excuses" isn't anything that Dobbie & Fryer lay out? What if it's something else, like self-selection & peer effects? Shouldn't we address this possibility before suggesting things like "a relentless focus on academic achievement" -- whatever that is -- makes all the difference?

**** It's worth noting that there are several large assumptions Davis and Heller make in their study. They use last names as a proxy for race, they've got a spotty dataset for tracking college attendance, and they make assumptions about where the control students attended school.

That's not to say this is a bad study -- it's not. But it is, like all social science research, limited. And before we jump to conclusions and use research like this to guide policy, we should stop and carefully consider its limitations.