CREDO Charter School Studies' "X Days Of Learning" Conversion: Still Unvalidated

This past week, the Center for Research on Education Outcomes (CREDO) at Stanford University released yet another report in a series on the effects of charter schools on test scores -- this time focusing on Texas.

Almost immediately, members of the the local media trumpeted the results as "proof" that charter schools are realizing meaningful gains in student outcomes:

For the first time, Texas charter schools have outperformed traditional public schools in reading and closed the gap in math, researchers at Stanford University have found.

Students at Texas charter schools, on average, received the equivalent of 17 additional days of learning per year in reading and virtually the same level of education in math when compared to traditional public schools, according to a study released Wednesday by the Center for Research on Education Outcomes, or CREDO.

Rather than looking at raw standardized test scores, CREDO researchers quantify the impact of a school by looking at student improvement on the tests relative to other schools. The researchers then translate those results into an equivalent number of "days of learning" gained or lost in a 180-day school year.

The center's staff produced similar analyses in 2013 and 2015, finding Texas charter schools had a negative impact on reading and math performance.

"The most recent results are positive in two ways. Not only do they show a positive shift over time, but the values themselves are both positive for the first time," the researchers wrote.

CREDO's studies of charter school performance are widely respected in education circles. The center compares students from charter and traditional public schools by matching them based on demographic characters -- race, economic status, geography and English proficiency, among others -- and comparing their growth on standardized tests. Scores from 2011-12 to 2014-15 were analyzed for the most recent report. [emphasis mine]

That's from the Houston Chronicle, which published just one paragraph suggesting the CREDO studies might have credible critics:

Skeptics of CREDO's study typically offer three main criticisms of the research: it focuses exclusively on standardized test results, incentivizing schools that "teach to the test"; it ignores other advantages of traditional public schools, such as better access to extracurricular activities; and it doesn't account for the fact that charter school students are more likely to have strong, positive parental influence on their education.

Sorry, but that's, at best, an incomplete description of the serious limitations of these studies, which include:

• The matching variables that create the counterfactuals are far too crude to do the job properly.
• The definition of the treatment -- enrolling in a charter school -- does not account for factors such as increased spending, peer effects, and other advantages which have nothing to do with "charteriness."
• The consistently small effect sizes have been pumped up by an unvalidated conversion into "days of learning" which has never been properly justified by the authors.

Here is how the CREDO Texas study reports its findings:

Stanley Pogrow published a paper earlier this year that didn't get much attention, and that's too bad. Because he quite rightly points out that it's much more credible to describe results like the ones reported here as "small" than as substantial. 0.03 standard deviations is tiny: plug it in here and you'll see it translates into moving from the 50th to the 51st percentile (the most generous possible interpretation when converting to percentiles).

I have been working on something more formal than a blog post to delve into this issue. I've decided to publish an excerpt now because, frankly, I am tired of seeing "days of learning" conversions reported in the press and in research -- both peer-reviewed and not -- as if there was no debate about their validity. 

The fact is that many people who know what they are talking about have a problem with how CREDO and others use "days of learning," and it's well past time that the researchers who make this conversion justify it. 

The excerpt below refers to what the eminent psychometrician Michael T. Kane coined a "validity argument." To quote Kane: "Public claims require public justification." I sincerely hope I can spark a meaningful conversation here and get the CREDO team to adequately and publicly justify their use of "days of learning." As of now, their validity argument is cursory at best -- and that's just not good enough.

I have added some bolding to the excerpt below to highlight key points.

* * *

Avoiding the Validity Argument: A Case Study

As an illustration of the problem of avoiding the validity argument in education policy, I turn to an ongoing series of influential studies of charter school effects. Produced by The Center for Research on Education Outcomes at Stanford University, the so-called CREDO reports have driven a great deal of discussion about the efficacy of charter school proliferation. The studies have been cited often in the media, where the effects they find are reported as “days of learning.”[1] Both the National Charter School Study (Raymond et al., 2013) and the Urban Charter School Study Report on 41 Regions (CREDO, 2015) include tables that translate the effect sizes found in the study into “days of learning.” Since an effect size of 0.25 SD is translated into 180 days, the clear implication is that an effect of this size moves a student ahead a grade level (a typical school year being 180 days long). Yet neither study explains the rationale behind the tables; instead, they cite two different sources, each authored by economist Eric Hanushek, as the source for the translations.

The 2015 study (p. 5) cites a paper published in Education Next (Hanushek, Peterson & Woessmann, 2012) that asserts: “On most measures of student performance, student growth is typically about 1 full std. dev. on standardized tests between 4th and 8th grade, or about 25 percent of a std. dev. from one grade to the next.” (p. 3-4) No citation, however, is given to back up this claim: it is simply stated as a received truth. 

The 2013 study (p. 13) cites a chapter by Hanushek in the Handbook of the Economics of Education (Hanushek & Rivkin, 2006), in which the author cites his own earlier work:

“Hanushek (1992) shows that teachers near the top of the quality distribution can get an entire year’s worth of additional learning out of their students compared to those near the bottom. That is, a good teacher will get a gain of 1.5 grade level equivalents while a bad teacher will get 0.5 year for a single academic year.” (p. 1068)

No other references are made within the chapter as to how student gains could be presented as years or fractions of a year’s worth of learning.

The 1992 citation is to an investigation by Hanushek of the correlation between birth order and student achievement, and between family size and student achievement. The test scores used to measure achievement come from the “Iowa Reading Comprehension and Vocabulary Tests.” (p. 88) The Iowa Assessments: Forms E and F Research and Development Guide (2015), which traces the development of the Iowa Assessments back to the 1970s, states:

“To describe the developmental continuum or learning progression in a particular achievement domain, students in several different grade levels must answer the same questions in that domain. Because of the range of item difficulty in the scaling tests, special Directions for Administration were prepared to explain to students that they would be answering some very easy questions and other very difficult questions.” (p. 55-56)

In other words: to have test scores reported in a way that allows for comparisons across grade levels (or, by extension, fractions of a grade level), the Iowa Assessments deliberately place the same questions across those grade levels. There is no indication, however, that all, or any, of the statewide tests used in the CREDO studies have this property.[2]

Harris (2007) describes the process of creating a common score scale for different levels of an assessment as vertical scaling. She notes: “Different decisions can lead to different vertical scales, which in turn can lead to different reported scores and different decisions.” (p. 233) In her discussion of data collection, Harris emphasizes that several designs can be used to facilitate a vertical scale, such as a scaling test, common items, or single group to scaling test. (p. 241) 

In all of these methods, however, there must be some form of overlapping: at least some students in concurrent grades must have at least some common items on their tests. And yet students in different grades still take tests that differ in form and content; Patz (2007) describes the process of merging their results into a common scale as linking (p. 6). He notes, however, that there is a price to be paid for linking: “In particular, since vertical links provide for weaker comparability than equating, the strength of the validity of interpretations that rest on the vertical links between test forms is weaker.” (p. 16)

So even if the CREDO studies used assessments that were vertically scaled, the authors would have to acknowledge that the validity of their effect sizes was at least somewhat compromised compared to effect sizes derived from other assessments. In this case, however, the point is moot: it appears that many of the assessments used by CREDO are not vertically scaled[3], which is a minimal requirement for making the case that effect sizes can be translated into fractions of a year’s worth of learning. The authors are, therefore, presenting their results in a metric that has not been validated and could be misleading.

I use this small but important example to illustrate a larger point: when influential education policy research neglects to validate the use of assessments, it may lead stakeholders to conclusions that cannot be justified. In the case of the CREDO reports, avoiding a validity argument for presenting effect sizes in “days of learning” has led to media reports on the effects of charter schools and policy decisions regarding charter proliferation that are based on conclusions that have not been validated. That is not to say these decisions are necessarily harmful; rather, that they are based on a reporting of the effects of charter schools that avoided having to make an argument for the validity of using test scores.

---

[1] Some example include:
http://www.chicagotribune.com/news/opinion/commentary/ct-charter-schools-public-teachers-unions-20161015-story.html
http://www.economist.com/news/international/21705697-liberating-schools-run-their-own-affairs-produces-some-great-ones-also-plenty
https://www.usnews.com/opinion/knowledge-bank/2015/03/19/new-study-shows-charter-schools-making-a-difference-in-cities

[2] Nor is there any indication the national and international tests Hanushek cites in his 2006 paper, such as the National Assessment of Educational Progress, share questions across grade levels. In fact, Patz (2007) notes: “NAEP attempted to vertically link forms for grades 4, 8, and 12, but abandoned the approach as the comparisons of students separated by 4 or 8 years were found to be ‘largely meaningless’ (Haertel, 1991).” (p.17)

[3] Some statewide assessments are vertically scaled, including the Smarter Balanced assessments; see: https://portal.smarterbalanced.org/library/en/2014-15-technical-report.pdf

References

Center for Research on Education Outcomes (CREDO) (2015). Urban Charter School Study Report on 41 Regions. Palo Alto, CA: Center for Research on Education Outcomes (CREDO), Stanford University. Retrieved from: http://urbancharters.stanford.edu/summary.php

Dunbar, S.& Welch, C. (2015). Iowa Assessments: Forms E and F Research and Development Guide. Iowa City, IA: University of Iowa. Retrieved from: https://itp.education.uiowa.edu/ia/documents/Research-Guide-Form-E-F.pdf

Hanushek, E. A. (1992). The trade-off between child quantity and quality. Journal of political economy, 100(1), 84-117.

Hanushek, E. A., & Rivkin, S. G. (2006). Teacher quality. Handbook of the Economics of Education, 2, 1051-1078.

Hanushek, E. A., Peterson, P. E., & Woessmann, L. (2012). Achievement Growth: International and US State Trends in Student Performance. PEPG Report No.: 12-03. Program on Education Policy and Governance, Harvard University.

ITBS Research Guide https://itp.education.uiowa.edu/ia/ITBSResearchGuide.aspx

Harris, D. J. (2007). Practical issues in vertical scaling. In Linking and aligning scores and scales (233-251). New York: Springer.

Kane, M. (2013). Validating the interpretations and uses of test scores. Journal of Educational Measurement 50(1), 1–73.

Raymond, M. E., Woodworth, J. L., Cremata, E., Davis, D., Dickey, K., Lawyer, K., & Negassi, Y. (2013). National Charter School Study 2013. Palo Alto, CA: Center for Research on Education Outcomes (CREDO), Stanford University. Retrieved from: http://credo.stanford.edu/research-reports.html

NJ Charter School Follies: Asbury Park & Paterson Edition

Given the last presidential election, I'm reluctant to make predictions about the outcome of the 2017 New Jersey governor's race. But it's looking increasingly tough for Kim Guadagno -- which means we'll have new leadership next year at the NJ Department of Education.

Among the consequences is an almost certain change in how charter schools are approved and renewed. Although Murphy's stance on charters is "complicated," he is clearly far more skeptical than Chris Christie. And now that Shiela Oliver has joined the ticket, it's even more likely charter schools will have to clear a much higher bar when seeking approval from the state.

Which means that for the next few months, the state's charter sector is getting while the getting is good:

When Gov. Chris Christie leaves office in six months, one of his clear legacies will be the growth of charter schools in New Jersey, with school enrollment more than doubling in his eight years in office. 

Yesterday, his administration finished the job, announcing the final approval of five more schools to open this fall. That brings to 89 the number of charters that will be open when Christie steps down in January. 

That number isn’t that big an increase from the 70 in place in 2010 at the start of Christie’s tenure, a number that jumped to over 90 in his first year. But his administration ultimately closed nearly 20 charter schools as well. 

Nevertheless, there will be close to 50,000 students enrolled in charters this fall, according to the state, up from less than 25,000 when he took office. More than 56,000 seats will be authorized with the latest approvals.

So, who's getting the nod in this latest round of charter growth?

The five new schools announced yesterday to open in the fall were: 

• Achieve Community Charter School, Newark, opening as K-1, growing to K-4.
• College Achieve Greater Asbury Park Charter School, Asbury Park/Neptune, opening as K-1 and grades 5-6, and growing to K-9.
• College Achieve Paterson Charter School, Paterson, opening as K-1 and grades 5-6, and growing to K-9.
• LEAD Charter School, Newark, opening and staying as grades 9-12.
• Ocean Academy Charter School, opening as K-2, and growing to K-5.

Three "Achieve" charter schools? Not quite: Achieve Community is aligned with the team behind Brick/Avon Academy in Newark (one of the main subjects of Dale Russakoff's The Prize). But the two College Achieve charters are both being founded by a fellow named Michael Piscal.

And that brings us to an interesting story that sheds a bright light on the New Jersey charter school approval process, and why it really needs to change...

Because Piscal already has a charter school in New Jersey: College Achieve Central Charter School in Plainfield. Fiscal founded the school after a career in school management in California -- a career Darcie Cimarusti, aka Mother Crusader, detailed in 2014:

In the very same month Pallas wrote this piece it became very clear that not only were ICEF schools not the miracles they were portrayed to be, there were very real problems at ICEF that threatened their very existence. The downward spiral was chronicled by Los Angeles Times writer Howard Blume. 

A group of the city's leading philanthropists, including billionaire Eli Broad and former mayor Richard Riordan, rallied Monday to save ICEF Public Schools, one of the nation's largest and most successful charter school companies, which was teetering on financial insolvency.

ICEF, which operates 15 schools in low-income minority neighborhoods of Los Angeles, was virtually out of cash, unlikely to meet its Oct. 1 payroll. The nonprofit faced a $2-million deficit in the current budget year as well as substantial long-term debt.

The collapse of ICEF would have been a blow to the charter movement and to the 4,500 students and several hundred employees of an organization whose results have impressed many observers. Charters are independently run public schools that are free from many regulations that govern traditional schools.

ICEF representatives and others said the group's budget problems were caused by insufficient reserves; an overly ambitious expansion — 11 new schools in three years — that resulted in costly debt; and a reluctance to make cuts affecting students. These factors were exacerbated by the recession, which sharply reduced state funding to schools, and this year's late state budget, which has delayed payments to schools.

The rescue plan that emerged Monday was less disruptive than one under discussion as recently as Sunday. That plan would have broken up ICEF, distributed schools and students among other charter schools and forced out founder Mike Piscal.

Instead, Piscal will remain to oversee academic programs.(emphasis mine)

Well, so much for Piscal's theory that ICEF was infallible. 

Read the whole thing, as they say. The point here is that, according to Cimarusti's research, Piscal's previous charter network had expanded too quickly, putting it in a financially untenable position. That's why David Rutherford, a member of the Plainfield Board of Education, argued back in 2015 that giving a charter approval to someone with Piscal's history was a real risk:

As reported in the Los Angeles Times, ICEF leaders took pride in their “secret sauce”, a balanced program of “academics and extracurricular activities.” According to some parents, the more troubling secrets at the ICEF were the day to day behavior of its enigmatic leadership.

In the words of former Los Angeles United School District Board of Education President Caprise Young, ICEF exhibited “substantial bad judgement and lack of transparency with [its] finances,” which lead Piscal’s organization to a financial near-death experience in 2010. This is all despite cutting corners.

Apparently, employees complained of supply shortages, and were worked long hours with no additional compensation. It’s no suprise that teacher turnover ranged from 10 to 50 percent each year in ICEF’s schools, which served 4,500 students.

An even larger corporate charter school outfit, Alliance, wished to purchase the struggling company, but the sale fell though. It was ultimately new investors and uber-rich donors sympathetic to the charter school movement that prevented the total dissolution of the Inner City Educational Foundation.

As for ICEF’s founder, Mike Pascal, he had to go. According to the Los Angeles Times, Piscal resigned amidst the controversy in October of 2010. Los Angeles social justice writer and education advocate Robert D. Skeels, who followed the situation closely, asked in early 2011 “Where’s the money, Mike?” concluding that even if there was no criminal malfeasance, Piscal would have committed “the most unconscionably incompetent accounting of all time”. [emphasis mine]

Let me add a little more context with the benefit of hindsight:

In 2013, Camden Community Charter School opened under the aegis of the NJDOE, even though the school's founders were, at the time, under investigation by the Pennsylvania State Auditor General. You might have thought NJDOE would show some restraint and wait until the investigation was closed before allowing CCCS to open its doors; alas, no. Guess what?

The NJDOE ordered it closed in the spring of 2017. Sue Altman has all the gory details, noting that NJDOE was not solely to blame, as backing CCCS was a bipartisan effort. Still, NJDOE is supposed to be the backstop; it's supposed to look carefully at the charter applications and determine, based on whatever evidence may exist of the founders' pasts, the chances of that charter succeeding.

Which begs the question: did NJDOE do proper due diligence on the new College Achieve charter approvals? Specifically, did the department look carefully at CACCS's record in Plainfield -- short though it may be -- and make an informed decision about how the expansion of the charter chain will affect Paterson, Asbury Park, and Neptune, its sending districts?

I'll bet you know the answer. And I'll bet you know by now I just can't resist a chance to take a data dive:

We'll start with the percentage of free-lunch eligible students in the charter and its hosting district. Plainfield has a large proportion of student in economic disadvantage. But CACCS's free lunch percentage actually decreased over the last two years.

Plainfield also has a large number of Limited English Proficient students (due to its large population of Spanish-speaking residents). CACCS's LEP percentage did go up last year, but, like almost every other charter in the state, it's still far behind its hosting district schools.

That goes for special education students as well: Plainfield City Public Schools has more than three times the proportion of classified students as College Achieve Central Charter School. Again, this is quite typical for the state's charter sector.

CACCS has a track record of serving a fundamentally different student population than the Plainfield public schools. Did NJDOE take this into account when approving their expansion into the other towns?

State statute requires: "The admission policy of the charter school shall, to the maximum extent practicable, seek the enrollment of a cross section of the community's school age population including racial and academic factors." The outcomes suggest CACCS has some work to do on this front. Did NJDOE require a plan from the applicants to address this issue, either in Plainfield or in the new locations?

And did NJDOE take into account the fiscal impact on Asbury Park, Neptune, and Paterson from the new charters? Let's look at the spending of CACCS compared to the PCPS:

Again, CACCS follows a pattern I've documented across the state: lower overall budgetary per pupil costs, but driven by less spending "in the classroom." I don't have good staff data for CACCS, but if they're like almost all other charters in the state, their teachers are likely less experienced and, therefore, less expensive than PCPS teachers. This creates a "free riding" problem for the area public schools, where CACCS gets the benefits of PCPS's higher salaries for more experienced teachers.

Another big factor here is how much less CACCS spends per pupil on support services: the services which largely benefit special needs students, like child study teams, nurses, physical and metal health services, etc. Of course, CACCS doesn't need to spend as much as PCPS, because their student population has a much smaller proportion of students who need those services.

Again: did NJDOE take this into account when giving the OK to College Achieve in their new districts? How does NJDOE expect Asbury Park's, Neptune's, and Paterson's public schools -- and, for that matter, Plainfield's -- to absorb the cost of concentrating more special needs students into their districts?

Notice also that CACCS spends much more per pupil than PCPS on administrative costs. Let's zoom in on that:

CACCS spends less than PCPS on instruction and support, but much more on administrative costs, including salaries. As Bruce Baker spells out, charters are generally too small to reach an optimal size for fiscal efficiency. High administrative spending isn't necessarily indicative of anything nefarious -- it's just a logical consequence of establishing inefficiently small schools that essentially operate as their own school districts.

Again, NJDOE had a clear example from CACCS that they can't, as of yet, lower their administrative costs to stay in line with the local public school district, even with a smaller proportion of costlier special needs students. Maybe CACCS will get better -- but why not, then, wait until they prove they can operate more efficiently? Why not wait until they've established their track record in Plainfield before allowing them to expand?

Especially when their outcomes aren't really that much better?

Here are the PARCC mean scale scores in English Language Arts from 2016 for Plainfield City Public Schools and CACCS. In Grade 5, CACCS is quite average: higher scores than some schools, lower than others. They do somewhat better in Grade 6, but only beat PCPS's lowest scoring school by five points on a test whose score ranges from 650 to 850.

Grade 5 math isn't much different. Grade 6 is better -- but you have to remember that CACCS has a different student population than Plainfield's public district schools. When you have fewer special needs, LEP, and at-risk students, it's easier to get higher test scores.

Even NJDOE acknowledges this, which is why they calculate Student Growth Percentiles (SGPs) for every school enrolling students in grades 3 through 8. There are significant problems with SGPs (I know I keep promising I'll have more to say about this -- later this summer, I promise...), but NJDOE has said themselves they are necessary because schools serving higher numbers of lower-performing students can still show "growth," and that should be acknowledged.

So how are CACCS's growth scores?

By any normal standard, CACCS is quite average in its ELA growth scores compared to the PCPS -- some might say even a little below average, although given the statistical noise in these measures I wouldn't go that far.

Again, math growth is not superior compared to PCPS; it's quite typical.

I've used my own "value-added" model before when evaluating charter school performance*, which adjusts mean scale scores based on demographic comparisons across the state. Here are the adjusted outcomes:

You can dispute my model, and that's fine: I would simply point out that it shows -- just like the unadjusted test scores and the state's own growth measures -- that on these four test score measures, CACCS is hardly superior to Plainfield's public district schools.

I am not one to ever make the case that a school's effectiveness should be solely measured by its test score outcomes, no matter how we calculate them. But if NJDOE is granting an expansion for College Achieve into new school districts, we ought to ask: why? What are they seeing that the data does not show? Because the data is clear:

Compared to the Plainfield Public Schools, College Achieve Central Charter School enrolls proportionally fewer free lunch-eligible, Limited English Proficient, and special education students; spends much less on instruction and support and much more on administration; and does not get test score outcomes that are superior.

So, based on the record in Plainfield, what benefit is there to Paterson's, Asbury Park's, or Neptune's families in allowing College Achieve to set up shop in these communities? More importantly: is that benefit worth the cost to these communities?

Board administrator Geoffrey Hastings said a total of $3,266,750 was cut from the Asbury Park School District budget and redirected to the College Achieve. This resulted in layoffs and the need to eliminate jobs. Some of the staff were rehired through attrition but others turned in their resignations and retirement notices, Hastings said.

Among them was Brian Stokes, who headed the College and Career Readiness Institute, known for transforming students’ focus on their future by administering workshops, panels with nationally recognized professionals and paid internships with city and area businesses. [emphasis mine]

Will College Achieve pick up the slack here? Will it be able, with its high administrative costs, to establish a similar institute, and provide all the other programming Asbury Park provides to all of the district's students? And do the same for Neptune and Paterson?

Again: NJDOE has approved charter schools with extensive out-of-state records that would suggest caution at the least. In the case of Camden Community Charter School, it turns out they should have looked at that record more carefully. Now comes College Achievement, looking to expand rapidly across the state.

Maybe Piscal was able to convince charter authorizers that his dubious past in Los Angeles was behind him, and that he had a solid plan for success in Plainfield. OK... But why, then, rush to allow expansion to other communities when we're just starting to get a picture of how College Achieve's first campus is faring? Why allow College Achieve to expand when it's student populations are different, when it's administrative expenses are high, and when it's outcomes aren't at all superior?

I have no doubt that CACCS has many dedicated educators committed to their students. I am sure the school is full of bright, hard-working students with supportive parents who should be proud of their hard work and achievements. But when a charter school comes into a community, it affects all students. NJDOE has an obligation to look carefully at what those effects may be.

There was no need to rush through College Achieve's expansion. Let's see what they can do in Plainfield first, and then let the citizens of Asbury Park, Neptune, and Paterson decide, openly and democratically, whether any potential benefits from this charter chain's proliferation is worth the costs.

h/t Rob Tornoe

ADDING: One additional point, based on a rather fawning article from mycentraljersey.com:

On Monday, a year after opening its first school in Plainfield, CAPS is opening its latest venture — the College Achieve Central Charter School (CACCS). The first charter school for the borough, and only the third in Somerset County, CACCS, will educate students with an instructional program that emphasizes writing and history as well as a challenging science, technology, engineering, math and arts (STEAM) curriculum. 

[...]  

The new facility at 107 Westervelt Ave. is being financed and developed by Building Hope, a nonprofit organization that has provided more than $200 million in facility funds to academically successful schools, so they can expand and grow their enrollments. College Achieve is the first facility in New Jersey to be developed by Building Hope. The school will lease the campus from Building Hope with plans to eventually purchase it. [emphasis mine]

Long time readers may remember my series of pieces, back from 2013, about Building Hope. The outfit basically gathers government and private foundation money (The Walton family is a big contributor), then gives out loans for charter construction at below-market rates. It's a good deal for the foundations because the loans are paid back, and the foundations get to count "imputed interest" as charitable giving without seeing their coffers actually diminish.

There are at least two points to be made here. First: the taxpayers of Plainfield, North Plainfield, and New Jersey overall are paying money to College Achieve, which is being used to purchase a building the taxpayers do not own. State tax records show Building Hope owns the building -- not the state, and not the local school district.

When College Achieve says they have "plans to eventually purchase it," what do they mean? That it will be owned by a private entity? A nonprofit or a for-profit? A group or an individual? Who determines the rental rates College Achieve currently pays? What will happen to those funds if they buy the building and the debt is retired? Will someone be paid to administer the property if it's sold? Who? How much?

As Bruce Baker and Gary Miron document in great detail, our current chartering system is set up so that taxpayer funds are used to purchase assets the taxpayers themselves do not own. This makes no sense.

I hope Piscal's New Jersey charters don't suffer the fate his Los Angeles charters did, because I very much doubt there will be a couple of white knights around this time to ride in and rescue them. But if they do, who will own the assets New Jersey taxpayers have paid for?

Second: time and again I hear from our state's charter cheerleaders that the sector is disadvantaged because it has to raise funds to finance its own facilities. Can we please stop with that?

Building Hope operates with a great deal of governmental largesse, and the foundations it relies on exist only because the tax code allows it. Further, over and over and over and over and over and over again we find that charter schools have access to all sorts of capital and financing that public district schools do not. Again, much of this financing is made available because the tax code provides incentives; by any standard, that is publicly-subsidized financing.


* One important point about these adjusted scores compared to models I've used before: I don't have school-level special education data for CACCS that is comparable to PCPS data. What I present above is district level data, which isn't what's needed for this model.

That said, I did take a look at the school-level special eduction rates for the three middle schools in PCPS for 2013-14. They are high: between 14 and 22 percent. I have little doubt the "value-added" would grow considerably for the PCPS middle schools if I were able to include this as a covariate.

As always: caveat regressor.

When "Miracle" Charter Schools Shed Students

A follow-up to yesterday's post:

As I noted, NBC's Sunday Night with Megan Kelly broadcast a story earlier this month about Boys Latin Charter School, a "successful" charter school in Philadelphia which claims to have ten times the college completion rate of its neighboring high schools.

To his credit, reporter Craig Melvin didn't swallow the claims of the school whole, and pushed back on the idea that Boys Latin serves an equivalent student population to those surrounding high schools. But he did miss two important points:

First, and as I documented in the last post, Boys Latin raises funds outside of the monies it collects from public sources. The amounts add up to thousands of dollars per pupil per year.

As Bruce Baker notes in this (somewhat snarky) post, you really can't make a comparison between two schools and call one "successful" without taking into account the differences in resources available to both. Philadelphia's public school district has been chronically underfunded for years. It's hardly fair for Boys Latin to collect millions in extra revenue, then brag about their college persistence rate compared to schools that don't have enough funding to provide an adequate education.

But there's another issue Melvin missed -- an issue that Boys Latin's founder, David Hardy, has been refreshingly candid about in the past:

Hypothetically speaking, say a charter school is authorized to serve up to 500 students, but, for whatever reason, 50 students leave through the course of a school year. A charter that "backfills" will enroll the next 50 kids on its wait list as space becomes available.

Other schools will replace those empty spots at the beginning of the next school year, including filling seats in the upper grades.

Charters that don't do this will watch their total enrollment in a grade dwindle year by year — retaining only the students tenacious enough to persist.

In contrast, district-run neighborhood schools and renaissance charters must enroll all students living within a prescribed catchment zone, no matter what time of year or grade, when they show up asking for a seat.

At first glance this difference may seem a subtle nuance, but Philadelphia educators say the policy difference tremendously affects school culture and performance. 

[...]  

David Hardy, CEO of Boys' Latin, subscribes to the same theory. He oversees a rigorous admissions process that begins well before the school year.

Boys' Latin asks prospective ninth-graders to submit letters of intent in November, nearly a year before they would enroll. Staff then interview students and parents to ensure that they understand the school's rigor -- classes run until 5 p.m., students must learn Latin, wear a uniform, and adhere to a strict code of conduct.

Those who commit attend a month-long freshman academy in July before the school-year-proper begins.

By September, he said, the kids are all on the same page.

"You introduce new people into that, and it can kind of mess up the environment," said Hardy. [emphasis mine]

This is an issue that comes up over and over again in charter school research: student cohort attrition. As a cohort of students (Class of "x") moves from freshman to sophomore to junior to senior year, it may lose students. Sometimes students drop out; sometimes they move. If a charter school "backfills," they then replace the students who left with new students who come into the school in later years.

Many charters have high student cohort attrition rates, meaning students leave the school before graduation -- often returning to the public, district schools, which must take them no matter when they arrive at the schoolhouse door. These same charters don't backfill, so their cohort sizes shrink as they move toward their senior years.

What are the patterns of cohort attrition for Boys Latin?

This is based on federal data as reported in our School Funding Fairness Data System. Each cohort or "class" at Boys Latin is tracked as it moves from freshman to senior year. Every year cohort*, Boys Latin loses at least one-third of its students, and never replaces them.

Again: Hardy is completely upfront about this:

So why do kids leave?

"Normally it's because a student doesn't want to take the volume of the work that we do," said Hardy. And when that happens, "there's a mechanism to kind of go after them."

At Boys' Latin, this process includes tutoring, probation, and Saturday school.

"If the student makes the decision that they really don't want to do the work, then what's the point?" Hardy said. "That normally is a family decision. They say, 'Why keep him in a school that he really doesn't want to be in?' That's the beautiful thing about school choice."

Hardy says his school doesn't actively push kids away.

"You can stay here and fail," he said. "But that doesn't make sense, does it?"

OK -- Why, then, would you ever make a comparison in your college completion rates to neighboring high schools?

I'll give Hardy this: he's at least more intellectually consistent than Steve Perry, who can't get his story about "ham shaving" straight. But can we at least acknowledge a few things?

- It's a lot easier to have high college acceptance and persistence rates when you don't have to educate everyone who shows up on your front steps.

- It's a lot easier to have high college acceptance and persistence rates when kids who can't cut it in your program decide it's a "beautiful thing" to leave.

- Why does a school like Boys Latin, which skims the cream, get all sorts of extra funding while the Philadelphia public schools, which must educate everyone, remain chronically underfunded?

I have no doubt the staff of Boys Latin is full of dedicated, caring professionals who are working hard every day to give their students a great education. But I see very little in their model that could be scaled up.

Do you?

h/t Rob Tornoe

* Stephen Ronan, commenting over at Diane Ravitch's blog, correctly points out my phrasing here was misleading: every cohort loses at least a third of its students from freshman to senior year, but they don't lose a third every year. My apologies.

School "Choice": A Failure of Honesty and Will

It's starting to feel like the Summer of School Choosiness.

Here's Nick Kristof, making the case that privatizing Liberia's schools is the only sensible way to ensure they are stocked with books.

Here's Gayle Tzemach Lemmon, telling her personal story of how her mother snuck her into a "better" school, which widened the "narrow funnel that leads to the nation’s elite institutions" for her.

Here's a story on Megan Kelly's new show (which, like the rest of America, you almost certainly missed) about a "successful" charter school in Philadelphia. Give reporter Craig Melvin credit: he does push back on the platitudes that are standard fare for such reports, even if he misses one of the biggest parts of the story (hang on...).

And of course we have the steady stream of laudatory essays about charters and school vouchers from outlets like The 74 and Education Post. Similar to the pieces above, they almost all revolve around stories of how students are being saved from "bad" schools because they are being given a "choice" to attend "good" schools.

The clever thing about this construction is that anyone who challenges the narrative is immediately put on the defensive: Why are you against helping people get a better education? Why don't you care about these children? It must be that you care about your own interests more than theirs...

And so on. It's a neat rhetorical trick, but it keeps us from asking and answering the questions that really matter. For example:

What, exactly, makes a "better" school? Lemmon says her mother risked arrest to get Lemmon into "... what she judged to be the best public elementary school in our area." OK -- on what basis did her mother make that judgment? Test scores? We know these correlate tightly to family socio-economic status. Peers? That's obviously problematic: peer effects matter, but you can't have everyone attend a school where their peers are above average. Resources? It's easier for a wealthy community to tax itself to provide more resources for its schools. Was Lemmon's new school more economically advantaged?

The easy answer we often get about what makes "good" schools from the "choice" industry is that charter and voucher schools are free from bureaucracy, which allows for innovation. But there's very little evidence these schools are engaging in practices that are truly "innovative."

It's becoming increasingly clear that those "no excuses" charters that get some gains -- and, understand, many of them do not -- do so because of longer school days and years, resource advantages, peer effects, "free riding" on public school teacher wages, student attrition (often exacerbated by extreme disciplinary policies), and a focus on test outcomes.

It's perfectly fine to argue in favor of these things; I happen to be in favor of some of them (resource advantages) and against others (student attrition when prompted by extreme discipline, overly focusing on test prep). Some are logically impossible to bring to scale (peer effects, "free riding"). But even if we agree on the benefits of some of these advantages, it's a leap to claim that "choice" is the only way to deliver them. Which leads to...

How, exactly, is "choice" creating "better" schools? Kristof claims that Bridge International Academies, the for-profit company delivering schooling in Liberia and elsewhere, hasn't yet turned a profit, so fears about monetizing children are "misplaced." So why, then, does BIA have to be structured as a for-profit company? 

"In fact, it’s a start-up that tackles a social problem in ways similar to a nonprofit, but with for-profit status that makes it more sustainable and scalable."

Sorry, but that answer is far too facile. Why would a for-profit be more sustainable? What about BIA's for-profit model makes their schools more "scalable"? Could it be related to this?

How do you improve education, make it cheaper and also make it profitable? May and Kimmelman have come up with an “innovative pedagogical approach.” The possibility of setting up a few thousand standardized schools within a few years is to be the first innovation. The profit made from each school may be low, but once half a million pupils are recruited — the number of enrollments that Bridge needs to break even — business really takes off. The plan is to reach two million pupils by 2018 and 10 million by 2025.

This rapid growth would be made possible by using Bridge’s second innovative method, namely its very own approach to the role of teachers and their salary scale. May believes that “qualities such as kindness” are more important than diplomas and this allows for significant savings. In Kenya, where the starting salary for qualified teachers is around US$116 dollars a month, Bridge teachers usually earn less than US$100 a month. However, as Kimmelman explains in a presentation, teachers can earn bonuses by recruiting new students themselves. Marketing is a core task for both teachers and school principals. [emphasis mine]

This is disturbingly similar to the charter/voucher "advantage" in the United States: teachers earning less while forced into marketing roles for their employers, all while being touted as superior in some unspecified way to teachers who bargain collectively for better wages within public district schools.

Gary Rubinstein, a Teach For America alumnus, has often described TFA's construction of the myth of the "miracle" teacher: that some select schools (and, by extensions, their leaders) have figured out the "combination to the lock" and that we should just all go along with their methods, never stopping to question what exactly is going on, or...

What, exactly, is the financial cost of "success" in "choice" systems of schooling? Again, I give Craig Melvin credit for pointing out that the "miracles" of schools like Boys Latin in Philadelphia are not available to all. What he failed to uncover, however, is that schools like Boys Latin almost always have resource advantages over their public school counterparts.

Remember that Philadelphia's public schools have been under state control for years, and are chronically underfunded compared to suburban schools a few miles away. Boys Latin, however, has found a way to make up the difference:

PHILADELPHIA >> Boys’ Latin of Philadelphia hosted its newest event, “Magnum Opus,” April 27 at the Please Touch Museum in Fairmount Park, Philadelphia. The benefit raised vital funds for the school so it can continue to offer world-class educational experience and pays tribute to the great work of David Hardy and Dick Williams. The achievements of the school would not have been possible without the leadership of these two men, both of whom are in their final year of service at Boys’ Latin. 

[...] 

“Magnum Opus” welcomed comedian Jay Pharoah, Saturday Night Live alumnus, as entertainment for the evening’s festivities. The fundraiser has already raised over $500,000 in sponsorships and ticket sales. The school relies on these funds to support areas of college counseling, enrichment programs in engineering and the arts, technology programming, academic interventions and athletics. [emphasis mine]

According to its 2015 tax forms (available at Guidestar), Boys Latin pulled in $1.1 million in government grants and $1.7 million in charitable giving in a single year.* That's in addition to the revenue they get from the Philadelphia school district. In 2014-15, Boys Latin enrolled 747 students; that's an extra $1,464 in government grants and $2,228 in philanthropy per pupil.

And again: extra funding is only one way to gain a resource advantage if you're a "choice" school. If you don't enroll many Limited English Proficient students, your costs stay lower. If the few special needs students you do enroll have less-costly learning disabilities, your costs stay lower. If you employ a less-experienced, less-credentialed staff, your costs stay lower. If you only enroll students who choose to attend, your costs probably stay lower.

Money matters, and in education it matters a lot. I can certainly sympathize with the parent Melvin interviewed who declared that she was willing to do whatever it took to get her child a good education. But let's be clear that part of what makes an education "good" is having adequate resources. 

Even Kristof acknowledges this: "It would be odd if schools with teachers and books didn’t outperform schools without them." OK, great... Are you saying that the only way to get teachers and books into African schools is to turn them over to for-profit companies that can't turn a profit?

That brings us to the last question:

What, exactly, must students, teachers, families, and citizens give up under a "choice" system to get more resources into their schools?

As a long-time observer of all things reformy, this is the one argument from school privatizers that frustrates me the most: that extra resources can only come if schools slip away from governmental control. That the hope for competent, corruption-free governance is so far-fetched -- be it in Nairobi or Philadelphia -- that the only chance schools have is to move away from democracy and towards market models dominated by providers with ties to Wall Street or Silicon Valley.

In fact, this is exactly the case reformy CEOs like Reed Hastings of Netflix make: school boards, which allow citizens a direct say in how our society's children are educated, are actually an impediment to educational improvement. In other words: we have no "choice" but to move school governance away from governmental structures and put it in the hands of the not-really-all-that-free market.

So how's that working? It hasn't helped Detroit, or the students of cyber charters of Pennsylvania, or the taxpayers who subsidize the charter school land grabbers in Miami. It hasn't helped the citizens looking for transparency from Gulen-linked charters, or the taxpayers of Ohio. It hasn't helped students and families who lose due-process rights when enrolling in charters. It hasn't helped those who care about good government in North Carolina, or those who want to control school administration costs in New Jersey, or students left in the lurch in Michigan, or all those affected by numerous examples of waste, fraud, and abuse throughout the nation's charter school sector.

Are there examples of bad behavior in our public schools? You bet. Are there decent charter schools getting good results? Sure. But it increasingly seems as if there are at least as many bad apples in the charter barrel as there are good.

Carol Burris, who spent years as a public school principal and now advocates for public education, has been putting examples of charter school malfeasance on her twitter feed lately with the intro: "Another day, another charter scandal..." She's almost literally right; I'll bet she could put up a new scandal a day if she really tried.

There's a very good case to be made that the "choice" system our reliquishy friends in the reform industry tout actually encourages this sort of behavior. Self-dealing, profit-taking, cherry-picking... it's not necessarily a part of "choice," but it does seem the odds of such behavior increase when the market becomes the primary mechanism through which we hold schools accountable.

* * *

Let's bring this all together:

There is little evidence that the fraction of "choice" schools that appear to get better results do so because they are "innovative" in their educational practices. But the "choice" schools that do get gains all seem to have structural advantages, starting with resource advantages -- gained through a variety of strategies -- that allow them to offer things like longer days, longer years, smaller student:staff ratios, and extended educational programming.

By all appearances, we seem to be able to adequately fund our schools in the affluent, leafy 'burbs, even as we shrug our shoulders at the prospect of doing the same for urban centers enrolling many students who are in economic disadvantage. Millburn has what it needs; Newark does not. Gross Point has plenty; Detroit doesn't. New Trier is fine; Chicago is not. Lower Merion thrives; Philadelphia withers.

It's a story that plays out across the nation. Somehow these affluent communities manage to scrape together enough to provide adequate educations for their children, even when burdened with unionized teachers and step contracts and democratically elected school boards. Somehow they manage to get their schools what they need without giving up transparency and governmental accountability and agency for all of their citizens through the democratic process.

School "choice" is the result of a failure of honesty and will. 

The failure of honesty comes from failing to fully acknowledge that structural inequities -- inequality, chronic poverty, racism, inadequate school funding -- lead to unequal educational outcomes. It also comes from failing to acknowledge that the advantages a select few "choice" schools have accrued to themselves are directly responsible for their outcome gains.

The failure of will results from a failure to act collectively in ways that would move adequate resources to all schools where they are lacking, without giving up democratic governmental control.

Neither Kristof nor Lemmon nor Hardy nor anyone else has given us any reason to believe that the only way to get more resources into schools that need them is to abandon governmental control. There is, however, plenty of reason to believe shifting school control to private entities will reduce transparency, student and family rights, and efficiency -- both here and abroad.

When children live lives free of want and attend well-resourced, government-controlled schools they do very well. Certainly, there are problems and room for improvement. But communities don't need to give up control of their schools if the pre-conditions for success are in place.

Instead of upending the entire system, why don't we try that?

ADDING: I've been looking at the data, and we need to talk about student attrition and Boys Latin. Stand by...

* At the same time, David Hardy, founder of Boys Latin and prominently featured in Melvin's piece, took in $200K in salary and another $66K in benefits; again, Boys Latin had 747 students in 2014. In contrast, William Hite, the superintendent of the entire Philadelphia district, reportedly makes $300K/year running a district with over 130K students.

Just saying.