Betsy DeVos's Florida Fantasy

Betsy DeVos's incoherent mess of a interview with Leslie Stahl (who I thought did a good job) on 60 Minutes Sunday night has got to be one of the most embarrassing performances by a sitting cabinet member in modern times.

I'm not sure my stomach could take a complete debunking of all of DeVos's nonsense. But I would like to focus on one exchange:

Lesley Stahl: Why take away money from that school that's not working, to bring them up to a level where they are-- that school is working?

Betsy DeVos: Well, we should be funding and investing in students, not in school-- school buildings, not in institutions, not in systems.

Lesley Stahl: Okay. But what about the kids who are back at the school that's not working? What about those kids?

Betsy DeVos: Well, in places where there have been-- where there is-- a lot of choice that's been introduced-- Florida, for example, the-- studies show that when there's a large number of students that opt to go to a different school or different schools, the traditional public schools actually-- the results get better, as well. [emphasis mine]

Stahl goes on to ask about how choice has worked Michigan -- as well she should. But I'd like to take a minute or two to examine what DeVos thinks she knows about Florida and school "choice."

Any time a policymaker talks about what "the studies show," watch out. Some of them are adept at reading and synthesizing research, but many are not; too often, they let their staffs, who tend to have cursory training in research methods (especially quantitative methods), assemble the evidence so that it matches their ideological predilections.

In DeVos's case, it's been clear for years that she supports school "choice," no matter what "the studies show." DeVos has publicly admitted that her advocacy for school vouchers is driven by her religious faith. In a way, that's not different than Milton Friedman's voucher advocacy, which also was not based on empirical evidence, but rather ideology.

The theory behind school "choice" has largely rested on the notion that competition forces improvements in schools; therefore, if we want to improve public, district schools, we should threaten them with losses of enrollments by introducing "choice" through a market-based system subsidized by taxpayers.

So, what do "the studies show" about school vouchers in Florida? Let's ask Patrick Wolf, writing here with Anna Egalite. Wolf is well-known within education policy circles as one of the foremost advocates for school "choice" in academia:

The competitive effects of Florida’s various voucher programs have been the subject of nine studies. All of them reported that the test scores of students who remained in public schools increased as a result of school choice competition. Although these positive effects of competition on public school achievement tended to be small, they were larger when school choice increased dramatically (Forster, 2008a). [emphasis mine]

I'm going to object to that last phrase. Forster's study -- which was not peer-reviewed, and has some serious methodological weaknesses (some of which are recounted in a review of a related study here) -- is hardly enough evidence to suggest that expanding school choice leads to better outcomes in public schools. Forster's study really does nothing to control for all kinds things aside from vouchers that might explain rising test outcomes.

That said, Egalite and Wolf are right: the effects of competition on Florida public schools were small.

Really small.

Let's take, as our best available example, the latest study on Florida's vouchers they reference: a 2014 peer-reviewed paper by David Figlio and Cassandra Hart. It's a clever piece of econometric work; not airtight by the authors' own admission (what is?), but still well worth considering. The authors exploit the fact that competition from vouchers varied considerably across Florida during the period under study: some schools, for example, have a private school nearby, while others have one further away. Some areas have a variety of private schools; some have only one type (say, evangelical).

Foglio and Hart looked at how this competition varied and correlated with outcomes for public schools. What did they find? If you're not used to reading this sort of research, it might be hard to grasp, so let's break it down:

Every mile the nearest private school moves closer, public school student test score performance in the post-policy period increases by 0.015 of a standard deviation.

The study excluded schools that were more than five miles from a private school, and the average public school was about 1.3 miles from a private school. What this study found was that if you put a private school a mile closer to a public school that already had one within five miles, the test scores at the local public school would increase from the 50.0 percentile to the 50.6 percentile.

Not impressed? Try this:

Adding 10 nearby private schools (just shy of a standard deviation increase in this measure) increases test scores by 0.021 of a standard deviation.

The average public school had 15.4 private schools within five miles. Add 10 more and you'll move the school from the 50.0 percentile to 50.8.

Each additional type of nearby private school is associated with an increase of 0.008 of a standard deviation. Adding an additional 100 churches in a 5 mile radius (a nearly one standard deviation increase) is associated with a 0.02 standard deviation rise in scores, and adding an additional 300 slots in each grade level in a 5 mile radius (just over a 1 standard deviation increase in this measure) increases scores by 0.027 standard deviations. Overall, a 1 standard deviation increase in a given measure of competition is associated with an increase of approximately 0.015 to 0.027 standard deviations in test scores. [emphasis mine]

In other words: increasing the competition measures by what is a very substantial amount results in moving test outcomes from the 50.0 percentile to between the 50.6 and 51.1 percentile. This is the most generous interpretation using this conversion.

While these estimated effects are modest in magnitude, they are precisely estimated and indicate a positive relationship between private school competition and student performance in the public schools even before any students leave the public sector to go to the private sector.

Well, yes, they are precisely estimated -- that's easy to do when you have a really big data set with over 9 million student-year observations.

But these results are not "modest" -- they are tiny. They represent no meaningful educational impact. To say, as DeVos does, that "the results get better" is just not accurate in any practical sense.

Now, voucher proponents could make the case, based on this study, that there is no evidence that the schools got worse. But I think that argument fails for at least a few reasons:

First, test score gains or losses are a very poor measure of whether a public school suffers fiscal stress due to the diversion of funds. Instruction in tested subjects would be the last thing a school district cuts if it's under competitive and accountability pressure. The question is what happens to instruction and programming not related to tests: extracurriculars, arts, history, science, student support services, etc. The truth is, we just don't know.

Second, if "choice" is introduced as a substitute for things like adequate and equitable funding, the overall progress of the system will be impeded. The sad fact is that the "Florida Miracle" has been grossly oversold; the state is a relatively poor performer compared to other states that make more of an investment in public education. Can that all be attributed to policy? No, of course not... but Florida is a state that makes little effort to fund its schools.

In any case, DeVos's contention that public, district schools see improvement when there is competitive pressure is just not held up in any practical sense by research like this. As I said in my last post, the effects sizes of things like this are almost always small. In this case, the effect is exceptionally small; in practical terms, it's next to nothing.

The idea that we're going to make substantial educational progress by injecting competition into our public education system just doesn't have much evidence to support it. I wish I could say that conservatives like DeVos were the only ones who believe in this fallacy; unfortunately, that's just not the case. Too many people who really should know better have put their faith in "choice," rather than admitting that chronic childhood poverty, endemic racism, and inequitable and inadequate school funding are at the root of the problem.

As always: I'm not saying we can't and shouldn't improve our public schools right now as best as we can. But DeVos's policy of expanding Florida-style school choice as a way of improving public schools makes as little sense as her policy of arming teachers to improve school safety. Neither policy has any empirical support, because both are clearly illogical.

School "Choice's" Best Friends