UPDATED Chartery Snake Oil: Tennessee Grade

UPDATE, 6/10/14: So Bluff City Education has responded to this post. I won't waste my time or yours arguing semantics; I'll only point out the author apparently has no argument to make against my data and/or analysis.

And I'll point out, once again, what really bugged me about Aycock's original piece:

Yes, reducing poverty is an essential piece of the puzzle in solving the achievement gap. However, schools don’t control housing, economic development, or social welfare policy. Schools only control what goes on inside the walls of the building. To argue that we can’t improve education until poverty is solved only gives a pass to schools that are failing our children. [emphasis mine]

As I say below: it'd be one thing if Aycock was just cheering on his school. But he went well beyond that: he called out other schools that are "failing." As far as I'm concerned, he asked for everything I gave him. 

If you're going to throw down a gauntlet, you'd better be ready for someone to pick it up and throw it back in your face. If you can't handle that, you'd do well to keep your mouth shut about everyone else's "failure."

* * *

You know what really bugs me these days? Especially in the world of education "reform"?

Smugness.

I'm not talking about pride. Pride is great (to a point). Everyone who works at a school should be proud of their students, their staff, their leaders. If you talk up the good things happening at your school -- charter or public -- that's fine with me.

And I'm not talking about confidence. Hey, if you can back up what you say, more power to you. Understand you've left yourself open for critique, but if you feel that you, as an educator, can show you are getting it done and you want to share that with the world, I say go ahead.

No, what irks me is this: reformy types who puff up themselves and their "miracle" schools to the point where they actually think they've discovered the cure for what's ailing our horrible American public schools (which, in reality, are pretty good, considering the circumstances).

Educating children -- especially children who are at-risk or have special needs -- is complex, difficult work. Most people who do this job are humbled by it; they know that there are no easy answers, and that techniques and strategies that may have worked once are never guaranteed to work again in the future. And, while we teachers have no problem with accountability, we also understand that it's really easy to tut-tut at the failings of others when your own grasp on success is so tenuous.

Not so the smug reformer. No such self-doubts plague him: he knows that his superior moral center and indomitable will are the keys to his absolute and unquestioned excellence. If only the rest of us would follow in his footsteps, we could transform the entire education system in a few short years. If only we excuse-makers would get out of his way, he could solve society's ills faster than a French economist could raise your taxes.

Often, these reformy, self-aggrandizing braggarts have credentials as phony as a carnival barker's: TFA, Broad, SUPES, TNTP, Relay, NLNS, etc. Invariably, they have some sort of link to a charter chain: KIPP, Uncommon, Success, ASD, Mastery, Academica, etc.

And, inevitably, they take to the media at some point to boast on their awesomeness:

Critics of education reform often blame poverty for poor academic achievement. In fact, many argue that you can’t fix education until you fix poverty. 

This simply is not true. 

Yes, reducing poverty is an essential piece of the puzzle in solving the achievement gap. However, schools don’t control housing, economic development, or social welfare policy. Schools only control what goes on inside the walls of the building. To argue that we can’t improve education until poverty is solved only gives a pass to schools that are failing our children. 

Schools can help reduce poverty, but only by increasing opportunity for children. 

And I’m proud to say that I’m involved in two schools doing just that. I’m involved in one as a parent and the other as a teacher.

So, even though reducing poverty is "essential" for raising educational outcomes, we don't have to actually do it. Yeah, that makes sense...

I'm going to skip over the school to which the author, James Aycock, sends his own children for two reasons. First, the rest of this post uses 2013 data from the Tennessee Department of Education, and KIPP: Memphis Collegiate Elementary doesn't report test scores; it only has a kindergarten and first grade (apparently, a school that only serves two primary grades can "prove" poverty doesn't matter - amazing, huh?). Second, it's probably better to leave Aycock's choices for his own kids out of this debate; let's concentrate on something just a little less personal.

We'll look, instead, at the vaunted school where Aycock works. What is this astonishing citadel of learning that proves that you don't have to "fix" poverty before you "fix" education?

There are several key takeaways that I have from my experiences with KIPP: Memphis Collegiate Elementary and Grizzlies Prep.

First and foremost, poverty does not have to be destiny; zip code does not determine potential. We can’t afford to use poverty as an excuse to continue the status quo. We can’t wait until poverty is solved. We have to set expectations high for all kids.

Secondly, parents should not be so afraid to educate their kids alongside kids of different socio-economic, racial, or other backgrounds. Specifically, white parents need to be okay with their kids being the minority. This is Memphis after all, the first majority African American metro area in the nation. And, since median household income in Memphis is $35,000, middle class parents need to be okay with their kids being around those with less income. [emphasis mine]

Wow - this Grizzlies Prep must be some school! Not only is it named after an NBA franchise; it proves that poverty just doesn't matter! And it teaches less enlightened parents, through example, that you can send your middle-class child to a school full of students with lower incomes without fear! Step back, haters: Grizzlies Prep is proving that it's your weak status quo excuses -- and not poverty -- that are holding our kids back! And I'm sure a look at Grizzlies, using publicly available data and standard statistical methods, will prove Aycock is correct.

Won't it?

Let's start with the obvious: poverty matters. Anyone who tells you otherwise is full of crap. Above, I've plotted every school in the Greatest State in the Land of the Free that reports Grade 3 through 8 test takers on a graph with two axes. The y-axis is the number of students at proficient or above on reading/language arts (RLA) tests. The x-axis is the number of children in the school who qualify for free lunch, a proxy measure for economic disadvantage.

What we find is the most obvious thing in the world: as poverty goes up, test scores go down. The correlation in Tennessee is so strong that over 70 percent of the variation in test scores can be explained solely by free lunch eligibility.

See that green line through the middle? That "regression line" is kinda-sorta an average: it's a prediction, based on all the points, of how many students will be proficient in a school if we know that school's free lunch eligible rate. The prediction here is very, very good, because the correlation between poverty and test scores is very, very strong.

This, by itself, is more than enough to dismiss Aycock's ridiculous claim that "poverty isn't destiny." It doesn't matter that there are outliers -- points above or below the line. The overall pattern is so compelling that the vast majority of the outliers don't vary much from the line. In Tennessee, as in every other school system in the world, poverty matters. 

But let's take a look at Grizzlies Prep anyway. Let's see if Grizzlies chows down on poverty like a bear scarfs a Sockeye.

Remember when I said the green line was a prediction? If a school is above the green line, that means it beats predicition: it over-performs where we would expect given its free lunch rate. But look at Grizzlies Prep: it's under the line. Grizzlies Prep underperforms prediction -- a very strong prediction.

I've marked out the other charters in Tennessee that reported Grade 3-8 proficiency rates in red triangles (using the TNDOE's markers for whether the school is a charter). And I've marked a few that are part of the vaunted Achievement School District, pet project of reformy blowhard and TN Commissioner of Education Kevin Huffman.

Does it look to you as if Grizzlies or ASD is proving that "poverty does not have to be destiny"? Or could it be that Tennessee's charter industry really proves exactly the opposite: all schools, charter or otherwise, follow a pattern where test-based outcomes are highly correlated to measures of economic disadvantage.

For kicks, let's look at math scores:

The correlation here is significant, but not as strong as in RLA. Grizzlies (and ASD), however, actually falls further behind prediction here: they are farther below the trend-line.

How, then, can Aycock claim with a straight face that "Grizzlies Prep is proving that you can educate both the more and the less prepared together successfully"? I suppose he might make the argument that he needs more time; that Grizzlies' kids are too far behind and need to catch up. OK... but don't go around bragging that you've proved poverty doesn't matter until you've actually done it!

Don't smugly look down your nose at "schools that are failing our children" when many of those schools, burdened with higher rates of student economic disadvantage than yours, outperform your school!

And don't attempt to take the moral high ground with middle-class parents about economic segregation without first looking in the mirror.

I don't know Memphis at all (which is a shame, because this Jersey boy loves him some barbecue). For this graph, I looked at a map, found Grizzlies Prep (zip code: 38103), and picked out a bunch of zip codes in the immediate area. I then plotted the free lunch percentage for all the equivalent schools: the ones that report valid test takers for grades 3 through 8.

I've switched over to free lunch, as opposed to free and reduced-price lunch, because central Memphis has many children in economic disadvantage, and FL is a better indicator of the variance in this disadvantage (you-know-who breaks this argument down here). The point is clearly made: Grizzlies Prep -- and, indeed, the entire Memphis charter sector -- serves fewer students in deep economic disadvantage than the rest of the city's public schools.

Let me be clear: a 65 percent free lunch rate is high, relative to the rest of the state. But it is not high relative to Memphis. I won't say this is a deliberate economic segregation; most likely it is not, at least in the sense that some students are being told outright they can't attend because the quota of poor children has been filled. It's far more likely deficits in social capital are playing a role here; however, deliberate or not, this segregation is real.

Like I said, I don't really know Memphis, but let's pull back a little: near as I can tell, these are all the schools (again, reporting valid test-takers for grades 3-8) within the "loop" of I-40, I-240, and the river. I've only marked Grizzlies Prep this time.

Still not looking like the rest of the city's schools, is it?

Let me stop here and give the caveat I always give in posts like this: I have no doubt that Grizzlies Prep is full of wonderful, deserving kids and hard-working, dedicated educators and staff. Every one of them should take pride in their school. My beef is only with Aycock -- and not with his work as an educator, of which I know nothing.

No, my problem here is that he (and the TN Charter School Center, which reprinted his piece) has polluted the conversation about America's schools and childhood poverty, and he's done so in a way that keeps us from having an honest discussion about what's really going on.

Aycock goes well beyond praising his own school; again, I really have no problem with that. But he takes it to the next level: he claims his school is better than the schools that "fail" our children and make "excuses." What Aycock fails to point out however, is that his school's outcomes are really no better than those schools he excoriates, even as his student population has significantly fewer students in economic disadvantage.

Poverty matters, James. Your own school proves this to be true, and it's not an "excuse" to say so, loudly and clearly.

It's probably a little unfair for me to come down so hard on Aycock: he's merely the latest in a long line of reformy swaggerers whose claims, upon closer inspection, almost never hold up to scrutiny.  But their voices have become so loud, so insistent, and so pervasive, that I and the few others who are attempting to stand up for real public education reform have little choice at this point:

It is time to state, in no uncertain terms, that the people who push the notion that poverty is an "excuse" for academic outcomes are little better than snake-oil salesmen. Their claims are pernicious and dangerous. They are providing an excuse for plutocrats to continue economic policies that concentrate more wealth into the hands of a few at the expense of millions of children, from both middle-class and working-poor families.

These "reformers" are the enablers of a level of inequity we have not seen for a century. And they are, through the tenets of "reform," embedding a system of social reproduction within our urban schools, all while claiming they are doing exactly the opposite. They are abetting a transition to privatized schooling that abrogates the rights of already disenfranchised families, often instilling a pedagogy of compliance that would never be tolerated in wealthier communities.

In short: the people who tell you that their miracle schools prove that "poverty is not destiny" need to stop.

Now.

Tennessee's finest grade.

H/t to the always excellent Gary Rubinstein for pointing this piece out.



Geek Notes: This is my first time working with Tennessee data; like all states, it has its quirks. You can get Excel tables with composite Grade 3 through 8 proficiency rates, or just Grade 3, or just Grade 7 (why not the others?). Proficiency is broken down to four levels, but I couldn't find scale scores. There are also tables with "3-year average NCEs" in a variety of subjects, but I'm not touching those without fully understanding the methodology behind them.

Proficiency is an incomplete measure; any test-based outcome is an incomplete measure. Again, I'm not saying Grizzlies, or any Tennessee charter, is a bad/good school: I'm saying the data doesn't back up the overblown claims Aycock makes in his piece.

I ran the regression and made the tables in Stata. I excluded any school that didn't have test takers in Grades 3 through 8 because I wanted to look at equivalent schools, but let's face it: there could be a lot of variation even within this group, and there's no guarantee we've got a clean apples-to-apples comparison here. Caveat regressor.