I recently found myself thrust into the orbit of a particularly confounding public institution: the local school board. To be specific, my kid’s school district has begun a process they are calling “resource alignment.” This is another way of saying the district will restructure by cutting and expanding various programs, and by combining and closing schools. Reading that last sentence, what images come to mind? Angry parents? Opaque guidelines? Stonewalling administrators? Perhaps endless committee meetings, town halls, and process, process, process?
The truth of it isn’t quite that bad, though overall the world of school board politics does seem to earn its reputation. That world is characterized by us parents as much as it is by the related institutions. Not surprisingly in such an emotionally charged, hyper-sensitive environment, opinions often masquerade as facts, passions dominate reason, and objectivity is in short supply. “We’re talking about our children!!!”
As I sit through these committee meetings, or overhear fellow parents sharing their fears, or read about this school board that regularly makes national news for its intriguing decisions, I also struggle to separate emotion and conjecture from truth and observable facts. How can any parent not? After all, we are talking about our children, and few relationships are more grounded in emotion than ours. But it is not the board’s business to be serving our emotional needs. Their responsibility is ultimately an institutional one, and institutions must act with reason and objectivity.
These messy circumstances compelled me to revisit a paper I wrote a few years ago regarding correlation versus causation. (I’ve posted the original paper under “essays” on this site.) The paper looks at the dangers of falling into the mental trap of confusing mere correlation for causation, and offers an introduction to use a widely lauded set of nine tools developed by epidemiologist Arthur Bradford Hill to assist researchers in uncovering genuine causal links among related data sets. Hill’s work was largely focused on medical research, and my paper also looks at examples from that field, so you may ask how this all applies to our local school board’s decision making? Quite directly, in fact.
“Middle schools students lag behind their K-8 peers.”
“DEI objectives compromise to academic outcomes.”
“Schools in wealthier neighborhoods perform better than schools in poorer ones because of higher parent involvement.”
Intriguing! But are these statements even true? And more importantly, if true, are these assertions meaningful without interrogating the underlying causes? Could the introduction of DEI objectives be a response to address already deteriorating tends in academic performance, and not the underlying cause of it? Might parent involvement be a corollary condition of wealthy schools’ larger resources and PTA budgets, and not a reason itself that students do well? I don’t know any of these answers, but they are the kinds of questions researchers and all responsible actors in these processes must ask.
For my part, I’ve spent some time recently trying to dig deeper into the research on differences between K-8 schools and their elementary/middle school peers. I won’t bore the lay reader with all the details, but one group, Education Northwest, did some research that particularly impressed me, as they specifically sought to examine how well other studies uncover causal links between observable correlated data sets. Their meta-analysis highlighted the same worrying trend that I discussed in my essay on the topic: researchers often reach conclusions from correlated data, but do not dig deeply enough to prove causal links. For example, Education Northwest highlighted some regularly cited studies on K-8 versus middle schools that make claims about links between school structure and performance, but those studies did not even account for income differences across school when examining data. They ultimately reach the conclusion that despite there being a sizable body of research available that compares K-8 and elementary/middle school structure, it is hard to reach a conclusion from the corpus of research that school structure itself is a meaningful determinant of academic performance.
The debate about restructuring our school district largely mirrors this trend. Well-intentioned researchers, school board members, and parents often highlight trends or findings that we should be concerned with. Some of the more data-driven ones hold up sets that point to conclusions in alignment with their goals or worries. But I’ve yet to sit through the meeting that examines the limitations of this data and the potential folly of incorrectly interpreting it. Without demonstrating causal links, how do we know that moving kids from this underperforming school to that leading school will improve things, or even that the increased rate of parents sending their kids to private schools can be reversed by improving the city’s public schools?
I may be speaking here only of how we treat data in a discussion of structuring a public school district, but confusing correlation for causation has implications for nearly every facet of your life. The next time you are fed some chart or statistic at work or school or the doctor’s office that shows how this action is tied to that outcome, do yourself a favor and challenge the data-pusher to go further and demonstration just how this action causes that outcome.
(For a brilliant and entertaining look at the pitfalls of confusing correlation for causation, check out www.tylervigen.com/spurious-correlations.)
…
There is something of a postscript to this piece, and it may sound more cynical than my thoughts above. While I would love to believe that the process of our school board realignment would be improved markedly through more attention to separating causation from correlation, even using correlated data is still an attempt to ground decision making in data, and not just people’s favorite means of conveying information, the beloved anecdote. I suspect though that in an emotional debate such as this, victory will not go to those who examined data the most rigorously, or even to those who most skillfully extrapolated meaning from that data. This debate, like most policy ones, will come down to story-telling.
And I’m not sure that’s at all a bad thing. But why I think the best rhetoric should win the day will be the subject of another post.
