Charles Manski: Seminar with a skeptic

Published: 22 October, 2013

Charles Manski Charles F. Manski

On the 21st and 22nd of January this year Charles F Manski was in Stockholm, invited by the Institute for Futures studies to hold three lectures on his newly published book Public Policy in an Uncertain World.

Charles Manski is Professor of Economy at Northwestern University, Chicago and his research spans econometrics, judgement and decision, and the analysis of social policy. He has written several books and over lunch after the lectures I ask him why he wrote this particular one.

– Because I want people to know, he answers.

And I can tell. Manski knows a lot, and he is eager to share. His words have been flooding over me and the other forty people in the room during the lectures, people who in their work deal with research in different ways. We are all there to know why we can’t trust researchers and what we can do about it.

The weak link

Being an economist, Manski begins by showing us an equation:

assumptions + data = conclusions

This equation will be with us for all three lectures and is the core of what he wants to teach us. But why then, why can’t we trust researchers? Manski reminds us of the equation. Data in itself can tell us very little, if anything. Not until you pose a certain question you could get an answer, and this question is always based on assumptions. What’s tricky with assumptions is that they are subjective, based on what someone believes to be credible. Even trickier is the fact that many assumptions are impossible to disprove.

We cannot disprove for example that the god of the Torah created the universe in six days and then rested on the seventh day, just as we cannot disprove that the universe was actually created by The Flying Spaghetti Monster, as Manski writes in the article “Unlearning and discovery” (The American Economist, vol. 55, No 1, Spring 2010). This is something Manski learnt as a teenager, realizing he was not a true believer and therefore left Judaism to become an agnostic. This experience also taught him to accept that the truth is unknowable and that decisions therefor must be made under ambiguity.

Manski has created a law describing the tension in the equation which he calls “the law of decreasing credibility”: The credibility of inference decreases with the strength of the assumptions.

<h4>The dirty laundry of research</h4>
Researchers in general prefer strong conclusions, and they are not the only ones. Strong conclusions are easier to understand, easier to communicate and easier to use in decision-making. Therefore the credibility is often sacrificed using non-refutable assumptions, or by using models by comfort or convention.

This is something every consumer of research should be aware of, says Manski. He encourages us to be specifically skeptic when someone says that “research shows…” and ask ourselves what assumptions are in the equation. We should still be skeptical if the researchers have used different assumptions to gain the same results – if the difference is marginal, the conclusions are still not sure to be trusted.

<h4>Time to grow up!</h4>
But Manski wants more. He recommends us to follow the path he himself chose as a teenage agnostic.

– It’s time to grow up! There is uncertainty, learn to deal with it, he says.

He encourages us to allow research to be more transparent, showing the uncertainties, and to rather trust ranges than point estimates. This is especially important when it comes to social sciences, since it deals with people and no individuals are the exact same. Working with public policy this is extremely important, since it means trying to predict the effects of future decisions.

– We can’t expect policy makers to make optimal decisions because that is impossible, all we can expect is for policy makers to make reasonable decisions, says Manski. And he recognizes the difficulties for policy makers to make these decisions.

In order to show us how important the choice of assumptions and models are to the end result, Manski gives us several examples. One of them is the examination of the deterrent effect of capital punishment in the United States. He shows that the data alone cannot give us an answer, and that differences in assumptions and models give us totally different conclusions about the efficiency of death penalty as a way to reduce murders.

He also uses decisions on national vaccination policies as an example, which more easily resonates with Swedes after the decision about mass vaccination against the swine flu in 2009. He shows how it is impossible to know all the facts and how scenarios can be used as a method to be able to avoid the worst case scenario and choose the least bad alternative.

When the lectures are over we leave with a whole bunch of methods to identify and deal with uncertainties. I feel more skeptical than ever, ready to scrutinize every piece of research or policy decision I come across. But how about my own decisions, what assumptions do I build them on? Manski says;

– It’s like a whole hill of sand which can just collapse.

Erika Karlsson

Public Policy in an Uncertain World, 2013. Harvard University Press
www.charlesmanski.com