Elegant Economic Theories Get Shoved Aside by Data

The number crunchers are taking over the field.

Let's run the numbers.

Photographer: Haruguchi Yamaguchi/Bloomberg

A number of writers, including my Bloomberg View colleague Justin Fox, have been following the big shift in the economics discipline. Where theorizing once dominated the journals and academia, empirics and data now comprise the majority. Let's hope this leads economists to be more careful about checking their theories against reality. In fact, the empirical revolution is already changing many policy debates, from minimum wages to immigration.

But how does empirical economics work? People in the public need to understand the basics. With all of the empirical econ results being hyped in the press (some of them by yours truly), it’s important to understand the power and the limitations of the research methods involved.

Everyone knows that you can’t put the economy in a lab and test it, like you can a semiconductor or a virus. Economics experiments exist, but they are mostly used to investigate people’s individual decision-making processes, kind of like in psychology. To see how the economy behaves in real life, we have no choice but to look out and observe the world in action.

That creates a number of problems. It limits the questions we can answer, since we can’t order the economy to try out this or that scenario just so we can see what happens. There is also the fact that correlation doesn’t equal causation -- a lot of times we observe  that A and B tend to go together, but we don’t know whether A causes B, B causes A, or both A and B are caused by some third factor, C.

But even more troubling is the problem of omitted variables, also called confounders. When you do an empirical study, there might be lots of important factors that you fail to take into account. That can lead to false conclusions. For example, suppose we observe that people with more education earn higher salaries. We might assume that education raises pay. But we might overlook that people with more education also tend to have richer parents. If rich parents help their kids earn bigger paychecks, then our rosy conclusion about education will be incorrect. No matter how many things economists try to take into account, there will always be the possibility that they left out something important, or omitted variables.

There are two basic ways that economists try to deal with these problems. In a nutshell, these are: 1) rely heavily on theory to help you, and 2) look for natural experiments. Kevin Bryan of the Rotman School of Management has a good post about these methods for anyone who is versed in the jargon of the profession. But let me try to boil it down.

The first approach -- relying on an assist from econ theory -- is called structural estimation. Basically, you make a theory of how the economy works -- how consumers behave, who owns what and what kind of costs companies face. Then you use available data to see how well the model fits the data, and to figure out the most likely values for the model’s parameters. Those parameters, or the specifics of the model, could include how risk-averse people are or how much it costs companies to change the rate at which they purchase new capital equipment.

The advantage of this method is that it allows you to pose all kinds of interesting questions. You can ask what would happen if racial discrimination suddenly vanished from the workplace -- even if that’s never actually happened in the past.

The downside is that your theory might be wrong and give you the incorrect answers. All those parameter values might be estimates of things that don’t even exist! Ideally, you can check whether the theory fits the data, and only use the theories that check out, but in reality very few models fit the data well enough to pass these tests. Structural estimation is powerful if you have a very good working theory of the world, but that’s a luxury we rarely have.

The second main approach is called the quasi-experimental technique, or natural experiments. This is when you look for a random variation in economic conditions or policy, and you observe the effect of that random variation. For example, suppose that a crazy dictator in a Caribbean nation suddenly decides to send a whole bunch of low-skilled refugees to Miami. You can use that random decision to get an idea of how an influx of low-skilled immigrants affects local wages and employment levels. Or suppose a city runs a lottery, and lets the winners go to whatever school they want. By comparing the lottery winners to the losers (who are stuck with their old schools by pure random chance), you can get an estimate of how much difference school choice makes.

This is a very powerful technique, because it doesn’t require you to have the correct theory. Just look at the effect of A on B. But it also has a severe limitation -- the more the world changes from when and where you did the study, the less useful the result is. Suppose you look at a minimum wage hike that raises hourly pay to $5.05 in New Jersey in 1992. What does that tell you about the likely effect of a plan adopted to raise the minimum wage to $15 in Seattle? Maybe a lot, but maybe nothing at all. Quasi-experimental research becomes less reliable the further you move away from the conditions where the experiment happened -- and you don’t even know how fast the reliability vanishes.

So economists are left with a choice -- rely heavily on possibly incorrect theory to interpret empirical results, or get results that don’t depend on theory but whose value is limited in time and place.

There is certainly some sniping between adherents of the two approaches. Joshua Angrist and Jorn-Steffen Pischke, two of the chief evangelists for the quasi-experimental approach, called their book “Mostly Harmless Econometrics” -- the implication being that structural estimation, by relying on incorrect theories, has the potential to do harm. Francis Diebold, a practitioner of the structural approach, was not impressed.

But this kind of food fight is common among academics. The truth is that both approaches are useful in their own way. A healthy economics profession needs both. The real problem isn’t any particular empirical technique -- it’s the reliance on theory with no verification whatsoever. Fortunately, economics that relies only on theory is in steep decline.

This column does not necessarily reflect the opinion of the editorial board or Bloomberg LP and its owners.