Stanford Economist Musters Big Data to Shape Web Future

Economics Professor Susan Athey
Economics Professor Susan Athey returned this year to Stanford University in the heart of Silicon Valley, where she got her Ph.D., to spearhead a vision that's emerged from her years with Microsoft Corp., where she still consults. Photographer: Marc Perrier/Bloomberg

It was late spring in 2007 when then-Harvard University economist Susan Athey received a call. Microsoft Corp. Chief Executive Officer Steve Ballmer had read about her and wanted to meet.

Athey, 36 at the time, had just become the first woman to receive the John Bates Clark Medal, joining a group that now includes 12 Nobel Prize winners and two White House chief economists. She flew to Redmond, Washington, and was ushered into a windowless conference room with Ballmer and several of his vice presidents. The company was at a crossroads, struggling to gain ground with a search technology that had long trailed Google.

“The obvious question was, would this become a one-firm show or was it possible to have two competing search engines?” recalled Athey, an authority on timber auctions who had just begun studying the market for digital advertising. “I believed then, based on economic theory and some data, and I believe more strongly now that it’s indeed possible to have two competing search engines and that it’s also incredibly important for the Internet ecosystem.”

The meeting with Ballmer would steer her from what was already an award-studded career into the nascent field of Internet economics. Taking on the role of consultant as chief economist for the world’s largest software maker, she focused her research on the real-world puzzles she witnessed in Redmond.

Advising Engineers

With access to information on a scale so vast she says “it takes hundreds of machines just to open up the dataset and take the first average,” Athey started guiding Microsoft’s executives and engineers with the latest theories and findings in economics, advising them on decisions that would affect millions of the world’s Internet users.

“There are people who build on other people’s research,” but Athey “does a single paper that moves the entire field along 10 years,” said Joshua Gans, a professor of strategic management at the University of Toronto and a graduate-school classmate with whom she has co-authored several studies. “Now she’s running an economic movement” and she’s “nearly starting from scratch. It’s the kind of thing you would want someone like Susan to do.”

Athey returned this year to Stanford University in the heart of Silicon Valley, where she got her Ph.D., to spearhead a vision that’s emerged from her years with Microsoft, where she still consults.

Creating Opportunities

“We’ve never had data on that scale before to study behavior,” which “creates opportunities for new science,” she said. “I want to help the economics field embrace those opportunities.”

In a profession that tends to encourage expertise in a single area, Athey’s research is distinguished for its diversity. When she won the Clark medal as the most influential U.S. economist under 40 -- one of economics’ most prestigious prizes -- the American Economic Association cited her work spanning fields from pure theory to the invention of new statistical tools and her analysis of real-world data.

“She’s very motivated by applied problems, but she has all the tools, abilities and firepower to go at those problems in a general way with deep theory,” said Robert Marshall, her undergraduate adviser at Duke University and now a professor of economics at Penn State University. “That combination is very rare.”

Extra Math

Born in Boston to a physicist and an English teacher, Athey spent her childhood asking her parents for extra math problems and reading textbooks “just for fun.” Taking the SAT college-admission test when she was 12, she scored so high in math that she was placed into a special cohort of gifted students tracked by the Study of Mathematically Precocious Youth. She also had high verbal marks.

After a brief rebellion “against all things academic” as an early teen, she convinced her parents and guidance counselor to let her graduate high school a year early, when she was 16. To help pay for college, she took a summer job as a receptionist at a startup company that sold computers to the government through procurement auctions. At Duke, in Durham, North Carolina, she wrote her economics thesis on this bidding process.

While she also majored in computer science and math, she chose to pursue economics for her graduate degree because of its potential to influence public policy. “It was incredibly powerful” watching her adviser Marshall testify in Congress based on research she did under his direction, she said.

Lifting Weights

At Stanford’s Graduate School of Business, where she got her doctorate at age 24, she lifted weights because she realized she was missing out on valuable networking conversations with her male classmates and professors who socialized at the gym, she said.

Her dissertation led to her first national media attention, with the New York Times profiling her as “the top draft pick in economics.” Working with models that predict how, say, an investor or company makes a risky decision, she developed a tool now taught in graduate economics courses that simplified the assumptions researchers use to make forecasts, allowing them to focus on the essence of the problem and not complicated math.

After graduate school, she chose to join the Massachusetts Institute of Technology as an assistant professor; it had competed with about two dozen of the country’s best economic departments to recruit her.

‘Crushing’ Expectations

With early achievement also came pressure to live up to what she says were sometimes “crushing” expectations. At MIT, Athey became known for pulling late nights and sleeping multiple times a week in her office -- equipped with a couch, refrigerator, freezer and microwave. To streamline a 40-page mathematical proof, she once spent three weeks secluded in her office, leaving only to shower, she said.

“I had the feeling that everyone was watching me and I was basically set up to fail, that there was no way that I could achieve,” said Athey during an interview at Stanford near Palo Alto, California. “My success on the job market was more based on potential than the things that I had accomplished at that point.”

She won the 2000 Elaine Bennett Research Prize, presented for “outstanding contributions by young women in the economics profession,” and then left MIT for Stanford in 2001. She joined Harvard’s faculty five years later.

Timber Auctions

Examining the results of U.S. Forest Service auctions to sell off timber, she stumbled upon results that defied conventional thinking. Among them: The way the rules were designed was crucial in making sure small bidders participated, keeping the proceedings competitive. She used her findings to help the Canadian government deregulate its timber auctions.

The principles she gleaned from the economics of forestry would prove invaluable for her later work. Auctions were becoming ubiquitous on the Internet, from consumers selling bicycles on EBay to companies bidding to have their advertisements appear next to search results on Google.

What Athey saw in these online marketplaces was a treasure trove of largely unexamined data. She had just started studying the auctions for ads when Microsoft contacted her. She quickly saw the bigger picture: a chance to inject more competition into the then-$8.8 billion U.S. search business, which people used to find not only commercial information but also “political information and information that’s relevant to a democracy,” she said.

Strong No. 2

“Having a strong No. 2 is what drives a lot of the investment decisions, the prices for the advertisers and the quality of the search for the user.”

The work for Microsoft meant Athey needed to study the company’s proprietary computer-programming language, along with the details of running a search engine. She says she already was exhausted from juggling responsibilities on academic committees, teaching full-time and raising two young children. She took a year off from teaching at Harvard in 2008, shuttling between Microsoft’s research center in Cambridge, Massachusetts, and meetings at the Redmond headquarters.

She wrote her own computer code to extract information on advertiser bids, often submitting computing requests to the company’s servers at 1 a.m. She also performed hundreds of Internet searches individually, checking the advertisements to see what was going right and wrong.

“Susan is one of the most respected leaders in her field and we’re fortunate to have the opportunity to work with her,” Rik van der Kooi, chief operating officer of the company’s Online Services Division, said in an e-mail. “She has provided valuable advice on search economics and helped us think through the dynamics of the search market, as well as opportunities to re-imagine advertising.”

Statistical Debates

Athey now has one boy and two girls with husband and fellow economist Guido Imbens. They occasionally collaborate on research, she said, which sometimes leads to debates about statistical methodology as they change their youngest daughter’s diapers.

While she no longer sleeps in her office, she works 75 hours a week, often starting at home in the morning to get her children ready for school, returning by 6 p.m. for dinner, then logging back online from 9 p.m. until midnight. She blocks off weekends to spend with her family, although she typically takes three hours each night on Saturday and Sunday to work after her children go to bed.

“She really cares about making sure you don’t take short cuts and that everything is done incredibly carefully,” said Jonathan Levin, an economics professor at Stanford who has written a number of papers with her. “She has a remarkable energy to get things done.”

Delegate Meticulously

Athey has learned to decline some opportunities, delegate meticulously and accept that some nonessential tasks won’t turn out “perfect the way I want them to be” -- all decisions she says she would like to see more women make. Halfway through a year in which her husband was preparing their children for school, she remembers asking him where the hairbrush was. He asked her if they had one.

“But you know what? That was fine. Who really cares if a five year-old’s hair is brushed?” she said. “His approach was different, not worse, and I had to let go.”

That letting go ensures she has time to build out her new big-data-meets-economics vision. She uses advanced techniques such as machine learning, developed by scientists using large sets of data that enable computers ultimately to program themselves. While such techniques have become commonplace at technology companies, they aren’t widespread among her colleagues.

New Generation

She’s helping to train a new generation of economists who can straddle the worlds of research and real-world market design. Just this month, Athey finished teaching her first class on the economics of e-commerce and the Internet, a Ph.D.-level course drawing on much of her own research from the last six years.

“Throughout her career, she tried to be where the action was,” said Glenn Ellison, an MIT economics professor who also was at Athey’s first meeting with Ballmer in 2007. “There are some people who have done the same thing for the past 20 years and you know they’re going to keep doing that for the next 20 years. It’s much harder to predict where Susan’s going to go.”

Before it's here, it's on the Bloomberg Terminal. LEARN MORE