How Microsoft will scrape and crunch Twitter
Jennifer Chayes, the chief of Microsoft’s new research lab in Cambridge, Mass., was telling me yesterday about her Twitter epiphany. A theoretical mathematician, Chayes was at one of Tim O’Reilly’s “Foo Camps.” These are open-ended brainstorming sessions for groups of 150-300 people. She was telling Laura Fitton, a Twitter expert, that she just didn’t get the microblogging, and didn’t see why people would follow so many others.
Fitton started telling her quickly information spreads on Twitter. Chayes says she suddenly saw that Twitter might have dramatically different dynamics than blogs. She became fascinated by the math involved. Perhaps it would show that information in a network of millions of micromessages would spread like an epidemic.
Chayes was standing next to Luis Von Ahn, a researcher at Carnegie Mellon and also, until that moment, a Twitter skeptic. But now they’re going to join efforts, “scrape” the data from Twitter and do the math on it.
By the way, Microsoft has opened the lab in Cambridge precisely to do this type of analysis on social behavior—the juncture between computer science, mathematics and the social sciences (and the laboratories of the people I call the Numerati). Chayes told me that it was hard to recruit top economists, anthropologists and other social scientists to Redmond. It’s easier in Cambridge.