Lunchtime on Wall Street. Let's say a trader hears a rumor of an impending buyout over his pastrami on rye. Back at his desk, he wants to see if anyone has talked about it -- especially in the past few hours. Where to look? Google (GOOG )? There, the top-rated link might date all the way back to 1999. No, if you're looking for today's buzz, the first place to click is a Web log search engine such as Technorati, Feedster, or Blogdigger. By picking up the latest posts, minute by minute, from some 12 million blogs, these sites showcase an ever-changing mosaic of what is on the world's mind. And from politicos to ad execs and plaintiff's lawyers, lots of people are hungry for just that.
Now a surge of traffic to blog search sites has raised expectations that giants Google, Yahoo! (YHOO ), and Microsoft's (MSFT ) MSN may soon plunge into blog search. Blogs would give them yet another vehicle for the search-based ads they sell. Just as vital, blog search promises to be a big traffic driver as more surfers -- some 40,000 to 50,000 a day -- take up blogging.
The time looks ripe. Industry incumbents appear vulnerable as they struggle to sort through the avalanche of new data pouring into their servers. Adam Hertz, Technorati Inc.'s chief engineer, says the company handles 850,000 daily blog postings, up 70% in three months. Adding servers is a snap, Hertz says. But revamping systems to digest all the data "is like changing a flat tire on a moving car."
Little surprise, then, that many blog search engines are moving at a crawl. Some drag on for 30 seconds or longer. And the blogosphere is echoing with loud lamentations. Bloggers, after all, count on these search engines to track down who's writing about them and their blogs -- a pressing concern for most. On July 8 blog entrepreneur Jason Calacanis asked for help. He petitioned on his blog for Google and Yahoo to lend their muscle to blog search. Bloggers quickly seconded his motion. Technorati CEO David L. Sifry responded with a plea for patience. "I'm sorry if we're not meeting your expectations," he wrote on Calacanis' blog. "But we're going to keep working our asses off to provide...the most relevant information on the blogosphere you can find."
While Yahoo, Google, and Microsoft's MSN decline to detail their plans, industry observers say it's likely a question of months before the big three dive into blog search -- and perhaps scoop up one or two of the venture-backed blog search engines. On July 8 a prototype Yahoo blog search page was briefly spotted before the company pulled it back. "If [the three giants] are not in blog search by the end of the year, I'd be very surprised," says Forrester Research Inc. (FORR ) analyst Charlene Li.
It stands to reason. Blogs are the fastest-growing segment of the Web, and the posts -- many of them dispatched directly to subscribers -- are perfect for the kind of targeted advertisements favored by search engines. Strategically, whichever company masters blogs gains an edge: It can serve up its archived treasures with timely updates from the blogs.
If blog search is so vital, why are the giants moving so deliberately? In part, say observers, because blending blogs into Web search is a knotty conundrum. It requires adding a new dimension -- time -- to traditional search. Currently, the search hierarchies used by Google and its rivals define the leaders and laggards in a vast library that changes little day by day. Sites rise in rank as they receive more visits and as other sites link to them.
The challenge is to marry this orderly Web with the blogosphere's never-ending flood of perishable posts. Searching the blogs for key words shouldn't be so hard. The challenge, though, is figuring out which posts should rank atop the results. Should it simply be the most recent? Or from a popular blogger? Or perhaps a blogger already bookmarked by the user, or even by those on the user's buddy list?
Search engines may initially wall off blogs from the rest of the Web. But the more ambitious goal is to provide readers -- and advertisers -- with just the right blend. This work will push the search wizards to spin their most dazzling algorithms yet. But those who pull it off stand to become powerhouses in the blog world, dispensers of every day's buzz.
By Stephen Baker in New York