Building a Smarter Search Engine

Vivisimo's Clusty aims to sort through the confusing welter of Web results and cluster them by topic, speeding users on their way

By Heather Green

Some of the best inventions are inspired by frustration. That was certainly the case with Raul Valdes-Perez' search-engine technology.

While watching an academic presentation of video-search technology at Carnegie Mellon University six years ago, Valdes-Perez, then a full-time computer-science professor, became exasperated with screen after screen of seemingly nonsensical results. "Wherever we looked, information seemed to be disorganized," says Valdes-Perez.

So, along with two other CMU researchers, he set out to come up with a smarter way to return search results. Armed with their research in using artificial intelligence to help organize scientific discovery, the three computer scientists founded a search startup four years ago. Called Vivisimo, it provides search technologies for organizing the computer networks of government agencies such as NASA and companies including Johnson & Johnson (JNJ ) and Cisco Systems (CSCO ).


  Now, Pittsburgh-based Vivisimo is trying to streamline search for consumers with a new service called Clusty. Since its launch three months ago, Clusty has generated buzz for its clean design and clever approach. Using artificial intelligence, Clusty groups search results into different categories. The idea is to help people quickly navigate through the flood of results a general search engine lists without any organizing principle beyond popularity.

Clusty provides a promising new approach to wading through the ever-increasing amounts of information available online. "It's a nice way to do breakdowns of searches," says Greg Notess, who runs the online guide Search Engine Showdown. "They are pushing for new ideas in search."

To understand how Clusty works, it's a good idea to first describe how other search engines work. When someone searches on the word "seal" in Google (GOOG ), the list of results jumbles together news and sites about Seal the musician, different kinds of certification seals, the Wet Seal clothing brand, the Navy SEALS, and, of course, the sea-faring mammal. That's because Google works by listing the most popular Web pages that contain the search terms for a general query.


  Clusty also provides this laundry list of results. But on the left side of its results Web page, it provides folders entitled Navy, Music, and Harbor Seal. By clicking on any of these groups, individuals drill down into more topic-specific results.

To pull together the results, Clusty uses metasearch technology, which means it searches the results of other search engines and indexes, including Microsoft's (MSFT ) MSN and Yahoo's (YHOO ) Overture. Then it applies the artificial intelligence to pick out the major themes found within the results for each search and organizes them into folders.

Despite its innovative technology, Clusty faces an uphill battle. Google, MSN, Yahoo, and AOL (TWX ) dominate the search market, and they're engaged in an ever-intensifying, extremely expensive war. They're adding more services and features, including desktop and multimedia search, at a dizzying pace. It's tough enough for more established search engine rivals, such as Amazon's (AMZN ) A9 and Ask Jeeves (ASKJ ), to gain mind share, let alone the new crop of upstarts that includes Jux2, Accoona, and Snap (see BW Online, 12/9/04, "Accoona: Should Google Worry?").


  To some analysts, Clusty's best strategy for becoming a mainstream alternative may be to start with a narrow focus. Instead of offering the general search engine it now provides, industry experts say it might do better to focus on providing niche consumer search engines, focused on health care, travel, or medicine.

By building a brand in different niches, Clusty could gain a steadily growing, loyal audience that could then be wooed to a more general service. "When you have the group of four that has a stranglehold on the majority of searches, you have to be innovative if you want to break in," says Fredrick Markini, CEO of search-marketing firm iProspect. "The best way to break in is with a stellar vertical engine."

Valdes-Perez argues that there's room for a new service for the masses that can take the work out of search. He's betting that, since Clusty cuts down on the hassles of crafting queries carefully and wading through long lists of results, it will attract converts and advertisers.


  Founded with a $1 million grant from the National Science Foundation, Vivisimo is drawing interest from outside investors. It has been approached by scores of venture-capital firms. But because it has been profitable for a couple of years, Valdes-Perez sees no need to bring in investors for the moment. In these days of battles between the big search players, analysts also speculate that Vivisimo could be an acquisition target for an established search company looking for new technology.

To gain a broader audience, Vivisimo, which already licenses the clustering technology to consumer online publishers such as the message-board search service and e-commerce site FatWallet, plans to push licensing to even more popular consumer Web sites. "We approach [the competitive market] by offering a very different product and trying to articulate it clearly," says Valdes-Perez, who is chief executive at the company he co-founded.

After years of working on a way to making search more productive -- and less frustrating -- Valdes-Perez wants the world to see what an easy-to-use search engine really looks like.

Green is Internet editor for BusinessWeek in New York

Edited by Jim Kerstetter

Before it's here, it's on the Bloomberg Terminal.