FireEye: Botnet Busters

Alex Lanstein stared at the 65-inch computer monitor in the living room of his Boston apartment. Streaming data lit up the screen, the actions of a cyberlord giving orders to his botnet, a zombie army of hijacked computers controlled from an unknown location . It was early in the morning of Mar.16. The 25-year-old cybersecurity analyst had spent months preparing for the events soon to unfold. His reddish hair still matted down from sleep, Lanstein stood up and poured another cup of coffee. Suddenly, the data stream flickering on the monitor became dark, and a smile curled across Lanstein's stubbly face. Operation Rustock had begun.

Lanstein's employer, FireEye, is a Silicon Valley company that defends corporations and governments against targeted malicious software, or malware. FireEye's clients include Fortune 500 companies—Yahoo! (YHOO), EBay (EBAY), and Adobe Systems (ADBE), among them—and members of the U.S. intelligence community. The company had recently shut down some of the highest-profile spam-blasting organizations, winning recognition for imposing order on a generally disordered and unpoliced world.

Now, Lanstein and FireEye were chasing their mightiest target to date, the Web's most sprawling and advanced spam machine, called Rustock—pusher of fake pills, online pharmacies, and Russian stocks, the inspiration for its name. Over the past five years, Rustock had quietly—and illicitly—taken control of over a million computers around the world, directing them to do its bidding. On some days, Rustock generated as many as 44 billion digital come-ons, about 47.5 percent of all the junk e-mails sent, according to Symantec (SYMC), the computer security giant based in Mountain View, Calif. Although those behind Rustock had yet to be identified, profits from it were thought to be in the millions. "The bad guys," is what Lanstein had taken to calling them.

For months, FireEye plotted a counterattack, along with Microsoft (MSFT) and Pfizer (PFE)—Rustock was peddling fake Viagra, as well as sham lotteries stamped with the Microsoft logo. Working from FireEye's intelligence, U.S. Marshals stormed seven Internet data centers across the country, where Rustock had hidden its 96 command servers. Microsoft lawyers and technicians were there, too, along with forensics experts. Another team had been deployed in the Netherlands to destroy two other servers.

The sting was executed flawlessly, with everyone pouncing at once. And yet Rustock somehow fought back. From an unknown location, perhaps in Eastern Europe, the botmaster remotely sneaked back into its spam network, locked out Microsoft's technicians, and began to erase files. Clearly, those behind Rustock didn't want anyone seeing what was inside those hard drives.

After a struggle lasting about half an hour, the technicians finally wrested back control of the server. Lanstein's cell rang. T.J. Campana, senior manager for investigations for Microsoft's Digital Crimes Unit, told him it was over. "The bad guys lost."

Global spam levels plummeted as Rustock was taken off line. At the same time, the cybergeek community saw that something significant had happened. Who killed Rustock? And how? For two more days, Lanstein was under order from a federal court in Seattle to keep silent as a way to defend against any leaks to the enemy. Even later, when he could talk, some of the biggest questions remained unanswered, such as who was behind Rustock, and what would he, she, or they, try next?

According to FBI data cited in U.S. Senate testimony, annual cybercrime profits and damages have hit a trillion dollars. That's an incredible number and impossible to itemize, but even conservative estimates place direct losses—and lost revenues—in the billions.

Botnets such as Rustock are increasingly the tools for these crimes. To build a botnet, hackers send out a program, often disguised as a link or hidden in an e-mail attachment, that infects the host computer and communicates invisibly with a command machine. Thus linked, this network generates hard-to-trace spam and goes "phishing" for user passwords and company secrets. They conduct denial-of-service-attacks, in which a hacker cripples a website with a flood of junk messages, and they can be designed to disrupt, for instance, national electrical grids. Symantec estimates there are about 3.5 million to 5.4 million botnets worldwide.

In May, Sony was hacked, as was Google, Lockheed Martin, and two of South Korea's largest banks. In early June officials from the International Monetary Fund disclosed that its computers had been breached. Citigroup made a similar announcement days later. "The amount of malware and malicious activity on the Internet is so prolific it's a daunting task just picking where to start," says Steven Adair at Shadowserver Foundation, a nonprofit that tracks the Internet's criminal activity.

That's where FireEye comes in. One of the world's most effective private cybercrime fighters, the 120-employee company is based in a technology park in Milpitas, Calif. While it won't disclose profits, it says it is cash flow positive. On a spring day, not long after the Rustock bust, FireEye founder and Chief Executive Officer Ashar Aziz sits in his office, explaining what keeps him up at night. "I worry about Cybergeddon—this virtual mushroom cloud," he says at one point. His eyebrows jump as he leans forward in his leather lounge chair. "Your bank has no money in it. It's all gone. What would the world look like?" Aziz thinks about such scenarios a lot. "I'm not paranoid," he says. "There are thousands of these infrastructures out there."

Aziz, 52, was born in Karachi, Pakistan. The son of novelists, he attended the Massachusetts Institute of Technology, completed a master's in computer science at the University of California at Berkeley, and then joined Sun Microsystems as an engineer. In 1999 he started a cloud networking company called Terraspring, which he sold to Sun in 2002.

When he launched FireEye two years later he had already come to believe most corporations, and even governments, were under electronic siege. Most used traditional antivirus software and firewalls based on the idea that if computer A gets infected the network will learn from it, and computer B will be protected. But that was the era of viruses and worms that, once identified, could easily be exterminated. Newer malware transformed itself each time it struck, defying simple pattern matching. "What cybercriminals did was they said, 'I'm going to change my form every time I come knocking on your door,'" Aziz explains. With several engineers he built a system that blocks out the known infiltrators and, in a virtual network of PCs running in parallel, sifts through real-time traffic for unknown attackers and blocks them, too. People at FireEye call it "YouTube for malware" because it produces a view of the malware's life cycle: how it behaves in the network, what it's looking for, which criminal servers delivered it, and which control servers it calls out to for orders.

"My thing is that you are insecure," Aziz says. "People don't like to hear that." Sequoia Capital and Norwest Venture Partners invested $6.5 million in Aziz's company. Soon after, the CIA's investment arm, In-Q-Tel, bought in as well (the amount is not public), and FireEye began defending the federal government. Over the last seven years the company deflected some of the most destructive online intrusions: Aurora, the China-originated hack attack that penetrated Google and other technology firms in 2009; Coreflood, the botnet that had been stealing millions from global bank accounts since at least the mid-2000s; and Zeus, a program that started using personal information to steal hundreds of millions of dollars from financial institutions in 2007.

FireEye has made itself an indispensable ally of U.S. business, providing security systems for $50,000 to $100,000 a pop. In addition to Yahoo and other tech companies, FireEye clients include U.S. universities such as Harvard and Berkeley.

While inextricably linked to FireEye's security business, the dismantling of botnets is unpaid work; Aziz says he's never made a cent from it. The botnets FireEye catches in its system are categorized, uploaded to a database, and then geolocated, producing a kind of intelligence few have ever seen—a massive real-time picture of the Web's criminal landscape. A crime cloud. Aziz says he feels an obligation to use this cloud to foil smarter and more brazen cybergangsters. "There's no business model for this," he says. "We are all citizens of the Web."

FireEye's first target was a 19-year-old Russian hacker named Nikolai (or Kolya) McColo. In 2004, McColo had opened a data center in a shiny tower in San Jose. All manner of online criminals based their activities there: phishing operators, child pornographers, spam engines. Nikolai himself may have been raking in hundreds of thousands of dollars a month, according to FireEye's estimates. In 2008, about a year after he had joined FireEye, Lanstein and then-Washington Post reporter Brian Krebs helped convince Internet providers to take Nikolai's servers offline. (Nikolai himself died in an auto accident in 2007.) Several of the largest botnets in the world, collectively controlling millions of computers, scattered for cover, looking for new servers.

Next, FireEye focused on a botnet called Srizbi that controlled half a million zombie computers, sending out millions of junk e-mails a day. Lanstein appealed to officials in Estonia, where Srizbi's control servers were based, and they shut them down. Then Aziz was able to purchase the botnet's backup domains, allowing FireEye to stop it completely. A year later, FireEye also shut down Mega-D, or Mega-Dik, which had taken over as the reigning blaster of male-enhancement and lottery-scam spam.

In November 2009, Lanstein started talking with Microsoft's Digital Crimes Unit. Both groups were obsessed with a spammer who, among other things, was abusing Microsoft trademarks and copyrights and overloading Hotmail. This was Rustock. Together, Lanstein says, "we decided to go after them."

Rustock's spam avalanche was more than just an annoyance. It was expensive for companies to filter out the daily deluge of junk and to clean systems infected by malware, not to mention the damage done to the brands associated with offers for fake Rolexes, Viagra, and whatever else.

As Lanstein and Microsoft studied it, they began to understand how Rustock functioned: Its malware slipped into unsuspecting PCs through pop-up ads and e-mails, exploiting a vulnerability in Internet Explorer and other desktop applications. Its rootkit, the software that makes contact with the controller's command servers, was a marvel. Botnet traffic on the logs of captured computers looked like innocuous postings on message boards. Lanstein also discovered that the botmaster had employed 96 different servers and spread them out to seven different data centers across the U.S. When one of the infected computers communicated with a command server in, say, Chicago, it would receive a lot less scrutiny than a call going from the same computer to a server in Ukraine or China.

Using the Lanham Act, a statute companies had employed to combat fake handbags and pirated DVDs, a Microsoft lawyer named Richard Boscovich filed a civil suit in the U.S. District Court for the Western District of Washington, alleging that Rustock was abusing its trademarks. With Pfizer now part of the team, Boscovich asked the court for the authority to seize the spammer's servers without alerting the owners ahead of time.

They got the green light, and on Mar. 15, the day before the raid, marshals did reconnaissance at the U.S. sites—in Dallas, Kansas City, Mo., Chicago, Denver, Scranton, Pa., Seattle, and Columbus, Ohio. The raid began promptly at 10:30 a.m., Eastern time, with Lanstein watching the botnet on the monitor in his apartment. Earlier that morning, Microsoft had quietly purchased 1,500 of Rustock's backup domains, allowing the team to seize the botnet once its main servers went dark.

"We have an order—step back!" the marshals yelled, badges out, as they entered the buildings, according to Boscovich, who participated in the Chicago raid. As the operation progressed, Microsoft noticed Rustock had remotely snuck into its Columbus servers and started wiping out files. "They came in the back door," recalled Boscovich. "It was freaky."

The technicians struggled with the enemy, which had wiped out their master passwords, locking them out. After 35 minutes of trying to break into their own computers, the technicians finally got fed up and literally pulled the plug on the whole system. Whatever was erased in the time it took Microsoft to finally reclaim control may never be discovered.

Cybersecurity experts believe the Rustock operation could be a technical and legal blueprint for future battles with online criminals. But much about the spam blaster and how it operated remains a mystery. "He made millions," Lanstein says, meaning whoever was behind it could probably afford to retire. "But I feel like the allure of cybercrime is too much for him."

Boscovich believes Rustock was probably overseen by several people, while Lanstein thinks it was just one person. Much of Rustock's equipment, it turned out, was leased to customers with addresses in Azerbaijan. Boscovich says the forensics analysis is leading them to Moscow and St. Petersburg. He also says Rustock had employed the handle Cosma2k to conduct business on the Net and had a WebMoney account under the name Vladimir Alexandrovich Shergin. Whether Shergin was a real name or not, no one knows yet, but WebMoney told investigators Shergin had listed an address in a small city outside Moscow.

On Apr. 6, Microsoft delivered its first status report to the federal court in Seattle in its lawsuit against Rustock. In the meantime the company purchased thousands more of Rustock's backup domains, further preventing its resurrection. On June 14, Microsoft published notices in Moscow and St. Petersburg newspapers, detailing the allegations against the spammer. The notices urged the individuals behind Rustock to respond to the charges or risk being declared guilty. By the time of its next court date in July, Microsoft may have enough information to reveal the identities of Rustock's masterminds.

Aziz predicts botnets will become smaller and more ephemeral. "They're being set up for one hit and then they disappear," he says. There are new threats emerging, to USB drives and SIM cards, among other hardware, which hackers are now using to deliver malware into networks. "The world is in this state of persistent insecurity," Aziz goes on. "It's just too easy for cybercriminals to bypass current defenses."

Complicating matters is the increasing involvement of organized crime and nation states, which bring vast resources, forcing FireEye and others in the security world to reimagine how they defend themselves and their clients. "This is one battle in a longer war in which the enemy will constantly morph," Aziz says. "But we definitely put them on notice."

    Before it's here, it's on the Bloomberg Terminal.