‘We Assume the Bad Thing Has Already Happened’

EMC is one of the world’s largest makers of data storage systems. Like other companies, it’s also being hacked with alarming frequency. Meet the team fighting back—before it’s too late.

Garrett Schubert leads a team tasked with defending EMC against cyberspies.
Photo Illustration: Jordan Hollender/Bloomberg Markets

Garrett Schubert leads a team tasked with defending EMC against cyberspies.

This story appears in the July/August special Rivalry Issue of Bloomberg Markets magazine.
This story appears in the July/August special Rivalry Issue of Bloomberg Markets magazine.

As colorful slides flash above him on a large screen, Garrett Schubert is talking quickly. It’s a habit he’s picked up on a job in which speed is the difference between stopping a computer network breach or getting there after the data is already gone.

Schubert and I are in a large, second-floor conference room, behind three locked security doors. Located in an innocuous glass-and-concrete building in a wooded suburb northwest of Boston, the conference room is deep inside EMC’s Critical Incident Response Center.

EMC, one of the world’s biggest makers of data storage systems, is a particularly juicy target for cyberspies. With revenue of $24.4 billion last year, the company is a Big Data icon, the leading provider of products and services for mass storage and analysis. Intruders see EMC as a potential gateway to the secrets of banks, technology companies, casinos, power plants, militaries, and governments.

Every day, devices protecting EMC’s 60,000 computers register 1.2 billion “events,” a broad term that includes probes by hackers looking for vulnerabilities to exploit later. Between 60 and 80 of those events are serious enough that they’re assigned to someone on the incident response center’s 28-person team for action.

About eight times a year, a breach is elevated to what EMC calls internally a “declared incident.” It’s the corporate equivalent of DEFCON 1. Hackers have been identified inside the network, possibly already stealing data. The company makes almost none of those white-knuckle events public.

EMC executives have agreed to lift the veil on their computer protection operations run by its security division RSA, and sandy-haired Schubert—who hunts for hackers in EMC’s computers as the center’s manager—is my guide on a recent afternoon. His slide deck comes to a finale with details of a previously undisclosed attack in 2014 by nation-state cyberspies. It was a stunningly complex operation. The hackers infiltrated a Korean-language news site, which an EMC engineer in South Korea had visited, infecting his laptop with sophisticated malware. When the engineer plugged his computer into the data centers of sensitive clients during his regular visits, the malware could have possibly jumped across the connection. EMC believes one of those clients, possibly linked to the South Korean government or military, may have been the intended target.

Schubert pauses to let that sink in: The hackers had to know something about the reading habits of a midlevel EMC employee and were willing to hack two other companies to get the data they wanted. “Some of these bad guys are brilliant,” says Schubert, 34. “The code that they write. The way they manage large compromised systems. I admire someone who makes my job difficult.”

  

 

 

Garrett Schubert
Garrett Schubert
Photographer: Jordan Hollender/Bloomberg Markets

 

Schubert and his colleagues face their adversaries from the confines of a 30-by-20-foot (9-by-6-meter) room—known as a security operations center, or SOC—where data from the company’s global computer network is piped to analysts who staff the center and a twin facility in Bangalore, India, around the clock. Next door to the SOC is the conference room with a mini-fridge in the corner and a large display screen on one wall. During declared incidents, managers are able to project data related to the crisis onto the screen. The space becomes a war room populated by a team of as many as 40 EMC and RSA employees, including the company’s general counsel and chief information officer, who are responsible for regularly briefing the board of directors.

The seriousness with which EMC takes those incidents shows that corporations are awakening to the huge vulnerabilities in an age when all our secrets are on servers somewhere, an awareness that has grown rapidly during the past 18 months.

The recent string of grim headlines in that time, involving Target, JPMorgan Chase, Anthem, and Sony Pictures, gives the impression of devastating, one-off attacks. Target, which was raided by hackers in December 2013, was also equipped with an SOC and high-end technology and, in fact, spotted the malware on its computers almost immediately. The alert lingered in an analyst’s work queue for several days, long enough for the hackers to remove 40 million customer credit card numbers. The company’s CEO resigned five months later.

In reality, what U.S. companies and their defenders, including EMC’s Schubert, are facing isn’t open warfare. It’s a siege.

 

 

Early on a Monday morning in April 2014, an EMC analyst working in the SOC stumbled onto an intruder’s faint digital footprint. It’s known among security experts as a web shell—a piece of computer code that gives hackers autonomous control over part of the network. This one was planted on a single server in one of EMC’s West Coast data centers. It took the global security team less than 30 minutes to declare an official security incident.

That decision was based less on the where than the who. There are all sorts of hackers. Criminals are after money. Competitors may target EMC’s product designs. But the ubiquity of EMC’s hardware makes the company a target of hackers in the employ of rival nation states. Knowing how EMC’s systems are designed or obtaining the source code can allow hackers to penetrate them once they’re inside the data centers of hospitals, banks, power plants, and military facilities. In the coming era of cyberwar, EMC sits squarely on contested terrain.

The web shell spotted by the EMC analyst was traced to one of about a dozen groups that most worry James Lugabihl, head of EMC’s Critical Incident Response Center and Schubert’s boss. He has his own rather ominous nickname for them: the Immortals. Employed by various nation-states, those 12 or so groups are the most closely tracked by EMC’s security team because of the enormous damage they can do in just a few hours. “When it comes to these guys, we aren’t going up against a double-A farm team,” Lugabihl says. “These guys are major-league.”

As the security team watched, beginning in the early hours of April 21, 2014, the intruders repeatedly signed onto the server, performed a few commands, and left again. It looked like the hackers were in the early stages of their operation, Lugabihl says, but the team’s members needed to be sure. After some deliberation, they made a decision: They’d let the hackers work so that EMC’s analysts could learn more.

Although it’s not an uncommon tactic, it’s a nerve-racking one. “It’s the right thing to do, but the board of directors is still going to look at you like you’re crazy,” says David Martin, RSA’s chief trust officer, whose relaxed manner suggests an unnatural calm compared with the hubbub around him. If the team were to miss something crucial, the hackers could bag the crown jewels while the defenders were looking the wrong way. “It’s like catching someone in your house stealing the silver in the dining room,” says Martin, “but you decide to let them keep going because it may lead you to the fact that there is also someone in the office rifling the safe.”

After two days, the hackers tried to remove data from the network but were halted by additional safeguards the defenders had by then put in place. By April 25, the team’s members decided they’d seen enough and shut down the server.

Even with all the preparation, it was an anxious week, Lugabihl says, filled with the kind of tension that pervades this line of work. “When the adrenaline wears off, you can literally just see people collapse,” he says. The company carefully monitored the network for another 60 days before EMC finally determined it had thwarted the attack.

  

 

A server room at EMC/RSA
A server room at EMC/RSA
Photographer: Jordan Hollender/Bloomberg Markets

EMC hasn’t always been so fortunate. In 2011, Chinese hackers infiltrated the servers of the company’s security division, RSA, and stole information related to a product widely used by banks and governments to protect their own data. EMC executives squirmed as the media rolled out details of the hacking of a company paid to fend off hackers, potentially leaving some of its most important clients vulnerable to further attacks. After that, EMC and RSA’s approaches to security were completely revamped. “We had to change our whole mentality,” Schubert says. “We honestly built our team not thinking that nation-states would ever target a place like EMC or RSA.”

Schubert, Lugabihl, and their colleagues know the intruders have most of the advantages. The more than 100 experts who cumulatively work on computer security at EMC and RSA have to be right every day, all day; their opponents have to be lucky only once—and are usually far better than that. “Our products and services are used in just about every part of critical infrastructure around the globe,” says Martin. “By its very nature, that puts us in the crosshairs.”

Although none of the defenders have ever met their adversaries, they do get to know some of them. Hackers have personalities that show up in the tactics they use—their digital habits, if you will. It’s like playing a high-stakes game of chess with an opponent sitting a continent away. That doesn’t mean they’re content to play nice. Schubert tends to speak in the language of the battlefield. Corporate secrets are “high-value targets”; hacking programs from random criminal attacks are “stray rounds.”

While the Immortals are the adversaries that most worry Schubert, they aren’t the only ones. Criminals have created a sophisticated hacking supply chain that rivals those of some nation-states. Russian malware engineers sell their wares to Ukrainian hackers, who buy space on hijacked servers from the Dutch criminal underground.

Much of what cybercriminals do is automated and aimed at stealing easily convertible data such as credit card numbers to maximize their take. One, in January, involved hundreds of EMC employees who received malware-laden e-mails to their corporate accounts. The company’s spam filters are designed to stop such attacks, so the hackers sent 23 different formats of the same e-mail to each account, slightly altering each in order to make it through the company’s digital defenses. But this particular malware also stole users’ security certificates, digital keys that identify them to other entities on the Internet as legitimate EMC employees. Those certificates could be sold later to sophisticated hackers or spies, making this attack more dangerous than the average criminal caper.

The company’s retooled approach to security is especially apparent when it comes to declared incidents, each of which is given a code name. In 2011, they were all the names of animals. Last year, in a nod to variety, the incidents were named for characters in HBO’s Game of Thrones. The April 2014 incident was called Daenerys Targaryen, or DT for short. The Korean attack was Eddard Stark.

 

'We have narrowed down a 12-hour analysis to 10 or 15 minutes,' says Schubert.
'We have narrowed down a 12-hour analysis to 10 or 15 minutes,' says Schubert.
Photographer: Jordan Hollender/Bloomberg Markets

It would have been easy for Schubert’s team to miss the Eddard Stark hackers. An analyst who had a few spare minutes was scanning outbound connections from company computers to the Internet and spotted something wrong. A laptop in South Korea was connecting to a site called mooo.com, which had been tagged by the intel team as dangerous.

The team worked quickly after spotting the connection, the past several months of preparation paying off. Typically, the defenders would have had an infected laptop shipped to Massachusetts by plane to examine the hard drive. But EMC had begun to install an RSA tool on company computers that allows its engineers to make the analysis remotely. “We have narrowed down a 12-hour analysis to 10 or 15 minutes,” Schubert says.

That didn’t give the hackers enough time to burrow into the company’s main network. After examining the laptop more carefully, though, the team found evidence that a file of documents had been removed, which could include sensitive information about clients.

Worse still, the engineer had spent two days working remotely prior to plugging into EMC’s network, when the breach was ultimately detected (engineers often work remotely, logging into clients’ networks as part of the job). The company contacted those clients to discuss the situation; Martin says those conversations are among the most difficult in his line of work. “Imagine sitting in front of your customer and explaining that you thought so little of them that you put their data at risk,” he says.

Schubert concedes that his team, as hard as it works, won’t always win. There are simply too many hackers, and they have far too many advantages. The best the defenders can hope for is to limit the damage.

“When I started in my career, the idea was, we wanted to stop a bad thing from happening,” Schubert says. “Now, we assume that the bad thing has already happened. Every single day, we walk in and we assume there is an active attack going on.”

This story appears in the July/August special Rivalry Issue of Bloomberg Markets magazine.