Feds, Industry, Battle the Biggest Bug
Four months after a public advisory warned of security vulnerabilities in a ubiquitous Internet remote management protocol, there have been no widespread attacks exploiting the holes. But technology companies and a special U.S. government panel are quietly evaluating the threat of related vulnerabilities in some of America's most critical electronic infrastructures, including the telephone network, the power grid, and the next generation of air traffic control systems.
On February 12th, Carnegie Mellon's Computer Emergency Response Team (CERT) issued a high-profile alert about serious security holes in dozens of implementations of the Simple Network Management Protocol (SNMP) -- the Internet's standard language for monitoring and controlling routers, switches and other devices. It was big news in itself, with nearly two hundred companies forced to evaluate, and in some cases patch, their products. Perhaps owing to CERT's careful behind-the-scenes advance coordination with vendors, months later there have been no reports of mass exploitation of the vulnerabilities.
But while the Internet-oriented CERT warned only about SNMP security holes, the research on which they based their advisory had farther reaching implications.
The CERT announcement was based on work performed last year by the Oulu University Secure Programming Group in Finland, a group that's perfected a technique of finding security holes in software by systematically flinging a wide range of unexpected values and illegally formatted data at it, and noting when, and how, it breaks. While their target was SNMP, the Finnish researchers' attacks actually hinged on manipulation of an even more fundamental and common language -- on which SNMP is built -- called Abstract Syntax Notation One (ASN.1).
Originally developed in 1984, ASN.1 is an internationally recognized standard for coding and transmitting complex data structures, similar to XML. The Oulu techniques worked by deliberately violating the rules of ASN.1 in a number of different ways -- lying about the amount of data being transmitted in a particular field, for example -- which would crash the vulnerable system, or in some cases, allow an attacker to overflow an internal buffer and execute their own instructions on the target machine.
It was the Internet and SNMP that got the press, but some experts, including high-level government officials, were immediately concerned that the same attack method might be equally effective against other networks and protocols relying on ASN.1. It's a long list, and includes some of the most critical systems in North America. The SS7 network that controls telephone call routing uses ASN.1 coded messages. Parcel delivery companies use ASN.1 to track their packages. Some credit card verification systems use it, as do digital certificates. And electric utilities use ASN.1 to control substations and transformers remotely.
So severe are the potential ramifications of widespread ASN.1 security holes, that President Bush was personally briefed on the matter, according to cyber security czar Richard Clarke, speaking at a meeting of the National Security Telecommunications Advisory Committee (NSTAC) last March. "When Howard [Schmidt] and I briefed the President on the ASN.1 vulnerability, he said to us, 'Don't wait for somebody to tell you that there's intelligence, or that there's a hacker group out there about to exploit the vulnerability because it will be too late then to fix it," said Clarke, according to a transcript of the meeting.
GOVERNMENT SECURITY AUDIT UNDERWAY. With that mandate, Howard Schmidt, former Microsoft security chief and newly-appointed vice chairman of the President's Critical Infrastructure Protection Board, created a full-time "Cyber Interagency Working Group" in February to examine the government's vulnerability to ASN.1 implementation holes. The group's initial goal, scheduled for completion this month, is to create an exhaustive inventory of vulnerable systems throughout the federal government. "The kind of information they're getting, it includes system name, system owner, type of system, vendor, name and version of the operating system, what patches are installed, and so forth," says a source familiar with the work. "It's a big effort."
At the March NSTAC meeting Schmidt described the working group as no less than "a tasking of a magnitude of something we've never seen before, either in private sector or in Government," according to the meeting transcript. Cabined by the National Communications System (NCS), a defense agency tasked with maintaining continuity of federal and emergency communications, the group's mission is in some ways akin to battling back the Y2K bug all over again, though on a smaller scale. The vendor of a particular product may no longer exist, forcing an agency to "remediate on-the-fly," said Schmidt. "We also have to look at the affected industries and build some consensus on what we're going to do, including public messaging. This has the potential to be very dramatic if we don't take the necessary steps."
Just how dramatic the holes might be at a practical level remains unclear -- the White House didn't return a phone call on the working group, and the NCS is mum on its current findings. "I don't have any authority to release any of that right now, because it's a White House dictate," says NCS spokesman Steve Barrett. But ASN.1 experts are taking it seriously. "There are things that one can do to defend against problems, such as putting rules in a firewall, but these are band aids in my mind," says Bancroft Scott, president of OSS Nokalvia, which makes ASN.1 programming tools. "The real solution is, you'll probably have to test these things and see if they have holes... Everything. This should have been done, of course, at day one. But here we are."
It's worth noting that most of the infrastructures cited by Schmidt rely on private networks, not the public Internet -- which at least throws up a small barrier to an attacker. And the same engineering blind spot that afflicted SNMP implementers might be less common in sectors where thorough testing is de rigueur. The Aeronautical Telecommunication Network, a next generation air-to-ground commercial aviation network, is built on ASN.1, but all the equipment and software has to meet the FAA's DO178B certification standard before deployment. "The tests are far more rigorous than what Oulu University created," says Scott. A spokesperson for ATN Systems, which is building the network, said he was unfamiliar with any ASN.1 issues, and that the system was scheduled for deployment this fall.
BORROWED CODE. In sectors already plagued by cyber security weaknesses, ASN.1 is just another item on an already long list. Electric utility companies use the protocol to remotely control some power equipment, and ASN.1 implementation is being examined as part of an ongoing cyber security program that grew out of Y2K remediation efforts, and took on urgency after September 11. "We're addressing that as part of a bigger effort to provide security enhancement for inter-control center communications protocols," says Massoud Amin, chief security researcher at the Electric Power Research Institute, the electric industry's think tank. "Existing communications protocols are being reexamined... all the way up to power plants, substations and control centers."
Meanwhile, supporters of ASN.1 are bracing for a public relations battle, as background noise from the government's remediation efforts sparks rumors that the standard itself suffers from congenital security flaws. In fact, there's nothing inherently wrong with ASN.1, except that so many programmers didn't plan for deliberately malformed messages. "There hasn't been a single person who's been able to identify a single problem aside from implementation problems," says Scott. "All this stuff about it being too complicated, it could be the simplest thing in the world, and if you don't implement it correctly, you'll have problems."
So why have the same security holes shown up in so many different implementations? Security experts offer a couple of reasons. Because of the standard's complexity, developers often use special compilers to generate the ASN.1 portion of their code, and a flaw in a compiler would pass like a bad gene to every application it creates. At least one commercial ASN.1 compiler was found to be vulnerable to the Oulu test suite, says Scott, though most, including his own company's, were immune. Additionally, programmers often borrow and reuse code from prior implementations of a protocol, or from open-source software, taking the flaws along with it.
But at its root, the problem may be that right people simply weren't looking. "ASN.1 is complicated, and the testing is never thorough enough," says AT&T researcher Steve Bellovin. "There were people who knew there were problems with the parse, but they weren't security people, so they didn't know it was a security problem." Counterpane CTO Bruce Schneier agrees. "You get what people look at and publish... and anything obscure isn't going to be looked at."
More efforts like Oulu University's might help, and one industry source says that the ASN.1 vulnerability has sparked discussions in Washington about the possibility of diverting some fraction of the supercomputing power at national laboratories like Los Alamos and Lawrence Livermore to the task of modeling and testing key communications protocols and the software that implements them. "There are a large number of people who share the administration's concern that the source of knowledge about the vulnerability was a Finish university," says the source. "Shouldn't it be a priority for the U.S. to generate that understanding and know-how from within?"
By Kevin Poulsen