A powerful storm knocks out power across Big City, U.S.A. Businesses are left scrambling. Without electricity, assembly lines, data centers, and office computers are useless. Every minute in the dark means thousands of dollars in lost revenue.
Such a dire scenario is why many companies, hospitals, and government agencies keep backup generators at the ready to power up their operations during outages. Although rarely used, generators can make all the difference in whether a business can function during a blackout.
But buying and installing a backup generator is just the first step. Ensuring that it will work properly when needed requires regular testing. Unused generators, like cars that sit parked too long, are prone to failure. Mistakes in fuel storage can also cause serious problems.
“Generators are a huge cost up front,” says Raouf Abdel, head of operations in the Americas for Equinix (EQIX), a company that has hundreds of generators at the data centers it operates across the globe. “They are very expensive to maintain and run, and you only use them in emergencies. When they don’t work, it’s frustrating.”
Hurricane Sandy, which walloped the Northeast in October 2012, exposed the consequences—some of them potentially life-threatening—of backup generators failing. Three New York hospitals wisely installed their backup generators above street level to keep them from being flooded. They had placed their fuel tanks and pumps in their basements, however, which were inundated by the storm surge. After the power went out, vital hospital equipment such as medical monitors and ventilators no longer operated. Hundreds of patients had to be evacuated during the height of the storm. A similar scenario played out in New Orleans hospitals in the wake of Hurricane Katrina in 2005, leading to a number of patient deaths.
Failed generators also played a big part in Japan’s Fukushima nuclear meltdown in March 2011. Backup power for the plant’s reactor cooling system kicked in, as it should have, just after the initial earthquake. But the tsunami that arrived an hour later destroyed the fuel tanks.
Arshad Mansoor, senior vice president for research and development at the Electric Power Research Institute, an industry-funded research group, estimates the failure rate for backup generators is as high as 15 percent. Human error, either from poor planning or inadequate testing, plays a major role, he says.
Testing generators only monthly or quarterly to make sure they start is not enough, Mansoor says. “Load testing,” in which generators are used to power facility equipment the way they would during a real blackout, is essential. Companies sometimes balk at such tests because switching to backup power adds an element of risk. Hospital administrators, for example, may not be keen on switching operating rooms to backup power during surgery if they can avoid it—particularly if there has been a glitch during testing in the past. “Unless you take the risk every month, your chance of failure will increase,” Mansoor says.
Power outages are a part of doing business. U.S. consumers lose power for an average of 214 minutes annually, according to a study by professors at Carnegie Mellon University. Some areas are more prone to outages than others, based on weather, terrain, and utility maintenance. For example, Northeasterners experience more than twice as many minutes without electricity annually as people in the Midwest, according to a study by the University of Minnesota. It doesn’t always require a blackout, however, for businesses to switch to backup power. Some turn on their generators if the electrical voltage from the utility fluctuates beyond acceptable levels.
Because of their critical role, hospitals and financial companies are required to have backup power available in case of outages. Essential government services such as air traffic control towers, jails, and sewage-treatment plants generally have similar equipment in place. As a precaution, many facilities also have backup batteries to tap for power, but they are usually only able to provide it for a few hours.
Industrial backup generators, some of which are as big as shipping containers, can cost hundreds of thousands of dollars. Facilities that use a lot of power, such as data centers, may need several of these generators to maintain normal operations. As with any piece of mechanical equipment, basic maintenance is important. Oil levels, fuel pressure, and drive belts are just a few of the items that require checking.
Failures are still possible, though, as illustrated by a brief blackout in San Francisco in 2007 that took down several websites, including Craigslist and Yelp (YELP). The sites housed their servers in a San Francisco data center that relied on 10 backup generators for emergencies. Three of the generators failed to start because of a technical glitch, the data center’s operator, 365 Main, said after a follow-up investigation. The sites affected by the outage took anywhere from an hour to 12 hours to get back online.
In most cases, backup generators work as intended during power outages. Equinix had eight facilities in the Northeast go on backup power during Sandy. The generators switched on automatically as they were supposed to when the blackout struck, says Abdel, the Equinix executive. The length of the power outage, however, added a dimension of complexity. Usually there’s only enough diesel stored on site for generators to operate for 48 hours. One of the facilities, in Secaucus, N.J., ran on generators for five days. Equinix had to arrange for daily diesel deliveries amid the widespread damage in the area. One of the generators broke down after several days of use, but the facility had extra generators in place that could take over, as is common practice in data centers.
“We have a backup for the backup,” Abdel says.