Does Asia have to rethink its Internet architecture after the Boxing Day earthquake?
"I was starting to get calls from 9 pm. When my guy called up and said the cable's broken, I thought, 'my holiday is gone'."
That's how the news of Asia's worst cable disaster broke for Andrew Kwok, vice president international business for Hong Kong telco HGC.
He wasn't the only one to spoil his holiday. Around the region, hundreds of engineers worked day and night to reconnect lost bandwidth after a once-in-a-century earthquake shredded Asia's trans-Pacific connectivity.
The impact on broadband-happy east Asia was catastrophic. Hong Kong and Singapore lost 80%-90% of connectivity, Taiwan nearly 100%.
For 48 hours Internet users had virtually no offshore connectivity. By the weekend the Web was back up, but extremely slow. After a week or so, with new paths running to the US via China, Europe or Australia, it began to approach normal.
Questions since have rumbled through the industry. Why were so many cables placed in the same location? How can we diversify our connectivity? Will it happen again? Can subsea capacity avoid quake-prone zones?
The story unfolded like this.
THE BREAKCables break slowly. At first some water gets in, then some fibers are torn, then some more and then soon enough the cable is completely out.
After the earthquake off Taiwan's southern tip at 8.30 pm on Boxing Day, the shifting seabed began to stretch some of the dozens of cable segment that run through it.
By lunch the following day, eight international cables had been severed in 16 places. As every Asian Internet user now knows, the prime role of those cables was to connect across the Pacific to the US, the largest single source of Internet content.
The result was Asia's biggest ever loss of Internet capacity. Users were unable to access major US sites such as Google, Yahoo, YouTube and CNN.
Echoing a universally-held view, VSNL vice president of global transmission services Byron Clutterbuck said it was fortunate that the event occurred in one of the quietest periods of the year.
But some did better than others out of the break.
Like Asia Netcom. The regional carrier lost two cable segments on its EAC system, which runs between Singapore and Japan, the first going down about 2 am on the 27th and the other at 11 am.
But the Hong Kong-based operator was the least affected. ANC president and CTO Wilfred Kwan attributed it to the multiple redundancy paths created by EAC's rings as well as recent completion of a 50% upgrade.
It was also a matter of luck, too, that the system was hit in two segments but could still operate.
A spokesman admitted ANC had been "lucky" with the upgrade, too, but noted it underlined ANC's advantage in managing its own infrastructure.
"The club carriers are also planning upgrades, but they take a long time to go through. Wilfred can sign off on an upgrade for ANC," he said.
ANC restored its lost capacity within 24 hours and went on to restore 100 Gbps around the region - mostly for its own customers, but also for other carriers and ISPs.
HGC's Kwok says the first 48 hours following a cable disruption are the most important period. "After more than two days you will end up piggy-backing on other carriers' restoration paths."
THE RESTORATIONRedundancy is built into cable systems. When a simple break occurs, the packets back up and go another way.
The Boxing Day disruption created a cascade effect, with packets defaulting to the next available path but eventually running out of options. It was then up to the networks and ops guys to find paths through.
The priority was voice, especially in the middle of the Christmas season.
That happened pretty quickly - within 24 hours at Reach, says operations director K.F. Larm.
Finding the data links takes longer. Larm says the key is good relationships with other carriers. Reach leased spare capacity from China Telecom, routing traffic up to Shanghai and then onto Japan.
"It just takes one call. I have their mobile numbers," he says of his contacts at Guangdong Telecom.
HGC, which has more than 60 carrier and ISP customers worldwide, says it lost 50%-60% of its capacity outside mainland China. It restored 98% of its corporate circuits within one and a half days.
Kwok said HGC also provided capacity to other carriers around the region, including Taiwan's Chunghwa Telecom.
HGC ran its replacement capacity through four ways. One, like Reach, is up through Shanghai and onto Japan, with some also going through Russia.
The other way is down through Singapore, and then either splitting off to the US via Australia or the long way back through Europe.
Kwok says HGC keeps basic commercial and technical agreements in place for just such an emergency. They are revisited and refreshed every month or two.
The disruption was less drastic for Indian-based VSNL, one of the world's biggest owners of subsea capacity, which opened up a "back-route" to the US via Europe.
"We are fortunate we own a lot of capacity on other systems. The cost side is more internal - the opportunity cost," said Clutterbuck.
He also dispelled the idea that satellite was a potential source of diversity, noting the latency on a 60,000-kilometer roundtrip and the much higher costs of satellite bandwidth.
But he complained that some carriers were exploiting the situation "to make a quick dollar," charging up to five times more than the regular prices for bandwidth.
Although Clutterbuck didn't single out any firms by name, other carrier executives confirmed that he was referring to firms such as Asia Netcom, which had capacity to spare after the earthquake struck.
He added: "This time we've seen some price-gouging - an opportunity to make some quick money in a short period of time. When something like this happens again, the carriers that worked together on this one will work together on the next one.
"Those guys have been lucky this time, they won't want to be unlucky next time."
However, others regard the price hikes as business as usual. Ovum analyst Matt Walker said they were "normal cases of price-gouging that result from an extraordinary circumstance. The telecom market is far from a perfect one, especially for services that require hundreds of millions of dollars of multi-year investment to create."
And an international carrier executive, who asked not to be named, said: "Everyone does it - Reach, VSNL, ANC, FLAG, they all put their prices up."
THE NEXT TIMEJust how did it happen that every Asian subsea cable got hit by a single event?
Consultant John Hibbard said the loss of bandwidth from a Taiwan earthquake should have come as no surprise.
"Three years ago we had an earthquake in the Taiwan area and we lost five [cables]," he said.
"The fact that it's come back suggests that there's significant risk, and therefore particular care has to be taken in both network design and cable implementation to make sure you don't get multiple failures.
"It could be another 50 years [before another big quake], but I would not be betting on it," said Hibbard. "Something could happen in the next five years, certainly within the planning horizon of the current systems and on the new systems that are being planned."
Hibbard said one solution might be to go terrestrial across the Philippines, where cables already land, and run a cable east to Guam. Other executives said this might work, but it meant relying on the fraught local telecoms environment, where the intense competition between PLDT and Globe Telecom makes operating difficult for foreign carriers.
In fact, most in the industry defend the current cable routes.
Kwok says the location is "very logical." Unlike the waters of the Taiwan Strait, which are shallow and full of fishing boats, the cables are laid in waters more than 4 kilometers deep. Not a single cable runs through the Taiwan Strait, which also carries some political risk.
"I don't think there's a better route," says Reach CEO John Wright. "Going to the US you don't really have a whole lot of options."
Some people, like Kwok, predict that this will accelerate fresh deployments. ANC, for one, announced in January a new trans-Pacific cable with design capacity of 2.56 Tbps which will link the Philippines directly with Japan.
But if there is a lesson from the Boxing Day earthquake, it might be that the best-equipped are carriers that are asset-light and use multiple cable systems.
Worst-placed are cable owners that rely on their own one or two cable systems.
"You put your traffic in as many cables as you can," says Kwok. "You plan a diverse network and plan for outages as well."
VSNL's Clutterbuck says he will keep a permanent back-route through Europe as backup for his trans-Pacific link, even though it's "a little more costly and a bit slower."
"It's a choice of having some delay or not having any connectivity at all."
IN THE QUAKE ZONENortheast Asia is one of the most active seismological zones in the world, making it a dicey area for telecom companies laying subsea cables.
Yet Taiwan's strategic location makes it unavoidable for cable builders wanting to connect Asean with northeast Asia and across the Pacific to the US.
Just as the Taiwan Strait between the island and the mainland is a critical shipping channel, the Luzon Strait to Taiwan's south is jam-packed with subsea fiber. It's also the shortest and most economic route between Asean and the US.
Most of Taiwan's seismic activity takes place in the center and north of the island.
The quake that hit at 8.26 pm on December 26 was in fact in Taiwan's most seismically stable area. On average Taiwan's south coast sees less than one significant quake a year - defined as Richter 5 or above by the US Geological Survey (USGS) - compared with four to five in north and central parts of the island.
The Taiwan Central Weather Bureau Board reported a quake of 6.7 on the Richter scale about 23 kilometers south-west of Hengchun Taiwan's southern tip.
Other agencies measured it differently: the USGS estimated it at 7.1, and the Hong Kong and Japanese authorities at 7.2. USGS said a second quake, measuring 6.9, and took place eight minutes later in the same region.
In any case, it was the largest quake in the area for 100 years, according to the Taiwan Central News Agency.