Link to Original Article

Ice Storm freezes operations

By Tim Ouellette and Thomas Hoffman
JANUARY 19, 1998

Cold temperatures are supposed to boost computer performance.

But the massive ice storm that recently blasted parts of New York, New England, Quebec and Ontario put a freeze on many computer operations in those regions.

That's because one- to two-week power failures forced many businesses to operate with skeleton staffs, shut down, move to remote disaster sites or rely on generators that in some cases are being used beyond their normal capacity.

There are cold, hard lessons to be learned for IT. Disasters can occur without major physical damage to computers, and you can never have enough generators, backup power systems and staffing strategies in place. The best plan should include building relationships with generator makers and disaster-recovery vendors so you are in priority order when disaster strikes, IT managers said.

Early estimates of overall residential and business losses are at nearly $1 billion in Canada and $200 million in the U.S.

The most pressing issue for many companies was the lack of information technology staff to keep computers and applications running, because of the number of roads closed by fallen trees, power lines and utility poles.

In Montreal, some downtown companies closed or ran on reduced power, while others let workers bring their families into the office for warmth because most of the outlying communities were still without power.

``Most of our problem was just getting people in,'' said Richard Cox, a project manager at Air Canada in Montreal. ``We had a very high absentee rate. A lot of our technical support came to a screeching halt for a solid week.'' To cover for the number of employees trapped at home, Cox set up a 24-hour coverage plan for the remaining staff to oversee Air Canada's applications and to make sure company operations ran smoothly across the country, even if Montreal's Dorval Airport was socked in.

IBM Canada closed its Bromont, Quebec, plant, which assembles most of the chips IBM sells. IBM wanted to save electricity in the fragile power grid and protect its 2,000 employees, most of whom live in communities that were blacked out.

The Inspector Generale for Financial Institutions, a Canadian government agency, shifted some of its computer processing to its Quebec City headquarters because its Montreal offices had been closed since Jan. 9, a spokesman said.

Hydro-Quebec, the province's power utility, doubled the number of technical staff on duty to maintain its computer network, which is crucial to tracking repair progress and customer service calls.

Many businesses relied on a combination of uninterruptible power supplies (UPS) and diesel generators to ensure that the computers kept running even when the power lines toppled.

At Kennebec Valley Health in Augusta, Maine, such a scheme worked well, though the hospital's Waterville, Maine, offices didn't have enough UPSs in place to keep all its servers running, said Bill Terrell, Kennebec's chief information officer. That put a crimp in computer applications, Terrell said, but the hospital had power. Medical staff were able to provide normal care and get caught up on related computer work when the power came back on.

Other companies weren't so lucky. Five Canadian firms opted to run their operations at Comdisco, Inc.'s disaster recovery sites in Montreal, Toronto and New Jersey. Seven other companies put Comdisco on alert status, said an official of the Rosemont, Ill., disaster recovery firm.

The long outages resulted in breakdowns and a shortage of parts for the many ride-through diesel generators that companies depend on for power. The generators are typically designed to run for a couple of days, not a couple of weeks.

At Domco, Inc., a floor coverings maker in Farnham, Quebec, backup generators failed after three days. Closed roads and the loss of all telephone service meant Domco's headquarters was cut off from its two other plants in the U.S. So IT staff switched operations to SunGard Data Systems, Inc.'s disaster recovery site in Philadelphia. But Domco's U.S.-based plants also had to shut down computer operations for three days until the hot site was online, because order and shipping applications are normally centrally managed from Farnham's AS/400.

Domco, like many other firms in the region, had a disaster recovery plan in place but hadn't taken considered an ice storm.

``You can plan for a disaster when a building is destroyed, but we never figured on a disaster where we couldn't communicate with our Farnham offices'' via roads, phone lines, network lines or cellular connections, said Guy Chamberland, Domco's IT director.

Net cuts through

Rescue workers in upstate New York and Vermont were able to ring up New York state disaster relief officials even after conventional phone service was completely cut off by the recent ice storm that has brought the region to a near standstill.

The isolated communities are linked to the state's disaster recovery headquarters via an advanced communications network, the Adirondack Area Network (AAN), that relies on frame-relay and Integrated Services Digital Network (ISDN) communications links.

The system provides direct voice and videoconferencing contact among state officials in Albany coordinating cleanup operations, and hospitals and schools acting as command centers in the outlying communities.

According to the network's developers, combining underground ISDN lines and IP networks that don't use regular phone lines helped to keep the system running, even in places where electricity and phone service had failed.

Most similar high-speed networks use such a strategy, but AAN's community-oriented effort and focus on an area with limited infrastructure is a departure from the norm. Because of its current geographic limits, the network won't be used for more distant areas in New England and Canada, which were also hard hit by the storm.

At press time, state officials still were relying on the AAN to contact workers in some areas.