Keyano teams saved critical IT infrastructure from floods

After days of concern, on Monday morning, April 27, a major ice jam on the Athabasca River caused floodwaters to rapidly inundate downtown Fort McMurray.

As Keyano College teams were alerted and rushed to the main Clearwater Campus, water was already threatening buildings and critical IT equipment, including vital servers the College relies on to function as an institution.

With the help of College Facilities staff, IT Director Dave Benoit and members of his team went to work to identify and disconnect essential equipment.

Using triage-like protocols, the team quickly selected the pieces they would need to move to higher ground at Keyano’s Suncor Energy Industrial Campus. Moving this much infrastructure would normally take weeks of planning. They had mere hours to work.

“It’s not as simple as unplugging and leaving,” noted Benoit. This had to be done in an orderly way, or we risked losing important data.”

As the team worked, floodwaters continued to rise and the Regional Municipality of Wood Buffalo was issuing mandatory evacuation orders. Several canoes were seconded in case they were needed as a means of escape.

The last pieces of equipment were being loaded into vehicles as floodwater filled campus parking lots and entered the interior of several campus buildings.

It was time to go.

A daunting challenge

Safely established at the SEIC Campus, Keyano teams reviewed the work ahead needed to bring systems back online.

Many valuable lessons had been learned from the devastating Horse River wildfire that ravaged major sections of Fort McMurray just four years earlier. That fire and evacuation resulted in a near complete interruption of services for many weeks.

This time, many important applications and systems, including the College’s website, were hosted within the Cloud and were not impacted in any way. Communication with students and staff also remained unbroken.

These were important advantages, but getting all critical IT systems back up and running would still require a lot of hard work and ingenuity. For a short time, the college’s remaining active IT systems relied on a small server, located more than 300 kilometres away at the Fort Chipewyan Campus.

IT and Facilities staff worked well into the night for more than a week as they built a temporary home for the critical servers and the other IT infrastructure needed to connect staff and students to college resources.

By May 9, all major systems were largely back up and running, allowing the Spring Semester to begin with only a week’s delay.

Rescuing and rebuilding the College’s IT Infrastructure on a new site within days represented a major challenge for Keyano, a critical undertaking now largely complete. Benoit says this feat could not have been accomplished without the support of the entire IT team, Facilities, the College’s Executive as well as other critical suppliers.  

“I really can’t say enough about how our people worked and how our teams came together,” he said. “We learned a lot from this experience and we’ll be even better prepared for the next crisis.”

While there remain challenges, including a need to enhance the available Internet bandwidth at the industrial campus, Benoit says they are already looking forward to the day when teams can start planning for their return to the main College.

-30-

Caption: Some of the key IT and Facilities team members involved in the rescue and rebuilding of the College’s IT Infrastructure, stand in front of the haul truck located outside the Keyano College Suncor Energy Industrial Campus.