Information Technology Services
Information Technology Services

Systems outage caused by AC unit leakage

On Sunday, July 26, 2015, the Queen's community felt the impact of an unplanned power shutdown at the Dupuis Datacentre. All systems and networks in the Datacentre were affected. Users were unable to access information systems such as the learning management system (Moodle), email and other services that depend on our infrastructure.

What happened?

The cause of the power shutdown was due to the triggering of the fire systems in the Dupuis Hall Datacentre. A manufacturer's defect in the second compressor of an air conditioning unit caused mineral oil to leak under the datacenter floor. The subsequent reaction of the oil mixing with dust present in the room created condensation that was interpreted by the fire system as smoke in the area. Standard operating procedure when the fire system alarm goes off is to automatically shut power down in that room in order to prevent further potential damage. Fire and emergency crews responded to the situation and the building was deemed safe for the ITS team to be present in the building in order to assess the situation.

The ITS team worked diligently late into the night in order to restore the servers and services to the campus community for Monday morning. Course registration was available for students by midnight. By 2:00 am on Monday, all systems were running except Moodle and ion-premise email. By 8:30 am, both Moodle and on-premise email were restored. Delays were experienced in restoring services due to some file corruption however all data was recovered from the previous night's backup. All services were fully up and running by Monday afternoon.

In most system outage situations, the ITS Notification Tool is used to disseminate information; however, during the first response process this tool was unavailable. Communications were made available to the campus users through ITS' Twitter account, @ITQueensU, until the ITS Notification Tool was restored.

Going forward...

A debriefing meeting concluded that repairs and improvements are slated for the future including greater maintenance and upgrades to the Datacentre server room and a better line of contingency communication notifying the end users of Queen's ITS' information networks and support systems. An incident report with full details is being prepared and will be available to the public from the ITS website.