ITS

Information Technology Services
Information Technology Services

Incident Reports

From time to time we experience unplanned system or service outages that cause a significant disruption to normal operations. In these instances, we will produce an Incident Report that summarizes the issue, impact, resolution, communications, costs, and resulting action items.

All incident reports are in PDF format and can be read using Adobe Reader.

 

Incident Number Incident Date Incident Name Brief Description
2017-207 26 May 2017 Web App Partial Service Outage (726 KB) From Friday, May 26 at 4:30 pm until Monday, May 29 at 8:30 am, a number of sites hosted on the Webapp Service became unavailable due to a configuration error.
2017-202 6 April 2017 Portal Outage (707 KB) On Thursday, April 6, from approximately 6:00 pm to 11:45 pm, the My.QueensU Portal service became unavailable to some users.  
2017-192 14 March 2017 Network Outage (714 KB) On Tuesday March 14, at approximately 6:30 am, a planned configuration change was made to one of the university border routers located in Fleming Hall. 
2017-190 7 March 2017 Campus Wide Network Outage (717 KB) On Tuesday March 7, at approximately 4:35pm, a campus-wide network outage was reported. A network loop was identified as the cause of the outage. 
2017-188 28 February 2017 Services Proxy Migration Issue (714 KB) After the migration to a new proxy server on the morning of Tuesday, February 28, there were issues reaching the Romeo Researcher application.
2017-187 23 February 2017 SSO Login to Some Sites Affected (744 KB) Queen’s Single Sign-On (SSO), the feature that allows for a single login across multiple Queen’s websites and services, experienced an issue that limited or denied access to some of these sites and services. 
2017-183 9 February 2017 OnQ Service Unavailable (679 KB) On the morning of Thursday, February 9, 2017, the OnQ service stopped responding to logins (users were unable to log in to the system). 
2017-182 3 - 8 February 2017 External Email Delay for Campus (601 KB) Some external email coming into Queen’s was not delivered. As the weekend progressed, delivery of external email continued to degrade. Late Sunday night, all external email stopped being delivered to Queen’s users.
2017-179 9 January 2017 OnQ Sync Issue (731 KB)  A number of users were not successfully enrolled in their classes. Investigation into the issue uncovered an inconsistency in how enrolment data was provided. 
2016-0175 29 December 2016 2016 Dec 29 Data Centre Power Failure (596 KB) One of two main data centre UPS units failed, causing power to be lost to many systems in the Dupuis Hall data centre.
2016-0172 15 November 2016 Deletion of Alumni Users (570 KB) Alumni accounts were erroneously removed from Active Directory, causing alumni users to be unable to access their email and incoming mail to be bounced back to senders.
2016-171 11 November 2016 HCDS Student Health TS Reboot Issue (579 KB) The Student Health server didn't restart properly after routine schedules Microsoft operating system patching.
2016-0169 21 October 2016 Degraded Campus Internet Service (585 KB) Queen's Internet service was degraded due to a failure of one of the redundant Internet Service Provider's links and a false bandwidth cap on the second link.
2016-0166 19 September 2016 SSO - uApprove Database Outage (586 KB) An error message displayed on specific single sign on Shibboleth websites.
2016-164 9 September 2016 Loss of Windows File Share (724 KB) Reports indicated that there was a problem accessing Windows Shares. An investigation uncovered that the incident affected all shares on a particular network.
2016-0160 20 August 2016 WebPublish2 Production Database Outage (576 KB) The WebPublish2 service was unavailable (live and authoring environments for approximately three hours.
2016-0157/158 15 August 2016 DCP Firewall Outage (585 KB) During a scheduled maintenance to replace a failed firewall, problems were identified; services were affected.
2016-0155 25 July 2016 Dupuis Data Centre Services Unavailable (701 KB) An unplanned power outage occurred in the Dupuis Hall Data Centre, making all systems hosted by that centre unavailable.
2016-0156 21 July 2016 PeopleSoft Open Registration Overload (568 KB) Open Registration web servers crashed due to system overload.
2016-0153 6 June 2016 New Medical Building Outage (322 KB) During a network switch update was found unresponsive upon reboot following the scheduled maintenance.
2016-0152 9 June 2016 Disruption on Multiple Services (239 KB) Services in VMware experience storage slowness for roughly three hours, affected Moodle, Queen's Portal, Wiki and PeopleSoft.
2016-0146 18 May 2016 Siteimprove Update Causing Wiki Service Failure (579 KB) Siteimprove had an update to the software's code which caused the Wiki system services to fail, rendering Wiki inaccessible.
2016-0147 17 May 2016 PeopleSoft Finance System Unavailable (233 KB) PeopleSoft Finance application was unavailable for five to ten minutes due to background changes.
2016-0144 6 May 2016 Portal Outage (239 KB) A brief service interruption occurred in the production Liferay Portal from the MyQueen'sU login page.
2016-0143 4 May 2016 IDM Outage due to bad deploy (248 KB) Inadvertent deployment of new code destined for Preview environment entered into Production environment.
2016-0141 11 April 2016 IDM Bonita Outage (325 KB) Staff and students were unable to access key functions of the IDM Bonita self-service account management portal.
2016-0140 15 March 2016 IDM Upgrade - Email Delivery (223 KB) Shortly after the email upgrade, it was noted that email was not being delivered from primarily off-campus accounts.
2016-0138 17 March 2016 Solaris Cluster Reboot (250 KB) A number of services hosted by the ITS legacy services environment became unavailable, affecting a number of services for approximately 20 minutes.
2016-0135 29 February 2016 TRAQ Romeo outage (229 KB) Incorrect performance of a manual copy of a database caused the production database for Romeo Researcher to be overwritten.
2016-0131 10 February 2016 PeopleSoft login issues (233 KB) PeopleSoft and Student Admin users were unable to login to PeopleSoft through MyQueen'sU.
2016-0130 2 February 2016 IDM - Alias Removal (229 KB) Following a testing exercise, many email aliases were improperly removed from production accounts, causing non-delivery messages to be sent to senders. 
2016-0129 1 February 2016 DNS resolving issue (319 KB) Inconsistent access to off-campus Internet resources.
2015-0121 14 December 2015 Issues with Film 240 online testing procedure (471 KB) On December 14, 2015, Film 240 students has a timed online exam however, due to unexpected issues, the exam was closed at approximately 9:30 am.
2015-0116 27 November 2015 Moodle outage (415 KB) Runaway database queries prevented backup utility from completing which in turn prevented all writes to the Moodle database.
2015-0112 16 November 2015 Moodle questionnaire (231 KB) Application servers restarted to reduce heavy load through Moodle.
2015-0110 20 October 2015 IDM Email Provisioning Failure (231 KB) A reboot of the IDM Windows Connector server caused an unexpected upgrade of the Windows Powershell language. Some users experience non-deliverable reports.
2015-0108 16 October 2015 File Server Outage After Hours (270 KB) One file share server did not return to operation following a Microsoft update.
2015-0106 13 October 2015 Slow response times for Enterprise applications (257 KB) Multiple services connected to Dupuis Hall Datacentre networks has intermittent and slow response times due to faulty stack interconnect cable.
2015-0102 6 October 2015 Shibboleth Single Sign-On service outage (257 KB) The Shibboleth Single Sign-On (SSO) service malfunctioned, preventing successful login for users attempting to access a Shibboleth SSO-protected production website or service.
2015-0101 27 August 2015 DNS turned off (321 KB) A domain name server in the production datacentre was accidentally turned off causing a widespread outage of the enterprise services.
2015-0095 1 September 2015 Office 365 application issues (234 KB) During the first week of September, random users of Office 365 online applications experienced various issues with accessing emails and/or applications.
2015-0099 22 September 2015 High traffic network outage (254 KB) An unknown and hidden hard limit of 100 DHCP relay conversations through the firewall at once caused a wireless network outage.
2015-0094 24 August 2015 Unavailable wireless networks (225 KB) Due to a spanning tree loop, a new wireless network and production wireless network went down.
2015-0092 26 August 2015 Low space on pool of disks at Dupuis Hall Datacentre (335 KB) A large number of services began paging down due to low space in a pool of disks at the Dupuis Hall Datacentre.
2015-0090 12 August 2015 Loss of power in Dupuis Hall (246 KB) Loss of power to power bar in Dupuis Rack 21 caused a brief loss of network services.
2015-0089 6 August 2015 Campus-wide network outage (256 KB) The campus core network was unavailable, affecting university productivity.
2015-0088 26 July 2015 Dupuis Hall Datacentre System Outage (415 KB) Services located in the main Queen's Datacentre were unavailable on Sunday, July 26, 2015 when the fire detection sensors in the main Queen's Datacentre activated the fire suppression system.
2015-0087 15 July 2015 PeopleSoft Performance Issues (251 KB) PeopleSoft experience greater than normal loads due to student registration causing login issues.
2015-0086 10 July 2015 Misconfiguration of Mail Exchange Record (228 KB) Mail from external sources was not being delivered to queensu.ca addresses caused by a misconfiguration of the Mail Exchange Record for queensu.ca.
2015-0084 13 June 2015 Unresponsiveness of TRAQ Millenium Fast (235 KB) TRAQ Millenium Fast became unresponsive on June 13, June 14 and June 27, 2015 causing denial of access to reports or data.
2015-0082 28 May 2015 Inaccessibility of Multiple Services (270 KB) Multiple address conflicts between the existing load balancers and the new appliances caused routing problems.
2015-0081 20 May 2015 Shibboleth Single Sign-On Service (SSO) Unavailable (249 KB) SSO service was unavailable affecting services such as Moodle, Liferay Portal (my.queensu.ca), SOLUS, PeopleSoft and Ensemble Streaming. 
2015-0080 10 May 2015 Storage Issues from Upgrade (323 KB) Failure of various services during storage unit upgrade to latest software revision.
2015-0079 11 May 2015 Network Outage at Various Campus Locations (249 KB) Network loop causing main Fleming core routing/switching device to have high CPU usage.
2015-0075 29 April 2015 Moodle Login Issues (229 KB) Issues experienced by students, faculty and staff accessing Moodle. 
2015-0074 20 April 2015 Shared Storage System Delayed (248 KB) Considerable delays in shared storage system due to server rebooting.
2013-0001 10 January 2013 LDAP/AD Corruption Error (285 KB) Corrupted student identities reduced or removed access to applications and services.
2013-0002 9 January 2013 Network Outage (275 KB) Network services unresponsive across campus due to interruption at Dupuis border router.
2013-0003 15 January 2013 Database Server Failure (185 KB) A motherboard fault caused a server to fail to restart during scheduled maintenance, causing an unplanned disruption to a number of services.

Help & Support

If you have any questions, please contact the IT Support Centre by calling 613.533.6666 during regular business hours or by filling out the online help form.