Post Incident Report | Resolved | North America | CRM Online | Some CRM Online organizations were disabled unexpectedly

Summary

On 15^th August 2014 at 6:55 AM Pacific time, a number of customers hosted in our North American data centers began to experience problems accessing their CRM Online organization. Microsoft engineers began investigating and determined a routine maintenance task resulted in affecting customers' ability to login to their organizations. A large number of customers were brought back online within a few hours of the issue beginning, however it took until August 17^th at 3:00 AM to restore access to each affected organization.  

Customer Impact

Impacted customers could not sign in to their CRM Online Organization. The majority saw an error message stating their Organization was disabled, and a small number saw a "Page Cannot Be Found" or a similar error message.  

Incident Start Date and Time

August 15^th, 2014 6:55 AM Pacific time

Date and Time Service was Restored

August 17th^th, 2014 3:00 AM Pacific time

Root Cause

The problem was caused by an operator error and a failure to capture an exception in a routine maintenance task. An erroneous execution of an administrative tool resulted in affecting multiple organizations, causing them to become disabled.

While we were restoring service to the disabled organizations, some customers experienced performance degradation and/or the inability to execute workflows. A large number of customers were restored within a few hours of the issue beginning, however it took until August 17th at 3:00 AM to restore access to each affected organization.  

Next Step(s)

Image may be NSFW.
Clik here to view.