Summary
During network maintenance, organizations were switched over to their alternate database instances. Once the network maintenance was complete, some organizations did not automatically switch back to their primary database instance. This imbalance eventually contributed to up to 50% of organizations in the region to receive timeouts when logging in or using their service.
Customer Impact
Impacted customers could not access their CRM Online organizations.
Incident Start Date and Time
February 8, 2013 6:47 AM PST
Date and Time Service was Restored
February 8, 2013 8:05 AM PST
Root Cause
Several organization databases were temporarily not balanced across available hardware after network maintenance. As traffic increased in the region, the imbalance resulted in some database servers being over utilized, causing timeouts to database queries. The Dynamics CRM Online team received alerts of this condition and manually switched the affected organizations back to their primary databases, alleviating load and restoring service.
Next Steps
Issue | Next Step | Team Owner | Timeline |
Failover | Investigation and resolution of why organization databases did not switch back to their primary database servers at the conclusion of maintenance | Dynamics Service Engineering (DSE) - CRM Online | Ongoing |
Capacity | Provision of additional database server capacity in North American data centers | Dynamics Service Engineering (DSE) - CRM Online | Ongoing |