Summary
On Monday, December 22, 2014, an issue in one of Microsoft's datacenters caused some customer in APAC to be unable to access their organizations. The issue affected customers who had IPv6 enabled on their network. Microsoft engineers investigated and took steps to mitigate the issue.
Customer Impact
After entering their user name and password, affected customers received a 404 error, or in some cases a certificate error in their browser. Some customers were able to use alternate browsers or change their proxy settings to get around the error. This only affected customers using IPv6 and who were in Asia.
Incident Start Date and Time
22 December 2014 13:45 UTC+08:00
Date and Time Service was Restored
23 December 2014 04:42 AM UTC+08:00
Root Cause
An incorrect IPv6 DNS entry was pushed to production in one data center in Asia for the authentication service used by CRM Online. When customers with IPv6 enabled attempted to log in, they were redirected to a non-existent server and received a 404 message. Using a computer without IPv6, or a proxy server that only accepted IPv4, would allow customers to work around the issue. Microsoft engineers removed the incorrect DNS entry and restored service.
Next Step(s)
Issue | Next Step | Team Owner | Timeline |
Change control procedures | Perform a thorough investigation of how this incorrect IPv6 entry was pushed to production | Microsoft Dynamics CRM Online Service Engineering jointly with the Microsoft DNS team | Underway |
Additional monitoring | Investigate additional monitoring that can be added to help the team pinpoint this kind of DNS issue in the future. | Microsoft Dynamics CRM Online Service Engineering | Tool identified and monitoring being configured |