Microsoft Azure
– Sebastian Moss/DatacenterDynamics

Microsoft Azure services in its West U.S. region were impacted for five hours by a networking issue.

The trouble started on June 23 during routine fiber maintenance. That service required isolating specific network paths, a process that converts requests into system-readable formats and verifies that at least one of the two redundant paths remains in order.

According to Microsoft, "a bug in the request conversion system incorrectly marked additional devices as a part of the maintenance event and caused a set of IP routes to be removed from more devices than intended."

"The routes were removed between our data center and wide-area network, impacting traffic entering or exiting the region," the hyperscaler added.

As a result, multiple Azure services began to detect and correlate service degradation, including Application Gateway, Azure Firewall, Azure Kubernetes Service (AKS), Azure VMware Solution, Virtual WAN, VPN Gateway, and its Microsoft Sentinel security service.

The issue initially presented as large-scale route churn in Microsoft's WAN. Microsoft later found that route removal was from one of its data centers in its U.S. West region. 

Microsoft said its next steps would include a comprehensive analysis focusing on safety checks and the automated maintenance request change process. It also recommended that customers with mission-critical workloads adopt a multiregion deployment strategy to ensure availability during single-region disruptions and evaluate their overall application reliability using its Azure Well-Architected Framework best practices guide.

Organizations were also advised to configure Azure Service Health alerts to guarantee that appropriate personnel receive timely notifications regarding any future service issues.