The Federal Communications Commission released a damning report about the cause and impact of a nationwide network outage that CenturyLink experienced late last year. The outage impacted as many as 22 million customers across 39 states, and at least 886 calls to 911 were not delivered, according to the FCC. 

The outage on CenturyLink’s fiber network, which lasted for nearly 37 hours, was caused by an equipment failure that was “catastrophically exacerbated by a network configuration error,” the agency concluded. As many as 17 million customers spread across 29 states lacked reliable access to 911 throughout the outage. Further studies determined that there was no harm to life or property resulting from the outage. 

When the outage occurred in the early morning of Dec. 27, 2018, communications service providers, business customers, and consumers who rely on CenturyLink’s transport services, including the routing of communications traffic from numerous service providers, experienced extensive disruptions to phone service, according to the FCC.

“Last year, CenturyLink had one of the worst network outages in U.S. history, affecting millions of consumers. For nearly two days, some 911 callers got a busy signal instead of help,” FCC Commissioner Geoffrey Starks wrote on Twitter. “FCC’s recent report suggests fixes but we must hold folks accountable so this doesn’t happen again.”

The trouble began when a switching module in CenturyLink’s Denver node “spontaneously generated four malformed management packets.” These occurrences are common and usually discarded immediately because characteristics indicate that the packets are invalid. However, that did not occur in this instance. 

CenturyLink and Infinera, which supplied the network operator with the affected nodes, conducted an internal investigation into the matter but drew no conclusions about how or why the malformed packets were generated. 

“The outage was caused by a network management card that generated malformed packets that unfortunately were retransmitted across parts of CenturyLink’s transport network,” a CenturyLink spokesperson told SDxCentral in a prepared statement. “CenturyLink has taken a variety of steps to help prevent the issue from reoccuring, including disabling the communication channel these malformed packets traversed during the event and enhancing network monitoring.”

The outage was a "rare service occurrence" that was "related to an earlier generation of technology," an Infinera spokesperson told SDxCentral. "We responded immediately and worked closely with CenturyLink to identify the root cause, rectify the problem, and fully restore the network. Preventive measures were immediately implemented, eliminating forward exposure from this issue to [CenturyLink] or any of our other customers' networks," the spokesperson wrote.

Each of the malformed packets, also referred to as Ethernet frames, shared attributes that contributed to the outage, including a broadcast destination address, a valid header and valid checksum, no expiration time, and a size larger than 64 bytes, according to the FCC. 

The malformed packets were sent as network management instructions to a line module, which subsequently caused them to flow through the network. The troubles continued when the malformed packets passed multiple conditions without being discarded, and a filter designed to stop such activity let the packets continue unimpeded because they were larger than 64 bytes. 

The malformed network management packets were eventually delivered to all connected nodes, and each node continued to retransmit them because the packets appeared valid and had no expiration time. “This process repeated indefinitely,” the FCC wrote in its report. 

“The exponentially increasing transmittal of malformed packets resulted in a never-ending feedback loop that consumed processing power in the affected nodes, which in turn disrupted the ability of the nodes to maintain internal synchronization,” the agency explained. 

Once CenturyLink was alerted to the problem by a customer near New Orleans, multiple alarms were triggered indicating an issue with Infinera’s control modules and the operator determined that the outage was widespread. CenturyLink’s network administrators were unable to connect to the nodes remotely because they were overloaded, which further exacerbated the problem.

Within hours, network engineers were on the ground in multiple cities to locate and diagnose the outage directly on the affected nodes. The module that had generated the malformed packets was identified and removed about 17 hours after the operator was first made aware of the problem. However, the malformed packets continued to replicate and transmit across the network until a network engineer disabled the proprietary management channel and began instructing nodes to disregard the malformed packets.

CenturyLink and Infinera brought the network back to normal function about 25 hours after the initial disruption occurred, but the operator didn’t regain remote access to all nodes on the affected network for another five hours. Finally, more than 31 hours had passed before CenturyLink restored visibility into the network and the backbone network was stabilized about an hour later.

At least 12,100,108 calls were blocked or degraded due to the outage, according to CenturyLink. The incident caused problems for many other network operators, too, including AT&T, Bluegrass Cellular, Comcast, Cox, General Dynamics, TDS, TeleCommunication Systems, Transaction Network Services, U.S. Cellular, Verizon, and West Safety Services. 

Infinera’s investigation into the matter is ongoing, but the company’s engineers have been unable to replicate the malformed packet creation, according to the report. Moreover, CenturyLink and Infinera have reconfigured nodes in the affected network by disabling proprietary management channel and established a network monitoring plan to help detect similar events more quickly in the future.

As of yet the FCC has not issued any enforceable actions other than drafting a list of best practices that could have prevented the outage or diminished its impact. Those guidelines include disabling system features when they’re not in use; improving early detection by regularly auditing network monitoring memory and processor utilization alarms; and standardizing operating procedures for network repair in circumstances where normal network monitoring procedures are inoperable. 

“This massive ‘sunny day’ outage was completely unacceptable and impacted millions of customers across the country. Americans expect and deserve reliable phone and broadband service — especially the ability to call 911,” FCC Chairman Ajit Pai wrote in a prepared statement. “It’s important for communications providers to take heed of the lessons learned from this incident.”

CenturyLink says it’s continuing to fully cooperate with the FCC’s investigation and regrets any inconvenience the outage may have caused its customers.