A nearly two-day, widespread outage of the CenturyLink cloud network that disrupted the emergency service in Washington and other states was caused by a faulty network management card. according to a customer message.
Brian Krebs, a veteran of the security journalist, sent a copy of a notification CenturyLink's "Core Customers" Saturday's Twitter feed, which blamed a map in its Colorado data center for "propagating invalid frame packets across devices," caused a series of problems that led the company to re-allocate much of its network equipment start. It took more than two days for CenturyLink to identify the issues when the all-clear came on Saturday morning, a time when 911 services in several states, including Washington, were fancy or spotty.
It came from CL itself to the core customer. Sorry, I can not share the source. pic.twitter.com/21T9x458gr
– Brian Cancer December 29, 2018
CenturyLink representatives did not immediately respond to a call for review notice.
With modern cloud service providers, a two-day outage is eternal. And it's not clear how a device could cause such a failure if cloud providers integrate redundancy into their systems.
An FCC investigation of the failure could raise some answers, unless CenturyLink is prepared to post a detailed post-mortem on the failure that is becoming a standard part of incident response.