The company said engineers made a change on Aug. 3 “that applied an erroneous policy to our North American object storage service.” This deleted some data. The problem has since been fixed, Cisco said.
“In the majority of cases, this issue will not impact network operations, but will be an inconvenience as some of your data may have been lost,” Cisco said in an advisory. “Your network configuration data is not lost or impacted — this issue is limited to user-uploaded data.”
Cisco said its engineers were working over the weekend to recover customers’ data and help them identify what had been lost. The company will provide an update by the end of day today “with the current status of what resources we will be making available to restore functionality.”
Cisco said that no customer data was compromised and that it is working to restore the deleted files.
The company will “continue to update customers in real time via a Meraki Support Page,” a spokesperson said in an email to SDxCentral. “Cisco will continue to actively engage with our customers to provide whatever support is needed to remediate this issue.”
The engineers’ mistake likely didn’t affect companies’ critical data. Cisco said impacted areas include splash themes, floor plans, logos, and phone-related data such as hold music and voicemail greetings.
But the cloud configuration snafu is yet another example of how easy it is to make an error that results in customer data loss.
Stronger Oversight Needed
In February, Amazon Web Services’ (AWS) engineers made a configuration error that caused its S3 cloud storage service to crash for more than six hours. Google Compute Engine cloud services experienced a similar outage in April 2016.
ACG Research analyst and CEO Ray Mota said the Meraki data loss incident illustrates the need for stronger oversight and governance models to prevent these types of blunders.
“Automation is great, but you have to have checks and balances with that as well,” Mota said.
“With this transformation that is happening with virtualization, SDN, and NFV, a lot of vendors and customers are making the mistake of only focusing on the infrastructure transformation,” Mota added. “But this transformation is touching other areas — the organization, operations, the processes, a lot of the skill sets — and to be successful you have to address those other layers. You can’t just throw technology at it and assume it solves the problem.”
Editor’s Note: This story was updated with Cisco’s response.