Networks have changed dramatically in recent years.

Analog switches of old couldn’t answer questions — they either worked or didn’t work — but today’s devices are sending out data all the time.

“Devices are getting smarter, chattier,” thus bringing all new complexities to network management, said Denny Petrie, infrastructure health director at Lumen.

In response to these new and evolving challenges, the global telecom company is increasingly relying on automation, machine learning (ML) and artificial intelligence (AI) capabilities.

“It’s taking all that noise that’s coming at us from the network and making sense of it, and making it actionable for somebody,” Petrie said in a recent virtual discussion with Thomas LaRock, principal developer evangelist at AIOps network monitoring and analytics company Selector.

Anomaly detection was the first logical use case for automation and ML at Lumen, according to Petrie.

The sheer volume of data coming in from the network is too much for humans to analyze, he said: How do they know what they’re supposed to be looking at or even where to look?

For this reason, engineers started out not by looking at what was in the data, but how much of it was coming in and comparing that to the norm.

For instance, they would typically receive a million messages a day from a particular device — then, all of a sudden, it began sending out 80 million.

“Let’s go look at that: Why is it talking to us so much?,” Petrie said. “Anytime a machine is telling us something, it’s telling us for a reason.”

The team discovered that a card on that particular device was failing and resetting so fast that the alarm system couldn’t catch up. It kept rebooting to try to fix itself, which was “absolutely impacting customer service.”

“Don’t always look at what’s in the data, look at the quantity of data,” Petrie advised.

Human analysis is critical to the process, because “you can have a poorly performing circuit that’s consistent,” he said, and machine-based anomaly detection won’t necessarily pick up on that because it doesn’t have reasoning skills.

Ultimately, “anomaly detection is by far the easiest and has proven to be the most consistent value over time,” said Petrie.

Networks are going to fail — that’s a given. But ML and automation can help teams react quickly, Petrie said.

Previously, Lumen had what it termed a “customer as an alarm” system. That is, customers would often experience an issue before the network knew about it and report it.

Now, with automation, an abnormal number of customer contacts in chats or calls can trigger an anomaly. Additionally, systems can identify when specific customers are no longer authenticated.

For instance, 50 customers have all of a sudden dropped authentication. Automation can help determine where they are located, whether they are on the same DSL pool or route, then alert a field team.

Through simulations, Petrie’s team determined that it would take a human a minimum of two hours to make those sorts of correlations, “where now it happens in minutes.”

He pointed out that Lumen’s fault management system processes between 60 and 200,000 transactions a minute, 24 hours a day.

“There's no amount of people you're going to have on staff figuring out what's going on in that mess,” he said.

The ability to find the “most common denominator” in a given group of customers is now an embedded, automated process that has saved the company tens of thousands of dollars, said Petrie.

“I'm never going to eliminate alarms,” he acknowledged. “In fact, they're probably going to continue to grow as devices get chattier and chattier. It’s really being able to pinpoint ‘What do I want to go fix and can I tie in automation so it doesn’t require a person and can happen very quickly?’”

Taking that a step further, automation systems can help predict failures before they happen (even if they can’t yet prevent them).

For instance, ML can look back to traffic patterns 15 to 30 minutes before an event occurred and map probes to determine commonalities. When a similar subsequent event occurs, humans then have more of an understanding of common routes preceding events, and can start there first.

“That's a tremendous amount of data,” said Petrie. “There's not enough people who could be looking through that and figuring out what's going on there.”

These insights can also help determine when a fault is going to occur in the network so that the company can have a plan to notify people. Similarly, if a card is set to fail, automation platforms can order cards and open up change requests to send technicians out at the appropriate maintenance windows.

Finding that right maintenance timeframe is particularly important in telecom — as there is no necessarily off-business hour, Petrie noted. Due to network fluctuations, Lumen can’t just send workers out at assumed slow periods (say, between midnight and 5 a.m.). Automation can assess network traffic to determine the lightest periods to schedule maintenance.

“These are things we would not have been able to do without the compute power that's at our fingertips today,” said Petrie.

Still, Petrie emphasized, organizations shouldn’t just assume that, while powerful and increasingly capable of more and more tasks, ML, AI and automation are always the answer.  They should start with business outcomes first, then build the solution — not the other way around.

“We didn't come with a solution looking for a problem,” he explained. “We came with, ‘you give us a problem, we'll find a solution.’”

Along with this, it’s important to have a realistic conversation around ML and set expectations around what it can — and can’t — do.

“We didn't want to get wrapped up in the buzzwords of ‘Oh it’s machine learning, it's going to solve everything,’ because people think of it as like a magic eight ball of data, you shake it up and it gives you the answer,” said Petrie. “That’s not the case.”

In the end, Petrie noted, let machines do what they are best at and leave humans to do the rest.

“Start simple, get small wins,” he advised. “Don't start out trying to do some very complex AI/ML.”