24 hours to set up new data infrastructure – that’s the reality facing Intel’s IT division today.

Intel IT manages the company's entire tech stack, handling more than 418,000 servers across 15 data center sites supporting its 100,000 employees. On top of that, the division is seeing 25% growth in demand for network capacity every year. When new capacity is received, Intel puts it into production within a day to optimize the value of the investment.

Starting in 2014, Intel began evaluating software-defined networking (SDN) to handle these data center challenges, eventually opting for an open, standards-based SDN architecture over supplier-centric solutions.

In 2019, the tech giant began migrating the majority of its data centers to a new SDN architecture employing a leaf-spine underlay combined with overlay networks, with over 90% migrated in the last five years.

Part of this transformation was an automated management plane, which hinged on a network source of truth (NSoT) built using Intel’s in-house resources. But as Intel senior network engineer Greg Botts explained to SDxCentral, the crunch point came with an influx of new equipment, specifically more than 2,000 new switches across Intel’s design and office/enterprise data centers.

“We landed a new platform right in our data centers. We landed a new solution, and with it, we introduced automation. About five to six years down the road, we needed to pivot.”

To provision the hardware, a turnkey commercial management and orchestration platform was enabled, based on a virtual machine (VM) image hosted on-site, backed by Intel’s own infrastructure servers and hosting-supported OS build. An open-source hypervisor (KVM) hosted 17 regional orchestration clusters, with three failover servers each, making for a total of 51 servers.

Each VM could only support a limited number of network devices, and as the network grew, Intel IT’s two-person network automation team was forced to spend much of their time on development rather than their core network design work. In addition, the per-device licensing costs for the commercial platform escalated into the millions.

“I always complained we need a full time sysadmin just to deal with our hosting of our automation system. That burden was really heavy,” Botts recalled.

The situation called for a new orchestration and automation framework. The criteria was to find an off-the-shelf product with a user interface (UI); a homegrown solution wasn’t possible due to the small size of Intel’s network automation team.

“It also had to be API accessible,” Botts explained. “The more abstraction that we could leverage, or that was in the product, the better, so things like GraphQL to be able to pull your data out were very attractive to us.

“Open source was another kind of key tenet, especially open source that had enterprise support; that option being there was key.”

Intel ultimately settled on Nautobot, an open-source network SoT that supplies an off-the-shelf data model for network inventory and relationships, reducing the need to build and maintain a bespoke schema. The tool is sponsored by New York-based network automation solutions provider, Network to Code (NTC), which has partnered with the likes of Cisco, Red Hat, and Cisco Meraki in Nautobot deployment.

With it, Intel created a central repository for network attributes backed by a database-as-a-service (DBaaS). The orchestration stack was containerized, eschewing the previous VM-based setup, while automation handled integration and continuous deployment (CI/CD) pipelines, alongside managed services for container images. Dynamic scaling allowed by container use removed the need for additional clusters, which were essential previously when the team was constricted by the physical resources of a bare-metal server.

The data delta

Through its API accessibility, engineers query Nautobot using GraphQL to retrieve attributes that feed into templates from the Jinja2 templating engine, which are rendered into device configuration files. As a note, Jinja templates are text files made up of static text and dynamic placeholders which are used to generate customized output such as network device configurations based on variable data.

The end result is that orchestration caller scripts can fetch device identity and all required variables from a single authoritative place, render configuration outside of the device using templating, and push validated configurations to devices.

Botts explained that the golden goose, if you will, of the automation solution is its Golden Config piece, short for golden configuration. The element enables a single, intended device configuration from the SoT and Jinja templates. Specifically, the Golden Config compares the intended configuration from Nautobot’s SoT and the templates, reports on compliance, and automates remediatory configuration plans and pushes fixes if needed. The end result is the golden configuration, general network parlance for the authoritative and ideal version of a network device's configuration.

“It’s our most popular application that adds on to the platform, a capability specifically designed to handle configuration compliance and configuration drift remediation as part of the application suite,” explained Jeff Bradbury, VP of marketing at Network to Code.

In his work with network clients over the years, Bradbury has noted many suffer from a vast gulf of difference between the golden configuration of intent in a network and the actual reality.

“The difference between what's in your network source of truth is your intended state and what you're seeing in your observed state really creates a delta of data. That delta becomes the data you need to then go do some of the remediations to fix and bring things back into compliance.”

Botts agreed on the deception of the data delta, having learned from mistakes during the first phase of Intel’s automation framework.

“I thought we were super clean, right? And we found out, even with a pretty standard solution, going to a 100% standard solution, there were some one-offs, and those are the things that have been missed.

“Historically, you see a one-off, it's almost like a landmine getting set off. And then a few weeks, few months later, somebody does something, somewhere else, which sets off the landmine.”

For the Intel engineer, this underlines one of the architectural tenets behind Intel’s network architecture strategy, standardization, as the backbone pillar.

“What everything kind of feeds off of is scalability through standardization … The more diligence you apply to that tenet, the better your results will be.

“In our case, you model all of our network data in a network source of truth, and then all your configs that you're generating are just a byproduct of all that data being modeled. So that's kind of the pillar … We know what our configs are going to look like, and the more you scale, the more you realize the agility benefits there.”

Having realized this in the phase automation framework, Botts and team could apply their learnings through Nautobot without input from NTC, meaning they could pick up the solution and run with it.

The future of network automation

As Intel's networking services seem uncertain with its dismantling of its Network and Edge Group (NEX), it’s natural to wonder what the firm's automation ambitions portend: an open source future of some services as it concentrates on chip production? An automation of services previously carried out by network teams?

In the short run, Botts believed Intel IT’s customers are benefiting from network automation.

“We recently did a massive rolling OS upgrade through our whole environment, with no downtime. We used to have to get these big windows of downtime … and now we’re much more effective if we don’t have to do that. And the customer is happy, too,” he told SDxCentral.

In the longer run, the engineer can see how Nautobot sets up Intel for an AI-centric future.

“With the mindset of getting all the data into a network source of truth, we've sort of set the table for AI, right? Because AI needs data, that's the fuel. So now we have our network-specific data, and we can start leveraging all the AI solutions that are coming out. Every day, there is something new, but now we have something to run that against, and really take our efficiencies to the next step.”

But neither Intel nor NTC sees AI and automation replacing network engineers, with Botts reflecting how the automation project was helped by the Nautobot open source community.

“We spent a lot of time interacting with the Slack channel. Especially when we got into the weird stuff, like, how do you do this, and how do you do this? It's a vibrant community that's very responsive.

“We had a pretty tough roadblock, but we went and found the code, made some changes, it got accepted and pushed into the next version, and we were off and running.”

In other words, human engineers were the real source of truth when it came to automation for Intel.