Having written a series of posts on the topic, I've been trying to avoid SD-WAN lately. However, it seems my ongoing conversations with service providers and vendors don't stray very far, and SD-WAN continues to be top-of-mind for many in our industry. The latest set of conversations centered around the universal CPE (uCPE) — the hardware on which SD-WAN software, along with other virtual network functions (VNFs), will run.

Given the multi-billion-dollar numbers that IDC projected for the uCPE and virtual CPE (vCPE) market in December last year, I would expect this year’s forecasts to remain the same or possibly increase. These predictions will again fuel the software and hardware vendors in the SD-WAN space, prompting them to increase their investment and capture a large slice of this presumably lucrative market. And yet, in the uCPE world, there are numerous challenges.

Lack of uCPE Standardization

While uCPEs are touted as the universal — after all, that's part of the name — white box on which software can easily be loaded or unloaded dynamically, the reality is that there isn't an agreed-upon set of specifications for what goes into them. Many manufacturers claim any edge device running general-purpose x86 or Arm CPUs are white boxes, while some service providers are open to having FPGAs or ASICs for cost-effective acceleration.

Verizon made an early launch into the fray, working with ADVA back in 2016 to define a uCPE white-box platform. AT&T has submitted edge device CPE specifications for both XGS-PON and G.fast, as well as an open CPE specification, to the Open Compute Project (OCP). AT&T's hope is to work through OCP to create a uCPE standard. However, the OCP Telco Project does not include any edge standard submissions from Verizon or other major telcos.

Intel is also pushing its own standardization around the Intel Select framework for uCPE and leveraging partners like Advantech and Lanner to push its version into the market. At the same time, Dell EMC is hoping to rally service providers as well as SD-WAN and other network solution providers to its standard uCPE for SD-WAN.

In spite of this, most service providers are taking matters into their own hands and driving their own standards. Their goal is to keep costs down and flexibility up, ensuring no lock-in to any SD-WAN or VNF vendor in the long term. If service providers have enough volume, they care less about whether their uCPE is standardized or not because they can benefit from manufacturing scale. In any case, the battle for the universal CPE will continue. One front will see a tussle between x86 and Arm, each claiming the better price, performance, and efficiency. The other front will see all the different vendors trying to balance participation in standards organizations, like the OCP, while still coming out as the winner on their own platforms.

uCPE and vCPE Architectural Dichotomy Plays Smaller Role

It is also important to consider deployment architectures. There are a variety of locations where SD-WAN and other VNF functions can reside. While the uCPE on customer premises is certainly one option, these functions can also be hosted on shared hardware via containers or virtual machines (VMs) in central office/point-of-presence locations (COs/POPs) or in cloud data centers. Some functions, like WAN optimization or caching, may require a presence on the local branch devices, but advanced security scanning could be done off-branch. Not only would this approach bring scale economies and cost-savings but also improved manageability.

ZScaler is a prime example. It has built a nice business running hosted security solutions for enterprises in its cloud and has partnered with a number of SD-WAN vendors. Along this line, some VNF vendors tout the ability to execute their functions anywhere: in the cloud, at COs/POPs, and on uCPEs. Others go further and claim the ability to coordinate between sites and migrate VNFs dynamically as needed. Recently, I’ve seen less discussion around how the right deployment architectures or hybrid approaches could make sense. Solutions are usually in the cloud or on uCPEs, and lightweight CPE with VNFs in COs/POPs seem to be less popular. Service providers tell me this is because enterprises desire functions that run better on uCPEs such as local cloud breakout or WAN optimization. For service providers, it's easier to focus on a single uCPE-centric architecture augmented with cloud over-the-top (OTT) services for advanced services. Perhaps the uCPE is the reality going forward, and the alternate architecture of lightweight CPEs with converged, high-density VNFs in COs/POPs will decline over time.

The Case of the Missing Open-Source uCPE OS

Moving from hardware to software, another major service provider gripe is the lack of an open source CPE operating system. Most CPE solutions use kernel-based VMs (KVM) for virtualization with some adding container support. However, the underlying OS still stymies them. Most people agree that it will be Linux-based, but the other necessary capabilities for device management, cloud-based management, and telemetry are still challenges.

In March 2018 AT&T contributed its  disaggregated network operating system (dNOS) codebase — including assets from its Vyatta acquisition from Brocade — to the Linux Foundation. This led to AT&T using its dNOS code to seed the Disaggregated Network Operating System (DANOS) project, which also takes advantage of other open source projects like Free Range Routing. DANOS is meant to be an open source operating system for edge devices. Yet, as I write, Github or other open repositories have yet to see any releases, even though code availability had been promised for the latter half of 2018.

In October 2018 AT&T submitted its specification for a white-box cell-site gateway router to the OCP. Tellingly in its launch of the platform, it loaded Debian-based Vyatta onto its routers, indicating the Vyatta OS was production-hardened — implying that DANOS wasn’t ready. Instead it stated DANOS would be used when available and reaches production grade. Let's hope we're not going to end up playing a “Waiting for Godot” game with DANOS.

I, for one, will be excited to see how DANOS drives the service provider-vendor dynamic when it shows up. Some SD-WAN vendors are leveraging ease-of-use of a fully integrated platform, which reduces complexity for service provider SD-WAN deployments. These vendors have invested in zero-touch provisioning, advanced remote troubleshooting and monitoring, and cloud-based CPE management. Unfortunately, that means service providers are locked into the SD-WAN vendor’s hardware and software platform. Recent service provider requests for proposals (RFPs) mandating that SD-WAN providers support the service provider’s preferred uCPE platforms is a step toward avoiding vendor lock-in. But DANOS would provide service providers even more control of their own destiny; allowing them to modify it for use on their uCPEs, limiting SD-WAN and other VNFs vendors to just software packages that can be replaced at will.

Bloated VNFs Add to Ongoing SP Indigestion

Moving from the OS to the network functions, we find that bloated VNFs, especially for SD-WANs, is a common service provider complaint. They are trying hard to keep the costs of uCPE boxes down. And when compared to security devices like next-gen firewalls (NGFWs), they need to eventually achieve price and performance in the same ballpark. My earlier post on NGFW versus SD-WAN didn't mention that service providers are seeing their $1,500 or $2,000 uCPEs barely matching $500 to 800 NGFWs in key aspects of networking performance. In addition, service providers are sore about having to manage 60 to 80 GB of SD-WAN and other VNF images and about sacrificing two to four CPU cores just for SD-WAN/NFVI management overhead. They worry that eight x86 cores on an edge box is insufficient, yet don’t want to go to 16 cores because of the high price. Over time, the situation will improve as SD-WAN and VNF vendors pare down their software footprint, but this is creating additional drag around uCPE deployment today.

Longer term, uCPEs are likely to be the preferred deployment model for many SD-WAN deployments, but there are significant issues to be overcome. And we haven't even covered other uCPE issues like security, standardized telemetry, and trusted platform modules — which may be necessary for a trusted WAN in an enterprise. What I’d like to see is industry-wide efforts at OCP, the Linux Foundation, and other open standards organizations to drive the market forward. Hopefully, the release of DANOS code to the public will shake things up and add momentum toward a more universal uCPE.