In a modern data center, servers account for nearly half of operations costs. Other data center costs include infrastructure, power consumption, and hardware networking (cabling, switches, patch panels, etc.). As a result, it is critical to optimize on dimensions such Gb/s per dollar and/or PPS (packets per second) per dollar.

One way operators are optimizing costs is by maximizing their data center resource utilization. Idle resources are the worst use of data center spending and are generally viewed as money going down the drain. One way operators are solving this problem is by introducing software-defined networking (SDN) into their environments in order to increase network agility. This is done through simplifying the network design to maximize bandwidth between data center servers, introducing programmability of the fabric through use of centralized control, and pushing more networking tasks to the servers. With this model, operators can gain programmatic control over all of their resources, which provides quick and easy control over how servers are utilized.

Using a virtual switch (vSwitch) on the host (server-based networking) enables the programmatic model to work effectively. The most commonly used vSwitch implementation is Open vSwitch (OVS). OVS hosts rule and flow entries responsible for directing traffic into and out of tunnels and virtual machines (VMs), translating or modifying packet headers, and enforcing access control and security. In an April 2016 survey of OpenStack users, 40 percent of respondents were in production with OVS, while another 20 percent were either in development or in proof-of-concept phase.

The challenge is that running OVS on the server is expensive from a CPU cycle perspective. The number of CPUs that are dedicated to the vSwitch to execute the workload directly impacts performance of the vSwitch. However, as more CPU cores are allocated to vSwitch operation, lesser numbers of cores are available for VM and application execution. This limits the effective output of an individual server, having negative effects on PPS/dollar. When applying the PPS/dollar penalty across data centers of medium and large scales, the cost impacts are extremely high. But by offloading OVS processing to an Intelligent Server Adapter (ISA), data center operators can reclaim lost server cycles and accelerate networking performance.

As an example, we will look at the capex and 3-Year opex for a virtualized data center built with ISAs accelerating the OVS data path, as compared to basic NIC adapters with OVS running in the host CPU kernel. The comparison assumes an Intel-based server with 16 physical CPU cores (8 cores per CPU with 2 CPUs per server) and a 40 Gb/s Ethernet network interface. The cost per server is $2,500 and the cost per top-of-rack switch is roughly $8,500.

The assumption in the model is that each VM in the data center can process 1 million packets per second (MPPS) per CPU core assigned to it. VM workloads can greatly vary when it comes to PPS output, but 1MPPS is a reasonable average. To support a data center output of 500 billion PPS with a VM capacity of 1MPPS, a total of 500,000 VMs must be deployed with one CPU core assigned to each VM. To support packet input/output (I/O) to the VMs, OVS must also be present, and its CPU consumption must be accounted for in the model.

The breakdown in footprint and costs is as follows:

• Total PPS rate output for data center: 500 billion PPS
• Total CPU cores needed for VMs: 500,000
• Total CPU cores needed for OVS: 1.67 million
• Total servers needed in data center: 135,417
• Total number of ToR switches: 6,771
• Capex: $396 million
• 3-Year opex: $304 million

These costs are most impacted by the resource requirement for OVS. Recall that OVS is required for network agility so that operators can achieve optimal utilization across the data center. Unfortunately the agility comes at a high cost penalty.

ISAs offer the ability to offload and accelerate OVS workloads at a fraction of the CPU consumption. This translates to major savings across data centers of any size. Because the ISA’s software integrates with OVS, original control plane interfaces are used, and the network-programming model is preserved. The benefit is retaining the network agility achieved using server-based networking but at higher performance and reduced CPU and server footprint. Consider the same model presented above, but with ISAs and software in use for offload (Table 1).

Graphic provided by author  

Table 1: Comparison of ISAs with basic NICs for OVS processing.

Because there are a fixed amount of CPU resources consumed by OVS with the ISA model, the cost savings in the comparison grow linearly as the data center capacity increases. This offload and reduction in footprint, allow data center business managers to decide if cost reduction and savings are desired, or if revenue expansion is desired through increasing VM density per server.

Viewing the model a different way, the basic NIC with host OVS suggests that with 135,417 servers, the total data center output is 500 billion PPS, or 3.7 MPPS per server. When using the ISAs and software, each server can generate 15 MPPS of output, a 4X gain. This output gain suggests using the original server footprint with Basic NICs and Host OVS (~135,000 servers) but with the output achieved through using ISAs, the total data center output would increase to 2 trillion PPS. This would translate to a 4X gain in VM capacity, and therefore, equal revenue gain with virtually no impact on data center cost.

OVS, the most commonly deployed Open vSwitch implementation, is a critical component of SDN deployments. However, data center operators must be careful to optimize their server resources at all times, and running OVS on host CPUs eats up valuable server cycles that could otherwise be used to process VMs and applications. As this example shows, offloading OVS to ISAs dramatically increases server output and maximizes efficiency in data center operations.