THE HAGUE, Netherlands — Intel and AT&T have launched an open source benchmarking initiative for network functions virtualization (NFV), hoping to replace the usual chip-based metrics with something more relevant to service providers.

Steve Vandris, Intel's director for NFV solutions planning, announced the VNF Industry Standard Benchmark (ISB) initiative during a session today at the SDN World Congress.

Hewlett Packard Enterprise (HPE) has also joined the project. Other vendors "have been participating" but are not yet being announced, Vandris said.

Benchmarks for virtual network functions (VNFs) exist but tend to be chip-geeky, measuring things like the number of packets per second that a VNF can process.

Service providers are more interested in metrics that reflect the bigger picture, Vandris said. They'd like to know how many mobile applications the infrastructure can handle, or how well the infrastructure reacts as workloads scale up.

"What the NFV platforms need is more efficient and real-world test infrastructure," he said. "The test architectures we use have to be able to benchmark at the service level, not at the packet-processing level."

Service providers could also use an industrywide set of benchmarking standards, replacing vendors' own test results, which result in apples-to-oranges comparisons.

The ISB plans to develop open source approximations of telco-grade VNFs — models that service providers could use to test an NFV infrastructure. In that sense, it's like SPECint, the benchmark for CPU processing power; the Standard Performance Evaluation Corporation (SPEC) provides files that companies like Intel can use to benchmark their chips.

So far, five of these reference VNFs have been created, representing virtual versions of an access control list, carrier-grade network address translation (NAT), a firewall, a provider edge router, and a service architecture evolution (SAE) gateway.

The initiative will also look at new ways to compare VNFs, he said. And long-term, ISB hopes to integrate NFV management and network orchestration (MANO) into its benchmarks.

For a test harness, ISB will use Yardstick, an open source project within OPNFV. And, of course, ISB plans to contribute code back to Yardstick.