MOUNTAIN VIEW, Calif. — If you think you've had a hard time migrating to software-defined networking (SDN), consider the Department of Defense.

Each day, the DoD Information Network (DoDIN) experiences 22,000 changes and generates 10 million alerts, almost all of them innocuous. And its services have a real-time aspect that your network probably doesn't face, as Major General Sarah Zabel of the Defense Information Systems Agency (DISA) explained at ONUG Spring 2016.

Her keynote talk Tuesday morning, kicking off the semiannual meeting of the Open Networking User Group (ONUG), covered some of her organization's progress — and struggles — with SDN, virtualization, and automation.

It's an outlier, but the DoDIN story exposes some of the difficulties that normal large enterprises will probably encounter as they try to adopt virtualization and automation.

That includes the struggle for organizational change. "I see the technological change coming, and what I'm struggling with is: How do we change our business processes to take advantage of it?" Zabel said.

As advanced as DISA is, it's still subject to some antiquated government processes. Some requests still have to be generated on paper, for example. "It's probably encapsulated in law somewhere," she said.

DISA isn't alone here. Later on Tuesday, Nick Lippis, ONUG's organizer, noted that JPMorgan Chase requires 100 sign-offs for a change to the network.

Because it supports "warfighter" activities, the network has a real-time aspect that's unusual and a lot more urgent than, say, streaming cat videos.

"We have found that the warfighters need a response quickly, on their timescale, not on ours," said Zabel. "This also means they need to have a certain degree of visibility and control into what we do," something that's not in place yet.

The desire for visibility also points to a cultural conflict: People relying on military communications are accustomed to having nailed-down circuits.

That's a telecom phrase meaning every hop in the network is predetermined and guaranteed. Unlike Internet connections, circuits are predictable. But that predictability goes away in a virtualized world, where you might not even know the location of the server you're using, and where virtualized network connections spring up out of nowhere.

"They want to see a wire," Zabel said. "They want to see blinking lights, and when they see those blinking lights in sequence, they know that their circuit is up."

Moreover, DoDIN is just plain large, serving as the Internet gateway for the entire Department of Defense. It's continually adopting the fastest pipe speeds — 100 Gb/s at the moment — but "we're finding that as network speeds increase, the computing and processing speeds for network-intelligent activities haven't increased at the same pace."

That's led to dropped packets and problems with load balancing. "I guess there's a limited number of folks operating an enterprise where they're pushing that upper limit."

One option would be for DISA to start developing its own stuff — assembling open source hardware and software and running established protocols over it all. That's not a route for everybody, though. DISA isn't interested in it, Zabel said when an audience member asked, and based on her comment, it seems no one in the organization has been pressing for a DIY approach.

None of this means DISA is standing still. Zabel listed a few of the proofs-of-concept that have been developed for DoDIN, including the use of data center interconnect (DCI) to move an entire job to a different physical facility. This could be useful for load balancing — or for outfoxing an adversary that's compromised a particular data center.

Possibly the most ambitious PoC has to do with modernizing DoDIN's information core. DoDIN doesn't have the luxury of going greenfield, Zabel said. The introduction of an automated network has to be done in slow steps, with the newer, virtualized network coexisting with the current one.

As with the circuit issue, the information core project shows that operational realities are an important factor in trying to become a software-defined enterprise. "It's great to draw a picture with lots of blocks and arrows," said Zabel, "but it's not terribly useful to get us to the future."

Photo: Wikipedia.