For years, Facebook has had a career development program called "hack-a-month," which allows engineers to switch to a different team for a month — and if they like it better than their original job, they can make the switch permanent.

"No software engineer ever came to network," Najam Ahmad, Facebook's director of network engineering, lamented in a talk at the Open Networking User Group (ONUG) Conference in New York on Thursday. At least, that was the case until a few years ago, when Facebook began to radically reconfigure its  approach to networking, he said.

Those efforts have been well-publicized, but here's a quick recap: In 2011, Facebook rolled out a network automation tool called FBAR, for Facebook Auto-Remediation.

Today, FBAR sifts through 3.37 billion notifications from network devices each month, filtering out noise down to roughly 750,000 alarms that need action, Ahmad said. Of those, FBAR resolves 99.6 percent of the alarms without human intervention, according to Ahmad.

To achieve that kind of automation, Facebook has put heavy emphasis on modular, standardized network hardware controlled by software. Its Wedge top-of-rack switch and 6-Pack modular switch use the same board for fabric and line cards (Facebook plans to open source both designs through the Open Compute Project.)

[caption id="attachment_25086" align="alignleft" width="300"] Facebook's Najam Ahmad speaks at ONUG.[/caption]

If all of this sounds like it drains the fun and magic out of network engineering, Ahmad begs to differ.

"The traditional network engineering job was more like systems integration," he said. "You were limited by what the vendors could build, and limited in what you could integrate between them using protocols."

Facebook's approach has the effect of "taking a systems integration job and converting that into an actual engineering job where you're creating things. The network is a large scale distributed system, and it's a really fun problem to solve," he said.

Ahmad's team now regularly draws software talent through the hack-a-month program, where they're paired with old-school network engineers for skills exchange.

"Pretty much everyone on the team is writing some level of code," he concluded.

"You need to attract software engineers to your organization."