Security is always tricky, but it creates a special paradox in an open source environment.

You want everything about the project to be open, often to the point opening all discussions to the public, but then, how do you report a vulnerability? If you wanted to prove that a flaw is serious, you'd provide an example of an exploit. Send that to a public mailing list, and you've just shown the world how to break into the open source platform. Swell.

These issues aren't new, and they aren't unique to the OpenDaylight Project. But they started coming to light last month when it was discovered that a security vulnerability had gone unfixed since August, partly because there was no formal way to report it. To their credit, OpenDaylight organizers assembled a security team and reporting process —and patched the vulnerability — in less than two weeks. But it shouldn't have taken that long.

OpenDaylight's case is particularly interesting to follow because the organization is so new, still figuring things out along the way. For me, it's a peek into what it really takes to build a software development organization.

To learn more, I went to Mountain View on Monday for a pizza-and-soda OpenDaylight meetup at Hacker Dojo, organized by Srini Seetharaman. The featured speaker, security engineer David Jorm of IIX, went over some of open source's biggest security issues and gave his view on how response teams should operate.

The OpenDaylight vulnerability, named Netdump, was discovered by Gregory Pickett of Hellfire Security. Finding no security contact within OpenDaylight, he notified the project's generic in-box. His note went undiscovered until December, when the issue flared up on the project's public mailing list, as first reported by SDxCentral.

What was missing was obvious: "You need a team that knows how to handle a security response project and knows how to keep it quiet," Jorm said. Now, researchers can report OpenDaylight vulnerabilities privately, using a newly created email list accessed only by the project's five members of the newly formed security team.

OpenDaylight also created a process for emergency code releases, so that patches won't have to wait for the usual six-month cycle of major code releases.

Finally, best practices for security involve lots of documentation, so OpenDaylight has instituted a template for security advisory pages that will describe vulnerabilities and list the status of fixes.

All of that constitutes the reactive side of security — best practices to use when something has already gone wrong. What's tougher, Jorm said, is making the code more airtight in the first place, a concept he calls secure engineering. Here, there's less of a template to copy from elsewhere. The key is coding with care and, more importantly, running quality testing before code goes live.

Security, in other words, is a lot of work, and it's not always the most glamorous work. To that point: Some OpenDaylight members had actually been thinking about security all along. One cross-project group produced a report about secure engineering in May. After Jorm's talk, Meenakshi Kaushik of Cisco presented some of the group's findings, including some security concerns that arise with OpenDaylight's adoption of Apache Karaf as a distribution vehicle.

The report, available on the OpenDaylight wiki, was "quite thorough," Jorm says. "I was really impressed with this." But it hadn't resulted in any progress until December.

Jorm also went into some detail about Netdump itself. The problem stemmed from a known fault in Java regarding external entities, which can inject new code into an XML document. Software platforms often turn off external entities; Java leaves them turned on because many standards rely on implementing them, Jorm said.

If network settings aren't configured strictly enough, you could set the external entity's value to a file — the one containing admin passwords, say — and essentially grab that file.

That ubiquity made the Netdump hole important. But at the same time, a hacker would require access to apply the network configuration API to OpenDaylight in the first place. That still left a hole, but one that a smaller percentage of users would be able to exploit. And the fix, as I understand it, was relatively simple, a matter of changing the default network configuration settings.

The point I took away from this is that OpenDaylight is facing the same issues as any other software development team. It's just that there are a lot of them, and some of them seem very small at times when nothing is going wrong.

And as we learn from OpenDaylight, it's looking to peer projects for ideas too. The Open Networking Foundation recently created an OpenFlow security analysis team that's coming up with guidelines similar to what Jorm discussed. A representative from that group was at the meetup, in fact.

And Jorm himself is part of the security group for ONOS, the open source network operating system — and a probable OpenDaylight competitor, many of us think — developed by On.Lab.

Other things Jorm sees on OpenDaylight's to-do list include a threat model, to help inform developers where the likely security problems lie, and a way to coordinate multiple vendors to respond to any vulnerability. The latter is going to become a political issue as OpenDaylight gets bigger, Jorm said; companies get reluctant to share with competitors, so any response team is going to need to be trusted to "embargo" information.

"You'll never get to the point where there are no security issues anywhere," Jorm said. "But you can reduce the risk."