LAS VEGAS – Hosted orchestration platforms currently provide a higher level of open source container orchestration security versus stand-alone options, according to a presentation by Capsule8 CTO Dino Dai Zovi at this week's Black Hat USA event.

Dai Zovi said there is a steep learning curve in terms of orchestration security, and that "there's a lot you need to do, and to get working for a reasonable level of security." But, despite the challenge, he said deploying security at the orchestration level provided a good return on that invested effort.

“Orchestration systems are an ideal place to install security,” Dai Zovi said. “They are the software fabric around everything,” adding it was “tedious and painstaking” to attempt to install security around all individual aspects of the container environment.

In proving his assertion, Dai Zovi used the Shellshock “bash bug” as a high-level example of an attack tool to target the three major orchestration platforms in Docker Swarm, Kubernetes and Mesos DC/OS. The results and accompanying commentary laid out a succinct hierarchy in terms of embedded platform security, but also reinforced Dai Zovi’s recommendation toward hosted solutions.

(Full disclosure: Capsule8 provides a hosted security platform targeting cloud-native deployments. However, follow-up conversations with others at the Black Hat event corroborated many of the main points touted by Dai Zovi.)

In parsing through his work, Dai Zovi said Docker Swarm was the least complex of the systems due to the background work conducted by Docker Inc. on the platform. He said Swarm security highlights included the default settings for untrusted nodes operating inside of a container, and containers only having access to declared links.

Dai Zovi said Shellshock was able to infiltrate a container, but noted “nothing else really happens.” He explained there was no avenue for lateral movement of the hack outside the container.

“Swarm is the gold standard in orchestration security today,” Dai Zovi said.

He was able to generate some mischief inside a Swarm deployment by compromising the platform’s Join Token. These tokens are required to add new nodes to a Swarm orchestrated container deployment.

Dai Zovi said it's possible to infiltrate a Swarm node and hijack its Join Token, which allows for the generation of a new container that could take over the IP addresses of legitimately created containers. This hack was not able to gain access to the cryptographic identity of the legitimate container, but could lead to interfering with network connectivity to the node.

To shut down the Join Token avenue, Dai Zovi said users should frequently rotate the token, adhering to the Docker recommendation of doing so every six months. It’s also possible to disable the Swarm autoscaling feature, but that was considered less desirable due to the deployment benefits.

In looking at Kubernetes, Dai Zovi said he was startled at the amount of ecosystem activity and growth, with many large companies having committed full-time engineers to the project. This focus could be due to what he said was currently a “more complex” environment compared with Swarm, though he did add that the complexity was also a benefit as it showed work toward more enhanced capabilities.

But, in terms of security, Kubernetes was viewed as being more of a challenge. He said a lot of security aspects of the platform were just a few months old, with a lot of work still waiting to be started.

Dai Zovi explained that more recent versions of Kubernetes include a larger focus on embedding security. But warned that open source options that claim to be “production ready” still lack clarity on security enabled by default.

Dai Zovi provided a slide full of tips for enhancing Kubernetes security. They included ensuring that the role-based access control (RBAC), which applies access control to Kubernetes application programming interface (API) objects, is properly configured; and that the PodSecurityPolicy function was set to prevent the running of privileged pods by possibly corrupted pods.

“This slide is a lot of work,” Dai Zovi said. “But, that’s where open source Kubernetes is today. … It feels like early days of Linux. It exists, but requires a lot of bootstrapping or using a commercial version.”

Mesos was a bit short-shifted in terms of focus, with Dai Zovi rather quickly running through the orchestration platform’s security chops. He did note Mesos was “the quiet giant” of the space as it was the only one configured to handle orchestration of hundreds of thousands of pods.

But, the often-cited “batteries not included” nature of Mesos means that those looking to use the orchestration platform should look toward a hosted solution with the necessary security instead of going it alone.

Dai Zovi ended with a number of takeaways from his work. This included making sure desired security options are available and enabled; that he felt orchestration systems were quickly converging on necessary security features; and his prediction that server-side endpoints in the cloud will become more dynamic via installation models similar to application stores, which will remove complexity from user endpoint devices.