Defending cloud workloads is big business with multiple vendor technologies in play.

Among the most common types of technologies is a cloud network firewall. Much like a regular firewall, this technology monitors and controls both inbound and outbound network traffic based on predefined security rules.

Nonprofit testing lab CyberRatings recently conducted a test of multiple cloud network firewall technologies with varying results from vendors including Amazon Web Services (AWS), Barracuda, Check Point, Cisco, Forcepoint, Fortinet, Juniper, Palo Alto Networks, Sophos, Versa Networks and Watchguard.

At the top end, CyberRatings gave the Palto Alto Networks VM-Series Next Generation Firewall with Advanced Threat Protection a security effectiveness score of 100%. Versa Networks next-generation firewall (NGFX) scored 99.90%, while Check Point CloudGuard and Forcepoint NGFW both scored 99.80%.

Perhaps the most shocking aspect of the ratings were not the high results of some vendors, but rather the bottom result. CyberRatings gave the AWS Network Firewall a security effectiveness score of only 5.39%.

The low score is not the result of a specific vulnerability or zero-day issue, according to CyberRatings.

“There's no zero days here,” Vikram Phatak, CEO of CyberRatings.org told SDxCentral. “This is just you know, the airbag in the car doesn't work, it's not that I can break into the car if I knock on the window three times.”

Phatak said that it reported its concerns to AWS and had some conversations with executives.

AWS however refutes CyberRatings review and assessment of its technology.

“AWS Network Firewall, which allows customers to define firewall rules that provide fine-grained control over network traffic, is working as designed,” an AWS spokesperson wrote in an email to SDxCentral. “This report is inaccurate and incomplete, and we recommend customers review the AWS Network Firewall Best Practices Guide to determine the deployment and rules appropriate for their environment.”

According to Phatak, CyberRatings tried multiple ways to get the AWS network firewall to work properly.

He noted that his firm followed the instructions Amazon support provided, but it still didn't work. CyberRatings also hired an Amazon-certified consultant to set it up, but it was still not working.

Phatak said that CyberRatings was also able to work directly with AWS engineers to try and get the firewall running.  When working directly with Amazon engineers, CyberRatings had to remove the stateless firewall rules and put the rule in the stateful firewall instead of how the documentation said it should work

Phatak said he thinks the main issue is that the handoff between the stateless and stateful firewall components is broken. This likely causes problems with blocking exploits and evasions, since rules are not being applied as intended based on the firewall's design.

Overall Phatak emphasized that his firm is just trying to do an evaluation of effectiveness of cloud firewall technologies.

“This is security, so there's a certain responsibility,” he said.