Remember open-book tests in school? Those were assignments that students enjoyed doing because all the answers were right in front of them; all they needed to do was to look up the facts and copy them into the test.
There are similar types of product tests in the IT world. In this case, we're discussing a no-brainer-type test of cloud network firewalls that's causing a stir.
Much like a regular firewall, a cloud network firewall monitors and controls both inbound and outbound network traffic based on predefined security rules. But it has different characteristics from firewalls that protect physical on-site data centers or edge IT systems.
In an evaluation of three mainstream cloud-native firewalls, CyberRatings.org conducted a focused assessment of offerings from Amazon Web Services (AWS), Microsoft Azure, and Google Cloud Platform (GCP). The results were strikingly poor, with AWS standing out as the least effective performer. The surprising data actually showed a loss of improvement from a similar test conducted six months previously (0.38 percent effectiveness in stopping standard hacker exploits, down from 5.39 percent).
These findings identify worrisome gaps in the security capabilities of cloud-native firewall solutions. They also serve as a wake-up siren for enterprises relying on these tools to revisit the protection of their digital infrastructure.
“The numbers were surprising on one hand and not so surprising on another,” Vikram Phatak, CEO of the non-profit testing lab, told SDxCentral. “I guess the first thing is that there's no polite way of saying this: None of these products is living up to any standard I remotely respect.” Do you basically claim they suck? “They do,” Phatak said.
Key findings The tests, described by CyberRatings as “mini-tests” designed to assess basic functionality, focused exclusively on known exploits, excluding more advanced techniques, such as evasions. This “open-book” testing approach, Phatak said, provided vendors with the equivalent of pre-disclosed questions and answers, yet the outcomes still were profoundly disappointing.
- AWS Performance: AWS scored a mere 0.38%, a regression from its already troubling score of 5.39% in a May 2024 evaluation, as reported by SDxCentral's Sean Michael Kerner. Despite having six months to address identified issues, AWS’s firewall failed to deliver meaningful improvements. According to CyberRatings: “This wasn’t a bug—it was a fundamental flaw in their approach to detection.”
- Azure and GCP Performance: While this was the first test involving Azure and GCP firewalls, their results were also alarmingly low. Azure’s and GCP’s firewalls demonstrated some level of functionality but also failed to meet even the most basic expectations for blocking known exploits.
Industry implications These findings highlight a troubling reality for businesses: Cloud-native firewalls, as they stand today, are ill-equipped to meet even baseline security standards. “It’s not unreasonable for organizations to rely on trusted brands to meet minimum expectations. Unfortunately, these products are falling far short of what the market demands,” Phatak said.
The lack of AWS improvement, in particular, has raised questions about internal processes and priorities, Phatak said. Despite being notified of deficiencies six months ago, the company has not addressed the fundamental issues in its network firewall product. This failure underscores broader challenges in cloud security, where architectural differences between cloud environments and traditional data centers complicate the deployment of effective security solutions.
The low score is not the result of a specific vulnerability or zero-day issue, according to CyberRatings. “There are no zero days here,” Phatak said. “This is just the airbag in the car doesn’t work; it’s not that I can break into the car if I knock on the window three times.”
Possible causes Several factors could explain the underperformance of these cloud-native firewalls:
- Misaligned priorities: Users may be prioritizing other product enhancements over security improvements.
- Resource allocation: Cloud providers might be optimizing for performance or cost efficiency at the expense of robust security capabilities.
- Organizational silos: Communication gaps between engineering and operational teams could lead to unaddressed flaws.
- Technical constraints: The unique architectures of cloud environments often require specialized solutions, which may not align with traditional security paradigms.
AWS' response AWS provided the following comment Nov. 25 when asked for a response by SDxCentral. It is identical to the response sent to SDxCentral's Sean Michael Kerner for last May's article:
“AWS Network Firewall, which allows customers to define firewall rules that provide fine-grained control over network traffic, is working as designed. This report is inaccurate and incomplete, and we recommend customers review the AWS Network Firewall Best Practices Guide to determine the deployment and rules appropriate for their environment,” the AWS spokesperson wrote to SDxCentral via email.
As of Nov. 1, AWS leads the world market in cloud infrastructure-as-a-service (IaaS) (as it has for several years) with 31% market share, compared to 20% for Microsoft Azure, and 11% for Google Cloud, according to Statistica.
Recommendations for businesses For organizations relying on cloud-native firewalls, these findings underscore an urgent need to reevaluate their security strategies, Phatak said. CyberRatings advises enterprises to:
- Consider third-party solutions: Established third-party firewall providers, such as Palo Alto Networks, Fortinet, Check Point, and Cisco have proven track records and should be evaluated for cloud deployments.
- Perform independent testing: Businesses using Azure or GCP firewalls should conduct their own assessments to understand the level of risk they face.
- Demand accountability: Enterprises should push cloud providers to prioritize security improvements and provide transparency regarding product capabilities.
Looking ahead CyberRatings has announced plans to conduct a more comprehensive test in the coming months, Phatak said. This next phase will include evaluations of third-party firewall solutions deployed across AWS, Azure, and GCP environments. The results are expected to provide deeper insights into how third-party solutions perform in comparison to native offerings and may reveal opportunities for enterprises to enhance their cloud security postures, Phatak said.
As cloud adoption accelerates, the reliance on native security tools should be tempered with caution. This report serves as a critical reminder, Phatak said, that robust cybersecurity cannot be assumed and must be rigorously validated to protect against the ever-evolving threat landscape.
Q&A with the CEO SDxCentral: What's the most important takeaway as you see it?
Vikram Phatak: The most polite way I can put it is that none of these cloud provider firewalls are living up to any standard I'd expect. They basically … well, they suck.
We flagged these issues with AWS six, seven months ago. They had plenty of time to fix it. And it's not some obscure bug. We're talking basic stuff (exploits) here. Their signature library? It's like they took a default list meant for home routers and thermostats and shoved it into a cloud firewall. We're talking preschool-level failures here, and they knew about it. We went through the Cybersecurity and Infrastructure Security Agency (CISA), the whole nine yards. Nothing.
SDxCentral: Any idea why they haven't taken action?
Phatak: No clue. We haven't spoken to them since the last test. Frankly, we felt a bit jerked around. It's not that they disrespected us, but … let's just say it wasn't a productive exchange.
SDxCentral: So only .38% of all exploits hitting the AWS firewall are caught and stopped. Sure looks like anyone using only an AWS network firewall is basically wide open.
Phatak: Unless all they care about is basic access control, pretty much. For any kind of deep inspection, exploit blocking … you need a third-party firewall. That's the bottom line.
SDxCentral: And these things aren't free, are they? How does pricing work?
Phatak: It varies. Enterprises usually get it bundled into their overall spending. Smaller businesses pay based on traffic volume. But one way or another, you're paying.
SDxCentral: What about Azure and GCP? Any improvement there?
Phatak: This was our first time testing them. We focused on AWS before because we were testing third-party firewalls on their platform, and it made sense to include their own offering as a benchmark.
SDxCentral: So this is a baseline for all three. You're planning a broader test later?
Phatak: Exactly. We'll include the major third-party players – Cisco, Checkpoint, Palo Alto Networks, the whole gang. See how they stack up across the different cloud platforms.
But it's not as simple as keeping everything out. This test focused on known exploits. We used Keysight's CyPerf tool, which everyone has access to. No evasions, no fancy tricks. Just basic stuff they should be blocking in the high 90s (percentage).
SDxCentral: Evasions?
Phatak: Think of it like this: There's a vulnerability in a web server. Someone writes an exploit to exploit it. Security vendors create signatures to block that exploit. But attackers get smart. They disguise their attacks, use obfuscation techniques to bypass those signatures. That's an evasion.
SDxCentral: So these cloud firewalls failed even this basic test?
Phatak: Yep. And it's not like we're some giant company with unlimited resources. We're a small, specialized testing firm. It's just … baffling.
SDxCentral: Do you think it's a matter of priorities at these companies? Maybe their security teams are not focusing on firewalls.
Phatak: It's possible. But I'd be surprised if they weren't testing this themselves, seeing these results, and not doing anything. Maybe it's a QA issue, or they're prioritizing performance over security. Or maybe they're running into architectural differences between their testing environment and the actual cloud deployment. It's like building software for Windows and deploying it on a Mac – it's just not going to work the same way.
SDxCentral: So many potential factors at play. Anything else that stands out?
Phatak: The main takeaway is that if you're relying on these built-in firewalls for security, you're not getting it. Use a third party. We haven't tested those on GCP and Azure yet, but we will soon. In the meantime, test them yourself, or wait for our next report. It's not unreasonable for people to rely on big brands like AWS. They expect a certain level of quality, a reasonable bar. These products aren't meeting it.
Comments