CyberRatings.org, the independent non-profit organization dedicated to putting confidence into cybersecurity product quality, perhaps can be likened to one of those little “conscience cherubs” who whispers in a cloud provider's ear, imploring it to do the right thing when it comes to maintaining cybersecurity services.

It's apparent, however, that some companies aren't paying full attention to some good advice.

The Austin-based CyberRatings team on April 2 released its Q1 2025 Comparative Test Report on Cloud Network Firewalls (CNFW), along with separate in-depth reports for each of the 10 top cloud firewall solutions tested. Security effectiveness results for all vendors ranged from 0% to 100%.

CyberRatings, which releases these reports periodically to the enterprise security community, included in this latest edition evaluations of third-party firewall solutions deployed across AWS, Azure, and GCP environments. As they have in the past, none of the Big Three performed well.

“The results should provide deeper insights into how third-party solutions perform in comparison to native offerings and reveal opportunities for enterprises to enhance their cloud security postures,” Cyberratings CEO Vikram Phatak told SDXCentral.

Use caution with native firewalls

As enterprise cloud adoption accelerates, reliance on native security tools should be handled with caution, Phatak said. “This report serves as a critical reminder that robust cybersecurity cannot be assumed and must be rigorously validated to protect against the evolving threat landscape,” Phatak said.

The main takeaway from this latest exercise: Security firewalls provided by all of the three major cloud providers are still flat-out ineffective and nowhere near the quality of those made by dedicated cybersecurity vendors.

For the third time, CyberRatings.org conducted focused assessments of Amazon Web Services (AWS), Microsoft Azure, and Google Cloud Platform (GCP). As they were last fall, the results were notably poor, with all three getting 0.00% effectiveness marks. In reality, first-class cybersecurity firewall services aren't the highest priority for hyperscale cloud providers, whose first orders of business are to store and distribute data and not lose it.

Key findings from the report are as follows: In the Cloud Service Provider Native Firewall test from November 2024, only 522 exploits were used in the Part 1 Mini-Test, but evasions weren't included. For this new round of testing, a greater number of exploits were deployed, and evasions were introduced into the test samples. In network security, an evasion is bypassing an information security defense to deliver an exploit, attack, or other form of malware to a target network or system, without detection.

• Third-party firewalls from Check Point, Fortinet, Juniper Networks, Palo Alto Networks, and Versa Networks demonstrated the highest security effectiveness blocking exploits and evasion tactics. Results ranged from 99.61% to 100% – all excellent ratings.
• Native cloud firewalls from Amazon Web Services (AWS), Google Cloud Platform (GCP), and Microsoft Azure offer a convenient alternative, but they all received 0% Security Effectiveness as they allowed attacks to bypass existing defenses.
• Google Cloud Platform’s Next Generation Firewall (next-generation firewall (NGFW)) service uses Palo Alto Networks technology. CyberRatings attributed the differences in security effectiveness and performance results between the two platforms to each provider independently selecting and deploying different software versions based on their own criteria.
• A total of six firewall solutions were recommended and four received “caution” ratings.

Here are results from the new round of tests: Behind the evaluation scenes

• False positives: 2,760 samples from various business-critical files and applications, ensuring security measures did not disrupt legitimate traffic.
• Exploits: 2,028 attack samples from widely exploited vulnerabilities in enterprise environments.
• Evasion techniques: 2,500 attacks spanning 27 evasion techniques tested across multiple network layers to bypass firewall defenses.
• Performance metrics: 46 different stress and capacity tests under diverse workloads.
• Stability and reliability: Seven extended tests simulating prolonged real-world attack and operational scenarios.

CyberRatings evaluates firewall security by testing for evasion detection at three separate layers of the Open Systems Interconnection model, specifically Layers 3, 4, and 7. Missing lower-layer evasions had the greatest impact on the overall score because these layers form the foundation of firewall security at the fundamental networking level, and when these lower layers are compromised, the firewall’s primary protective function is undermined.

Points were deducted based on the firewall’s ability—or inability—to detect evasions: Layers 3 and 4 evasions enable attackers to bypass security at a fundamental level, allowing them to deliver any type of malicious traffic to their target. Since all modern applications rely on IP and transmission control protocol (TCP), vulnerabilities at these layers can be exploited across a wide range of systems—from cloud services to enterprise applications.

• A missed evasion from the Layer 3 level resulted in a 50% deduction per category, up to a potential category maximum reduction of 100%.
• Missing a Layer 4 evasion led to a 20% deduction per category, up to a potential category maximum reduction of 60%.
• A miss at Layer 7 incurred only a 1% deduction per category, up to a potential category maximum reduction of 10%.

No movement on improvement 

The lack of improvement by AWS, the world's largest cloud services provider, has raised questions about internal processes and priorities, Phatak said. Despite being notified of deficiencies last year, the company has not yet addressed the fundamental issues in its network firewall product. This failure underscores broader challenges in cloud security, where architectural differences between cloud environments and traditional data centers complicate the deployment of effective security solutions.

The low scores are not the result of a specific vulnerability or zero-day issue, according to CyberRatings. “There are no zero days here,” Phatak said. “This is just the airbag in the car doesn’t work; it’s not that I can break into the car if I knock on the window three times.”

Note for users of Microsoft Azure

Phatak added a note of caution for users of Microsoft Azure's firewall. “The single biggest problem with the Azure firewall was its inability to decrypt traffic,” Phatak said. “Eighty percent of the internet is encrypted with HTTPS. They (Microsoft) have a third-party proxy to do that job, but basically they're saying, 'Offload your decryption over here, and then let unencrypted traffic go through the firewall,' which is an architectural choice.”

Summarizing it all

Until cloud service provider native firewalls provide better protection, customers should be looking to third parties for their cloud security needs, Phatak said. “Traditional third-party security vendors have demonstrated that they bring significant value to customers,” he said.

The cloud firewalls were tested using Keysight’s CyPerf v5.0 software testing platform alongside CyberRatings’ in-house developed test tools. Enterprises can easily perform similar testing with a two-week free trial from Keysight. Further details of the CyPerf strike library can be found here.

The full test reports are available at cyberratings.org.