Several security vendors touted perfect threat protection or detection scores in the recent MITRE Engenuity ATT&CK Evaluation. However, MITRE recommends an independent interpretation of the results and that interested consumers create their own scores based on specific needs.  

This year’s MITRE’s ATT&CK Evaluations simulated real-world scenarios and tactics seen from threat groups Wizard Spider and Sandworm’s attacks. They tested security detection or protection capabilities of 30 security vendors, including Palo Alto Networks, Fortinet, CrowdStrike, Cisco, Microsoft, and VMware.

“The evaluations basically came from ... how can we use attacks to understand how these products work [and] improve them,” Jamie Williams, principal adversary emulation engineer at MITRE, told SDxCentral. “Not only spread awareness of these products but make sure there's a way that we can track and measure and make sure that they're getting better over time.”

Vendors and users interpret the evaluation results differently, with some presenting a participant ranking of the results. However, MITRE does not rank participants. 

MITRE encourages anyone who is interested in the evaluation to check its website and explore those products, Williams said. “Most importantly [to] not see a winner or not see a score or ranking, but [ask] what can this product really do for me?”

The evaluation is designed to enable users to look for answers to those questions and what they need to better their security postures.

“It starts with understanding your own needs — as a defender, what problems am I trying to solve, and how am I going to solve those,” Williams explained.

Since the human elements in cyberdefense differ in each organization, he recommended users take in various inputs such as those result analysis and summaries from the participants, “and then you can really start to dive into the data and start to define your criteria, and eventually you might be able to create a score … that score is going to be your score and your criteria.”

Palo Alto Networks in a blog post claimed its product blocked 100% of attacks in the protection evaluation and detected 100% of the 19 attack steps. And Cynet in a blog post put together its own version of the overall rankings.

“​​One of the pros and cons of the data release is it's up for interpretation of the user,” Williams noted. “So when I look at those blogs, I understand what they're saying. But it's very advantageous for every user to go and look at the data themselves and really understand the structure so they can come to their own conclusions.”

He used the claim of 100% protection as an example. The emulated test was structured with 10 steps with each comprising multiple techniques, so “100% protection of that scenario isn't necessarily inaccurate, but it all depends on how you count.” 

“To the best of my knowledge, I don't think every vendor blocked every single technique of every single test, because as a user you wouldn't really want that,” he added.

In addition, quantifying these results in terms of counting is under the assumption that all adversary behaviors are equal. In reality, organizations face different risks and challenges, Williams pointed out. “Coming into it with your own biases and priorities is also going to change the way you actually see the results.” 

He suggested users remain open to different interpretations of the results. 

“What we're doing is if we could tell you what product was best or what was the No. 1 way to score things we would, but we've come to the realization that we can't. It really is a decision that starts with you,” Williams said. “So listening, trusting, and just buying into anything anyone else tells you is going to be inaccurate."