Pete Nicoletti, Field CISO, Americas
The MITRE ATT&CK framework and the Enginuity ATT&CK Evaluations that the company offers to security vendors is one of the most valuable frameworks and testing results to consider as you monitor the competitive landscape and assist you in making critical product and partnership decisions. However, there are a significant number of issues to be aware of as you are bombarded by the recent vendor messaging of “We are the best!” Let’s take a deeper dive into the very recent “2022 Wizard Spider and Sandworm” product evaluation and explore some items that will help you navigate this report, understand the results a bit better, and help you architect your own security event prevention plan.
30 vendors evaluated:
All the traditional and legit players submitted their products as these tests are pretty much table stakes to play in the market. Quick Overview: There were some dramatic improvements from last year’s tests, there were some serious failures of some well-known products and the top vendors are all grouped very tightly together in a close competitive battle! One of the first things to start looking at is the testing trends. This year is the fourth evaluation, so look at your target vendors results over time. Note that MITRE does not assign scores, ranking or ratings so it critical that you understand your own use case(s) and map those to the vendor(s) that offer the capabilities that you desire.
Screenshots and workflow captures is one of the best features:
One of the nicest features of the tests is the effort the evaluation team spent to organize all the tool’s screenshots as they respond to the testing. CISO’s and their teams should spend some time scrolling through the screens of the tools. Make sure you walk through the “Steps”, the “Pattern” the “Detection Type” and the “Detection Note” with all the associated screen captures attached. Some vendors use “error codes” that require a look up in a manual, don’t settle for that! Also look for clear Tactic & Technique identification, a detailed timeline, business impact levels, remediation steps, the entry point identified, the processes impacted, and most importantly: a complete mapping of the forensic evidence to the ATT&CK matrix! The last item is no longer just a “nice to have” as this framework is now the “lingua franca” of the security community and should be the way you and your teams discuss and share information.
Vendors are very motivated to get the best results, some cautions!
Each vendor is motivated to achieve the best results so there is the desire to pull out all stops. The detail oriented CISO will need to review the configuration modes, the use of additional tools that some vendors used or other considerations that may not work well in your environment.
“Out of the box” default config verses turning all the knobs to max:
End Point tools offer a very wide selection of capabilities and protection options. When end point protection tools are deployed in the field, each customer has the ability to tighten, loosen and specify various security controls. Each vendor gives advice on the configuration parameters that should be appropriate to manage targeted risks. To achieve unrealistic results, some security vendors are specifying configurations that are not recommended for production…as those configs will “break things” and cause excessive false positives. Some vendors’ documentation even explains the risks to production environments, but they still used those configs to achieve results that are not normally achievable with real world configurations. Check Point offered an “out of the box” default config and walked away with no knob turning!
An apple vs. the fruit salad:
Who knew that our Harmony Competitive team in Israel has comedians? Yes, it’s a funny twist on the old Apple vs. Orange comparison analogy when 2 different things are being compared… now the analogy is improved to represent one tool vs multiple tools. In the evaluation, Check Point offered its one single tool and we read that multiple competitors supplied multiple tools to achieve their results. The competitors “Fruit Salad” approach changes the economics and administrative overhead of their offering considerably. In one case the vendor had to supply the end point tool, a firewall/security gateway and their treat feed: 3 separate tools. The “cost per protected user” dramatically goes up in this case and administrators are challenged to manage additional tools and consoles to achieve results.
Focus on understanding “Detection” vs. “Prevention”
There is a big difference to security outcomes in this area. These days, it is only seconds or minutes from the initial compromise, to privilege escalation then to East-West propagation and then the exploitation of a vulnerability executing ransomware that results in a disruptive event. Check Point provided immediate alerts with zero delays in all its preventions. Some competitors have slow sandboxes, or let the malicious file thorough to the user inbox with just an alert, or worse, miss it entirely. Some vendors had detection alerts showing up hours after the event. This is no way to run a railroad! You can not count on employees to make security decisions that could be catastrophic for your company, you must not allow the turd to be delivered to the inbox in the first place.
Now, a little bragging on Check Point 2022 Evaluation Results:
Check Point has the only single console comprehensive security suite that covers cloud, network AND endpoint.
Check Point is one of the top leaders with 100% detection across all attack steps tests (19/19, from Initial Access to impact) 103 of 109 attacks sub-steps identified, rated in the top five.
Harmony Endpoint delivered the most extensive visibility and context across all detected attack sub-steps.
In 98% of detected sub-steps, Harmony provided the highest technique detection level providing additional data enrichment to help users thoroughly understand the attack.
(This is critical to the conversation as it differs the “amount of” conversation from the “quality”. Many vendors “detected” things. When Check Point detected, most of the time, it was accompanied by a high level of enrichment of the event)
Check Point provided immediate alerts with zero delays in all its detections with 98% detection for financial advanced persistent threat (APT) Wizard Spider, responsible for notorious malware such as Emotet, Trickbot, and Ryuk.
Look into your Security Vendors “Sense of Urgency” to determine if it is built into their DNA: Secure Code Development:
- Investigate how many vulnerabilities your vendor has
- Response to a vulnerability: look at how long the vendor took to remediate
- Zero Day responses and SLA for updates: Critical stat here, how many ZD’s did your vendor prevent in the last year?
Bottom line:
MITRE evaluation results are currently the most detailed and through tests available to assist the CISO and their team in making an informed security architecture decisions. But, dig into the report, investigate how your target vendors achieved results and certainly check out the screen captures and process flows to ensure tools making your short list achieve “prevention” verses “detection.”
For more information on MITRE and the framework, please see the companion paper written by EU’s Field CISO, Deryck Mitchelson, named: Advancing security with the MITRE ATT&CK framework.
Find the MITRE ATT&CK 2022 Wizard Spider + Sandworm Evaluation Results here: https://attackevals.mitre-engenuity.org/enterprise/wizard-spider-sandworm/