Mitre Round 4 involved detailed testing of endpoint security (XDR and EDR) protection technologies against simulated attack scenarios based on the Wizard Spider and Sandworm threat groups. These evaluations covered both detection and protection (endpoint prevention) capabilities for each participating vendor.
For MITRE Round 4, twenty-one security solution vendors were evaluated in the protection test, and of those one vendor failed to fully complete the testing sequence (due to blocking an action that MITRE deemed should not be blockable). In total seven vendors claimed a 100% success rate, in which 9 of 9 tests (or 8 out of 8 for vendors who did not participate in the Linux portion of the evaluation) included at least one correct block.
MITRE Round 4 - Combined Prevention Test and Detection results
While a number of vendors claim results of 100% protection in the MITRE evaluation, their evaluation results for blocking malicious activity are not equal, and the differences between them can have a significant impact on an affected organization. We encourage you to take a close look to understand when in the attack sequence the protections occurred, knowing that blocking earlier is better.
The protection portion of the evaluation included a total of nine tests, each of which involved a number of attack substeps. In order to obtain a successful result for each test, vendors were required to block at least one of the attack substeps, and were prohibited from blocking actions which were noted as valid actions. Failure to block at least one substep would result in failure of that test, and blocking an action that was noted as valid by MITRE disqualified the vendor's results for that step.
Each test result also includes specific details in the sequence where the prevention took place. This additional data is critical in enabling security analysts to gauge how effective each vendor is at preventing malicious activities and minimizing impact to the organization. This detail also highlights where in each attack sequence prevention actually occurred, and enables analysts to more accurately determine which vendor(s) were most effective at handling the attack and minimizing impact; this also illustrates significant differences in overall efficacy, even among vendors claiming "100% coverage".
To illustrate this concept, we will explore two of these test scenarios, and compare the MITRE Round 4 performance of three vendors (CrowdStrike, Microsoft and Palo Alto Networks), all of whom successfully completed this test, and all of whom attained a result of 9 out of 9 or ‘100%’ for the overall protection evaluation.
The scenario for test #4 is described by MITRE as follows: “Compromise Domain Host”. This test contains a total of 2 steps and is the simplest test in the protection evaluation when measured by the total number of steps. This step utilizes native windows utilities found within the operating system, and uses valid functions within those utilities to perform these actions. This is a “living-off-the-land” attack sequence which involves no external code. There are no unblockable steps in this test; both of the steps are valid block points. The summary of actions which occur are as follows:
While this is a short sequence, each step can provide immediate impact and value for the attacker, since each step yields highly valuable information.
For test #4, Palo Alto Networks prevented the attack sequence at the first step, denying the attacker any data of value that would have otherwise been gleaned. By terminating the hostile process tree, the subsequent action did not take place, and no data was gained by the attacker.
When we compare this result to that of a different vendor, in this case Microsoft, we see that in the evaluation they achieved prevention on the SECOND step of test #4, but allowed the first step to complete which permitted the attacker to create a volume shadow copy of the C:\ volume and thereby obtain a copy of the ntds.dit database it contained. NTDS contains user account information, including account names, as well as group membership. Most critically, it also stores the password hashes (encrypted text) for each account.
Technique/Substep | Blockable | Microsoft Blocked | Palo Alto Networks Blocked |
---|---|---|---|
OS Credential Dumping: NTDS | YES | NO | YES |
OS Credential Dumping: Security Account Manager | YES | YES |
Obtaining these hashes enables an attacker to either crack the hashes at their leisure via brute-force means or employ the hashes directly via pass-the-hash techniques, wherein the hashed value of a password can simply be substituted for the password itself. Armed with this data, an attacker can know which accounts to target (for example, privileged accounts or application service accounts) and can gain additional high value access with this information. Therefore, even though the attack sequence was ultimately prevented, with Microsoft Defender, prevention did not occur in time to stop the simulated attacker from gaining access to critical information, and the prevention that does ultimately occur only stops further actions from taking place via this attack chain, and does not address the information that attacker had already gained.
The scenario for test #8 is a much more extensive compromise attempt of a target domain host, utilizing both keylogging and browser credential harvesting methods in conjunction with an attacker-installed command and control (c2) backdoor for use in issuing instructions to the endpoint and to facilitate secure data exfiltration. This sequence contains a total of 21 steps and is the largest test by number of steps in the entire prevention evaluation. The summary of actions which occur in this test sequence are as follows:
This attack totals 21 steps from beginning to end. The first 2 steps in this test are considered benign and not valid for blocking by MITRE. The remaining 19 steps are all considered valid block points for this test and a block that happens on any of these remaining steps will yield a pass for that test.
In test #8, Palo Alto Networks detected every step in the attack sequence up to the point of prevention. Prevention occurred on step 3 in the sequence (the first blockable step). As a result, the endpoint remained unimpacted by the attack sequence, and no further remediation actions are required to address elements of this attack
Conversely, if we look at the performance of a different vendor, namely CrowdStrike, we see that while their product detected each step in the test sequence up to the point of prevention (process kill and binary quarantine). Prevention occurred at step 7 in the attack sequence (the 5th blockable step).
Technique/Substep | Blockable | CrowdStrike Blocked | Palo Alto Networks Blocked |
---|---|---|---|
Valid Accounts: Domain Accounts | NO | N/A | N/A |
Remote Services: SMB/Windows Admin Shares | NO | N/A | N/A |
Ingress Tool Transfer (wsmproav.exe) - exaramel malware | YES | NO | YES |
Ingress Tool Transfer (service binary) - exaramel malware | YES | NO | |
Create or Modify System Process: Windows Service - exaramel malware | YES | NO | |
Boot or Logon Autostart Execution: Registry Run Keys / Startup Folder | YES | NO | |
System Services: Service Execution | YES | YES | |
Remaining Steps (8 - 21) | YES |
As a consequence, the test system suffered the following impacts before the attack chain was ultimately prevented:
In summary, despite preventing the attack from completing, the CrowdStrike response for this test left a backdoor fully installed and configured to start automatically upon a future reboot. This leaves the system in a compromised state and at risk of suffering future impact by the attackers unless it is promptly remediated by analysts.
In order to fully remediate this attack, not only must the hostile process chain be terminated (it eventually was), but now the changes that were made by the attacker must also be discovered and corrected. In scenarios where the attack is widespread across an organization, the full extent of these remediations must be discovered and corrected at significant scale. While prevention did ultimately stop this attack chain from fully completing, it left numerous hostile artifacts and potentially damaging changes within the environment that must be separately addressed in order to achieve full remediation.
This attack sequence involves the installation and enabling of a number of attacker utilities, including a command and control module, a keylogger, and the use of a utility to dump and read the contents of various local database files, which are then utilized to collect information vital to the attackers and then exfiltrate that information to an external destination. Throughout the course of the attack, successive degrees of impact occur as these tools are installed, activated and then employed. While a delay in prevention may disrupt the attack before it can fully complete, any significant delay, as was observed in the CrowdStrike result, leaves the machine in a compromised state, and potentially open to further actions by the attackers via the elements which were installed and configured ahead of the prevention action. Early and accurate prevention is critical in minimizing this impact and in reducing the overall recovery efforts undertaken by the response team.
These collective steps described in these examples represent a significant number of security impacts that will occur if each are allowed to complete, and each step that is completed results in negative impact to the organization that must ultimately be remediated and may include business disruption, financial losses and other business impacts. Therefore it is critical that the security tool not only block the attack, but block this attack as early as possible in the attack sequence to minimize the impact to the organization. This is why it becomes important to gauge the point at which each vendor blocks the attack; the earlier the better. When we apply this lens to different vendors, we can quickly see that they perform very differently, and those differences carry significant implications for the security team.
MITRE Round 4 - Combined Prevention Efficacy and Detection Results
Chart 2: Y-Axis tracks out-of-the-box Technique Detections (config changes removed). X-Axis tracks substep level prevention results (97 blockable substeps).
These examples highlight the importance of not just IF prevention happens, but WHEN prevention happens during an attack; delays have significant consequences. We will be posting additional blogs in the coming weeks to analyze the MITRE results and what they mean and how you can use the results to better understand the efficacy of different security solutions.
Learn more about how to unpack the MITRE results by joining our ‘Dissecting the 2022 MITRE ATT&CK Evaluations Webinar and visiting our MITRE webpage.
By submitting this form, you agree to our Terms of Use and acknowledge our Privacy Statement. Please look for a confirmation email from us. If you don't receive it in the next 10 minutes, please check your spam folder.