Unit 42の最新のクラウド脅威レポートには、弊社のあるお客様（SaaSプロバイダ）のCI/CD（継続的インテグレーション/継続的デリバリー）パイプラインに対して実施されたレッドチーム演習の概要が含まれています。このお客様からのご依頼は、弊社リサーチャーに攻撃者の考えにならってSaaSプロバイダの開発運用(DevOps)プロセスに潜む脆弱性と設定ミスをあぶりだしてほしい、ということでした。レッドチーム演習では、現実の脅威を模倣し、既知の攻撃手法に対するセキュリティ能力を評価するため、SolarWindsのOrionサプライチェーン攻撃で使用された戦略と手法を手本としました。リサーチャーがインサイダー脅威役を演じます。目標はCI/CDパイプラインのコンポーネントを改変するに足るアクセス権を取得し、最終的にサプライチェーン攻撃を行える状況を作り出すことです。
Unit 42のリサーチャーはレッドチーム演習を立案する際に、SolarWinds Orionのセキュリティ侵害を手本としました。この攻撃を成功に導いた要因を学び、リサーチャーが演習を実施する上での知見を得てから、SaaSベンダのCI/CDパイプラインがどのように侵害される可能性があるかを実演したのです。最終的にこの種のセキュリティ侵害は、下流に存在する無数の組織が自社環境に有害なバックドアを意図せず受け入れてしまうことにつながります。
The most recent Unit 42 Cloud Threat Report contains the high-level results of a red team exercise performed against a SaaS customer’s continuous integration and continuous development (CI/CD) pipeline. In other words, a customer asked our researchers to think like attackers, with the aim of revealing vulnerabilities and misconfigurations in their development operations (DevOps) processes. During the red team exercise, researchers took guidance from the strategies and techniques used by the attackers behind the SolarWinds Orion supply chain attack, in order to emulate a real-world threat and assess the security practices against known attacker techniques. The researchers posed as insider threats with the intent of gaining sufficient access to modify components of the CI/CD pipeline, resulting in a potential supply chain attack.
Researchers were able to achieve administrator access, aka “God Mode,” within the organization’s cloud environment due to the hardcoding of 26 identity and access management (IAM) key pairs being stored within an internal GitLab repository. (IAM key pairs are a set of security credentials – a public key and a private key – that can be used to prove identity when connecting to a public cloud instance. Since these are literally keys to a cloud environment, access to them should be safeguarded, especially if they correspond to privileged roles.)
In the case of the SaaS customer, every repository within the GitLab environment was accessible to any of the organization’s developer accounts. Some of the key pairs stored in the GitLab repository allowed researchers to escalate their permissions within the organization’s cloud environment to the extent that they would be capable of compromising the CI/CD pipeline, potentially resulting in hundreds, if not thousands, of downstream clients being affected.
Only one out of more than 50 total AWS accounts identified by the researchers was being monitored by Amazon GuardDuty (an AWS security tool that continuously monitors configured AWS accounts for malicious activity) and the Prisma Cloud platform. The organization did not have these security measures enabled on their other 49+ AWS accounts. Luckily, the organization’s security operations center (SOC) was able to detect a portion of the researchers’ actions through that one monitored AWS cloud account.
We are keenly aware of recent real-world supply chain attacks such as SolarStorm, the attack on SolarWinds, which targeted a SaaS provider’s CI/CD pipeline and compromised thousands of downstream clients. Given that we have seen major attacks of this nature in the wild, the findings from this red team exercise are significant and should be conveyed to all organizations that operate a CI/CD architecture.
The following sections will give readers insight into how a supply chain attack targeting a cloud-based SaaS provider could be carried out. By outlining the strategies and techniques Unit 42 researchers performed during their own red team exercise (as was described within the Cloud Threat Report), and which was itself modeled in part on the SolarWinds supply chain attack, this hopefully gives readers and architects of cloud-based CI/CD pipelines a view into how their own pipelines can be targeted and mined for sensitive information leading to the compromise of their environments. After the description of the attack, the final section provides guidance for how to mitigate attacks on the CI/CD pipeline.
Compromising the CI/CD Pipeline
In the red team exercise, our researchers showed they could compromise the customer’s CI/CD pipeline – but what exactly does that mean? What happens behind the scenes when a vendor’s supply chain, or its CI/CD pipeline, is compromised? To answer this question, let us first understand why CI/CD pipeline development channels are considered the industry standard.
SaaS vendors use CI/CD pipelines to provide rapid deployment capabilities for their services and applications. This allows the vendor to quickly innovate and build market-leading platforms, but it also allows the vendor to quickly fix or patch components of the application should vulnerabilities or misconfigurations be identified. Using the CI/CD pipeline allows organizations to build smaller, more modular components that function congruently with other modular components, forming a larger and more robust application ecosystem. Additionally, building in a modular fashion, and by using the CI/CD pipeline workflow, allows the vendor’s engineering teams to quickly build, update or patch singular components of the application instead of requiring them to completely rebuild a more monolithic application.
In order for customers to use applications as intended, vendor organizations will offer their product/application as a set of containerized entities. Each of these containers holds a singular component of the larger application that, when combined, creates the application’s functional ecosystem. Typically, vendors will also containerize an application’s supporting infrastructures such as network mesh infrastructure configurations; user, roles or policy IAM configurations; or even user interface options. Each of these individual containerized application plug-ins is then packaged and uploaded to an external repository, where thousands of customers can download the application and any of the associated plug-ins they require to build their customizable application within their own cloud environments.
This brings us to malicious attackers. Some attackers deliberately target SaaS vendors with the specific mission of compromising that vendor’s CI/CD pipeline to insert malicious code into a portion of the application’s containerized ecosystem. Figure 1 illustrates an eight-step operation an attacker could use to compromise a SaaS vendor’s CI/CD pipeline and ultimately gain a foothold in the vendor’s cloud environment and what’s worse, use that vendor’s environment to send poisoned applications or plugins to downstream organizations resulting in the exponential growth of compromised victim environments, as was seen within the fallout of the SolarWinds attack.
First, attackers need to identify how an organization operates. Attackers will find and then scan any known Infrastructure as Code (IaC) repositories associated with the customer and any employees who work for them. Common IaC repositories are GitLab/GitHub, ArtifactHub, Docker Hub, Quay, Google Container Registry (GCR) and Amazon Elastic Container Registry (ECR). Each of these registries, either internal or external, can contain information that can be used during an offensive operation. During the red team exercise carried out by the Unit 42 researchers, we were able to uncover 26 hardcoded AWS IAM access IDs and their associated key pair or session token. While most of the key pairs were inactive, there were some that allowed the researchers to gain access to the cloud environment. Properly securing IAM credentials is a key concern for organizations as Unit 42 researchers have also found significant increases in the number of cloud environments not implementing secure IAM configurations.
Step 2 – Initial Access
The attackers will attempt to gain access to the vendor’s cloud infrastructure by leveraging weaknesses in the vendor's known cloud infrastructure. Or – as was the case with the red team exercise – attackers may take advantage of an easier way in. In the red team exercise, the researchers were able to simply log into the cloud environment using the hardcoded IAM access key pairs identified during the reconnaissance phase. Initial access will traditionally include leveraging misconfigurations within the vendor’s IaC templates, or vulnerabilities within the cloud-hosted applications and containers.
Step 3 – Lateral Movement
Once attackers have gained access to a cloud-hosted system, they will begin the process of expanding that access by attempting to locate the organization’s CI pipeline. Lateral movement within a cloud environment is similar to lateral movement within a traditional on-prem infrastructure, aside from the presence of container escape operations. For these operations, instead of moving laterally to physically separate systems, the attacker is moving between a virtual system and the physical system hosting that virtual system.
During a container escape, the attacker will attempt to move laterally from the container process to the container host system, allowing them additional privileges making future operations both possible and easier. For example, this could mean moving from a public-facing web server process running within a Kubernetes (k8s) cluster to the system which is hosting that k8s container. The escape technique used depends upon several factors, such as misconfigurations or vulnerabilities within the k8s cluster, but it could include leveraging shared volumes, libraries or namespaces between the host and the container, or it could involve taking advantage of overprivileged container rights. Additionally, the container itself may host IAM key pairs which could allow the attacker to simply log directly into the cloud platform hosting the cloud systems. By either escaping the container, discovering IAM key pairs or connecting to other containers within the same hosting machine, attackers can move laterally and discover new cloud infrastructure.
Through this process, attackers can find their way to the CI pipeline.
Step 4 – Target CI Pipeline
Once attackers locate the organization’s CI pipeline, they will need to acquire access to the CI pipeline management service. Similar to step 3’s lateral movement operations, this will require one of the following approaches:
The exploitation of a vulnerability or misconfiguration within the CI management service.
The ability to masquerade as a legitimate user with access to the CI platform by using compromised or stolen credentials obtained during previous operations (steps 1-3).
Step 5 – Poison CI Pipeline
The attackers now have access to the CI pipeline and can affect it in various ways, including “poisoning” components of the SaaS application. A poisoned component can be used to perform malicious actions such as building backdoor functionality into one or more of the application’s components, allowing attackers to enter downstream client environments. In the case of the SolarWinds attack, the attackers built a malicious backdoor into the networking plug-in module for the Orion application. By building network connectivity operations into the application’s components, attackers have an opportunity to integrate their malicious code into the vendor’s official application packaging. In the case of the SolarWinds attack, the attackers were even able to sign their malicious package with a legitimate digital signature, which further masked their malicious content. The poisoned package is then made available to the vendor’s customer base, and all the attackers need to do is wait.
Step 6 – Client Uploads Poisoned Images
The poisoned package is downloaded by a client. Since the customer is likely to perceive it as legitimate, the customer will build that package into their cloud or on-prem infrastructure – giving attackers the access they’ve been working toward.
Step 7 – Malicious Beacons to C2 Infrastructure
Once the poisoned package has been installed, the attackers’ malicious code will send a beacon to the attackers’ malicious command and control infrastructure (C2) and also open a backdoor within the compromised customer’s cloud infrastructure.
Step 8 – Attacker Gains Keyboard Access
The beacon signals to the attackers that there are new victims of their supply chain attack. The attackers will enter the compromised cloud instance or container through the backdoor. From here, the attackers will begin another round of reconnaissance, lateral movement and privilege escalation; however, now they will be looking within the compromised customer’s infrastructure. The attackers will perform any number of activities depending upon their mission set. This can include intellectual property theft, extortion (such as through ransomware), data destruction and so on.
These eight steps show how, from a seemingly innocuous mistake such as insecure storage of IAM key pairs, attackers can build to a far-reaching attack on the cloud infrastructures of many organizations.
SolarWinds – Failure to Secure the Build Environment
Unit 42 researchers took the SolarWinds Orion compromise as the north star when designing our red team operation. Understanding how the attackers successfully performed that attack informed how the researchers performed our operation – demonstrating how a SaaS vendor’s CI/CD pipeline can be compromised. A compromise of this nature could ultimately lead to tens of thousands of downstream organizations unwittingly becoming the recipient of malicious backdoors into their environments.
To offer an illustration of how, starting by compromising one SaaS vendor, attackers could compromise at least 18,000 unique organizations, see Figure 2. This lays out a model for an exponential growth pattern, following the steps described above, through which a malicious modification of a single SaaS vendor can spread to downstream organizations.
Figure 2. Exponential growth model.
The attack on SolarWinds can be viewed through the lens of this model. On Dec. 13, 2020, FireEye released information related to a breach and data exfiltration originating from an unknown actor FireEye called UNC2452. Unit 42 researchers tracked this event and related activity and labeled the group SolarStorm. Unit 42 published the observed techniques, indicators of compromise (IoCs) and relevant courses of action in the Unit 42 ATOM Viewer. According to FireEye, SolarStorm compromised organizations across the globe via a supply chain attack that consists of a trojanized update file for the SolarWinds Orion Platform (see Figure 3).
Figure 3. Path of attack against SolarWinds Orion.
SolarStorm specifically targeted supply chain operations for SolarWinds’ Orion project, singling out their IT performance and statistics monitoring software. From there, SolarStorm threat actors modified the legitimate and digitally signed .dll file, SolarWinds.Orion.Core.BusinessLayer.dll. This .dll behaved as a SolarWinds Orion plug-in which was opened by the SolarWinds process, SolarWinds.BusinessLayerHost(x64).exe. The .dll file, having now been digitally signed, was added to the legitimate HotFix #5 package for SolarWinds Orion (CORE-2019.4.5220.20574-SolarWinds-Core-v2019.4.5220-Hotfix5.msp). The embedded “legitimate” .dll file essentially became a trojanized software supply chain technique. Once the supply chain infection was propagated to SolarWind customers, it provided backdoor access to infected environments. The backdoor operations went largely unnoticed, from as early as March 2020 when the HotFix was first released, as the C2 traffic masqueraded as legitimate Orion Improvement Program traffic.
How to Mitigate Attacks on the CI/CD Pipeline
In order for other organizations to combat and remediate the actions taken during this red team exercise, Unit 42 recommends the following steps:
Limit developer access to internal CI repositories to only those repositories which will be directly used by the developer working on that repository.
Limiting the access of a developer account to only the repositories required for a specific job will assist in preventing sensitive data leakage. Limiting access could have prevented the red team operators from identifying misconfigured IAM key pairs.
Scan containers and IaC templates for misconfigurations and vulnerabilities throughout the development cycle.
Scanning all containers and IaC templates for misconfigurations or vulnerabilities using tools like Prisma Cloud’s BridgeCrew, or the open-source tool Checkov, can assist organizations in building secure cloud infrastructure and prevent malicious actors from finding footholds in their environment.
Implement drift detection across all IaC and container images.
Drift detection is the process of alerting security teams to the modification of containers or IaC templates over time. Prisma Cloud now has drift detection built into its BridgeCrew scanning operations.
Monitor CI/CD infrastructure for behavior changes from developers and network traffic patterns.
In the red team exercise, the SaaS vendor’s SOC did identify some of the researchers’ activities thanks to a single account configured with Amazon GuardDuty and Prisma Cloud. However, there were several additional detection mechanisms that could have been used to catch the researchers – aside from being sure protections are enabled on all accounts. By using anomaly detection from Prisma Cloud, the security team could have been alerted to network reconnaissance operations or unusual user activity.
To learn more about what our red team exercise revealed – and how to protect your cloud infrastructure – download the Unit 42 Cloud Threat Report, 2H 2021.
Related Blogs
Our library of online content is here to help you learn more, no matter what format you prefer or which topic interests you most.
Sign up to receive must-read articles, Playbooks of the Week, new feature announcements, and more.
By submitting this form, you agree to our Terms of Use and acknowledge our Privacy Statement. Please look for a confirmation email from us. If you don't receive it in the next 10 minutes, please check your spam folder.
Get the latest news, invites to events, and threat alerts
