Anatomy of a Cloud Supply Pipeline Attack
Written by: Nathaniel Quist
Supply chain attacks targeting third-party vendors, open-source software, or other components within a supply chain have been on the rise. As organizations increasingly rely on complex supply chains and external partnerships, attackers are exploiting these connections as potential entry points. The increased adoption of open-source software and cloud-based services has also expanded the attack surface for supply chain threats. High-profile incidents, such as the SolarWinds attack in 2020, have demonstrated the potential impact and reach of supply chain attacks, compelling organizations to prioritize supply chain security and improve their defenses against such threats.
Unit 42® Conducts Red Team Exercises
Unit 42 conducted red team exercise against a SaaS customer’s continuous integration and continuous development (CI/CD) pipeline. In other words, a customer asked our researchers to think like attackers with the aim of revealing vulnerabilities and misconfigurations in their DevOps processes. The high-level results of these exercises, as well as detailed analysis explaining how to avoid common supply chain issues, can be found in Unit 42’s Cloud Threat Report: Secure the Software Supply Chain to Secure the Cloud.
During the red team exercise, researchers took guidance from the strategies and techniques used by the attackers behind the SolarWinds Orion supply chain attack to emulate a real-world threat and assess the security practices against known attacker techniques. The researchers posed as insider threats with the intent of gaining sufficient access to modify components of the CI/CD pipeline, resulting in a potential supply chain attack.
Due to the hardcoding of 26 identity and access management (IAM) key pairs stored within an internal GitLab repository, researchers achieved administrator access, aka “God Mode,” within the organization’s cloud environment.
Every repository within the SaaS customer’s GitLab environment was accessible to the organization’s developer accounts. Some of the key pairs stored in the GitLab repository allowed researchers to escalate permissions within the organization’s cloud environment to the extent that they would be capable of compromising the CI/CD pipeline, potentially impacting hundreds, if not thousands, of downstream clients.
Only one of more than 50 AWS accounts identified by the researchers was monitored by Amazon GuardDuty and the Prisma Cloud platform. The organization didn’t have these security measures enabled on their other 49+ AWS accounts. Fortunately, the organization’s security operations center (SOC) was able to detect a portion of the researchers’ actions through the one monitored AWS cloud account.
Compromising the CI/CD Pipeline
In the red team exercise, Unit 42 researchers showed they could compromise the customer’s CI/CD pipeline — but what exactly does that mean? What happens behind the scenes when a vendor’s supply chain, or its CI/CD pipeline, is compromised? To answer this question, let us first understand why CI/CD pipeline development channels are considered the industry standard.
CI/CD Pipeline in Modern DevOps Workflows
SaaS vendors use CI/CD pipelines to provide rapid deployment capabilities for their services and applications. This allows the vendor to quickly innovate and build market-leading platforms, but it also allows the vendor to quickly fix or patch components of the application should vulnerabilities or misconfigurations be identified.
Using the CI/CD pipeline allows organizations to build smaller, more modular components that function congruently with other modular components, forming a larger and more robust application ecosystem. Additionally, building in a modular fashion, and by using the CI/CD pipeline workflow, allows the vendor’s engineering teams to quickly build, update, or patch singular components of the application, instead of completely rebuilding it, as DevOps teams would with a more monolithic application.
For customers to use applications as intended, vendor organizations offer their product application as a set of containerized entities. Each container holds a single component of the larger application that, when combined, creates the application’s functional ecosystem. Vendors typically containerize the application’s supporting infrastructures — network mesh infrastructure configurations, user, user roles, or policy IAM configurations, even user interface options. Each of these containerized application plugins is then packaged and uploaded to an external repository where thousands of customers download the application and the associated plugins required to build their customizable application within their cloud environments.
Enter CI/CD Pipeline Threat
Attackers sometimes target SaaS vendors with the mission of compromising the vendor’s pipeline to insert malicious code into a portion of the application’s containerized ecosystem. Figure 1 illustrates an eight-step operation an attacker could use to gain a foothold in the vendor’s cloud environment, poison the CI/CD pipeline, and ultimately use the environment to send poisoned applications or plugins to downstream organizations. This type of supply chain attack, as incurred by SolarWinds, results in an exponential radius of compromised victim environments.
Eight-Step Supply Chain Compromise Operation
Step 1: Reconnaissance
Attackers first need to identify how an organization operates. They’ll find and scan infrastructure as code (IaC) repositories associated with the customer and its employees. Common IaC repositories are GitLab/GitHub, ArtifactHub, Docker Hub, Quay, Google Container Registry (GCR) and Amazon Elastic Container Registry (ECR). Each of these registries, either internal or external, can contain information to use during an offensive operation.
During the red team exercise conducted by Unit 42, researchers were able to uncover 26 hard-coded AWS IAM access IDs and their associated key pair or session token. While most of the key pairs were inactive, some allowed the researchers to gain access to the cloud environment. Properly securing IAM credentials is a key concern for organizations, as Unit 42 researchers also found significant increases in the number of cloud environments failing to implement secure IAM configurations.
Related: CICD-SEC-2: Inadequate Identity and Access Management
Step 2: Initial Access
The attackers will attempt to gain access to the vendor’s cloud infrastructure by leveraging weaknesses in the vendor's known cloud infrastructure. Or attackers may take advantage of an easier way in.
In the red team exercise, the researchers simply logged into the cloud environment using the hard-coded IAM access key pairs identified during the reconnaissance phase. Initial access will traditionally include leveraging misconfigurations within the vendor’s IaC templates, or vulnerabilities within the cloud-hosted applications and containers.
Step 3: Lateral Movement
Once attackers gain access to a cloud-hosted system, they’ll begin expanding access by attempting to locate the organization’s CI pipeline. Lateral movement within a cloud environment is similar to lateral movement within a traditional on-premises infrastructure, aside from the presence of container escape operations. For these operations, instead of moving laterally to physically separate systems, the attacker will move between a virtual system and the physical system hosting that virtual system.
During a container escape, the attacker attempts to move laterally from the container process to the container host system, allowing them additional privileges that makes future operations both possible and easier. This could entail moving from a public-facing web server process running within a Kubernetes cluster, for example, to the system hosting the Kubernetes container.
The escape technique used depends on several factors, such as misconfigurations or vulnerabilities within the Kubernetes cluster. It could, though, include leveraging shared volumes, libraries, or namespaces between the host and the container — or perhaps taking advantage of overprivileged container rights.
Additionally, the container may host IAM key pairs that allow the attacker to log directly into the cloud platform hosting the cloud systems. Either by escaping the container, discovering IAM key pairs or connecting to other containers within the same hosting machine, attackers move laterally and discover new cloud infrastructure, eventually finding their way to the CI pipeline.
Step 4: Target CI Pipeline
Once attackers locate the organization’s CI pipeline, they’ll need to acquire access to the CI pipeline management service. Similar to the lateral movement operations in step 3, locating the CI pipeline will require one of the following:
- Exploitation of a vulnerability or misconfiguration within the CI management service
- Ability to masquerade as a legitimate user with access to the CI platform via compromised or stolen credentials obtained during previous operations (steps 1-3)
Step 5: Poison CI Pipeline
The attackers now have access to the CI pipeline and can affect it in various ways, including “poisoning” components of the SaaS application. A poisoned component can be used to perform malicious actions such as building backdoor functionality into one or more of the application’s components, allowing attackers to enter downstream client environments.
By building network connectivity operations into the application’s components, attackers have an opportunity to integrate malicious code into the vendor’s official application packaging. The poisoned package is then made available to the vendor’s customer base, and attackers need only wait.
In the case of the SolarWinds attack, the attackers built a malicious backdoor into the networking plugin module for the Orion application — and even signed their malicious package with a legitimate digital signature, further masking the malicious content.
Step 6: Client Uploads Poisoned Images
The client downloads the poisoned package. Since the customer likely perceives it as legitimate, the customer will build the package into their cloud or on-premises infrastructure — giving attackers the access they’ve been working toward.
Step 7: Malicious Beacons to C2 Infrastructure
After the poisoned package has been installed, the attackers’ malicious code sends a beacon to the attackers’ malicious command and control infrastructure (C2) and opens a backdoor within the compromised customer’s cloud infrastructure.
Step 8: Attacker Gains Keyboard Access
The beacon signals to the attackers that new victims have succumbed to their supply chain attack. The attackers will enter the compromised cloud instance or container through the backdoor.
From here, the attackers begin another round of reconnaissance, lateral movement, and privilege escalation. Now, though, they’re looking within the compromised customer’s infrastructure. The attackers then perform any number of activities — depending on whether their mission involves intellectual property theft, extortion (such as through ransomware), data destruction, corporate espionage, etc.
One seemingly innocuous mistake, insecure storage of IAM key pairs, and attackers breach myriad organizations.
Failure to Secure the Build Environment: Lessons from SolarWinds
Unit 42 researchers took the SolarWinds Orion compromise as the north star when designing our red team operation. By understanding how the attackers performed that attack, Unit 42 revealed the path to a SaaS vendor’s CI/CD pipeline compromise.
Related: SolarStorm Supply Chain Attack Timeline
Make no mistake — a compromise of this nature could result in tens of thousands of downstream organizations becoming the recipient of backdoor invasions, as illustrated in figure 2. Through the eight steps described above, the malicious code modification of a single SaaS vendor can propagate wildly across company lines.
Viewing the SolarWinds Attack Through the Exponential Growth Model
On Dec. 13, 2020, FireEye released information related to a breach and data exfiltration originating from an unknown actor FireEye called UNC2452. Unit 42 researchers tracked this event and related activity and labeled the group SolarStorm.
Unit 42 published the observed techniques, indicators of compromise (IoCs) and relevant courses of action in the Unit 42 ATOM Viewer. According to FireEye, SolarStorm compromised organizations across the globe via a supply chain attack that consisted of a trojanized update file for the SolarWinds Orion Platform (see figure 3).
SolarStorm specifically targeted supply chain operations for SolarWinds’ Orion project, singling out their IT performance and statistics monitoring software. From there, SolarStorm threat actors modified the legitimate and digitally signed .dll file, SolarWinds.Orion.Core.BusinessLayer.dll. This .dll behaved as a SolarWinds Orion plugin opened by the SolarWinds process, SolarWinds.BusinessLayerHost(x64).exe.
The tampered-with .dll file — added to the legitimate HotFix #5 package for SolarWinds Orion (CORE-2019.4.5220.20574-SolarWinds-Core-v2019.4.5220-Hotfix5.msp) — essentially became a trojanized software supply chain technique.
Once the supply chain infection was propagated to SolarWinds customers beginning in March 2020 with the HotFix release, it provided backdoor access to infected environments. The backdoor operations largely went unnoticed, as the C2 traffic masqueraded as legitimate Orion Improvement Program traffic.
How to Mitigate Attacks on the CI/CD Pipeline
To secure your CI/CD pipelines and effectively combat the maneuvers undertaken during the Unit 42 red team exercise, implement safeguards that fortify your stronghold and support resilience.
Limit Engineer Access
In accordance with the principle of least privilege, limit developer access to only those internal CI repositories the developer directly needs for a specific job. Role-based access control (RBAC) can be employed to assign appropriate access levels based on job responsibilities, reducing the risk of unauthorized actions and insider threats.
By limiting the access of a developer account, organizations go a long way in preventing sensitive data leakage. Limiting access could have prevented the red team operators from identifying misconfigured IAM key pairs.
Scan Containers and IaC Templates
Scan containers and IaC templates for misconfigurations and vulnerabilities using tools like Prisma Cloud, or the open-source tool Checkov, to build secure cloud infrastructure and prevent malicious actors from finding footholds in their environment.
By regularly scanning and validating both containers and IaC templates, organizations can detect potential vulnerabilities early in the development lifecycle, reducing the risk of attacks and maintaining the integrity of the CI/CD pipeline.
Implement Drift Detection
Implementing drift detection is an effective strategy to mitigate attacks on the CI/CD pipeline. By detecting configuration drift, organizations can quickly identify unauthorized changes, misconfigurations, or potential security risks in their deployed infrastructure.
Monitor CI/CD Infrastructure
Observing anomalous activities, such as unauthorized access or excessive resource usage, can indicate security risks. Organizations need to monitor CI/CD infrastructure for behavioral changes in developers and network traffic to quickly identify and remediate risks. Regular access reviews and automated audits help maintain an up-to-date understanding of traffic patterns and access permissions.
In the red team exercise, the SaaS vendor’s SOC identified some of the researchers’ activities via a single account configured with Amazon GuardDuty and Prisma Cloud. Several additional detection mechanisms could have been used to catch the researchers. With anomaly detection, the security team could have been alerted to network reconnaissance operations and unusual user activity.
Related: Defending Against the OWASP Top 10 CI/CD Risks
