What Is Ungoverned Usage of Third-Party Services?

Ungoverned usage of third-party services — an OWASP Top 10 CI/CD Security Risk — involves the unregulated integration of external services into an organization's CI/CD systems. The simplicity of third-party integrations often bypasses the need for special permissions or approvals. A resulting lack of oversight and visibility hinders the implementation of role-based access control (RBAC) within CI/CD systems, exposing the organization to a host of security vulnerabilities. To counteract risk, AppSec and DevSecOps teams must implement governance controls throughout the lifecycle of third-party service usage — from approval and integration to ongoing usage monitoring and eventual deprovisioning.

CICD-SEC-8: Ungoverned Usage of Third-Party Services Explained

Ungoverned usage of third-party services in the context of continuous integration and continuous deployment (CI/CD) refers to the unchecked and unregulated use of external services granted access to an organization's CI/CD systems and resources.

DevOps teams frequently leverage third-party services for tasks such as code analysis, artifact storage, or monitoring. The simplicity of these integrations and the immediate value they provide make third parties essential to daily engineering operations. But as the diversity of integration methods increases and their associated complexities decrease, organizations face the challenge of maintaining control.

Challenges in Controlling Third-Party Integrations

Consider GitHub SaaS, a common source code management (SCM) system, for example. Third-party applications can connect to GitHub through several methods — GitHub applications, OAuth applications, access tokens, SSH keys, webhook events. Each method requires only seconds to minutes to implement and can grant third parties a wide range of capabilities, which can extend from reading code in a single repository to administering the GitHub organization.

Integration of Third Parties in Build Systems

Build systems also facilitate easy integration of third parties. Adding a few lines of code to the pipeline configuration file or installing a plugin from the build system’s marketplace can integrate third-party functionality. The build process then imports and executes the functionality, providing full access to the resources available at the pipeline stage.

But the lack of governance and visibility around third-party implementations can prevent organizations from maintaining role-based access control (RBAC) within their CI/CD systems. The permissive nature of third parties means that an organization's security is only as strong as the security of its third parties. Minimal governance and diligence around third-party implementations, coupled with insufficient implementation of RBAC and least privilege, can significantly increase an organization’s attack surface.

Third-Party Services Defined

Third-party services streamline complex tasks, enabling DevOps and AppSec teams to focus on core competencies. Source code repositories like GitHub or Bitbucket offer version control, while Jenkins or CircleCI automate builds and tests. Container orchestration via Kubernetes or Dockers ensures scalability, and cloud platforms like AWS or Azure provide infrastructure. The significance of these services lies in their contribution to productivity, enabling faster deployment times and encouraging resource optimization.

The selection of external services hinges on factors such as functionality, compatibility, cost, and vendor reputation. Teams often lean toward services with extensive community support and comprehensive documentation. In a healthy CI/CD pipeline, third-party services interoperate seamlessly, each contributing to a cohesive, efficient, and secure software delivery process. Regular updates, maintenance, and monitoring ensure these services perform optimally.

Components of Effective Governance of Third-Party Services

Effective governance of third-party services involves policy formulation, oversight, risk management, and performance monitoring to ensure that the use of services adheres to the organization's security standards. It entails defining acceptable use, tracking compliance, and maintaining an inventory of third-party services. But several influences can challenge effective governance. Examples include:

• Lack of visibility into third-party service use
• Ineffective policies
• Inattention to service terms
• Lack of understanding of the associated risks
• Insufficient internal resources for management
• Rapid adoption of new technologies

How CICD-SEC-8 Happens

Let's imagine a scenario that illustrates how an attacker exploits an insecure system configuration — and underscores the importance of effective governance controls.

Identifying the Target

The attacker identifies an organization that uses a third-party static analysis platform within its CI/CD pipeline. This platform is granted an access token, allowing it to read and write code in the organization's repositories.

Exploiting the Vulnerability

The attacker discovers a vulnerability in the third-party platform's security, specifically in its token management system. The platform doesn’t adequately monitor or limit the usage of its access tokens. The attacker launches a phishing attack, targeting an employee who has the ability to generate or manage access tokens for the third-party service. At the same time, the attacker uses a brute-force method to predict a poorly generated access token.

Gaining Unauthorized Access

With the stolen access token, the attacker impersonates the third-party service and gains the same level of access to the organization's repositories. The CI/CD systems don’t have monitoring in place to detect the unusual activity, which allows the attacker to operate undetected.

Lateral Movement

Using the permissions of the compromised third-party service, the attacker moves laterally within the organization's systems. They target a repository that contains the organization's pipeline configuration files. Within these files, they find credentials for the build system, stored in plaintext. The attacker uses these to gain access to the build system where they can modify the build scripts and pipeline configuration directly, bypassing the need for the third-party service.

Injecting Malicious Code

The attacker writes malicious code that opens a backdoor in the system, giving them remote access even after the initial access token isn’t valid. The malicious code is committed to a repository, triggering a build in the CI/CD pipeline. The build system, unaware of the malicious code, executes it as part of the build process, which gives the attacker control over the build system.

Data Exfiltration

The attacker uses their control over the build system to initiate a data leakage. They modify the build scripts to include commands that dump the production database and send it to an external server under their control. This server is set up to receive and store the sensitive customer information extracted from the database.

Maintaining Persistence

The attacker ensures their continued access by embedding a backdoor within the system. They modify the CI/CD pipeline to automatically include this backdoor in every build, ensuring their access persists even if the initial backdoor is discovered and removed. Using the stolen build system credentials, they compromise another third-party service connected to the CI/CD systems, further expanding their reach within the organization's infrastructure.

Importance of Governing Third-Party Services in CI/CD

Third-party services in a CI/CD pipeline introduce an external variable into a controlled environment. While these services empower development, they can extend a potential attack vector to an adversary. Given the highly interconnected dynamics of the CI/CD system and environment, compromise of a single third-party can ripple well beyond the scope of the system the third party is connected to.

Risks Associated with Ungoverned Usage of Third-Party Services

Ungoverned usage can lead to several risks, including data breaches, compliance issues, and exposure to vulnerabilities within the third-party service. Moreover, it can cause system instability, service unavailability, or unauthorized access. The overall risk is a diminished trust in the organization's ability to safeguard its digital assets.

Case Study 1: From Compromised DeepSource Account to GitHub Application Permissions

In July 2020, GitHub's security team alerted DeepSource about potential malicious activity related to the DeepSource GitHub application. The activity, which began in mid-June, involved a large number of requests from unusual IP addresses for various DeepSource users. DeepSource promptly rotated all user tokens, client secrets, and private keys. Further investigation revealed that an employee's GitHub account had been compromised by the Sawfish phishing campaign, granting the attacker access to DeepSource GitHub app's credentials.

Case Study 2: Unauthorized Use of GitHub OAuth Token Behind Waydev Breach

In July 2020, Waydev, a software analytics company, faced a security incident involving unauthorized use of a GitHub OAuth token. The attack was reported by a user in their trial environment. Waydev's security team, in collaboration with an independent company, Bit Sentinel, identified multiple attacks, exploratory activities, and automated scanners. The company promptly fixed the issues, revoked all GitHub OAuth tokens, and took additional security measures.

Case Study 3: Codecov’s Docker Image Creation Process Allows Credential Extract

In April 2021, Codecov, a leading provider of code coverage solutions, discovered unauthorized alterations to their Bash Uploader script. The alterations, which began in January 2021, allowed a third party to export information from users' continuous integration environments to an external server. The breach was due to an error in Codecov's Docker image creation process. Upon discovery, Codecov secured the affected script, initiated an investigation, and advised affected users to reroll all credentials, tokens, or keys used in their CI processes.

Preventing Ungoverned Usage of Third-Party Services

AppSec teams with an understanding of the risks associated with ungoverned usage of third-party services are better equipped to manage risks through governance controls at every stage in the third-party usage lifecycle. Primary recommendations include:

Approval

Establish vetting procedures to ensure third parties granted access to resources are pre-approved and granted only the level of permission that align with the principle of least privilege.

Integration

Introduce controls and procedures to maintain continuous visibility of all third parties integrated to CI/CD systems, including:

• Method of integration. Make sure all methods of integration for each system are covered (marketplace apps, plugins, OAuth applications, programmatic access tokens, etc.).
• Level of permission granted to the third party.
• Level of permission in use by the third party.

Visibility over Ongoing Usage

Ensure each third party’s access is limited and scoped to the specific resources required and remove unused or redundant permissions. Third-parties services integrated as part of the build process should run inside a scoped context with limited access to secrets and code and with strict ingress and egress filters.

Deprovisioning

Periodically review your inventory of integrated third parties and remove those no longer in use.

Industry Standards for Governing Third-Party Services

Standards such as ISO/IEC 27001 offer guidelines for third-party service governance, including implementing service level agreements (SLAs) and in some cases, business associate agreements. Additionally, standards like OWASP ASVS define controls for using third-party services within application security.

Policy Formulation for Acceptable Use

Craft clear policies to prevent ungoverned usage of third-party services. Policies should define acceptable use and establish requirements for adopting new technologies, as well as covering aspects like data privacy, security standards, and compliance requirements. Policies must also outline the process for evaluating and integrating new third-party services, including security assessments, compatibility checks, and performance evaluations.

Oversight and Resource Allocation for Third-Party Service Management

Effective management of third-party services requires dedicated oversight and sufficient resources. Organizations should assign a designated team or individual the responsibility of managing these services, ensuring compliance with policies and addressing issues that arise. The oversight team should possess the necessary skills and resources to effectively manage the integration and use of third-party services, including understanding their functionality, monitoring their performance, and addressing any security or compliance issues.

Performance Monitoring and Compliance Tracking

Maintain an inventory of all third-party services and regularly monitor their performance, promptly addressing issues. Regular audits should be conducted to identify noncompliance issues, according to the organization's policies and relevant regulations.

Tools Supporting Governance of Third-Party Services

Several tools assist with the governance of third-party services and help organizations by automating the process of checking third parties for compliance with policies and standards.

Software composition analysis (SCA) tools, for example, can scan the codebase for third-party components, identifying their versions, licenses, and any known vulnerabilities, as well as creating and maintaining a software bill of materials (SBOM).

Static application security testing (SAST) tools can analyze source code for security vulnerabilities, including those introduced by third-party services. Dynamic application security testing (DAST) tools, like OWASP ZAP or Nessus, can identify security vulnerabilities in running applications, including those resulting from third-party services.

For containerized environments, a container security solution can scan container images for vulnerabilities, ensuring third-party services within containers meet security standards. As well, configuration management tools like Ansible or Puppet can enforce consistent configurations across third-party services, reducing the risk of misconfigurations that could lead to security vulnerabilities.

A cloud-native application protection platform (CNAPP) offers comprehensive security for cloud-native applications, seamlessly integrating many of the above tools into its platform. In addition to providing visibility into application components — including third-party services and their interactions — CNAPPs can detect anomalies, flagging potential security threats or policy violations. They also offer automated remediation, reducing the time to respond to security incidents.

By integrating a CNAPP into the CI/CD pipeline, organizations can continuously monitor third-party services for compliance with security policies, identify and address vulnerabilities, and ensure secure configurations. The CNAPP's ability to provide real-time security insights and enforce policies across the application lifecycle makes it a powerful tool for governing third-party services.

Ungoverned Usage of Third-Party Services FAQs