Abusing Repository Webhooks to Access Internal CI/CD Systems at Scale

With the increasing adoption of CI/CD systems, organizations tend to adopt a common CI/CD architecture. This architecture combines SaaS-based source control management systems — such as GitHub and GitLab — with an internal, self-hosted CI/CD solution like Jenkins.

Many organizations using this architecture allow their CI/CD environment to receive webhook events from the SaaS source control vendors to trigger pipeline jobs. For the webhook requests to pass through the organization's firewall and access the internally hosted CI/CD system, SaaS-based source control management (SCM) vendors need to supply the IP ranges from which their webhook requests originate.

On the surface, these repository webhooks seem secure. But we recently tested these webhooks to identify security issues and discovered that anyone on the internet can overcome the IP restriction, access data and execute code on internal CI/CD pipelines at scale.

Let’s walk through how we abused repository webhooks and then discuss practical ways you can protect yourself from this CI/CD security risk.

Source Control Management Webhooks Are Common Targets

The IP range of the SaaS SCM webhook service is shared between all organizations using the SCM. Any webhook event sent from the SCM, regardless of the tenant, will have a source IP in this range. Bad actors see this as an opportunity to successfully send packets through any firewall allowing this IP range via webhook events.

^{Learn how attackers abuse repository webhooks to trigger pipelines, send malicious payloads, and access hundreds of internal CI systems in scale.}

The structure and contents of SCM webhook events are strict. Each event is sent as an HTTP POST request with predefined headers and a structured JSON in the body of the request. Users are limited in their ability to modify the content of the webhook event. They’re only allowed to set the URL of the event target, modify specific non-harmful headers and control some of the JSON field values without changing its structure.

Because of this limited flexibility, there aren’t many opportunities for exploitation. One known attack scenario includes triggering CI/CD systems from the internet by sending a request from a foreign SCM organization. But this type of attack can be easily mitigated because the configuration of a pipeline is usually bound within a specific SCM repository and organization. If not, a secret can be attached to the webhook and verified when the pipeline executes.

Adversaries who attempt to manipulate webhook events face multiple limitations, including:

• Minimal control around the content and structure of the event
• Dedicated IP ranges used exclusively — or nearly exclusively — by the SCM webhook service in GitHub and GitLab
• Protections in the pipeline systems around pipeline triggers

Most organizations feel comfortable allowing their internal CI/CD systems to receive webhook events from the SaaS SCM providers.

A skilled adversary, however, can bypass these limitations. Let’s discover how.

Unauthorized Access to CI/CD Endpoints

Because organizations typically have countermeasures to prevent bad actors from triggering pipelines, more advanced attackers will take a different approach to access internal CI/CD systems. Let’s pretend we’re a bad actor who wants to find an alternative way to abuse repository webhooks.

The IP range of the SCM vendor webhook service was opened in the organization’s firewall to allow webhook requests to trigger pipelines. But webhook requests can still be directed toward other CI/CD endpoints besides the ones regularly listening to webhook events. We can try to access these endpoints to view valuable data like users, pipelines, console output of pipeline jobs.

Or, if we’re lucky enough to fall on an instance that grants admin privileges to unauthenticated users — yes, it happens — we can access the configurations and credentials sections.

In Jenkins, the endpoints can be accessed using the HTTP GET method to retrieve data and the POST to add or modify resources.

GET: Webhooks only allow us to send POST requests, which isn’t helpful for us.

POST: We can send POST requests using webhooks, but we face two other challenges. We can’t control the body of the request. As well, Jenkins requires adding a CSRF token to POST requests, but we don’t have the CSRF token.

So where do we go from here?

How to Abuse Jenkins Login

Let’s look for a way to get around the roadblocks we discovered above, starting with the Jenkins login page. We can try to brute force user credentials for one key reason. It’s common to see Jenkins users managed in its own user database or other user management methods. The other methods typically lack basic protections, such as a password policy or protections against automations, which gives us an opportunity to abuse the login.

The login requires us to send a POST request. Choosing to target the login endpoint solves the challenge of holding CSRF tokens because this request doesn’t require a token. We’re still limited in our ability to modify the body of the request, though.

A Jenkins login request looks as follows:

We need to send the credentials we brute force somehow. Fortunately, the Jenkins login endpoint accepts a POST request with the fields sent as query parameters:

Let’s create a new webhook in GitHub and set the Jenkins login request URL as the payload URL. We can then create an automation using the GitHub API to brute force the user account’s password by modifying the password field, triggering the webhook and inspecting the response in the repository webhook event log.

Then we can fire off the webhook. All SCM vendors display the HTTP request and response sent through the webhook in their UI. If the login attempt fails, we’re redirected to the login error page.

A successful login will set a session cookie¹ and direct us to the main Jenkins page.

While we did get a session cookie, we can only send one stateless request each time, and the cookie can’t be attached to our request because we can’t control the headers.

Is there a way to skirt this limitation?

We could obtain a Jenkins access token, which can be attached in the URL and used to send POST requests to Jenkins without the need of adding a CSRF token. This option is more complex because it requires an attacker to somehow find both a self-hosted CI/CD system only accessible from SCM IP ranges and a valid access token to that CI/CD. For this exercise, we’ll focus on more practical scenarios.

Abusing GitLab Webhooks

Now let’s send the same request using GitLab. Like GitHub, we have limited control over the content of the payload sent in the webhook event, so we send the same POST request and add the credentials as query parameters.

We trigger the request, but the response is 200. As with our previous example, we used GitLab’s webhook service to brute force a user and obtain a session cookie. This time, the contents of Jenkin’s response were relayed back to the GitLab UI, providing us with the full content of the Jenkins main page:

So what happened? When a webhook sent from GitLab returns a 302 response code, GitLab automatically follows the redirection. Because GET requests follow 302 redirections, we’re able to leverage GitLab to bypass the POST request limitation and send GET requests to targets from the GitLab webhook service. We couldn’t do that with GitHub.

We send the next event with the cookies set in the first response. As you can see, the response we received contains internal Jenkins data, such as the pipelines and their execution status.

In summary, we can:

• Brute force users and discover valid credentials
• Use the valid credentials against the login page to log in successfully
• Get the contents of the internal Jenkins main page

Like many other login mechanisms, Jenkins login accepts the redirection parameter “from.” This parameter redirects users to the page they aimed to reach after they log in. But the parameter also serves as a feature we can abuse to send a GET request attached with a session cookie to an internal Jenkins page of our choice. Let’s see how.

Step 1: Set a webhook with the following URL:

A POST request is sent to Jenkins, and the authentication succeeds.

Step 2: We get a 302 redirect response with a session cookie and a redirection to the job console output page.

Step 3: The GitLab webhook service automatically follows the redirection with a GET request sent to the job console output page. The session cookie is added to the request:

Step 4: Job console output is sent back and presented in the attacker’s GitLab webhook event log.

Keep in mind that Jenkins can be configured to allow access to internal components without authentication, or in a way that restricts access to authenticated users. How does that affect us?

• If there’s no authentication configured, we can make the GitLab webhook service access any internal page in the CI/CD system, capture the response and present it to us.
• If authentication is configured, we can try to brute force a user. We can then use the credentials to access any internal page, as described in the bullet above.

Vulnerability Exploitation: Another Access Path

We can’t always obtain an access token or password, but it’s still possible to gain access via vulnerability exploitation.

Jenkins is an ecosystem of plugins — with each plugin created by a different maintainer — which means that one vulnerable plugin has the potential to affect the entire system. That’s why hackers scour Jenkins for new vulnerabilities.

And because organizations use Jenkins for sensitive operations — building, testing and deploying to production systems — maintaining up-to-date patches and updating to secure core and plugin versions is challenging. These updates can sometimes affect the system’s stability or availability, so active production Jenkins installations frequently have unpatched and out-of-date Jenkins plugins or Jenkins core versions. Additionally, Jenkins typically resides inside the perimeter, creating a false sense of security that can lull organizations to further deprioritize Jenkins security.

As we explore Jenkins security, one question comes up: how much damage can an attacker inflict on Jenkins through a simple webhook request?

In 2019, Orange Tsai discovered a vulnerability in Jenkins that allowed executing code by sending one unauthenticated request. At a high level, the attacker triggers Jenkins to download a jar file from a remote location, leading to code execution on the instance.

Let’s set up a malicious webhook to exploit an unexposed vulnerable Jenkins instance located behind a firewall. Our goal is to establish a reverse shell on that instance.

The exploit payload is sent as a GET request and requires an attacker to do the following:

Step 1: Set a server that will:

• Receive a POST request with a redirection parameter and respond with a 302 redirection.
• Host the malicious jar file that is fetched by the Jenkins instance.
• Listen to the traffic arriving from the reverse shell we’ll run on the Jenkins instance.

Step 2: Create a webhook in a GitLab project and set its URL as the attacker’s server, with the redirection parameter containing the payload:

Step 3: Trigger the webhook to send an event.

Step 4: The webhook event arrives at the attacker’s server, which responds with a 302 redirection to the Jenkins instance along with the payload.

Step 5: The GitLab webhook follows the redirection with a GET request containing the payload that is sent to the Jenkins instance.

Step 6: The exploitation process starts. The Jenkins instance downloads the jar file from the attacker’s server, which runs a reverse shell on the instance and allows the attacker to execute commands remotely.

Accessing Internal Jenkins Instances at Scale

So far, we’ve discovered that even internal Jenkins instances can be accessed from the internet through SCM webhooks. We also know it’s possible to gain full control over a Jenkins instance with a single webhook event. A natural question emerges — how many Jenkins instances are susceptible via abused webhooks?

We scanned a fraction of the internet to identify Jenkins instances not accessible from the public internet but accessible from the GitLab webhook service. Though not as common as GitHub and other vendors, we chose GitLab because it allows more flexibility, which can be used to abuse an instance.

This was our approach:

Step 1: Use passive DNS services to discover potential Jenkins subdomains. We used a set of 900,000 records. For obvious reasons, many of those records were invalid, meaning they’re not real records or there was no Jenkins — or any other service — behind them.

Step 2: Verify which of these addresses can be resolved and filter out inactive subdomains.

Step 3: Access all subdomains from a standard IP address through HTTP[S] and keep the inaccessible ones.

Step 4: Send a request to each of the 800,000 subdomains we have left through the GitLab webhooks service. It’s safe to assume that the vast majority of these subdomains were not Jenkins instances.

From this list, we found 115 Jenkins instances accessible through the GitLab webhook service. We didn’t attempt to perform any action against them, but all of the earlier techniques to extract information or execute code are potentially valid for these instances.

We assume it’s possible to identify hundreds of additional Jenkins instances through the GitLab webhook service — and more through popular vendors. After all, plenty of self-hosted CI/CD vendors and other types of systems are accessible from these webhook services — artifact registries, for example — and can potentially be accessed and abused.

How to Protect Your Environment from Malicious Webhooks

You can harden your environment against malicious webhooks using one of several methods, as well as in layers, depending on your organization’s needs.

One solution involves denying inbound traffic from the SCM webhook’s IP range and stopping the reception of webhook events directly from the SCM. Alternatively, you could periodically poll changes by the CI/CD system or implement a proxy between the SCM and CI/CD system. Both of these methods, though, can involve high start-up costs and may fail to meet engineering needs.

If you need to keep receiving webhooks directly from the SCM, make sure to follow these guidelines:

• Only allow the inbound traffic to reach the CI/CD system and not any additional internal service.
• If possible, only allow inbound traffic to reach specific endpoints of the CI/CD system.
• Implement a secure authentication mechanism in the CI/CD pipeline that provides security controls aligned with industry best practices, such as integrating with the organization’s single sign-on (SSO) solution.
• Update your CI/CD system and its installed plugins to the latest available version.
• Disable anonymous access to the CI/CD pipeline.
• Enable an audit log in the target system and send all relevant logs to your security information and event management (SIEM), or alternative logging aggregation solution, to identify malicious actions.

Getting Started with CI/CD Security

Because of the data they store and the workloads they run, CI/CD systems are among the most critical and sensitive assets in your organization. Don’t rely on your SCM vendor’s webhook services to protect them. Safeguarding your delivery pipelines requires a comprehensive approach to CI/CD security

If you’re interested in learning more practical tips to get started with CI/CD security, read the CI/CD Security Checklist. You’ll find six best practices to help you embrace CI/CD security and harden your pipelines over time.

We’d like to thank Yaron Avital, Tyler Welton and Daniel Krivelevich for their contribution to this research.

References

1. In Jenkins version 2.266 the Acegi security library used for authentication was replaced by Spring Security. The Spring library provides indication on successful login attempts when logging in by sending a POST request with the credentials attached as query parameters, however it doesn’t return the session cookie. According to Jenkins statistics, around 30% of all instances use prior versions to 2.266.