Securing Golden Images at Build Using Prisma Cloud

Organizations often have a preferred base image from which they create virtual machines or instances. Base images are generally made up of a specific OS build and patch level, along with any software, configuration and security settings required by the organization or a specific use case.

These base images are bundled into a “golden image” and shared as a standard across the organization. As time goes on, the requirements may change — security patches need to be applied or configuration settings have been updated. Tools exist to help manage the creation of these images, and DevOps teams have been leveraging their skills with automation to ensure a consistent and efficient build process.

Security can’t be an afterthought when creating golden images. It’s a best practice to scan images during the build process. If vulnerabilities or compliance issues are found, the image pipeline should be abandoned before the image is published. Shifting left and scanning for compliance issues and vulnerabilities in the image build process can help eliminate runtime issues when instances are instantiated.

Let’s walk through a tutorial and see how we can secure golden images built by a popular system and container build automation tool: HashiCorp Packer.

Securing Builds with HashiCorp Packer

HashiCorp Packer is a utility that allows for the creation of identical images across multiple platforms using a single-source configuration. This could be your organization’s tool of choice if you have a multicloud deployment. Packer can easily integrate with different CI/CD software to create a build pipeline for golden images. It is extensible and can be integrated with custom utilities.

Packer can execute from a central build server, either in your data center or in the cloud.

As shown in figure 1, when the build process begins, Packer first connects to HashiCorp Vault to get the Prisma Cloud credentials and other sensitive information. Then, Packer determines the latest Amazon Linux 2 AMI and creates an Amazon EC2 instance to perform the build. Once the instance has been created, Packer connects to it via SSH and executes the build scripts.

Once the build scripts complete, Packer executes the Scan script — which will connect to Prisma Cloud — downloads and installs the agent, executes a scan and evaluates the results. Then, it compares the results to a policy to see if the compliance or vulnerability threshold was crossed. If the scan results don’t meet the predefined thresholds, the build fails and no image is created. If the scan succeeds, a golden image is created. Then the build instance is terminated.

Let’s walk through a demo of how you can use Prisma Cloud and HashiCorp Packer to secure your golden images.

Getting Started with Prisma Cloud and HashiCorp Packer

To work through the demo on your own, follow these steps to ensure you have all necessary prerequisites:

Step 1: Obtain access key credentials from Prisma Cloud. If you don’t already have a key, open the Prisma Cloud console and then navigate to Settings → Access Control → Access Keys to create one.

Step 2: Locate the Prisma Cloud Compute Console URL by navigating to the Prisma Cloud console and then to Compute → System → Utilities. Use the value for “Path to Console” as the Console URL.

Note: For the purposes of this blog, you’ll need the ability to install software either on your local workstation or a build server, depending on where you are testing.

Step 3: Make sure that you have network connectivity from your local laptop or build server to AWS, and then from AWS to Prisma Cloud. Packer will create a host in the default VPC; there should be an outbound network path to Prisma Cloud if you are using the AWS-created default VPC. If you want to use a nondefault VPC, we’ll cover how to do that later in the demo.

Step 4: Ensure the build user has the appropriate permission in AWS. You’ll need to create a user — or assume a role — that has the permissions documented here. Additionally, you’ll need to configure your AWS CLI profile with the appropriate credentials.

Tutorial: Securing Your Golden Images

To start, we’ll need to configure Prisma Cloud. First, we want to create a Collection in Prisma Cloud so that we can limit the scope of the vulnerability and compliance policy rules to only the instance that Packer manages.

Create a collection by following these steps:

1. Log in to Prisma Cloud.
2. Navigate to Compute → Manage → Collections and Tags.
3. Click “Add collection.”
4. Give the collection a name — for example, “packer-demo-collection” — and description.
5. Change the “Hosts” field to specify the hosts that Packer will create. The Packer build file specifies these will be named “packer-demo-build” — specify the Hosts field as “packer-demo-*.”
6. Click Save.

Now, we want to create a vulnerability rule. The vulnerability threshold for the Packer build process will be set in the Prisma Cloud console. This provides for separation of duties so that the security operations team can identify the appropriate threshold for the build to fail, and the build administrators can create and execute builds that automatically adhere to that policy.

To create a vulnerability rule that identifies the failure threshold, we’ll need to:

1. Log in to Prisma Cloud.
2. Navigate to Compute → Defend → Vulnerabilities → Hosts.
3. Click “Add Rule.”
4. Name “packer-host-vulnerability-rule".
5. Scope it to the collection created above.
6. Select a severity threshold: Off, Low, Medium, High or Critical.
7. Click Save.

Once we’ve created the vulnerability rule, we can create a compliance policy in the Prisma Cloud console. This enables the security operations team to identify critical configuration checks that will fail the build if they’re missing. To create a compliance policy, we need to:

1. Log in to Prisma Cloud.
2. Navigate to Compute → Defend → Compliance → Hosts.
3. Click “Add Rule.”
4. Name “packer-host-compliance-policy."
5. Scope it to the collection created above.
6. Under compliance actions, select the appropriate checks and related action: “Ignore” or “Alert” will disregard the check and “Block” will fail the build.
7. Click Save.

Now we can configure the build. To start, we need to download the artifacts for this blog and extract them to our build server. Then, we can install HashiCorp Packer and HashiCorp Vault according to the instructions for your build server’s platform.

Then, we can install the latest AWS CLI and ensure a profile is created with the appropriate AWS credentials: aws configure --profile packer-demo. Once we’ve created that profile, we can start the Vault server. Since this is a demonstration, run Vault in development mode with command: vault server -dev. Note the Root Token value displayed as output.

Then, we need to open a new terminal and set the VAULT_ADDR and VAULT_TOKEN environment variables:

Now, we can add the Prisma Cloud credentials to the local Vault. First, create a file named secret.json with the template below — replace with your Prisma Cloud access information:

Then, add the secret data to the vault: vault kv put secret/prisma_cloud @secret.json and delete the file containing the secret data: rm secret.json.

In the command prompt, navigate to the directory containing the downloaded artifacts and validate the Packer file using the command: packer validate goldenimage.pkr.hcl.

Now we can review the variables within the goldenimage.pkr.hcl file:

• aws_profile — the name of the profile used when configuring the AWS CLI. The default is packer-demo.
• region — the AWS region in which to source and build the image in. The default is us-east-1.
• vulnerability_rule_name — the name of the vulnerability rule we created earlier. The default is packer-host-vulnerability-rule.
• compliance_policy_name — the name of the compliance policy we created earlier, with a default name of packer-host-compliance-policy.
• scratch_dir — the temporary directory for file downloads, script execution and scan results. The default is /var/tmp/scan.
• vpc_id — the VPC in which to place the build server. If blank, Packer will attempt to place it in the default VPC. Note that if vpc-id is set, subnet-id must also be set. The default is blank.
• subnet_id — the subnet in which to place the build server. If blank, Packer will attempt to pick the most available subnet. The default is blank.
• show_scan_output — set this to yes to display the full results of the host scan. The default is no.

These can be modified within the file or on the command line. To override the default variable values on the command line, use the -var flag as in the example below.

To start a packer build, execute the command packer build goldenimage.pkr.hcl:

After running the command above, a build should begin:

If the build is successful, an AMI will be created:

Using a Custom VPC

The configuration provided above will use the default VPC in AWS. Many organizations prefer a different VPC for testing. If your organization has a dedicated VPC that meets the networking requirements listed above, pass in the appropriate subnet ID in the variable “subnet-id” when executing the build — e.g., -var "subnet_id=subnet-123456".

Packer supplies many different options for the EBS-backed AMI builder, including AMI naming and tagging, AMI distribution and encryption. Check out the documentation to learn more about different builder options.

Customizing Compliance Checks

The predefined Prisma Cloud compliance checks cover a wide range of configurations, but your organization may have custom configurations that need to be enforced at build time.

Prisma Cloud enables you to create custom compliance checks to help ensure golden images meet your compliance requirements. The custom compliance checks will run against the instance that Packer creates before an AMI is created. Let’s walk through how to enable custom compliance checks.

To configure custom compliance checks for hosts, follow these steps:

1. Log in to Prisma Cloud.
2. Navigate to Compute → Manage → Defenders → Settings.
3. Enable “Custom compliance checks for host.”

Once the option is enabled, any agents that require custom checks must be reinstalled. No action is required for the build process described for this solution. But if existing agents in the environment want to make use of custom compliance checks, they need to be reinstalled.

To define a custom compliance check, we’ll need to do the following:

1. Log in to Prisma Cloud.
2. Navigate to Compute → Defend → Compliance.
3. Select the Custom tab.
4. Click Add Check.
5. Specify name, description and severity.
6. Add a script. The script’s exit code determines the result of the check, where zero is pass and one is fail. For example, the script below returns an error if a particular file exists:

To attach a custom compliance check to a host’s policy, we’ll need to:

1. Log in to Prisma Cloud.
2. Navigate to Compute → Defend → Compliance.
3. Select the Hosts tab.
4. Modify the “packer-host-compliance-rule” created above.
5. In the Compliance actions section, search for the custom compliance check and set the action to “Block.”
6. Click Save.

Testing the Custom Compliance Check

Now we can rerun the Packer build to execute the custom compliance check. We’ll need to review the build output to ensure the compliance check is being evaluated. For example, if the custom compliance check described above successfully executes, the build will fail and the log will look like this:

Wrapping up the Demo

Now that we’re done with the demo, we need to undo the changes we made. To start, we recommend that you remove any AMIs that were used for testing because they could incur cost. The AMIs created during testing are named “packer-demo-*.”

To deregister them, follow these steps:

1. Log in to the AWS console.
2. Navigate to EC2.
3. Select AMIs under the Images heading.
4. Ensure the “Owned by me” filter is set and search for “packer-demo.”
5. Select the images, click Actions, then Deregister AMI and then confirm.
6. Delete any snapshots associated with the deleted AMIs.

Now, we can remove the scripts that we downloaded earlier and delete the binaries and path entries from your build server. To do this, uninstall Packer and Vault from your system using the appropriate package removal procedures for your platform and installation method.

Then, delete the AWS CLI profile by editing your ~/.aws/config and ~/.aws/credentials files. Once that’s done, we need to go back into the Prisma Cloud console and remove or delete the following:

• Compliance policy “packer-host-compliance-rule”
• Vulnerability rule “packer-host-vulnerability-rule”
• Collection “packer-demo-collection”
• Any agents named “packer-build-*”
• The access key created as a prerequisite for this process

Next Steps

Now that you have your secure image, you are ready to start deploying instances across your environment. You have effectively “shifted left” and added security to your golden image pipeline. However, please keep in mind that shifting left and enabling security in the build process isn’t enough. Organizations need to be vigilant and ensure visibility for vulnerabilities and compliance during runtime as well. Prisma Cloud can help achieve this with both agentless and agent-based solutions.

We hope you have enjoyed this post on integrating HashiCorp Packer with Prisma Cloud for scanning images in the build pipeline. Stay tuned for more ideas and demos on how to integrate Prisma Cloud and HashiCorp technologies to help secure your environment.