“This is How We Do It: The True Story of How Palo Alto Networks Runs Security Operations” is a new video and blog series that features interviews with various members of our SOC team. We discuss how we run our own SOC and apply our own products while openly sharing our practices.
At Palo Alto Networks, our SOC is highly optimized because we actively choose to break away from the traditional four-tier SOC approach. This ranges from tier 1 analysts who monitor, prioritize and investigate SIEM alerts, to tier 4 SOC managers responsible for recruitment, security strategy and reporting to management. Taking more of a hybrid approach, the Palo Alto Networks SOC team follows a general philosophy:
- 50% of the SOC staff has previous SOC experience while others are skilled in various technical areas.
- Cross-train the SOC team in all domains, including alert triage, incident response, threat hunting and others.
- Provide a well-funded annual training budget for all analysts.
Our Rationale:
- Maintain a nimble team, able to pivot between responsibilities.
- Support business continuity.
- Provide a more engaging atmosphere and reduce staff burnout.
- Promote an environment of continuous learning.
- Provide greater coverage with less staff by relying on the right technology to get the job done.
Episode 1: “Hasta la Vista Human Powers-Automating the Automation” – An Interview with Devin Johnstone, SOC Operations Specialist at Palo Alto Networks
Devin Johnstone shares how the SOC team handles the large volume of security alerts that they receive every day. Devin reveals that the Palo Alto SOC ingests nearly 56 terabytes of raw log data per day, and more than half of that comes from the cloud. Devin and his team take this raw data and filter it down to a manageable number of alerts. They achieve this by using machine learning in their products, as well as their own knowledge to reduce the number of important alerts that require a ticket.
Palo Alto Networks has tens of thousands of companies where we help protect millions of people from cyberthreats and data compromise. Devin believes our company's responsibility is to protect the infrastructure behind the services we offer:
“As much as we are working for Palo Alto Networks specifically, our
responsibility is protecting all of the infrastructure behind the services that we
offer. Our SOC is really focused on making sure everything behind the scenes
is safe, as well as our employees, and monitoring what they do on a
day-to-day basis."
Devin says that every single alert that comes into the SOC is automated in some way. The goal is to fully automate as many alerts as possible, so the team can focus on more important tasks, such as threat hunting. They use Cortex XSOAR to automate the investigation and response to security alerts.
Devin explains further:
“Once those 130-ish alerts get into Cortex XSOAR, which is both our ticketing
system and running the XSOAR playbooks to help us with the SOC response,
there's a portion of them (about 15%) that are fully automated end-to-end.
So the playbook picks it up, does all of the background research it needs to do
and then closes.
That is marked as something the SOC handled, but we didn't put any hands
on it. Every single other one of those alerts has some automation to help it
along…which will run proactively, start the investigation, and response in the
SOC will finish it, or vice versa. Sometimes we'll start an investigation and
then we'll reach a decision path where we can hand it over to XSOAR and say,
‘close this off for me.’”
Even with this workflow, Devin believes there will always be a need for human analysts to understand the context of a situation. Overall, the SOC's approach is to embrace automation to help them handle the large volume of alerts they receive every day.
“I get asked often: Is this automation ever going to take your job? And my
answer is, I hope not. I think there's still going to be an aspect where we need
to be focused on threat hunting, because that's where we provide value as
humans — understanding the context of a situation, thinking like the attacker
and giving the repetitive stuff to the automation. I think our jobs are safe, but
they're going to get even more interesting, because we're going to be able to
focus on more important stuff rather than just looking at the same tickets
every single day.”
With the sheer volume of alerts and events coming into the SOC each day, it's essential to have a system in place that can handle as much of the low-level work as possible. This would leave the analysts to focus on the more complex and nuanced threats, such as those seen from attacks, like SolarWinds:
“When the SolarWinds attack happened, we had already been using
SolarWinds for some time. There was a signed, trusted update that was
pushed down and it started trying to call out to a command & control. There
were multiple analytics-based detections… without us having to configure
those detections in advance.
This is one of the areas where we excel because now it's not up to the SOC to
imagine all of these potential scenarios and try to predict the future. We have
machine learning today that can do that type of behavioral detection and
prevention for things we have never seen before.”
One thing that's clear from speaking with Devin – automation isn't seen as a threat to his team's jobs, but rather as a tool to enhance their capabilities. By leveraging the power of machine learning and AI, they're able to analyze vast amounts of data and identify potential threats faster than ever before. By automating many of the repetitive tasks, they can free up time for their analysts to focus on what they do best – using their knowledge and expertise to outsmart attackers.
As the threat landscape continues to evolve and cybercriminals become more sophisticated and aggressive, it's clear that the role of the SOC is more critical than ever. By embracing automation and applying the latest technologies, teams like Devin's can stay one step ahead of the attackers and protect their organizations from even the most advanced threats.
Watch the full interview on the Cortex YouTube Channel.