NL2XQL: Turning Natural Language into Powerful Cybersecurity Querying

The Challenge: Making XQL More Accessible

XQL (Extended Query Language) is a domain-specific query language designed for cybersecurity investigations, offering powerful data interrogation capabilities within Cortex XSIAM^®. However, writing effective XQL queries requires prior knowledge of the language’s structure, syntax, and relevant dataset fields. For newcomers, this learning curve can hinder efficiency and delay critical security operations.

To solve this, NL2XQL leverages large language model (LLM) driven algorithms to enable seamless natural language to XQL translation, unlocking new levels of accessibility and productivity for security teams.

Introducing NL2XQL – Now Available in Private Preview

NL2XQL is designed to bridge the gap between human language and XQL. This innovative feature—now a part of the Cortex Copliot Early Access Program—is set to ease investigation and data analysis by allowing users to effortlessly translate natural language into precise XQL queries.

As part of Cortex XSIAM, NL2XQL strives to lower the entry barrier for security analysts, enabling efficient threat detection and data analysis without requiring deep expertise in XQL syntax. Whether you’re an experienced threat hunter or a security analyst new to XQL, NL2XQL empowers you to extract critical insights from massive datasets using simple, intuitive language.

The NL2XQL Research Journey: Overcoming Complexity

Developing an LLM-powered XQL generator introduces several challenges:

• Domain-Specific Complexity – Unlike SQL, XQL is proprietary, meaning LLMs lack prior exposure.
• Limited Training Data – High-quality natural language–XQL query pairs are scarce, requiring innovative data synthesis techniques.
• Strict Query Requirements – The generated queries must be compilable, syntactically correct, and contextually relevant.
• Evaluation Ambiguity – Correctness is difficult to assess as multiple valid queries could return the same results.
• Dataset and Field Characterization – Understanding dataset structures and field relevance is crucial for generating meaningful and contextual queries.

To address these, we have developed a multi-phase strategy combining synthetic data generation, fine-tuning, and multi-perspective evaluation methodologies.

The Algorithm: A Multi-Phase Funnel for LLM Grounding

NL2XQL employs a structured, multi-phase funnel pipeline to ensure robustness and efficiency. A funnel-like architecture progressively reduces uncertainties at each phase and grounds LLM-generated results in real-world cybersecurity context.

1. Understanding the Query Intent – Classifies the user’s input into specific tables, views, and event types (e.g., network, file, process events).
2. Determining Relevant Fields – Based on the event types and relevant datasets, selects candidate data fields (table columns) that comply with the user’s intent.
3. Planning Query Structure – Using a fine-tuned LLM, structuring a pseudo-query which breaks the query implementation task into smaller steps.
4. Step-by-Step Implementation – Parallel implementation of each step by leveraging the relevant data fields and the unique XQL syntax.
5. Query Assembly and Validation – The final query is validated for compilability and correctness, ensuring it aligns with XQL syntax.

This structured approach minimizes hallucinations, enhances query correctness, and ensures that security teams receive meaningful, actionable results.

The Evaluation: Multi-Perspective Methodology

Performance evaluation of an LLM-powered algorithm is a well-known challenge in the literature and the industry as a whole. To overcome this challenge and obtain a meaningful and reliable performance assessment, we have combined several approaches whose unification yields a comprehensive, multi-perspective, view.

More specifically, the following approaches are used:

1. Manual evaluation - Ensuring overall correctness through an expert verification.
2. Static KPIs - Evaluating compilability as well as the Jaccard similarity considering different query attributes like fields, stages, and operators.
3. LLM as a judge - Direct and indirect (via SQL translation) assessment in scale.

Combining the three approaches, we are able to attain a context-aware LLM-grounding evaluation that may also be extended to scale.

The Future of NL2XQL

As we continue developing NL2XQL, our focus remains on expanding and refining its capabilities. Translating natural language into meaningful, context-aware XQL queries is a complex challenge, and we have several planned enhancements to further improve accuracy and usability:

Expanded XQL Capabilities – Supporting more complex query structures.
Broader Table Support – Extending functionality to additional datasets.
Improved Query Understanding – Enhancing the model’s ability to capture nuances for more precise query translation.
Deeper Cybersecurity Context Awareness – Increasing relevance and effectiveness in security investigations.

Palo Alto Networks is committed to integrating advanced AI tools into its products, enabling customers to leverage AI for more effective and efficient security operations. NL2XQL is an important step in this direction, and we look forward to its continued evolution.

Looking to empower your research? Submit a registration request to the Cortex Copliot Early Access program: dl-cortexcopilotsupport@paloaltonetworks.com