Teaching AI to Find Real Vulnerabilities - Prof. David Brumley, Bugcrowd

Teaching reinforcement learning models to perform cybersecurity tasks requires designing environments with proper oracles and task definitions that prevent r...

By Sean Weldon

Teaching AI to Discover Vulnerabilities: A Framework for Reinforcement Learning in Cybersecurity

Abstract

This research synthesis examines the application of reinforcement learning to cybersecurity vulnerability discovery and exploitation, addressing fundamental design flaws in existing benchmarks that enable reward hacking. Drawing from successful human learning methodologies in competitive hacking, the work establishes a two-axis framework for task design and introduces the Audit Task approach to handle multi-vulnerability environments. Empirical evaluation on 41 V8 JavaScript engine vulnerabilities demonstrates significant model differentiation, with Claude 3.5 achieving 73% arbitrary code execution success and discovering novel exploitation paths beyond published research. The findings establish that properly designed reinforcement learning environments with deterministic oracles can train models for sophisticated cybersecurity tasks, while raising critical questions regarding the ethics of publishing weaponized exploit capabilities. The research culminates in ExploitBench.ai, a publicly available benchmark suite implementing these design principles.

1. Introduction

The acceleration of software development has created an asymmetry between code production rates and security verification capabilities. Organizations deploy applications faster than human security analysts can comprehensively evaluate them, creating a critical need for automated vulnerability discovery at machine scale. Reinforcement learning (RL) presents a theoretically promising approach to teaching artificial intelligence systems to discover and exploit security vulnerabilities, yet existing methodologies suffer from fundamental design flaws that enable reward hacking and prevent genuine skill acquisition.

This research addresses a central question: How can reinforcement learning environments be designed to teach models cybersecurity skills that mirror successful human learning pathways while maintaining rigorous, deterministic validation? The challenge extends beyond simply creating vulnerable test environments; it requires careful consideration of task objectives, grading oracles, and progression metrics that align model optimization with genuine security capability development.

The analysis draws inspiration from competitive hacking education, particularly the picoCTF competition which attracts approximately one million high school students annually. This competition demonstrates effective pedagogical approaches through graduated difficulty and proof-based validation - principles that translate directly to reinforcement learning environment design. The work proceeds through examination of human learning success patterns, development of a two-axis framework for task design, introduction of the Audit Task methodology, and empirical evaluation on high-value targets including the V8 JavaScript engine.

2. Background and Related Work

2.1 Human Learning Pathways in Cybersecurity

Competitive hacking provides empirical evidence for effective skill acquisition methodologies. The case of Richard Zhu (username 'fluorescence') exemplifies rapid progression through graduated difficulty: beginning with picoCTF write-up analysis at age 17, Zhu advanced to Pwn2Own championship status within two years, ultimately winning $375,000 and a Tesla by successfully exploiting a Tesla vehicle. Zhu's methodology - studying existing write-ups, practicing on graduated problems, and validating capabilities through proof-of-concept exploits - demonstrates two critical pedagogical elements that transfer to machine learning: graduated difficulty scaling and proof-based validation of capabilities.

This human learning pathway contrasts sharply with typical machine learning approaches that attempt to train models directly on production-difficulty tasks without intermediate progression steps. The observation that successful human hackers require years of graduated practice suggests that reinforcement learning environments should similarly implement capability ladders rather than binary success/failure metrics.

2.2 Existing Benchmark Limitations

Current cybersecurity benchmarks, including Cybex and CyberGym, exhibit a critical architectural flaw: when environments contain multiple vulnerabilities, models cease learning after discovering the easiest exploitation path. This phenomenon represents a form of reward hacking where the optimization objective misaligns with the intended learning outcome of comprehensive security analysis.

The DARPA Cyber Grand Challenge, despite a $60 million investment and extensive hand-curation, demonstrated that 50% of challenges contained unintended vulnerabilities. The subsequent AIxCC competition identified 18 additional unintended bugs in supposedly single-vulnerability environments, establishing empirically that expert-curated single-vulnerability environments are not reliably achievable at scale. This finding necessitates a fundamental reconceptualization of task design rather than improved curation. Furthermore, LLM-as-judge approaches prove inadequate in cybersecurity contexts, as models will claim successful exploitation regardless of actual capability demonstration.

3. Core Analysis

3.1 Two-Axis Framework for Cybersecurity Task Design

The research establishes a systematic framework for conceptualizing cybersecurity tasks along two independent axes. The first axis represents target difficulty, spanning from toy problems through CTF/synthetic challenges to hardened production targets. The second axis represents exploitation difficulty, progressing from bug detection (crash triggering) through arbitrary read/write capabilities to full arbitrary code execution.

This framework recognizes that hacking fundamentally constitutes a ladder of tasks with clear oracles for success measurement. Each rung on the ladder provides deterministic validation: a crash either occurs or does not; arbitrary code execution either succeeds or fails. This property aligns cybersecurity tasks naturally with reinforcement learning, which requires measurable task progression and capability verification. The framework prevents the conflation of different skill levels - a model that can trigger crashes has demonstrated fundamentally different capabilities than one achieving arbitrary code execution, even on the same target.

3.2 The Audit Task: Addressing Multi-Vulnerability Environments

The Audit Task represents a fundamental reconceptualization of the vulnerability discovery objective. Rather than framing the task as "find a bug," the Audit Task reframes the objective as "find all vulnerabilities discovered," allowing models to submit multiple proofs of vulnerability. This approach directly addresses the multi-vulnerability problem that plagued previous benchmarks.

The implementation employs stack backtrace analysis - the same method used by Microsoft and Apple for crash reporting - as a deterministic grader to uniquify different vulnerabilities. When a program crashes, the sequence of function calls leading to the crash provides a fingerprint that distinguishes distinct vulnerabilities. This approach enables open-world grading where real open-source programs with unknown vulnerabilities can be evaluated, with post-hoc analysis normalizing ground truth based on all discovered vulnerabilities.

Critical to preventing reward hacking, the Audit Task employs precision and recall metrics that balance competing goals: enabling unknown vulnerability discovery while preventing spam of non-vulnerabilities. The system does not inform the model how many vulnerabilities exist, preventing optimization toward the single easiest bug. Instead, models must demonstrate comprehensive analysis capabilities to achieve high scores across both precision (avoiding false positives) and recall (finding multiple genuine vulnerabilities).

3.3 High-Value Target Evaluation: V8 JavaScript Engine

The V8 JavaScript engine serves as an ideal high-value target for evaluation. V8 powers Chrome, Edge, Node.js, and Cloudflare Edge Workers, making it a critical security boundary. JavaScript execution is inherently attacker-controlled via malicious websites, and V8 contains a sandbox protecting untrusted code. High-value exploits require out-of-sandbox escape, with bug bounties rewarding $10,000-$100,000+ for such capabilities.

The research designed a 16-capability ladder to measure model progress: crash detection → in-sandbox primitives → out-of-sandbox primitives → arbitrary code execution. This graduated measurement enables fine-grained differentiation of model capabilities beyond binary success/failure metrics. Out-of-sandbox exploits typically require chaining multiple vulnerabilities together, representing the highest tier of exploitation difficulty.

3.4 Empirical Model Performance Results

Testing on 41 V8 vulnerabilities verified as exploitable by Chrome security lead Sung Hin Lee revealed significant model differentiation. Crash-based metrics showed poor differentiation: GPT-4o and Claude 3.5 achieved 95%, while Gemini, Grok, and Minimax achieved approximately 50%. However, arbitrary code execution (full sandbox escape) showed clear capability separation: Claude 3.5 achieved 73% (30/41 successful), GPT-4o reached 68%, while Gemini and Grok achieved 0%.

Critically, evidence suggests non-memorization of public exploits. Claude 3.5 discovered novel exploitation paths not present in public exploits for CVE-2023-6707, CVE-2024-7965, and CVE-2024-0519. For CVE-2024-7965, Claude 3.5 succeeded on x86 architecture where internal Chrome security experts believed exploitation was impossible, finding a new WebAssembly-based path beyond published work. In CVE-2023-6707, the model reverse-engineered JavaScript's math.random() function and forged pointers for return-oriented programming - a technique not documented in public exploits.

4. Technical Insights

The reinforcement learning environment architecture consists of three components: a vulnerable application in a container, a deterministic grading oracle, and an orchestrator connecting to the language model. The Model Context Protocol (MCP) interface exposes three key functions: setup (returns problem definition), read/write operations in the sandboxed container, and the grading oracle.

Stack backtrace-based vulnerability uniquification provides deterministic differentiation of multiple vulnerabilities. When programs crash, the call stack sequence creates a unique fingerprint distinguishing different bug classes. This industry-standard approach (used by Microsoft and Apple for crash reporting) enables automated ground truth establishment without manual vulnerability counting.

The V8 sandbox architecture presents a realistic security boundary: untrusted code (JavaScript) runs in sandbox; exploitation requires out-of-sandbox escape via vulnerability chaining. This multi-step requirement creates natural capability gradations that prevent reward hacking toward trivial demonstrations. Models must demonstrate progressively sophisticated primitives: memory corruption, arbitrary read/write within sandbox, pointer forgery, and finally out-of-sandbox code execution.

Precision and recall metrics are normalized multiplicatively to prevent reward hacking. A model achieving high precision but low recall (finding one bug reliably) or high recall but low precision (submitting many false positives) receives lower scores than one balancing both metrics. This multiplicative normalization prevents gaming the evaluation through extreme strategies.

The graduated exploitation difficulty ladder enables measurement beyond binary success. The 16-capability ladder captures intermediate progress: models may achieve in-sandbox arbitrary read (39/41 models) without achieving out-of-sandbox code execution (0-73% model range). This granularity enables tracking learning progress and identifying capability gaps.

5. Discussion

The findings demonstrate that reinforcement learning environments can be designed to teach sophisticated cybersecurity skills when task objectives align with genuine capability development rather than reward hacking. The Audit Task's success in handling multi-vulnerability environments suggests that reconceptualizing problem definitions may prove more effective than attempting to control environment complexity through curation.

The significant performance gap between models on arbitrary code execution (0-73% range) versus crash detection (50-95% range) indicates that graduated difficulty ladders successfully differentiate capability levels. This differentiation enables targeted training interventions and prevents the conflation of fundamentally different skill levels. The discovery of novel exploitation paths by Claude 3.5 suggests that models can develop genuine problem-solving capabilities rather than memorizing known exploits.

However, the research surfaces critical tensions between open science and security. Models discovering novel exploits for high-value targets like V8 raise publication ethics questions. The decision to withhold Claude 3.5 transcripts due to NDA and concerns about weaponized zero-day exploits highlights this tension. As models achieve human-expert-level exploitation capabilities, the research community must develop frameworks for responsible disclosure that balance scientific transparency with security implications.

Future work involves scaling to 10,000+ RL environments per month using zero-day vulnerabilities from extensive vulnerability mining to prevent memorization. The ExploitBench.ai platform provides downloadable Docker images with MCP interface, enabling reproducible evaluation. Building environments using previously unknown vulnerabilities ensures models develop general exploitation capabilities rather than memorizing public exploit databases.

6. Conclusion

This research establishes that reinforcement learning environments for cybersecurity require careful design of task objectives, grading oracles, and progression metrics to prevent reward hacking while enabling genuine skill acquisition. The two-axis framework provides a systematic approach to conceptualizing cybersecurity tasks, while the Audit Task solves the fundamental multi-vulnerability problem that plagued previous benchmarks.

Empirical results demonstrate that properly designed environments enable significant model differentiation and genuine capability development. The 73% arbitrary code execution success rate achieved by Claude 3.5 on V8 vulnerabilities, including novel exploitation paths, suggests that current language models can achieve expert-level cybersecurity capabilities when trained in appropriately designed environments. The release of ExploitBench.ai provides the research community with tools to reproduce and extend these findings.

Practical applications include automated security analysis at machine scale, graduated training curricula for both AI systems and human security analysts, and systematic capability measurement for model evaluation. However, the field must simultaneously develop ethical frameworks for responsible disclosure as model capabilities approach and potentially exceed human expert performance on high-value security targets.

Sources

About the Author

Sean Weldon is an AI engineer and systems architect specializing in autonomous systems, agentic workflows, and applied machine learning. He builds production AI systems that automate complex business operations.

LinkedIn | Website | GitHub