This paper presents the first large-scale empirical evaluation of prompt injection attacks against agentic AI coding editors — tools with system-level privileges such as terminal command execution, as distinct from earlier prompt-injection research that mostly targeted sandboxed chatbots or text-generation quality. It studies attacks delivered through poisoned external development resources, specifically coding-rule files that developers commonly import into their IDE workspace (e.g. Cursor’s “.cursor/rules”), which the paper argues developers rarely review line by line and which, coming from untrusted external sources, usually lack extensive security vetting.
To conduct the study at scale, the authors built AIShellJack, an automated evaluation framework containing 314 distinct attack payloads that cover 70 techniques drawn from the MITRE ATT&CK framework, adapted from the open-source atomic-red-team security-testing library. They evaluated it against Cursor and GitHub Copilot running Claude 4 Sonnet and Gemini 2.5 Pro across five real-world codebases in different languages.
Key Points
- Across the main evaluation, attack success rates (the fraction of the 314 payloads that both executed and achieved their intended malicious effect) ranged from 41.1% to 84.1% depending on the editor/model combination, with Cursor’s Auto mode showing the highest rate (83.4-84.1%) and GitHub Copilot with Gemini 2.5 Pro the lowest (41.1%).
- Command execution itself (regardless of whether the intended malicious effect succeeded) was reported as consistently prevalent: 74-89% of trials across editor/model combinations on one scenario (ts-lep), and 75-88% across the five development scenarios when holding Cursor’s Auto mode fixed — which the paper reads as showing terminal command execution is a routine part of how these editors respond to injected instructions, not a rare edge case.
- Broken down by MITRE ATT&CK category for Cursor in Auto Mode specifically, average success rates across the five scenarios ranged from 55.6% (Exfiltration) to 93.3% (Initial Access) and 91.1% (Discovery), with Defense Evasion techniques succeeding 67.6% of the time on average.
- The paper’s automated attack-detection methodology was validated by manual review of a 339-case random sample (a 95%-confidence, 5%-margin sample of the 2,826 total test results): the two human raters agreed with each other 98% of the time on whether an attack succeeded (Cohen’s kappa 0.96), and, separately, the automated tool achieved 99.1% accuracy in identifying successful attacks against the resolved human ground truth, with 89.6% of automatically-flagged successful attacks confirmed to have fully executed their intended malicious action.
- The paper reports its attack construction deliberately avoided explicit adversarial language (e.g. removing words like “exfiltration” or “adversary” from source security-test descriptions and replacing them with neutral technical framing) to better simulate a realistic attacker while still triggering unauthorized command execution.
- The paper cites two real CVEs as evidence the risk is not hypothetical: CVE-2025-65099 in Claude Code and CVE-2025-62222 in GitHub Copilot, both described as allowing attacker-specified commands to run before a startup trust dialog appears.
- The paper’s threat model assumes the attacker can modify an external resource a developer imports (such as a coding-rule file) but cannot access the developer’s machine directly, and assumes the developer has granted their coding editor permission to auto-execute terminal commands without per-action confirmation — which the paper describes as common practice for productivity reasons.
- For defenses, the paper reports that existing measures such as terminal-access restriction and command allow/deny-listing can block the most direct unauthorized command forms but are insufficient given the observed success rates, and recommends more context-aware input filtering and model training aimed at recognizing disguised malicious instructions, alongside developer-side practices (vetting external resources, limiting editor permissions, monitoring agent-made changes).
Notes
The study evaluates specific named products and versions (Cursor v1.2.2, GitHub Copilot/VS Code v1.102, Claude 4 Sonnet, Gemini 2.5 Pro) as of around May 2025, and the paper itself notes that attack success rates may shift as these products and models are updated, while arguing the underlying vulnerability class (editors struggling to distinguish legitimate from injected instructions) is likely to persist. The paper explicitly limits its attack payloads to relatively direct, unobfuscated language and does not test advanced evasion techniques a real attacker might use, and it tests only coding-rule files as the injection vector, not other external resources such as MCP servers or third-party libraries.