This paper, presented at the 48th IEEE/ACM International Conference on Software Engineering (ICSE 2026) by researchers at Singapore Management University, introduces AgentSpec, a domain-specific language for specifying and enforcing constraints on LLM agents while they run. The authors argue that existing mitigations fall short: LLM-powered risk evaluation such as ToolEmu can identify risks but lacks interpretability and offers no mechanism for enforcement, rule-based safeguards are either too rigid or do not generalize across agent architectures, and most approaches assess risk before execution rather than constraining what an agent does while executing. AgentSpec lets a user write rules made of a triggering event, predicate checks and enforcement actions, and intercepts the agent’s decision loop to apply them.
The framework is implemented on LangChain and evaluated on three agents chosen to cover different domains: a code agent built on the CodeAct framework, an embodied agent controlling a robotic arm, and an autonomous-driving agent built on the Apollo driving stack, using the RedCode-Exec, SafeAgentBench and FixDrive datasets respectively. The paper also investigates whether an LLM (OpenAI o1) can generate AgentSpec rules automatically, and measures the framework’s runtime overhead.
Key Points
- The authors describe AgentSpec as, to the best of their knowledge, the first framework that systematically enforces customizable safety constraints on LLM agents at runtime.
- A rule consists of a trigger (a state change, an event before an action, or the agent finishing), a conjunction of predicates to check, and one or more enforcements; the predefined enforcements are user inspection, LLM self-examination, invoking a predefined action, and stopping.
- The LangChain integration hooks the agent’s iteration step at three decision points — before an action executes, after an action produces an observation, and when the agent completes its task — without altering the agent’s core logic; rules are parsed with ANTLR4.
- The authors state the design is framework-agnostic and describe how it could be applied to Microsoft’s AutoGen and to the Apollo autonomous-driving stack, though LangChain is the primary integration.
- On RedCode-Exec, the paper reports AgentSpec detecting and intercepting risky code executions in over 90% of cases; the code agent attempted risky Python execution in 24 of the 25 risk categories.
- In two RedCode categories (“miss default case” and “duplicate key in associative list”) no programs were flagged, which the authors explain on manual inspection: the agent invoked Python but did not actually follow the risky instruction.
- For the embodied agent, the share of tasks leading to hazards fell to 0% in all ten unsafe categories with AgentSpec enabled, while successful completion of safe tasks dropped only from 58.62% to 54.26%.
- For autonomous driving, rules adopted from FixDrive achieved 100% compliance across the tested law-violation scenarios.
- Rules generated by OpenAI o1 with few-shot examples enforced 87.26% of risky code cases and reached 95.56% precision and 70.96% recall for the embodied agent; generated zero-shot for autonomous driving, they prevented law-breaking in 5 of 8 scenarios.
- The authors attribute LLM-generated rule failures to overfitting to the provided examples, failure to recognize complex object properties, and over-rigid rules for vague requirements — one generated rule banned pouring entirely, including watering a houseplant.
- Measured overhead is in milliseconds: about 1.42 ms to parse rules and 2.83 ms (code) or 1.11 ms (embodied) to evaluate predicates, against average agent execution times of 25.4 s and 9.82 s.
Notes
The paper contrasts AgentSpec with NVIDIA’s NeMo, which it describes as applying natural-language constraints at the dialogue level, and with GuardAgent, which relies on an LLM to interpret and apply safety constraints; AgentSpec instead keeps enforcement external to the model and developer-defined, which the authors argue improves verifiability and auditability. It is an instance of the deterministic, pre-execution style of Guardrails.
As limitations, the authors note that AgentSpec enforces deterministically at discrete checkpoints and does not reason about the long-term consequences of an action sequence; they propose probabilistic enforcement — for example, learning a discrete-time Markov chain from historical agent interactions — as future work. Among threats to validity they name the risk of overfitting rules to the dataset, which they mitigated by using 10% of each risky dataset for rule development and the rest for testing, and human involvement in judging user-inspection outcomes.
The paper is also available as an arXiv preprint (arXiv:2503.18666), first submitted on 24 March 2025 and last revised on 31 July 2025 (version 3); its journal reference is Proc. ICSE’26, pages 2938–2950 (ACM, 2026).