This survey, by a large collaboration whose first and corresponding authors are at Beihang University and whose contributors come from many universities and companies, sets out to synthesize research on large language models for code across the whole model lifecycle and to act as a practical guide. Its abstract frames the motivation as the commercial adoption of tools such as GitHub Copilot, Cursor, Trae and Claude Code, and the rise from single-digit to over 95% success rates on benchmarks like HumanEval. The authors argue that existing surveys are either panoramic or focused on earlier model generations, and that the research–practice gap — code correctness, security, awareness of large codebases and integration with development workflows — deserves systematic treatment.
The survey is organized in nine parts: code foundation models (general and code-specialized LLMs, their architectures, pre-training tasks and open pre-training datasets); code tasks, benchmarks and evaluation metrics at statement, function, class and repository level and for agentic systems; alignment through supervised fine-tuning, reasoning data construction, multilingual and multimodal code, and reinforcement learning including reinforcement learning with verifiable rewards; software engineering agents organized by lifecycle phase (requirements, development, testing, maintenance and end-to-end); code for generalist agents (code as interaction protocol, as agentic capability and as environment interface); the safety of code LLMs; training recipes; and applications. It opens with a six-stage account of how programming has evolved, from manual and tool-assisted coding through framework-based and AI-assisted development to AI-driven and, as a possible future, AI-autonomous coding.
Key Points
- The survey organizes software engineering agents by the phases of a waterfall-style lifecycle — requirements engineering, software development, testing, and maintenance — and describes a layered architecture for issue-resolving agents that runs from an agent–computer interface layer (as in SWE-Agent and OpenHands) through architecture, knowledge and semantic layers to a self-evolving intelligence layer.
- It treats code as the medium for generalist agents along three dimensions: interaction protocols (tool use, the Model Context Protocol and agent-to-agent coordination), agentic capabilities (thinking, acting and memory in code, as in CodeAct), and environment interfaces (simulation gyms and computer-use agents).
- On evaluation, it traces a shift from string-matching metrics such as CodeBLEU to execution-based metrics such as pass@k and on to LLM-as-a-Judge approaches, and catalogs repository-level and agentic benchmarks including SWE-Bench and its variants and Terminal-Bench.
- In the authors’ pre-training experiments across seven programming languages, interpreted languages such as Python show larger scaling exponents than compiled languages such as Rust and Go, and multilingual pre-training mostly helps — syntactically similar pairs such as Java and C# reinforce each other — with Python as a target language as the main exception.
- In their supervised fine-tuning experiments, global batch size is the dominant sensitivity factor (accuracy degrades once it exceeds roughly 256), the dense Qwen2.5-Coder-14B is more robust to hyperparameter changes than the mixture-of-experts Qwen3-30B-A3B, and datasets with executable or test-based supervision transfer best under a fixed data budget.
- In their reinforcement-learning experiments on code with a verifiable reward, the best choices depend on the target metric: RLOO gave the best pass@1 and pass@5 among advantage estimators while REINFORCE++ with a baseline converged faster and was adopted as the default; 16K-token responses gave the best pass@1 and 2K tokens the best pass@5; and 16 rollouts per prompt is recommended as a default.
- On safety, the authors describe code LLMs as insecure by default because they learn from public code in which flawed patterns are common, and organize mitigation into safety pre-training, safety post-training, red-teaming, and a defense-in-depth approach for coding agents built on secure execution environments, pre-execution validation and runtime oversight.
Notes
The authors’ reinforcement-learning experiments were run with the VERL framework on the codecontest_plus dataset and evaluated on the lcb-v6 benchmark; the authors state that they plan to compare results across further training frameworks, because infrastructure differences may change stability and outcomes.