Type: url
Original: https://arxiv.org/pdf/2401.16310
Status: generated
Summary
The full text of an empirical study of LLMs in security code review: seven LLMs under five or six prompts are evaluated on 534 code files containing security defects identified by human reviewers in OpenStack and Qt, and compared with static analysis tools. It finds LLMs significantly outperform the static analysis tools, with DeepSeek-R1 best followed by GPT-4 via ChatGPT, analyses the quality problems and hallucinations in the two best models’ responses, and uses regression analysis to show detection is better in files with fewer tokens and security-related annotations.