Type: url
Original: https://arxiv.org/pdf/2403.07974
Status: generated
Summary
The full text of the paper introducing LiveCodeBench, a continuously updated benchmark for code LLMs that collects new problems from LeetCode, AtCoder and CodeForces contests and tags each with its release date, so models can be evaluated only on problems published after their training cutoff. Besides code generation it evaluates self-repair, code execution and test output prediction; across 52 models it reports stark performance drops on LeetCode problems released after some models’ cutoffs, indicating contamination, and a group of fine-tuned open models that likely overfit HumanEval.