Data contamination is the situation in which the problems of an evaluation benchmark are included in the massive training corpora of a large language model, so that the model may have encountered the exact problems it is later tested on. Because the model’s score then partly reflects memorised material rather than its ability on new problems, contamination can paint a skewed or misleading picture of its capabilities; LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code names it, alongside overfitting, as a shortcoming of code benchmarks such as HumanEval, MBPP and APPS.
Usage
The term is used in LLM evaluation, including evaluation of code models. The LiveCodeBench paper notes that earlier work tried to decontaminate training data with exact and fuzzy matching, that this can be non-trivial, and that it can be evaded by simple strategies such as rephrasing. Its alternative is time-segmented evaluation: LiveCodeBench tags each problem with its release date so a model can be scored only on problems released after its training cutoff. The same dates make contamination visible — the paper reports stark drops in DeepSeek, GPT-4-O and Codestral performance on LeetCode problems released after each model’s cutoff or release, which it reads as evidence that the earlier problems were likely in their training data. It also cites related work on detecting contamination, such as methods based on edit distance and AST-based semantic similarity for code.