The Outdated Rate is a metric introduced in BitsAI-CR: Automated Code Review via LLM in Practice for evaluating automated code review comments. It is the percentage of comments seen by code committers within a one-week measurement window that became outdated, where a comment counts as outdated if any line within the code range it flagged is modified in a subsequent commit. It was proposed to complement precision, which the paper argues cannot show whether developers actually accept and act on review comments and which requires manual assessment too costly for large-scale or sustained evaluation; the Outdated Rate, by contrast, can be computed automatically.
Usage
In BitsAI-CR the Outdated Rate is tracked weekly for each review rule and used together with precision to decide which rules to keep, a combination the paper presents as what drives its data flywheel. A rule with high precision but a consistently low Outdated Rate is read as producing comments that are technically correct but practically superfluous — its example is a magic-number warning that follows Go coding standards but that a user disliked — and may be decommissioned. The same measure can be applied to human reviewers: the paper reports a human Outdated Rate at ByteDance of 35–46%, which it uses as a baseline for effective review impact.
The paper is explicit about the metric’s limit: an outdated comment does not definitively prove the code was changed in direct response to it. It is offered as a signal that helps a system improve continuously in large-scale deployment rather than as proof of causation.
Related Terms
- Code Review Agent — the kind of automated reviewer whose comments the metric evaluates