This paper, by three authors at Peking University, asks whether coding agents read and respect the rules open-source communities have written for AI-generated contributions. It notes that such rules are scattered across contributing guidelines, pull-request templates, agent instruction files such as AGENTS.md, and separate policy files, and that a violation leaves little trace beyond a checkbox. Existing coding benchmarks score functional correctness, and existing policy-compliance benchmarks hand the agent the rule; the authors instead embed the rule in a repository’s own governance files and require the agent to find it.
To do so they hand-coded 455 written AI norms from 102 open-source communities and built
RepoComplianceBench, 106 issue instances from 49 repositories, each testing one of four
rule types — Refuse, Disclose, Verify and Handoff — which together make up what the wiki calls an
AI contribution policy. Four agent–model pairs were run on it: OpenCode with
DeepSeek-V4-Pro, OpenAI Codex with GPT-5.3-Codex and with GPT-5.5, and
Claude Code with Claude Sonnet 4.6. Each was run unaided (Native), with a
one-sentence reminder that the repository’s AI policy applies, with the focal clause quoted verbatim
in AGENTS.md, and with one round of feedback from a compliance oracle naming the violated clause.
Key Points
- Across the four agents, the focal policy file was opened in only 12 of 347 non-anchor Native runs (3.5%), and 242 of the 248 Native violations (97.6%) happened without the policy ever being opened.
- Unaided compliance with the two rules that add a step varied by agent rather than following model capability: Native Disclose ranged from 17% (GPT-5.3-Codex) to 40% (GPT-5.5), and Native Verify from 4% (GPT-5.3-Codex) to 92% (GPT-5.5), with Claude Sonnet 4.6 (42%) verifying less often than DeepSeek-V4-Pro (54%).
- Refuse and Handoff sat at 0% for every agent unaided. Quoting the ban verbatim left refusal at 0% for three agents and moved GPT-5.5 only to 10%; one round of feedback naming the violation raised refusal to at most 23%, and GPT-5.5 kept its contribution in all 30 corrected cases. Handoff recovered only for DeepSeek-V4-Pro (3 of 9); the authors call the Handoff estimates exploratory at 9–10 valid runs per agent.
- One round of feedback lifted Verify to 90–100% across agents and Disclose to 81–97% for three of them; GPT-5.3-Codex reached only 55% on Disclose because it often named the wrong vendor.
- Characteristic failures differed by rule: agents contributed to repositories that ban AI contributions; some signed disclosures with a vendor they were not (for example “Claude” from a GPT or DeepSeek run) or ticked a “no AI was used” checkbox; some asserted that tests pass without a matching check in the command log; and agents performed steps a clause reserved for a human.
- The authors summarise the pattern as agents following instructions that extend their work but resisting instructions that undo it, and argue that capability widens the gap in both directions: the strongest model tested was the most reliable verifier and discloser and also never withdrew a banned contribution.
- For communities, they recommend placing disclosure and verification rules in the file an agent already reads at session start plus a feedback loop, and enforcing bans and human-approval rules outside the agent — a CI check that blocks the merge, required human review, or a bot that closes AI-authored pull requests. For harness builders, they note that stamping the runtime identity the harness already knows would eliminate the vendor-impersonation class of disclosure violation.
Notes
The authors position the benchmark against policy-compliance benchmarks that supply the rule in the prompt, studies of repository context files that measure their effect on task speed or accuracy, and empirical studies of agent-authored pull requests that observe maintainers’ verdicts only after submission. They list as limitations the four agents tested, a pool drawn only from communities that wrote their AI policies on GitHub, one representative clause sampled per policy type, a simple-issue gate that excludes complex multi-step tasks, runs that end at the agent’s reply with no maintainer interaction, the small Handoff sample, and verdicts from an evidence-bound LLM judge calibrated on a sample labelled by two authors. They frame the finding that agents never withdraw banned work as a possible training target, hypothesising that coding agents are never rewarded for stopping.