This post is a firsthand account, by an engineer in CyberAgent’s anime Tech STUDIO, of implementing Agent as a Judge for coding agents and wiring it into their own local development flow as a feedback loop — the post is explicit that this is where it currently runs, and that the trial is an individual one. Its argument is that a class of failure is invisible in the diff and can only be found by looking at how the agent worked, so an evaluation agent is triggered on every completion claim to inspect the session’s execution history.
The post is explicit that this sits on top of code review rather than replacing it. The team already runs AI code-review tools — it names Greptile as the one it uses centrally alongside others — with the AI performing a first pass before a human makes the final check, and reports that this improved review accuracy. What it adds is aimed at what that flow could not catch once coding agents became the normal way of working.
Its closing claim is about where work now gets stopped: the author reports being able to filter out work with insufficient verification or process before the stage where a person reads the pull request, which they name as the significant change. The post states this is still a personal, individual trial rather than a team-wide result.
Key Points
- Four failure modes are given as the motivation, all of them invisible in a diff: skipping verification in a real environment while local tests pass; claiming completion for work not actually done; circumventing process, its example being bypassing pre-commit hooks with
--no-verify; and not following the intended procedure, such as writing no tests where TDD was expected. - A second, separate motivation is that parallel workers dispatched by an orchestrator make a session’s time and token spend hard to reconstruct afterwards, so the same report also visualizes the work process.
- The judge is triggered by a Claude Code
Stophook and runs as a skill in its own forked context, separate from the agent being judged. - It works in two steps: first collect facts from the transcript without judging (deduplicated Bash commands, files read, tool-use counts, token cost and elapsed time), then work out what was claimed and go verify each claim.
- The post’s stated evaluation principle is to check rather than infer — its examples are reading CI status with
gh pr checkswhen the agent claims tests passed, and looking for an execution log when it claims it verified behaviour. It states the verification method differs with each task’s context and that fixed checking patterns need not be defined in advance. - Six axes are scored as OK / needs-confirmation / reject, with the weakest taken as the overall verdict; token cost, model and working time are recorded but explicitly excluded from the verdict.
- Where no evidence can be found, the axis is held for human confirmation rather than passed or failed.
- Output is deliberately two-format: a Markdown report for a person including a phase timeline, and a JSON object for the feedback loop that carries each axis’s verdict along with what should be done next. In the sample shown, the axes that passed carry a verdict alone, while the held ones also carry a reason and a suggested action.
- The worked example the post shows is a needs-confirmation verdict on an API implementation task: prerequisites, dangerous operations and misreporting passed, while real-environment verification, security scanning and test-first evidence were held.
- The claims are the author’s own account of their own setup, with no measurement, baseline or error rate reported for the judge itself.
Context
The post’s stated future work is about scope rather than accuracy: building a team-wide AI gateway to centralize the work histories of multiple agents and have every pull request’s creation process checked, and observing token and cost trends over time to inform model selection and task sizing.
It closes with a reference section naming the Agent-as-a-Judge paper and a Japanese commentary on it. The post does not discuss either in its body, so what those documents themselves argue is not established here.