Agent as a Judge is an evaluation arrangement in which an agent judges another agent’s work by examining that agent’s execution process — the record of what it actually did — rather than only the artifact it produced. The single account available here is one engineering team’s implementation of it for coding agents, so what follows describes that implementation and the problem it was built for, not a general or consensus definition. The term’s origin is not established by this source, which cites an academic paper of the same name in its reference list without discussing it.
The case the source makes for looking at process rather than diff is that a class of failures is invisible in the artifact. It names four: skipping verification in a real environment while local tests pass, declaring a task complete when it was not (with nothing in the diff to show otherwise), circumventing process — its example is bypassing pre-commit hooks with --no-verify, which leaves a trace in the process but not in the change — and not following the intended procedure, such as writing no tests where test-driven development was expected or not using a skill the team wanted used. All four, the source argues, can only be seen by looking at how the agent worked.
A second motivation given is unrelated to correctness. As the team moved to orchestrators dispatching tasks to several workers in parallel, it became hard to reconstruct afterwards what a session had spent its time and tokens on, so the judge’s report also serves to make that process visible.
Usage
In the implementation described, the judge is invoked whenever the coding agent declares completion, through a Claude Code Stop hook that runs the evaluation as a separate skill in its own context. It works in two steps. First it collects facts from the session transcript without judging them: the Bash commands that were run (deduplicated), the files that were read, tool-use counts, and token cost and elapsed time. Then it works out what the agent claimed and goes to check each claim against evidence — the source’s stated principle is to verify rather than infer, with its examples being checking CI status with gh pr checks when the agent says tests passed, and looking for an execution log when it says it verified behaviour. Where no evidence can be found, the axis is marked as held for human confirmation rather than passed or failed. The source states the verification method varies with the task and that no fixed set of checking patterns needs to be defined in advance.
Six axes are scored, each as OK, needs-confirmation or reject, with the weakest taken as the overall verdict: whether the prerequisites were read (specifications, design documents, issues, related code), whether there is evidence of verification beyond tests alone, whether dangerous operations occurred (judged in context — --no-verify, git push --force, rm -rf), whether there are signs of misreporting (does the claim match the log), whether the recommended development method was followed, and whether a security check was run. Token cost, model used and working time are recorded but explicitly do not contribute to the verdict.
Output takes two forms for two audiences: a Markdown report for a person, containing the verdict, the reasoning behind it, and a timeline of how long each phase of the session took; and a JSON object for the feedback loop, in which each axis carries its verdict together with what should be done next — in the example shown, a held axis also carries a reason and a suggested action while a passing one carries the verdict alone.
When It Applies
- Applies where the concern is whether work was done properly rather than whether the resulting code is correct. The source positions it as complementary to code review rather than a replacement: the team already ran AI code-review tools as a first pass before human review and reports that this improved review accuracy, and built this on top for the failures that review could not reach.
- Assumes an execution transcript exists and can be read, and that the judge has tools to go and check claims against systems outside that transcript. The described setup depends on hook support in the coding agent — a
Stophook to trigger on completion — and on running the judge in a separate context from the agent being judged. - Its stated effect is on what reaches human review: the author reports that work with insufficient verification or process is now caught before the pull-request stage that a person looks at, which they describe as the significant change. No measurement is given — the author states this is still a personal, individual trial.
- Reported limits are about scope rather than accuracy. The arrangement runs only in the local development flow; the author raises centralizing work histories across multiple agents through a team AI gateway, and checking every pull request’s creation process, as things being considered rather than done.
- Where evidence cannot be found, the design deliberately returns the needs-confirmation verdict rather than passing or failing the axis, so the output is a list of points for a human to confirm as much as a decision. The worked example the source gives is a task that passed on prerequisites, dangerous operations and misreporting while being held on real-environment verification, security scanning, and test-first evidence.
Related Terms
- LLM-as-a-Judge — the same delegation of evaluation to a model, applied to an output rather than to a process
- Trajectory Evaluation — evaluating an agent by its path rather than its result
- Agentic Code Review — the artifact-side review this is described as sitting on top of
- Agent Hooks — the mechanism the implementation uses to trigger on completion
- Human-in-the-loop — the held axes are surfaced for a person to resolve
- Verification Debt