Seven AI Reviewers Became One, Run Twice
Every change in our delivery pipeline is checked by an AI code review agent before it merges, and a person stays responsible for whether it does. How that works depends on the mode:
- Interactive. The review gives me a verdict and its reasons, then asks before merging anything.
- Unattended. The person decides up front. I approve the plan, choose which work enters the queue, and set how far the run may go on its own. The review applies that decision rather than making a new one.
The five steps of our AI delivery pipeline describe in more detail where people sign off. This post is about the reviewer itself, because until 14 September it was seven different agents, and today it is one definition that runs twice.
Seven AI reviewers that did not agree
We built the review the way most teams build one, by adding a specialist whenever a gap appeared. By September we had seven:
- a general code reviewer
- a security reviewer
- five language reviewers, for Java, C#, Python, C++, and Swift
Together the seven files ran to about 1,250 lines. Most of those lines were the same rules written seven times: the confidence threshold, what counts as part of the change, how a finding is written down.
Copies drift, and these had drifted on the one rule that matters most. The general and security reviewers blocked a merge only on a critical finding. The five language reviewers blocked on critical or high. The same high-severity defect could stop a change or pass as advice, depending on which reviewer happened to raise it.
Nobody chose that inconsistency. It accumulated, one reasonable addition at a time.
We measured the obvious fix before rejecting it
The open question at the time was whether a change mixing several languages should get several language reviewers at once. It sounds like more safety for little cost, so before deciding we measured it. ADR-0132 records the result, taken over every pull request merged in the repository between 20 May and 6 September:
- 787 merged pull requests examined
- 1 with no single dominant language
- 0 containing two languages that each had a reviewer of their own
- 43,000 to 57,000 tokens per change for one extra reviewer, run live against real diffs
- 0 findings that the existing review had not already covered
We kept one language reviewer and wrote down what would reopen the question: a mixed-language rate above roughly 5% in any repository using the skill. The decision also records its own limit. The evidence came from one repository that is mostly Python, and a polyglot codebase could tell a different story.
One definition, run twice
The rebuild that followed changed the question rather than answering it. Today our reviewer is a single file, agents/reviewer.md, and it holds no review rules of its own. It is a worker with a role and read-only tools. It can read and search the code, and it cannot run a shell, edit a file, or touch git.
The rules live once, in the review skill that dispatches it, and a test fails if a second severity-to-verdict rule appears anywhere in the repository. Language knowledge moved into rule files that load for each changed file. A change touching four languages is reviewed against four rule sets, with no extra agents and no contest over which language wins.
Security did not disappear into a single pass. The same definition runs twice for every pull request, in parallel, each run in its own context:
- Correctness and standards. Does the change do what it is for, and does it meet the standards for every file it touches?
- Security. Does the change open or widen an exposure: secrets, injection, authentication and authorization, untrusted input crossing a boundary?
The security pass never sees what the correctness pass concluded, so it cannot be anchored by it. What we removed was the separate security agent with its own copy of the rules. The separate look stayed.
Every report names the standards it loaded for each file, and every fallback where no standard applied. A review that cannot show what it checked cannot be told apart from one that checked nothing.
One rule decides the verdict
The verdict rule is now short enough to quote:
- A critical or high finding inside the change means changes requested.
- Medium and low findings are recorded and do not hold the merge.
- A problem the change did not cause never holds it, at any severity, and is filed as its own issue so it is not lost.
Four guardrails keep that from becoming a loophole:
- When in doubt, a finding counts as part of the change.
- A defect on a changed line counts even when the pattern around it is old.
- Anything the change makes reachable counts even when the code is old.
- An incomplete review never approves.
What the reviewer caught that we missed
Three catches from the weeks around the rebuild show what two independent passes add.
A guard that would have reported false safety. We wrote a test to prove the repository held only one verdict rule. Both review passes, independently, found the same flaw: the test recognized only the two wordings the old rules had used, so a third rule phrased differently would slip past while the test stayed green. A guard that reports safety it does not provide is worse than no guard. One of the retired agents already stated its verdict as a table row the test would have missed, so the gap was real rather than theoretical.
A merge path with no security exclusion. A change to how the review handles a failing test check let a merge proceed past a failing required check when the failure was judged unrelated to the change. Review found that the judgment was unverified and that nothing excluded security checks. “Unrelated” now has a mechanical meaning, the same check failing on the base branch, and security checks can never take that path.
Two rules that cancelled each other. The next day, a follow-up change pinned each merge to the exact commit that had been reviewed. Both passes raised the same high-severity finding: the new re-check would also have blocked the narrow, recorded exception introduced the day before, quietly making that path unreachable. It is the kind of interaction that is easy to miss when you read one change at a time.
Where a person had to step in
The review is very good at what is in front of it: the diff, the standards, the code around the change. Four cases show where that stops being enough.
A rule that was right in substance and wrong on paper. The failing-check change above existed because of something the review did not see on its own. Five unattended reviews had merged past the same failing test check. Each judgment was right, because the failure was unrelated to the change. But the rule said never to merge on a red check, and none of the five recorded why it had decided otherwise. It took a person reading back over the runs to see that the rule was wrong and the record was missing, and to change both.
A sentence that read as true for months. Our review repeatedly passed a sentence in a reference document claiming an interface returned data it never returned. Every reading found it plausible, because nothing in the change contradicted it. A test that compared the document with the code found the error in under a second.
Knowledge the team shared and nobody wrote down. In August an unattended run picked up an issue labeled manual and spent a full delivery unit on it before handing it back as undeliverable. To everyone on the team, manual meant documentation work a person does by hand outside the repository. That rule existed, but only in one layer the run could route around and in prose nobody enforced. The machine did exactly what the written system allowed. What it lacked was what we all knew, so we turned that knowledge into a guard at the point of use.
A judgment that belonged to a person. The same month, under our earlier design, a change to permission logic stopped at a hold that routed it to a person. I read the specific risk the hold named against what I knew of that change, merged it, and recorded the merge as a one-time decision rather than a precedent. Over the following weeks we replaced that kind of standing hold with a heightened bar inside the review itself. The judgment did not move to the machine. It moved earlier, into the plan and the run profile, where a person makes it once instead of being woken for it.
The pattern behind these cases is the one we design around. The reviewer does not yet carry the history of why a project is the way it is. That knowledge still sits with the people who were there, and the process is built so they decide.
The author gets a self-check, a reviewer gets a verdict
The review behaves differently depending on who runs it. GitHub does not allow you to approve your own pull request, so when I review a change I authored, the review reaches the same verdict and posts it as a comment rather than an approval. When it reviews someone else’s work, it posts a real approval or a request for changes.
In both cases it says why. Every report opens with:
- what the change was for
- the verdict
- the finding counts, by severity
- the standards applied, and any fallbacks
- the state of the checks
Even an approval leaves a record a person can audit afterwards. And when the review approves but I have not told it to merge, it merges nothing. Earlier this month it declined a merge instruction passed along by another agent, approved the change, and waited until I said merge myself.
What this means if you are adding reviewers
When review quality slips, adding another reviewer feels like the responsible move. In our case it mostly added another opinion about what blocks a merge. What improved the review was:
- putting each rule in one place
- measuring before adding capacity
- keeping a second, independent look where independence matters, which for us is security
- keeping the decision with the people who know the project
If you are building AI-assisted review into your delivery, or deciding how much a machine verdict should carry in your process, contact us at A-CX. We are glad to walk through how we set it up and where people still make the call.
