My AI Code Reviewer Is Only Allowed to Say Two Things

Either a blocker-grade finding, or "LGTM! 👍". Nothing in between. Here's the policy stack that got my AI reviewer's comments read instead of skimmed — and the one question about it I still can't answer.

I sit on both sides of AI code review. On one side, I run AI reviewers over pull requests as part of my own tooling. On the other, my repos receive automated reviews from vendor bots — Codex, Copilot — whose comments I then have to triage.

Living on the receiving end teaches you something the producing end desperately needs to know: most AI review comments are noise, and noise is not free. Twenty comments where two matter doesn't feel like nineteen bonus insights. It feels like a reviewer you stop reading.

So over time I've hardened my own AI reviewer's output rules into something almost insultingly simple. It is allowed to say exactly two things:

1. A High-severity, blocker-grade finding, as a line comment.
2. "LGTM! 👍" — optionally with one suggestion. One.

Everything else is silence. Let me defend that, because the first reaction is always "but you're throwing information away!" Yes. Deliberately. On the floor.

The Asymmetry That Justifies It

The whole policy rests on one operating thesis:

A false-positive blocker comment costs more reviewer trust than a missed minor issue saves.

When an AI reviewer posts a confidently wrong "this will break in production," the author loses a little faith. The second wrong one, more. And once trust is gone, something worse than annoyance happens: the correct comments stop being read too. Every subsequent finding — including the genuinely dangerous one — gets skimmed with the same skepticism. The value of the entire review channel collapses at once, not comment by comment.

A missed naming nitpick has no comparable failure mode. Nobody ever lost faith in a reviewer for not mentioning that a variable could be more descriptive.

The corollary: an AI review pipeline should optimize precision over recall, structurally. Prefer silence to a wrong assertion. If that sounds familiar, it's the producer-side twin of a rule I apply everywhere else in my tooling: no source, no claim.

The Three Filters

Concretely, a finding has to survive three gates before a human ever sees it.

Gate 1 — the confidence floor. If the model's confidence in a finding is below the threshold (I use 50%), the comment is dropped. No hedged version, no "might be worth checking…" — dropped. And here's the part people find counterintuitive: a High-severity but low-confidence finding is dropped too, not downgraded. Uncertainty about a blocker is itself a signal — the right move is to ask the human directly, not to post a scary comment you can't stand behind.

Gate 2 — the severity filter. Only High-tier findings get posted, where High means: security vulnerabilities, data-loss risk, auth/payment logic inconsistency, breaking changes without a migration path, race conditions, null dereferences, incorrect native configuration (the signing-key-mismatch, wrong-Firebase-environment class of disaster).

Medium (missing tests, missing error handling, questionable abstraction) and Low (naming, formatting) are dropped silently. Not compressed into a "minor nits 🧹" list. Not stored as TODOs. Not appended after the LGTM as a passive-aggressive coda. The premise is that the marginal Medium finding is worth more as silence than as text — and that a genuinely important Medium issue will recur until it either crosses into High or gets caught by a different gate entirely (CI, a quality audit, a human with taste).

When zero High findings survive, the entire review collapses into a single PR-level "LGTM! 👍", with at most one suggestion attached. Why max one? Because three suggestions after an LGTM quietly reconstruct the noise the severity filter just eliminated. (The noise always tries to come back. Policy is how you keep the door shut.)

Gate 3 — the tone constraint. A finding that survives both filters still gets rewritten before posting: when the author's intent isn't fully clear, the comment is phrased as a question — a verification request, not a verdict. Raw model output never gets pasted as-is, and no AI-attribution footer gets appended. The severity filter decides whether to speak; this gate decides how it sounds when it does.

The Consumer Side: Trust No Bot, Including Mine

The same asymmetry governs the reverse direction — when vendor bots review my PRs. Two failure modes show up constantly:

1. Stale-commit citation. The bot quotes a hunk from an earlier commit that a later commit on the same branch already fixed. Implement its suggestion and you re-introduce the exact regression the later commit removed.
2. The plain-wrong claim. The quoted snippet is current, but the bot's mental model of the behavior is not.

So bot output gets treated as a source to verify, never a directive to implement. Before acting on any finding: read the file as it stands at HEAD — not the bot's quoted hunk. Already fixed in a later commit? Reply with the commit SHA and move on. Claim doesn't hold against the live code? Reply with why.

Knowing how often I have to do this as a consumer is exactly why the producer-side filters exist. I know precisely how it feels to triage a confident wrong comment. I decline to manufacture more of them.

The Honest Caveat

Is 50% the right confidence floor? Is High-only the optimal severity cut?

I don't know, and I want to be precise about the shape of that not-knowing: the direction of the asymmetry — false positives cost more than missed nits — I'd defend anywhere. The numbers are my working position, not a benchmarked result. I have a standing research question about where measured false-positive rates for AI review tools actually converge, with candidate sources collected and — in the spirit of this entire post — unread sources get no citations. Until that research is done, the thresholds stay as stated preferences that have survived daily contact with real PRs.

Lessons Learned

1. Trust is the actual output of a code reviewer. Findings are just the mechanism. Optimize for the thing that compounds.
2. Uncertainty about a blocker is information — route it, don't post it. "Ask the human" is a better channel than a comment you half-believe.
3. Dropped means dropped. Half-measures (nit lists, TODO dumps, post-LGTM suggestions) reconstruct the noise through the side door.
4. Apply your own standards to yourself in reverse. If you verify every bot comment before acting on it, your bot's comments should be built to survive that same verification.

What's your floor? If you're running AI review in your team and posting everything the model emits — genuinely, how's that going? That's not rhetorical; the comment section is open, and unlike my reviewer, you're allowed to say more than two things.