Why the first rollout fails
A team turns on AI code review, and within two weeks every pull request has fourteen comments. Three are good. Eleven are style opinions, a suggestion to add a comment above an obvious function, and a warning about an error path that cannot happen.
Engineers do the rational thing. They stop reading. Now the three good comments are lost too, and you have a tool everyone resents.
The problem is not model quality. It is that the default configuration reviews everything at once, with no sense of what your team already handles elsewhere.
Give it the jobs humans are bad at
Humans review well for design, naming, and whether the change should exist. They review badly for the tedious mechanical things, because attention runs out around file six. Point the bot there:
- Changes to a function's contract where a caller was not updated.
- A new code path that has no test, when every sibling path has one.
- Secrets, keys, and connection strings in the diff.
- An error swallowed silently, a context or cancellation token dropped, a lock acquired without a matching release.
- Migrations that are not reversible, or that lock a large table.
- The diff not matching the pull request description. This one is quietly the most valuable, because it catches the change nobody meant to ship.
And take jobs away from it. Formatting belongs to the formatter. Lint rules belong to the linter. If a comment could have been a config line in .eslintrc or ruff.toml, the bot should never have said it.
Make it earn its place
Write the reviewer instructions as if you were onboarding a contractor. Ours tend to start with what to skip, not what to check: skip style, skip test naming, skip comment density, skip anything the CI pipeline already fails on. Then a short list of what this codebase gets wrong repeatedly. Every team has three of these, and everyone knows what they are.
Then set a comment budget. Five per pull request, ranked. A reviewer that must choose gives you its best five instead of everything it noticed. This single constraint does more for adoption than switching models.
Run it in a non-blocking mode first. If the bot can block a merge on week one, you have handed a veto to something with no accountability, and the first false positive will end the trial.
The metric that matters
Track the share of bot comments that get acted on. Not comment count, not lines scanned. If the act-on rate is above roughly a third, engineers keep reading. If it drops into single digits, the tool is dead and no amount of encouragement revives it.
Check it weekly for the first month. When it falls, the fix is almost always to remove a category of comment, not to add prompt text.
What AI code review will not do: judge whether the feature is right, know the incident from March that explains the odd retry logic, or take responsibility. Keep a human approver on every pull request. The bot is a first pass that clears the mechanical noise so your reviewer has attention left for the design question.
What to do next
Pick one repo with steady pull request traffic. Turn the reviewer on non-blocking, cap it at five comments, and delete every category your linter already covers. After two weeks, count how many comments changed a line. If it is under a third, cut a category and measure again. Roll to a second repo only after the first one has engineers who would complain if you took it away.
We set this up with teams as part of our workshops, tuned against your codebase and your review culture rather than a template.