Codex auto review: setup, strengths, and limits

Auto-review catches syntax drift and misses permission drift. We turned it on, the merge queue sped up, and the audit a week later showed exactly that split. Codex-auto-review is the practice of letting Codex review diffs automatically while a human keeps the merge decision. It is worth running, but the green checkmark guarantees less than it looks like, and the gap is always around scopes and permissions.

So treat auto-review as a filter, not a verdict. It clears the boring stuff so your eyes land on the parts that actually need a human: who can touch what, and why.

Trust the transcript, not the pass rate

The first sprint, we made a quiet mistake. We audited only the diffs auto-review flagged and never the ones it waved through. The waved-through pile is where every permission problem was hiding.

That is survivorship bias in plain clothes. The diffs the tool catches leave a report behind. The ones it misses leave nothing, so the evidence pile flatters the tool by design. A clean dashboard told us things were fine while a connector quietly gained scopes it never should have had.

The fix is cheap. Ask every run to end with a transcript snippet a reviewer can replay: the actual commands, the actual output. Confidence should track that snippet, not a screenshot and not the pass count.

Four gaps to guard before you widen autonomy

These are the four places auto-review gets quiet when you most want it loud. Each one has a small written contract that closes it.

Verification bypass. Exec shortcuts skip tests, a regression slips back in, and the automated pass makes the shortcut feel safe. Require a transcript snippet showing tests ran after the code was generated. Now a green merge means the ritual happened, not that someone felt good about it.

Headless versus browser drift. Chrome workflows diverge from CLI habits, so the reviewer sees two versions of the truth and the demo contradicts CI. Write the staging URLs and credential boundaries down next to any browser task.

Model mismatch. People swap models casually, and different models carry different risk appetite. Pin the default model and the escalation rule inside AGENTS.md so a lead can still reason about the blast radius of a change.

MCP privilege creep. Connectors accumulate, each server widens what an agent can reach, and this is the permission drift the trial actually exposed. Keep a Markdown roster of connectors checked into the repo root, grounded in the MCP specification.

Here is the snippet we keep at the bottom of AGENTS.md. Paste it, adjust the wording, ship it.

# AGENTS.md verification snippet

- Every Codex CLI run ends with the transcript snippet reviewers can replay.
- Pair browser evidence with the project's normal CLI checks before merge.
- If MCP servers are enabled, list allowed actions beside each connector name.

Keep four gates on every merge

When a model is the reviewer, the human questions do not change, they just get more deliberate. These four cover most of what auto-review cannot prove on its own.

And the short scope receipt a person still signs off on:

• Red-folder paths received explicit human acknowledgement.
• Scopes in the PR body match the folders in the diff.
• Primary-doc links were smoke-checked after publishing edits.
• MCP connectors mentioned (if any) list owners.

If you want the tool's side of the loop, the Codex CLI docs, the features page, and the slash commands reference cover it, and OpenAI Plugins repository shows where packaged behaviors are heading. The architecture judgement stays with the humans either way. Use Codex CLI workflows for the broader operating model, Codex code review training for the review practice, and repo governance for Codex workflows when the connector roster has to live in the files. Goal Mode and Appshots is the same loop on a higher-level objective. If MCP reach is the failure, start with whether Codex CLI gives MCP too much reach.

Start with the connector roster

If you only do one thing this week, write the connector roster, because permission drift is the failure auto-review demonstrably missed. To build the habit on your team, review the Codex training formats or book a 15-minute engineering-team sync.