Microsoft’s Claude Code and Copilot CLI Study
Microsoft researchers published a new arXiv paper about the company’s early 2026 use of Claude Code and GitHub Copilot CLI from GitHub. Whether AI coding tools change actual engineering output, or just make the work feel faster. The paper’s useful signal is narrow: adopters merged roughly 24% more pull requests over four months, but PR count is not the same as shipped value. Claude Code hooks are shell commands that run at defined points in a Claude Code session, and a practical side question is what are Claude Code hooks good for after a result like this: they can mark where you collect evidence, not declare victory.
Read the PR number carefully
The number that got developers talking was simple. In the study, people who used the tools merged roughly 24% more pull requests over a four-month window.
That is a real signal. It is also an incomplete one.
A merged PR can be a bug fix, a docs edit, a refactor split into five tiny changes, or a feature that should have been one commit. PR count rewards smaller batches. That can be healthy. It can also make output look cleaner than it is.
The paper appears useful because it studies actual workplace behavior, not a toy benchmark. That matters. Many coding-agent claims still come from demos, tasks, or anecdotes that end before code hits the main branch.
Don't treat merged PRs as a proxy for customer value. The study does not prove that the extra PRs reduced incidents, increased revenue, improved retention, or paid down debt. It shows more merged change. That is worth knowing, but it is not the finish line.
A better reading is this: AI coding tools may increase the tempo of code movement. If your review, testing, and ownership habits are weak, that tempo can amplify noise.
Ask what changed after merge
The developer critique was fair. More PRs is not the same as more useful software.
For a product repo, the next questions are boring and important. Did escaped defects rise? Did review time fall? Did reverts increase? Did cycle time improve without dumping work onto reviewers? Did the extra changes land in areas users care about?
A concrete example helps. Suppose an API team merges 24% more PRs in a billing service. That could mean the team shipped overdue invoice fixes. It could also mean the agent split a migration into many safe PRs. Or it could mean reviewers rubber-stamped generated churn that later caused a reconciliation incident.
Those are different stories. PR count cannot tell them apart.
This is where Claude Code conventions matter. A concise repo rule can say that database migrations need a rollback note, a test plan, and an owner review. A slash command like /review-ready can ask Claude Code to prepare those items before a human reviewer sees the PR. The rule does not make the work good. It makes the evidence visible.
Keep this light. A giant CLAUDE.md file that tries to encode every policy will go stale. Put durable repo rules there, then use hooks, permissions, and review receipts for the parts that need runtime evidence.
Follow the cost question too
The uncomfortable question is cost. A faster workflow can still be a bad trade if it burns too many tokens, too much reviewer attention, or too much CI time.
One objection developers raised was extreme token volume. At $5 per million tokens, 281 billion tokens is about $1.4 million. That example is not a conclusion from the Microsoft paper. It is a useful stress test for any claim that output went up.
Cost is not only model spend. It is also the cost of larger diffs, slower review, duplicate tests, and cleanup work. A coding agent can move work from typing into supervision. That may be great. It may also hide the bottleneck.
Watch for arguing from vibes in either direction. It is lazy to say agents are magic because PRs rose. It is also lazy to dismiss them because PR count is imperfect.
A better habit is to measure one workflow edge. For example: when Claude Code touches a payment path, record whether the PR had a test plan, a rollback note, and a reviewer with domain ownership. That gives you a small truth you can inspect later.
For more examples of Claude Code team conventions, team conventions is the right place to keep patterns like this organized.
Use hooks as a measurement boundary
When someone asks what are Claude Code hooks good for in this story, the answer should be modest. Use them to define a boundary where a session must record or check something.
Claude Code hooks can run around tool use and session lifecycle moments. The exact hook types and event names should come from the official Claude Code hooks documentation, because the Claude Code hooks events list 2026 may change as the product evolves.
A good hook boundary is small. It does one job. It does not try to turn an agent into a compliance department.
For example, a repo can use a pre-tool boundary for risky shell commands. If Claude Code wants to run a migration command, the hook can require an issue ID or block the command outside a dev database. A post-session hook can write a small receipt with changed files, tests run, and commands skipped.
That receipt is more useful than a chat transcript. It lets a reviewer ask: what did the agent change, what evidence exists, and what still needs a human?
There is a trap here too. Do not hook everything. Hooks that nag on every file read or harmless command will train people to bypass them. Start with one dangerous boundary, such as production credentials, schema migrations, or write-capable MCP tools.
A related example is claude-meseeks Adds Meeseeks Waiting Alerts, where the interesting part is not the alert itself. It is the small workflow edge where an agent session becomes easier to notice and review.
Try a small hook policy table
Here is a copyable table for one repo. It is not a rollout plan. It is a way to keep the measurement honest while you experiment.
| Boundary | Allow | Record | Block | Why it matters |
|---|---|---|---|---|
| Read-only repo inspection | grep, rg, file reads |
Nothing by default | Never block unless secrets are exposed | Keeps normal exploration fast |
| Test execution | Unit tests, focused integration tests | Test command and result | Commands that require production services | Separates evidence from narration |
| Database migration commands | Local dev database only | Migration file, rollback note, issue ID | Production or shared staging without approval | Prevents fast PRs from becoming slow incidents |
| MCP write tools | Issue comments, draft PR notes | Target system, object ID, summary | Writes to customer data or billing systems | Makes external side effects visible |
| Session stop | Always allowed | Changed files, tests run, skipped checks, open risks | Do not block session stop | Produces a review receipt without trapping the user |
A tiny receipt can be enough:
Claude Code session receipt
- Changed files: api/invoices.ts, api/invoices.test.ts
- Tests run: pnpm test api/invoices.test.ts
- Skipped checks: full integration suite, requires staging fixture
- External writes: none
- Human review needed: billing owner for rounding behavior
This is the practical takeaway from the Microsoft study. If output rises, create one place where you can inspect quality and cost. Hooks are a good boundary for that because they sit at the workflow edge, not in someone’s memory.
Further reading
- Claude Code hooks
- Claude Code product page
- A Study of Microsofts Early 2026 Rollout of Claude Code and GitHub Copilot CLI
Next step
Pick one risky workflow edge this week and write the hook policy table for it. If the table feels too big, the boundary is too broad.