17K Runs of Claude, Codex, and Cursor
Review an agent's dependency and service choices before accepting its code. A working implementation can still introduce a second test runner, bypass an existing database wrapper or select a provider the team cannot operate.
Armature's tool-choice study gives a reason to inspect those decisions. The publisher reports 16,893 sessions across Claude Code, Codex and Cursor, with 5,292 sessions included in its first published results. The full-run count and the published-result count are different denominators.
What does the study establish?
Armature describes synthetic repositories, varied task prompts and a simulated human choosing among agent recommendations. This is an experiment about choices under those conditions, not an observation of every production team. The publisher also sells services intended to influence which developer tools agents select. That commercial interest belongs beside the results when interpreting them.
A popular choice in this study is not automatically the right dependency for your application. Check the experiment's conditions before applying any reported preference. This article does not reproduce the study or claim that our workshop ran those sessions.
How can you audit a dependency decision?
Use one recent agent-generated patch from your own repository. Begin with the manifest, lockfile, scripts and configuration, before reading the agent's explanation. List anything that changes how the application is built, tested, hosted or connected to another system.
For each addition, find the existing path for the same job. If the repository already has a test helper, database client wrapper or validation library, ask what prevented the agent from using it. The answer should name a missing capability or a concrete incompatibility. A general claim that a new tool is modern is not enough to accept ongoing maintenance work.
This is a proposed review exercise. Its output is a list of choices you can verify in a diff, not a model quality score.
| Change in the patch | Question for the reviewer |
|---|---|
| A new package | Which required capability is missing from existing dependencies? |
| Another test command | Does this replace a command or create a second convention? |
| A direct database import | Is the normal application boundary being bypassed? |
| A new external service | Who will own its credentials, operation and removal? |
| A lockfile change without a manifest change | Is this an intentional version update? |
What should the agent receive next time?
Write down the local path that the failed attempt missed. For example, tell the agent to use the repository's existing test helper and database wrapper, and to explain a new dependency before adding it. Keep these instructions beside the relevant code or in a narrowly applicable Cursor rule. The Cursor rules documentation describes the configuration mechanism.
A rule is guidance, not an installation restriction. If dependency installation must be prevented, enforce that in the build or execution environment as well. Review a subsequent patch to see whether the instruction helped; do not assume that adding a file solved the problem.
Use a short note in the next review:
New dependency or service:
Required capability:
Existing alternative considered:
Reason the existing path was insufficient:
Verification performed:
Owner if accepted:
Skip the form when nothing changed in the toolchain. Keep the ordinary code review and tests: justified tool selection does not prove the resulting implementation is correct.
Editorial update
Updated 20 September 2026: distinguished the full experiment from its published subset, disclosed the publisher's commercial interest, and replaced repeated skills advice with a concrete dependency-review exercise.