Evaluate Prompt Quality in Claude Code and Codex

This research library uses AI-assisted source research and drafting. Linked sources support product claims; analysis and proposed exercises are our interpretation. Unless an article documents a test and its results, do not read it as a hands-on review or an independently verified benchmark.

Koukyosyumei’s article describes an offline prompt-text metric in h5i. It scores how clearly the text states an objective, a target and completion conditions, with additional signals for context and readability. It does not run a model or measure whether generated code is correct.

That distinction matters. A well-specified prompt can still produce a bad patch, and a vague prompt can succeed on an easy task. A static score can help identify missing instructions, but it cannot establish engineering quality. The author also notes that a public rule-based metric can be gamed.

Add an outcome test separately

The exercise below is our proposed complement to that text metric, not a result reported by the source. Keep the repository state, client, model, permissions and acceptance criteria fixed while comparing prompt variants. Inspect actual changes and test output, not only the final response or a prompt score.

Turn one coding request into a repeatable test

Start with one boring repo task. Good eval tasks are small enough to run repeatedly and real enough to expose agent behavior.

For example, use a failing test in a TypeScript service:

mkdir -p .prompt-evals/runs

git switch -c eval/prompt-quality

# Start this attempt in a separate disposable checkout at the recorded commit.
# Preserve existing changes; do not reset a working repository.

printf '%s\n' 'Fix the failing user-settings test. Keep the public API unchanged. Run the focused test and explain any tradeoff in the final note.' > .prompt-evals/prompt-a.txt

codex 'Fix the failing user-settings test. Keep the public API unchanged. Run the focused test and explain any tradeoff in the final note.' | tee .prompt-evals/runs/a-01.log

npm test -- user-settings | tee .prompt-evals/runs/a-01-tests.log

git diff --stat > .prompt-evals/runs/a-01-diffstat.txt
git diff > .prompt-evals/runs/a-01.patch

If your Codex CLI setup uses an interactive session, paste the exact same prompt and save the transcript manually. The important part is not the shell trick. The important part is that Prompt A and Prompt B face the same repo state and leave comparable receipts.

Run the next prompt in a second disposable checkout at the same recorded commit, with a fresh agent session:

# Start this attempt in a separate disposable checkout at the recorded commit.
# Preserve existing changes; do not reset a working repository.

printf '%s\n' 'Repair the user-settings failure with the smallest safe code change. Do not rename exported functions. Run the focused test before finishing.' > .prompt-evals/prompt-b.txt

codex 'Repair the user-settings failure with the smallest safe code change. Do not rename exported functions. Run the focused test before finishing.' | tee .prompt-evals/runs/b-01.log

npm test -- user-settings | tee .prompt-evals/runs/b-01-tests.log

git diff --stat > .prompt-evals/runs/b-01-diffstat.txt
git diff > .prompt-evals/runs/b-01.patch

This is a small Codex CLI workflow, but it is enough to learn something. If Prompt B fixes the same bug with a smaller patch, a passing focused test, and fewer invented assumptions, it probably beats Prompt A for this task type.

Watch for changing two things at once. Do not change the prompt, repo state, test command, and agent permissions in the same comparison. You will not know what caused the difference.

For more CLI patterns, keep this close to CLI workflows, where repo-local instructions and verification loops matter more than one perfect prompt.

Keep repository memory separate from the prompt

AGENTS.md is where Codex reads durable project guidance. Put stable rules there: architecture boundaries, test commands, formatting expectations, and safety notes.

Do not hide those rules inside the prompt you are scoring. If Prompt A includes a secret paragraph about the test command and Prompt B does not, you are not measuring prompt quality anymore. You are measuring missing context.

A tiny AGENTS.md boundary for eval work can look like this:

# Agent instructions for prompt eval runs

- Prefer the smallest code change that fixes the named failure.
- Do not change public APIs unless the prompt explicitly asks for it.
- Run `npm test -- user-settings` before claiming success.
- Treat GitHub, Jira, Slack, and production databases as read-only during eval runs.
- If an MCP server exposes write actions, ask before using them.

For the comparison, AGENTS.md should define the playing field, while the prompt under test should define the task. That separation makes the result easier to compare.

Nested AGENTS.md files are useful when only part of the repo has special rules. A frontend package can say which Playwright command to run. A backend package can name the database fixture policy. Local scope beats a giant root file that every task has to carry.

Don't let memory files become a second prompt arms race. Keep them short. If the instruction would change from task to task, it probably belongs in the evaluated prompt or the issue, not in AGENTS.md.

Score the diff, not the conversation

A good scorecard is dull on purpose. It should reward the things a reviewer already cares about.

Use five checks for a first pass:

Keep the maximum small. A twelve-point score is easier to use than a fake-scientific hundred-point scale.

Run each prompt more than once if the result matters. Coding agents can vary across runs because context, tool calls, and model behavior are not perfectly deterministic. Three runs per prompt is often enough to reveal an obvious bad prompt; it is not enough to publish a universal benchmark.

This is also where sandboxing matters. If your eval lets an agent write to external systems through MCP, the measurement can cause real damage. For heavier experiments, the same idea behind Sanbox Gives AI Agents MicroVM Sandboxes applies: isolate the agent before you trust the result.

Watch for scoring the prettiest explanation. A fluent final answer with a messy diff is still a bad coding run.

Try it safely with a fit/not-fit table

Use this table when deciding whether a prompt-quality eval is worth running. Copy it into an issue before spending an afternoon on prompt variants.

The smallest useful experiment is two prompts, one task, three runs each, and one scorecard. That is enough to find bad wording, hidden assumptions, and instructions that make the agent over-edit.

Don't turn the first eval into a benchmark platform. Do the tiny version first. You can always add more tasks after the scorecard catches one real difference.

Further reading

• AGENTS.md guide
• Codex CLI docs
• How to Quantitatively Evaluate Prompt Quality in Claude Code and Codex

Next step

Pick one failing test and run two prompt variants against the same clean repo state. Keep the better prompt only if the diff, test log, and review score agree.

Updated 21 September 2026: distinguished the source’s static prompt metric from our proposed outcome experiment, removed destructive reset commands and unrelated search-query text.