self-bench Turns Private PRs Into Evals

self-bench is an MIT-licensed open-source project from mupt-ai that turns completed GitHub pull requests and local coding sessions into private coding-agent benchmarks. Public coding evals can look impressive while saying little about whether an agent can work in your repository. Measure agentic coding on work your codebase has already accepted, then keep the tasks held out.

A private coding-agent eval is a benchmark built from your own repository history, with hidden tests and a known reference solution. self-bench matters because it tries to make that benchmark from normal engineering exhaust instead of from hand-written toy tasks.

Start with the thing self-bench actually builds

self-bench builds Harbor evals from a GitHub repo. It scans completed work, reconstructs each task from the commit before the change, creates hidden tests, stores the original implementation as the reference solution, and checks that the task fails before the solution and passes after it.

That last step is the important bit. A benchmark task that already passes at the starting commit is not a task. A task that cannot be validated after the original PR is applied is not useful evidence either.

As of August 14, 2026, the repository was mainly TypeScript, MIT licensed, and small enough to still feel like an experiment rather than an ecosystem. The README describes a local control plane using Bun, Docker Compose, Temporal workflow state, GitHub CLI access, and Modal sandboxes for disposable task generation and validation.

Don't treat the generated tarball as objective truth. It is better than vibes, but it is still shaped by your merge history, your tests, your review habits, and whatever work the tool can successfully reconstruct.

Read the benchmark like a lab sample

The interesting claim is not that self-bench beats public evals. The interesting claim is that private repo history can become a repeatable lab sample.

SWE-bench-style tasks are attractive because they begin with a real broken-or-incomplete repository state and ask the model to produce a patch. self-bench borrows that shape, but points it at your own accepted changes rather than a public benchmark corpus.

That helps with saturation. A model can overfit public tasks through training data, benchmark-aware prompting, or plain ecosystem repetition. Your boring internal bugfix from three months ago is less likely to be memorized, and more likely to exercise the exact framework seams, naming conventions, and test patterns your agents will face tomorrow.

Don't assume privacy alone makes an eval good. A tiny repo with thin tests will produce thin signals. A codebase full of flaky integration tests will punish agents for test noise as much as reasoning failure.

Notice what it does not prove

self-bench can tell you whether an agent solves a reconstructed task under the runner's conditions. It cannot prove that the same agent will behave safely in a live branch with half-written requirements, messy chat context, and a reviewer who changes their mind.

That difference matters for Claude Code and for similar terminal-based coding agents. A benchmark run is a clean race. A real session is a conversation with file edits, shell commands, tool permissions, failed assumptions, and sometimes a human steering halfway through.

Use the result as a model comparison signal, not a moral ranking. If Model A solves 18 of 30 private tasks and Model B solves 11, that is useful. It still does not say which one should touch migrations, secrets, billing code, or release scripts without extra review.

Watch for collapsing eval score into permission level. Evals should inform where you ask the agent to work. They should not replace code review guardrails.

Run a tiny internal version before believing a big one

You do not need a grand measurement program to learn from the self-bench idea. Pick three merged PRs from one repository: one bugfix, one refactor, and one test-backed feature. Recreate the pre-change commit, ask your coding agent to solve the issue from the PR description, and compare the result to the merged diff.

For Claude Code users, keep the harness dull. Put task instructions in a short slash command, run from a clean worktree, and require the agent to stop after tests and a diff summary. A concise CLAUDE.md can help only if it contains durable repo facts, such as test commands and architecture boundaries, not task spoilers.

Here is a small command file shape that works well as a smoke test:

# .claude/commands/private-eval-task.md

You are solving a held-out repository task.

Start by reading the issue summary below and inspecting the current tree.
Do not inspect later commits, merged PRs, or branch history for this task.
Make the smallest patch that satisfies the task.
Run the relevant tests.
Stop with:
- files changed
- tests run
- remaining uncertainty

Task summary:
$ARGUMENTS

This is not self-bench. It is the hand-rolled version you can run in under an hour to see whether the idea fits your repo. If it feels noisy at three tasks, it will probably feel noisy at thirty.

For a neighboring terminal-agent measurement story, Mole Puts a Budget on Terminal Research is a useful contrast: it focuses on constraining research cost, while self-bench focuses on turning past code changes into eval tasks.

Try a one-hour private eval smoke test

Use this when you want a fast read on whether your repo history can produce meaningful coding-agent evals. It is deliberately small.

If this smoke test produces clean tasks, self-bench is worth a closer look. If it produces ambiguous tasks, fix the task selection process before blaming the model.

Common questions

Is this just SWE-bench for my codebase?

Roughly, yes, but that shorthand hides the hard parts. self-bench uses a SWE-bench-style shape by reconstructing tasks from earlier commits, adding hidden tests, and validating against the original implementation, but the quality depends on your repo history and test coverage.

Further reading

• self-bench on GitHub
• Claude Code docs
• SWE-bench
• MCP specification
• Codex on GitHub

Take the small measurement win

Start with three old PRs and one clean agent command. If the tasks are clear, self-bench gives you a path from anecdote to evidence without pretending the evidence is perfect.