Oqoqo Measures Agents on Real Tasks

Oqoqo is an eval and custom benchmark builder from the Oqoqo team, posted as a Show HN project in August 2026. Benchmark scores from clean labs often do not predict whether an agent can use your real product surface, CLI, MCP server, or code review flow. For Cursor users, Test the work you actually ask agents to do, including a cursor code review, instead of trusting a generic leaderboard. An agent eval is a repeatable task with a starting state, instructions, evidence, and a pass/fail judgment that can be rerun after product or prompt changes.

A review of Cursor, the AI code editor, can tell you how it feels to use; Oqoqo's angle is whether an agent completes the same job reliably. That distinction is why developers cared about the Show HN post. The pitch was not another abstract model scoreboard. It was a way to measure agent-facing interfaces in the messy places where agents actually operate.

Understand what Oqoqo is measuring

Oqoqo is aimed at realistic evals and custom benchmarks for tasks users care about. In the Show HN description, the authors say it can measure how agent-friendly product surfaces are across coding agents and IDE agents, including Cursor Claude Code and GitHub Copilot, GitHub's AI coding assistant.

That matters because the agent surface is now part of the product. A documentation page, CLI flag, MCP tool response, or repo rule can be clear to a human and still confusing to an agent. Oqoqo's promise is to turn that fuzziness into something you can rerun.

Don't treat this as a universal ranking. As of August 2026, the useful claim is narrower: Oqoqo is about making your own work reproducible. A benchmark that proves your billing CLI is easier for an agent to use does not prove your whole stack is agent-friendly.

Separate real signal from leaderboard comfort

The strongest part of the Oqoqo idea is regression testing. The authors explicitly call out MCP, CLI, skills, SDKs, and other agent-facing interfaces. Those are exactly the surfaces that change quietly and break agent workflows in boring ways.

Take a real example. Suppose your repo has a packages/billing service, a Cursor rule for security-sensitive changes, and an MCP server that exposes invoice fixtures. A useful eval is not: can an agent solve billing bugs in general? It is: can the agent inspect a tax rounding change, read the right rule, call the read-only fixture tool, and leave a review with evidence?

That result does not prove the agent is smart. It proves the path is navigable. For Cursor users, that is often the more valuable finding, because cursor rules, cursor skills, and cursor subagents can improve the path without changing the model.

Watch for overfitting to one happy prompt. If the task only passes when you phrase it perfectly, you measured prompt luck. A good custom benchmark has a fixture, a known expected answer, and enough variation to catch brittle behavior.

Try a tiny Cursor review eval

You can borrow the Oqoqo lesson without building a full benchmark harness on day one. Pick one review task that already causes friction in Cursor. Keep it small enough that a human can verify it in ten minutes.

A good first task is reviewing and refactoring real code in Cursor, based on an actual pull request. For instance: review a change that replaces duplicated date formatting in packages/billing, confirm no invoice totals changed, and suggest only safe refactors. That gives the agent a realistic job with a clear evidence trail.

Use Cursor rules to make the boundary visible. Put repo-wide conventions in AGENTS.md, and put local review constraints in a scoped .mdc rule. If you are experimenting with Subagents and skills, this is also where a Cursor custom subagent or Cursor skill earns its keep: it should make the eval easier to pass for the right reason.

Here is a small rule stub you can adapt:

---
description: Use during review evals for billing changes in packages/billing
globs:
  - packages/billing/**
alwaysApply: false
---
When reviewing billing changes:
- Check tests or fixtures before suggesting refactors.
- Do not approve behavior changes without invoice evidence.
- Mention the exact file and line for each blocking issue.
- Separate correctness findings from style suggestions.

Don't let the agent grade itself. Save the transcript, diff, command output, and final review. Then have a human compare the answer against a short answer key. Agents are very good at sounding done before the evidence is done.

Copy this one-hour mini-eval plan

Use this as a tiny internal version of the Oqoqo idea. It is not a replacement for a benchmark platform. It is a safe way to learn what your current Cursor workflow can and cannot measure.

Task fixture

• Choose one merged pull request with a known review issue.
• Reset a branch to the pre-review state.
• Keep the diff under 300 lines if possible.
• Add one scoped Cursor rule or AGENTS.md boundary that should affect the review.

Agent instruction

• Ask Cursor Agent to review the diff, not rewrite it first.
• Tell it to cite files, tests, and commands used.
• Allow read-only inspection commands.
• Disallow pushing, deleting files, or changing fixtures.

Evidence checklist

• Did the agent read the scoped rule or repo boundary?
• Did it inspect the changed files rather than summarize the PR title?
• Did it run or request the smallest relevant test?
• Did it find the known issue?
• Did it avoid inventing failures that are not in the diff?
• Did it separate blocking findings from refactor suggestions?

Pass condition

• Pass: finds the known issue, cites evidence, and avoids unsafe edits.
• Partial: finds the issue but lacks evidence or mixes review with broad refactoring.
• Fail: misses the issue, changes code without permission, or fabricates a test result.

Run the same task twice: once with only the raw prompt, and once with the Cursor rule enabled. If the rule improves evidence quality without making the agent noisy, you learned something concrete. If nothing changes, your rule description may not be activating clearly enough.

For a broader example of why shared context matters across coding agents, see Vibsync Shares Memory Across Cursor, Claude, Codex. Memory and evals are different problems, but they meet at the same question: did the agent use the right context at the right time?

Further reading

• Oqoqo
• Cursor Agent docs
• Cursor rules

Next step

Pick one real review your team already understands, turn it into the mini-eval above, and run it once in Cursor. The first useful result is not a score; it is the first place your agent lacked evidence.