Foresight-SWE mascot: an owl looking through binoculars

Do coding agents catch their own bugs? (They don't.)

Leaderboard

Cost vs. foresight

Pareto plot

  • Harness. Closed models run in their providers' own agents (Claude Code, Codex); open-weight models run in mini-swe-agent.
  • Settings. Proactive prompt asks the agent to consider unstated edge cases up front; self-review runs the harness's own review command (/self-review, /review) after the patch.
  • Cost, turns. Averages per instance; turns are model calls. Codex cost uses OpenAI list prices. Review settings include the first round; hover a value for the split.
  • Best values. Bold marks the best value in each column.

About Foresight-SWE

Why we built it, how we collected the data, and how we score an agent.

Why foresight?

Coding agents are usually evaluated on fully specified tasks, where every requirement is written down. Real software tasks rarely are, and a large part of engineering is foreseeing the problems a change will cause that nobody has described yet. Foresight-SWE measures exactly that: given an ordinary task, does the agent avoid the defect a human later had to fix?

A pytest timeline: day 0, PR1 merged; day 23, an issue reports that pytest collects every package's tests inside conda environments; day 25, PR2 merged with explicit conda detection
A real case from pytest. On day 0 a tested and reviewed PR1 changed how virtual environments are detected. By day 23 users reported substantial breakage, because conda environments do not create the marker file it relied on. On day 25 PR2 repaired it and closed the issue. The tests PR2 added fail on exactly the code PR1 shipped. Foresight-SWE holds those tests out.

Dataset collection

Each instance pairs two pull requests from the same repository. PR1 implemented a change, passed review and merged. PR2, a median of 88 days later, fixed a defect that PR1 left behind. We mine pairs from open-source repositories on GitHub. We look for maintainers identifying PR1 in PR2's discussion, changelogs, Django's Trac tickets and security advisories via CVEs. We also use the SZZ algorithm to identify PR1 where maintainers have not attested to it in discussion. We then use LLM-as-a-judge to keep genuine pairs and build dockerized environments via agents. See Section 3 of our paper for the full details.

How we measure foresight capability in coding agents?

The agent is given the repository as it was just before PR1, together with a self-contained task statement for PR1. It is told nothing about PR2, and nothing suggests a defect is coming.

The agent works on PR1's task and submits a patch. We then run two sets of tests on it. PR1's own tests check that it did the stated task without breaking anything. The tests PR2 added check whether it also avoided the defect.

  1. Resolve Rate: The fraction of instances for which an agent passes all PR1 tests.
  2. Foresight Resolve Rate (FRR): The fraction of instances for which the agent passes all PR1 tests and all PR2 tests.
  3. Anticipation Score: The mean fraction of PR2 tests the agent passes. This is a soft metric to measure partial repair.

Get in Touch

Have suggestions or ideas? Please open an issue or pull request on our GitHub repository. You can also email

Citation

If you use Foresight-SWE in your work, please cite our paper!

@misc{patel2026codingagentscatchbugs,
      title={Do Coding Agents Catch Their Own Bugs? (They Don't.)}, 
      author={Shaswat Patel and Michel Liao and Peter Henderson and He He and Valerie Chen},
      year={2026},
      eprint={2611.01181},
      archivePrefix={arXiv},
      primaryClass={cs.SE},
      url={https://arxiv.org/abs/2611.01181}, 
}

Browse instances

Pick an instance to read the task statement the agent sees and the defect it was expected to anticipate.

What the labels mean impact, reach, failure mode, severity, difficulty, time to fix, security
  1. Impact: We capture the effect of the defect on the user via this label. We sub-categorize it as:
    1. Corruption: persisted wrong data, data loss or a security bypass.
    2. Broken: a feature gives wrong or missing output, or stops working.
    3. Degraded: results stay correct but get worse, such as slower or noisier.
    4. Cosmetic: formatting, messages or listings.
  2. Reach: This label captures how many users will be affected by the defect. A defect on a commonly used setup is more detrimental than one on a setup used by a narrow user base. We sub-categorize it as:
    1. Broad: default settings or mainstream usage.
    2. Narrow: a specific niche setup; generally the defect surfaces when the user uses non-default flags.
    3. Rare: multiple unusual conditions need to be met for the defect path to be triggered.
  3. Failure mode: We also capture whether the user notices the defect when it happens:
    1. Silent: the code returns a wrong result, or an expected error never appears.
    2. Loud: the code raises an error the user sees.

Severity: It is a mixture of the above three, based on the rule defined in the table below:

Impact \ ReachBroadNarrowRare
Corruptioncriticalmajormajor
Brokenmajormajor if silent, else minorminor
Degraded / cosmeticminorminorminor
  1. Difficulty: How much time a human developer would need to implement the change. It is sub-categorized into three types, shown for the task (PR1) and for the fix (PR2):
    1. Easy: under 15 minutes.
    2. Medium: 15 minutes to an hour.
    3. Hard: over an hour.

Time to fix: Days between PR1 and PR2: how long the defect stayed in the code before it was fixed.

Security: A separate tag for fixes that close an attack, such as injection or an authorization bypass. It does not change severity. PRs mined via CVE security advisories are flagged as security too.

An LLM judge assigns impact, reach, failure mode and difficulty for every instance. See the appendix of our paper for the prompts.