Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan created SWE-bench. They described it in "SWE-bench: Can Language Models Resolve Real-World GitHub Issues?", posted to arXiv in October 2023 and published at ICLR 2024. The strongest model in that first paper, Claude 2, resolved 1.96% of the issues.
Agentic Coding Benchmarks: What They Reveal About AI Agents
Learn what agentic coding benchmarks like SWE-bench and Terminal-Bench measure, where leaderboard scores mislead, and how to test coding agents yourself.
:quality(80))
Key takeaways about agentic coding benchmarks
Agentic coding benchmarks measure whether an AI agent's code change passes the repository's tests on real tasks, such as GitHub issues or terminal jobs.
A resolved rate counts tasks where every test passes, while a passed rate averages partial test success and can hide unfinished work.
Claude Opus 4.5 resolves 74.4% on one SWE-bench Verified leaderboard run but 11.0% of FeatureBench’s full feature-task set in the paper’s Claude Code (routing) setup.
SWE-Bench+ (2024) found leaked solutions in 32.67% of one agent’s successful SWE-bench patches, which is why agentic coding workflow evaluation should include private, recent tasks alongside public benchmarks.
BAND records each recipient agent's status on a handoff message as processing, processed, or failed, giving multi-agent coding evaluations a per-handoff record without scoring code.
What Are Agentic Coding Benchmarks?
An agentic coding benchmark is a fixed set of real repository tasks, each packaged with a runnable environment and tests that decide whether the agent's change resolved it. The agent gets a repository and a task description, then edits files and runs commands in the agentic coding loop until it submits a patch.
The catch is scope. The FeatureBench paper (ICLR 2026) notes that existing agentic coding benchmarks "cover a limited task scope, e.g., bug fixing within a single pull request (PR)." A SWE-bench-style score tells you how an agent fixes one reported bug in one repository, checked by tests from the pull request that fixed it.
Take a platform team choosing between two coding agents for a framework migration across its monorepo. Both vendors cite strong leaderboard numbers. Neither number came from a migration, which spans many commits and packages, so the team is holding scores for a different job from the one it wants done.
Key Metrics for Evaluating AI Coding Agents
Six numbers describe a coding agent run, and the first two come from tests. Fail-to-pass tests fail before the reference fix and pass after it, and pass-to-pass tests already pass and must keep passing.
Metric | What it counts | Blind spot |
|---|---|---|
Resolved rate | Share of tasks where every fail-to-pass and pass-to-pass test passes | Partial progress: a task with one failing test counts as zero |
Passed rate | Average share of fail-to-pass tests passed per task | Whether any task is finished |
Input and output tokens | Context the agent reads and text it writes per task | Whether the output is correct |
Cost per task | Token spend at the provider's prices on the run date | Later price changes |
Wall time | Time from task start to submitted patch | Review time after submission |
Scaffold and settings | Harness, model version, and reasoning effort used for the run | It is a condition, and every other row depends on it |
Resolved rate versus passed rate
FeatureBench reports both. On its full 200-task set, using the paper’s Claude Code (routing) setup, Claude Opus 4.5 passes 43.29% of fail-to-pass tests on average and resolves 11.0% of tasks, reading about 7.5 million input tokens per task. The authors describe agents that "often produce seemingly plausible solutions with a large underlying gap from truly solving the problem."
For the migration team, a high pass rate beside a low resolved rate means patches that look nearly done and still need an engineer to finish them. Read the two as a pair, with the token count beside them, since failed attempts cost money too.
Popular Benchmarks: SWE-bench, Terminal-Bench, and More
SWE-bench Verified, SWE-Bench Pro, Terminal-Bench, and FeatureBench each define a coding task differently, so a score on one does not forecast a score on another. Versions below were checked on November 2, 2026.
Benchmark | What a task is | Size and scope | Current version |
|---|---|---|---|
Resolve a real GitHub issue in a Python repository | 500 human-filtered issues from the original 2,294 in 12 repositories | Live leaderboard | |
Complete a long-horizon software engineering task | 642 tasks in the V2 public split | V2, paper posted October 16, 2026 | |
Finish a job in a terminal, with its own environment, reference solution, and tests | 66 tasks | 4.0, released August 28, 2026 | |
FeatureBench | Implement a feature traced across several commits and pull requests | 200 tasks from 24 open-source repositories | First version, ICLR 2026 |
The usual misreading treats a bug-fix score as a forecast for feature work. FeatureBench cites a 74.4% SWE-bench Verified score for Claude Opus 4.5, which matches a medium-effort mini-SWE-agent run on the leaderboard dated November 24, 2025. On FeatureBench's full set, run in Claude Code, the same model resolves 11.0%. Task type and scaffold both changed between those runs. When FeatureBench limits its tasks to repositories SWE-bench also uses, the model resolves 5.2%, which makes unfamiliar codebases an unlikely explanation.
The paper does not name the scaffold behind that 5.2%, and the SWE-bench leaderboard lists the same model as high as 79.2% under other scaffolds and settings, so read the drop as a direction rather than an exact size. None of these figures rank one agent above another.
How to Evaluate Multi-Agent Coding Workflows
Agentic coding workflow evaluation scores each handoff between agents and the final patch, because public benchmarks grade one agent on one task and never observe a handoff.
Suppose the migration team splits the work across a planner agent, a coder agent, and a reviewer agent. If the migration fails its tests, a resolved rate tells the team it failed. It cannot say which agent introduced the regression, or whether the coder ever picked up the planner's second batch of packages.
Measuring each handoff in the migration
Record four things per handoff:
Whether the receiving agent picked up the work, and how long after it was sent
How many attempts the handoff took before it finished or failed
Which agent's change first broke a passing test
How many round trips the work made between coder and reviewer
Collect the counts during the trial, before adoption. Watching agents after they ship is a separate job, covered in the BAND guide to monitoring AI coding agents.
Best Practices for Benchmarking Agentic Coding Systems
Reliable benchmarking of agentic coding systems fixes the conditions of every run and tests agents on work they cannot have memorized.
Public tasks leak. SWE-Bench+ (Aleithan et al., 2024) screened the successful SWE-bench patches of SWE-agent with GPT-4. In 32.67% of them, the solution sat in the issue report or its comments, and 31.08% of passing patches looked suspicious because the tests were too weak to verify them. Filtering out those issues cut the agent's resolved rate from 12.47% to 3.97%. Maintainers prune for the same reasons: Terminal-Bench 4.0 removed two tasks with public solutions, and SWE-Bench Pro V2 repaired or retired 480 of 731 tasks and re-grades every patch in a clean container.
Fix the scaffold, model version, and reasoning settings, and write them beside every score.
Date every figure, including scores copied from a public leaderboard.
Build an internal task set from recent changes unlikely to have appeared in the candidate models’ training data.
Run each task more than once and report the spread alongside the average.
Re-grade each patch in a clean environment, so changes outside the patch earn no credit.
For the migration team, that means replaying past migration commits from its own monorepo through both agents in one fixed harness, several times each. How the agents should coordinate during real work is a separate question, answered in agentic coding best practices.
How BAND Records the Delegation Trail That Workflow Evaluation Needs
A multi-agent trial needs a second record beside the benchmark score: who asked which agent to do what, and whether that agent finished. Without it, a failed migration has a resolved rate and no explanation.
When the planner hands packages to the coder, the evaluation needs the addressee on record. In a BAND room, work moves by @mention, and an agent that is not mentioned never receives the message, so each handoff names its recipient. When a handoff stalls, the evaluation needs to show where. Each recipient agent marks its delivery status processing when it starts work, which opens a numbered attempt, and later processed or failed. If an agent crashes mid-task, the message stays processing, so the stall shows up in the record.
People with access to the room see every message and event, so the room history doubles as the trial's delegation trail, and the multi-agent lifecycle with BAND shows how to tie a retry back to the message that failed. BAND does not score code or replace a benchmark, and prompt traces and model evals stay with tools like LangSmith and Arize. Your team still writes the tests. To see where that layer fits, explore the BAND platform.
FAQs about agentic coding benchmarks
Leaderboards differ in task set, scaffold, reasoning settings, and run date, and headline scores blend different ingredients: the Artificial Analysis Coding Agent Index averages three separate benchmarks. In contrast, LiveBench averages its own task categories. Scaffold moves the number even when the tasks stay fixed: on FeatureBench's full set, Claude Opus 4.5 resolves 11.0% in Claude Code and 10.5% in OpenHands. Compare two scores only when every condition matches.
A benchmark is a fixed task set with a published scoring method so that different agents can be compared under the same conditions. An eval is a team's own test set, graded on its own criteria and often kept private. A benchmark helps you build a shortlist of agents. An eval on your repository decides which one you adopt.
Nobody has established a fixed number. FeatureBench offers one data point: a 30-task Lite subset of its 200 tasks ranked models with high consistency against the full set. A few dozen well-chosen tasks may separate agents that differ a lot. Close candidates need more tasks and repeated runs, and how many is hard to know in advance.
Yes, as a baseline for single-issue bug fixing in Python. It helps less for separating frontier agents or predicting feature work. FeatureBench notes that success rates on SWE-bench, whose Verified subset became the standard test, rose from below 10% to over 70% within a year, and SWE-Bench+ also identified leakage and weak-test concerns that complicate interpretation of SWE-bench-style results. Pair it with a harder or fresher benchmark.
Sign Up For The Band
A short and to the point summary of what we've been up to, delivered once a month to your inbox.
By submitting this form, I agree to be contacted by Band and receive occasional offers & product updates via phone or email, in line with Band’s Privacy Policy.
:quality(80))
:quality(80))
:quality(80))