Agentic Coding Benchmarks: What They Reveal About AI Agents

Learn what agentic coding benchmarks like SWE-bench and Terminal-Bench measure, where leaderboard scores mislead, and how to test coding agents yourself.

A robot holding a checkmarked scorecard on a running track, between a cleared low high-jump bar labeled bug fix and a much taller bar labeled feature

Key takeaways about agentic coding benchmarks

  • Agentic coding benchmarks measure whether an AI agent's code change passes the repository's tests on real tasks, such as GitHub issues or terminal jobs.

  • A resolved rate counts tasks where every test passes, while a passed rate averages partial test success and can hide unfinished work.

  • Claude Opus 4.5 resolves 74.4% on one SWE-bench Verified leaderboard run but 11.0% of FeatureBench’s full feature-task set in the paper’s Claude Code (routing) setup.

  • SWE-Bench+ (2024) found leaked solutions in 32.67% of one agent’s successful SWE-bench patches, which is why agentic coding workflow evaluation should include private, recent tasks alongside public benchmarks.

  • BAND records each recipient agent's status on a handoff message as processing, processed, or failed, giving multi-agent coding evaluations a per-handoff record without scoring code.

What Are Agentic Coding Benchmarks?

An agentic coding benchmark is a fixed set of real repository tasks, each packaged with a runnable environment and tests that decide whether the agent's change resolved it. The agent gets a repository and a task description, then edits files and runs commands in the agentic coding loop until it submits a patch.

The catch is scope. The FeatureBench paper (ICLR 2026) notes that existing agentic coding benchmarks "cover a limited task scope, e.g., bug fixing within a single pull request (PR)." A SWE-bench-style score tells you how an agent fixes one reported bug in one repository, checked by tests from the pull request that fixed it.

Take a platform team choosing between two coding agents for a framework migration across its monorepo. Both vendors cite strong leaderboard numbers. Neither number came from a migration, which spans many commits and packages, so the team is holding scores for a different job from the one it wants done.

Key Metrics for Evaluating AI Coding Agents

Six numbers describe a coding agent run, and the first two come from tests. Fail-to-pass tests fail before the reference fix and pass after it, and pass-to-pass tests already pass and must keep passing.

Metric

What it counts

Blind spot

Resolved rate

Share of tasks where every fail-to-pass and pass-to-pass test passes

Partial progress: a task with one failing test counts as zero

Passed rate

Average share of fail-to-pass tests passed per task

Whether any task is finished

Input and output tokens

Context the agent reads and text it writes per task

Whether the output is correct

Cost per task

Token spend at the provider's prices on the run date

Later price changes

Wall time

Time from task start to submitted patch

Review time after submission

Scaffold and settings

Harness, model version, and reasoning effort used for the run

It is a condition, and every other row depends on it

Resolved rate versus passed rate

FeatureBench reports both. On its full 200-task set, using the paper’s Claude Code (routing) setup, Claude Opus 4.5 passes 43.29% of fail-to-pass tests on average and resolves 11.0% of tasks, reading about 7.5 million input tokens per task. The authors describe agents that "often produce seemingly plausible solutions with a large underlying gap from truly solving the problem."

For the migration team, a high pass rate beside a low resolved rate means patches that look nearly done and still need an engineer to finish them. Read the two as a pair, with the token count beside them, since failed attempts cost money too.

How to Evaluate Multi-Agent Coding Workflows

Agentic coding workflow evaluation scores each handoff between agents and the final patch, because public benchmarks grade one agent on one task and never observe a handoff.

Suppose the migration team splits the work across a planner agent, a coder agent, and a reviewer agent. If the migration fails its tests, a resolved rate tells the team it failed. It cannot say which agent introduced the regression, or whether the coder ever picked up the planner's second batch of packages.

Measuring each handoff in the migration

Record four things per handoff:

  • Whether the receiving agent picked up the work, and how long after it was sent

  • How many attempts the handoff took before it finished or failed

  • Which agent's change first broke a passing test

  • How many round trips the work made between coder and reviewer

Collect the counts during the trial, before adoption. Watching agents after they ship is a separate job, covered in the BAND guide to monitoring AI coding agents.

Best Practices for Benchmarking Agentic Coding Systems

Reliable benchmarking of agentic coding systems fixes the conditions of every run and tests agents on work they cannot have memorized.

Public tasks leak. SWE-Bench+ (Aleithan et al., 2024) screened the successful SWE-bench patches of SWE-agent with GPT-4. In 32.67% of them, the solution sat in the issue report or its comments, and 31.08% of passing patches looked suspicious because the tests were too weak to verify them. Filtering out those issues cut the agent's resolved rate from 12.47% to 3.97%. Maintainers prune for the same reasons: Terminal-Bench 4.0 removed two tasks with public solutions, and SWE-Bench Pro V2 repaired or retired 480 of 731 tasks and re-grades every patch in a clean container.

  • Fix the scaffold, model version, and reasoning settings, and write them beside every score.

  • Date every figure, including scores copied from a public leaderboard.

  • Build an internal task set from recent changes unlikely to have appeared in the candidate models’ training data.

  • Run each task more than once and report the spread alongside the average.

  • Re-grade each patch in a clean environment, so changes outside the patch earn no credit.

For the migration team, that means replaying past migration commits from its own monorepo through both agents in one fixed harness, several times each. How the agents should coordinate during real work is a separate question, answered in agentic coding best practices.

How BAND Records the Delegation Trail That Workflow Evaluation Needs

A multi-agent trial needs a second record beside the benchmark score: who asked which agent to do what, and whether that agent finished. Without it, a failed migration has a resolved rate and no explanation.

When the planner hands packages to the coder, the evaluation needs the addressee on record. In a BAND room, work moves by @mention, and an agent that is not mentioned never receives the message, so each handoff names its recipient. When a handoff stalls, the evaluation needs to show where. Each recipient agent marks its delivery status processing when it starts work, which opens a numbered attempt, and later processed or failed. If an agent crashes mid-task, the message stays processing, so the stall shows up in the record.

People with access to the room see every message and event, so the room history doubles as the trial's delegation trail, and the multi-agent lifecycle with BAND shows how to tie a retry back to the message that failed. BAND does not score code or replace a benchmark, and prompt traces and model evals stay with tools like LangSmith and Arize. Your team still writes the tests. To see where that layer fits, explore the BAND platform.

Book a demo

FAQs about agentic coding benchmarks

Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan created SWE-bench. They described it in "SWE-bench: Can Language Models Resolve Real-World GitHub Issues?", posted to arXiv in October 2023 and published at ICLR 2024. The strongest model in that first paper, Claude 2, resolved 1.96% of the issues.

Leaderboards differ in task set, scaffold, reasoning settings, and run date, and headline scores blend different ingredients: the Artificial Analysis Coding Agent Index averages three separate benchmarks. In contrast, LiveBench averages its own task categories. Scaffold moves the number even when the tasks stay fixed: on FeatureBench's full set, Claude Opus 4.5 resolves 11.0% in Claude Code and 10.5% in OpenHands. Compare two scores only when every condition matches.

A benchmark is a fixed task set with a published scoring method so that different agents can be compared under the same conditions. An eval is a team's own test set, graded on its own criteria and often kept private. A benchmark helps you build a shortlist of agents. An eval on your repository decides which one you adopt.

Nobody has established a fixed number. FeatureBench offers one data point: a 30-task Lite subset of its 200 tasks ranked models with high consistency against the full set. A few dozen well-chosen tasks may separate agents that differ a lot. Close candidates need more tasks and repeated runs, and how many is hard to know in advance.

Yes, as a baseline for single-issue bug fixing in Python. It helps less for separating frontier agents or predicting feature work. FeatureBench notes that success rates on SWE-bench, whose Verified subset became the standard test, rose from below 10% to over 70% within a year, and SWE-Bench+ also identified leakage and weak-test concerns that complicate interpretation of SWE-bench-style results. Pair it with a harder or fresher benchmark.