You get
what you measure

1.1 Goodhart in agent evals

Goodhart’s law: when a measure becomes a target, it ceases to be a good measure.

In agent evals that looks like:

1. The SWE-bench leader is often the team that spent the most money—not the team with the cleverest harness. High-price models + high effort + many retries. SWE-bench does not control cost, so “best agent” ≈ “most expensive agent.”
2. After HumanEval saturates, discrimination dies—when five models sit at 90%+, you cannot rank them. It is giving a PhD student a primary-school arithmetic test: everyone scores 100, nobody learned anything.
3. After contamination, a high score may be recall—more than 94% of SWE-bench issues predate the models’ training cutoffs. The model may be remembering, not reasoning.

1.2 Four layers

The higher you go, the less it looks like an exam and the more it looks like a job. Human scores fall with the layer. That is not because items get “harder” in the contest sense. Reality carries ambiguity, multiple solutions, and environmental noise—none of which a contest item contains.

HumanEval

How it was built: OpenAI in 2021 hand-wrote 164 Python items. Each item gives a signature (def has_close_elements(numbers, threshold):) and a docstring ("""Check if any two numbers in the list are closer than threshold."""). The model fills the body. Unit tests decide right/wrong—about ten asserts; all must pass.

Why it is still used in 2026: the number is simple and comparable. Everyone knows the limits. Five years of scores are a baseline.

Why it is not enough:

• Single function, single file, Python only—real development edits tens of files
• The items have been seen N times—they are probably in training data
• pass@1 is unreliable once you give a sampling budget—the model can guess differently 100 times and hit once

The core critique in AI Agents That Matter (Kapoor et al., 2025, TMLR):

They ran an experiment: no fancy agent architecture, just “on failure, change parameters and retry.” That strategy beat many then-SOTA agents on HumanEval. HumanEval ships public tests—the agent can run the code, see fail, rewrite. That is not “smarter.” That is “more patient (and more tokens).”

How to read a HumanEval number: treat it as a smoke test. A new model at 30% does not have the primitive skill; do not waste SWE-bench budget. Two models both at 90%+ cannot be ranked by HumanEval.

Other L1 suites

The essential difference from L1

L1 (HumanEval) is “here is a blank page; write a function.” L2 (Aider Polyglot) is “here is a project with many files; change some lines in some files; the project must still run.”

The first measures “can you write code.” The second measures “can you be a programmer”—read existing code, locate the edit, match the format, do not break the rest.

How Polyglot scores

Item source: 225 Exercism tasks covering C++ / Go / Java / JavaScript / Python / Rust. The filter is deliberate: seven frontier models each ran all 697 candidates; only the 225 where ≤3 of the 7 succeeded were kept. An item all seven solve adds no discrimination.

Protocol:

1. The model receives the task → edits files
2. Language-specific tests run (pytest, cargo test, …)
3. First failure → the model sees the test error → one more edit (at most two attempts)
4. Everything runs in Docker—LLM-generated code does not execute on your real machine

Three metrics (they measure different things):

Why format compliance is its own metric: a coding agent is not done when the code is right. The diff must be machine-applicable. If the agent writes “insert between lines 37 and 38…,” a human still has to apply it. That is a suggester, not an agent.

Why it matters

SWE-bench is the first widely used eval that pulled agents from “doing items” to “doing work.” Humans did not write contest questions. The authors mined bugs that were actually fixed on GitHub, gave the agent the issue text plus a snapshot of the repo at that time, and asked for a patch. The patch must pass the full unit-test suite inside Docker to count as resolved.

SWE-bench = things that happened. Each item is a real person, a real project, a real bug.

How an item is built

Take a real Django issue:

1. June 2022, issue #33892: “when using GenericForeignKey, prefetch_related_objects sometimes raises AttributeError”
2. A Django developer finds the bug, opens PR #15000, changes 3 lines in django/db/models/query.py, adds 15 lines of tests
3. SWE-bench turns that into:
4. Agent sees: the text of #33892 + Django at the pre-PR snapshot
5. Agent must: find the bug in query.py, edit correctly, pass all tests (including the new 15 lines)
6. Hidden: the developer’s actual patch and tests (for scoring; the agent does not see them)

SWE-bench Verified vs the original

Original SWE-bench has 2,294 items. OpenAI with Princeton filtered a Verified subset (500 items). The filter was strict:

• 93 professional software developers, not crowd workers, not students
• Each item labeled three times (not once)—agreement across independent labels
• Dropped: unclear issue text; tests that cannot reliably separate right from wrong; serious defects
• Result: 68.3% of original items were dropped

A leaderboard “SWE-bench 80%” on a non-Verified slice may be riding items that are themselves broken.

Scoring

“The model emitted the right code” is not enough. Two suites must pass:

1. FAIL_TO_PASS: failed before the fix, must pass after. Did you actually fix the bug?
2. PASS_TO_PASS: passed before, must still pass. Did you avoid a regression?

Both green → the item is “resolved.”

Known limits (findings from 2025–2026)

These are not “SWE-bench is bad.” They showed up because many people used it and audited it:

On SWE-bench-Live (fresh items), scores drop from >43% to 19–22%—evidence that contamination and leakage are real.

Reference scores

Rate of change: ~33% to ~74% in a year. That says three things—models improved quickly + harness engineering improved a lot + some of the lift may be eval-specific optimization rather than general skill.

GAIA: easy for humans, extremely hard for AI

How it was built: Meta, 2023, 466 items. Each needs multi-step reasoning—search, browse, files, light math—without specialist knowledge.

Typical item: “Find the 2023 Nobel physics laureate’s city of birth; the answer is the number of letters in the city name.”

Solution: search “2023 Nobel Prize Physics” → name → search name + “born” → city → count letters → emit a number.

Humans average 92%. Best AI at end of 2023: 15% on GAIA.

Why the gap is so large: GAIA does not measure “do you know.” It measures “do you know how to look it up.” The model may “know” the laureate; if 2023 is after its cutoff, it must search, read, extract, compute. The chain perceive → search → read → reason → emit can break at every hop.

WebArena: real operations on fake but complete sites

How it was built: four fake but fully functioning sites (e-commerce, forum, maps, CMS). The agent completes 812 tasks in a browser.

Typical item: “On OneStopMarket, search ‘wireless keyboard’, add the cheapest one to the cart.”

Human: ~78%. Best AI (OpenAI Operator): 58.0%. Best open source: 48.0%.

OSWorld: real software on a real computer

How it was built: 369 tasks across Windows / macOS / Ubuntu, driving Excel, image editors, a terminal, a browser.

Typical item: “Open Excel, read sales.xlsx, make a bar chart of quarterly sales by region, export PNG.”

Human: >72%. Best AI: Agent S3 w/ bBoN 63.5%. That 63.5% is late-2025; early-2025 was 38%—the field is moving very fast.

Where the remaining gap sits (why 63.5% is not “almost there”): AI is strong at structured operations (“fill a form in a browser, search a product”). Humans are strong at “parse a vague instruction, handle a surprise.” OSWorld has a lot of the latter, so a close total score is not equal skill. AI is already 80%+ on programmable tasks (forms) and under 30% on tasks that need common-sense recovery.

Problem 1: uncontrolled cost = scores bought with money

AI Agents That Matter (Kapoor et al., 2025) is the sharpest argument: a simple retry-with-temperature strategy beat complex agent architectures on HumanEval. HumanEval gives you the tests; the agent only has to sample. Spend 10× the tokens, pass@1 rises. If the board says “Agent X 85%, Agent Y 82%” but X spent 10× Y’s tokens per request, X is not better. X is more expensive.

What to do: every agent eval must report total token spend. The denominator is not “number of items.” It is “items × spend.”

Problem 2: irreproducibility

OAgents (EMNLP 2025) checked several systems claiming “open-source SOTA”—many cannot reproduce the paper’s score from the public code. Two runs of the same code can swing enough to matter. That is not necessarily fraud—agents are stochastic—but if the protocol does not control seeds and average multiple runs, the ranking is not a ranking.

Problem 3: evals age

HumanEval is five years old and saturated. SWE-bench is two years old and its limits have been itemized. Any fixed suite is on a path to expiry—models are optimized against it, data leaks, item defects surface. Evaluation has to evolve with the models.

A four-layer strategy

Layer 1: smoke (run on every change)

• HumanEval as smoke only—a collapse means you broke something. A flat score is normal
• Controls: temperature = 0, one run, record token spend

Layer 2: editing (run per version)

• Aider Polyglot—not “can it write code,” but “can a machine apply the diff”
• Watch format compliance—correct code in a mess of formatting is not an agent

Layer 3: real tasks (run at milestones)

• SWE-bench Lite (300 items, cheaper than Verified) or an internal bug suite you built
• Record spend: total tokens / items resolved. Compare agents on “dollars per resolve,” not “resolves”

Layer 4: your own real tasks (the one that matters)

• Collect problems you actually needed an agent for; make them replayable
• Steal CodeWhale’s issue-notebook move: each time the agent fails → write it down → write a test → next run must pass that test

A practical rule

Do not optimize for the board. An agent you overfit to a board is usually only good on the board. Spend the time making it good in your setting. Scores are someone else’s job. Usefulness is yours.