# SWE-bench

> SWE-bench is a benchmark that scores language models on resolving real GitHub issues; SWE-bench Verified is its human-validated 500-task subset.

SWE-bench, from Princeton NLP, gives a model a codebase and a GitHub issue and checks whether its patch passes the fail-to-pass tests. The original has 2,294 Python tasks; SWE-bench Verified (500 tasks, validated with OpenAI) is the variant vendors quote; Scale AI's SWE-bench Pro adds harder copyleft and private repositories.

**SWE-bench is a benchmark that evaluates language models on real software issues collected from GitHub: given a codebase and an issue, the model must generate a patch, and the task counts as resolved only if the patch passes the repository's tests.**

Princeton NLP released it in October 2023 with 2,294 task instances from 12 Python repositories. Scoring is the percentage of tasks resolved: the patch must pass the fail-to-pass tests that reproduce the issue and the pass-to-pass tests that guard existing behaviour. The first baseline solved 1.96% of tasks. Three subsets followed: Lite (300 tasks), Multilingual (300 tasks across nine languages) and Multimodal, whose V2 release of September 1, 2026 has 480 tasks.

SWE-bench Verified is the number on model cards. OpenAI's Preparedness team and the SWE-bench authors released it on August 13, 2024 after 93 developers annotated 1,699 samples; 68.3% were filtered out for underspecified problem statements or unfair tests, leaving 500. The swebench.com default view, "Bash Only," runs every model in the same mini-SWE-agent environment so scores are comparable; its top entry as of October 2026 is Claude 4.5 Opus (high) at 76.80%, dated February 2026. Scale AI's SWE-bench Pro (September 2025) is the harder successor: 1,865 tasks from 41 repositories, with a 642-task validated public split and private and held-out sets built on copyleft and proprietary code to limit contamination.

The criticisms come from the maintainers themselves. OpenAI's filtering rate shows how noisy the original was, Scale cites "Data Contamination" and "Oversimplified Problems" as its reasons for building Pro, and the paper "The SWE-Bench Illusion" found models can identify the buggy file path from the issue text alone up to 76% of the time. Vendor cards also run different harnesses (Claude Code, mini-SWE-agent, their own scaffolds), so a Verified number only means something next to the harness that produced it. [LLM Benchmarks Explained](/guides/evaluation/llm-benchmarks-explained-2026) covers how to read those numbers next to [Terminal-Bench](/glossary/terminal-bench) and [pass@k](/glossary/pass-at-k).

---

_Source: https://agentscamp.com/glossary/swe-bench — Term on AgentsCamp._
