// benchmark explainer
SWE-bench (Verified), explained
By the TrueIQ editorial team at Ahzy Capital · Last reviewed Oct 15, 2026 · How we work
SWE-bench gives an AI system a real GitHub issue from a popular Python project and checks whether its code change makes the project's tests pass.
| Made by | Princeton researchers (2023); Verified subset released with OpenAI (2024) |
|---|---|
| Format | Generate a code patch; graded by running unit tests |
| Size | 2,294 tasks from 12 Python repositories; Verified is a 500 task human checked subset |
| Paper | arXiv 2310.06770 |
What it measures
It tests realistic software engineering: understanding a codebase, finding the right files and writing a fix that passes hidden tests. The Verified subset removed tasks with unclear issues or unfair tests, so it's the version most people quote.
Where it falls short
- Python only, and drawn from a small set of well known repositories that models may have seen in training.
- Scores depend heavily on the agent scaffold (tools, retries, time limits), not just the model.
- Passing tests isn't the same as writing code a reviewer would accept.
How to read a score
Compare models only within the same scaffold or harness. A score with a custom agent isn't directly comparable with a 'bare model' score.
Where to see current scores
We don't republish scores. Leaderboards change weekly, and numbers depend on how the test was run. Check these instead: