SWE-Bench tests AI models on real GitHub issues. Learn how scoring works, what the leaderboards mean, and where to find official results at swebench.com.
SWE-Bench is the definitive benchmark for evaluating how well AI models can solve real software engineering problems. Instead of synthetic coding challenges, it uses actual GitHub issues from popular open-source repositories—complete with codebases, bug reports, and test suites.
If you’re choosing an AI coding assistant, SWE-Bench scores are one of the best predictors of real-world performance.
What SWE-Bench Tests
Each SWE-Bench task gives an AI model:
- A code repository — the full source code to navigate
- A problem description — the original GitHub issue text
- The goal — generate a patch that fixes the issue
The model doesn’t see the solution or the tests. It has to understand the problem, find the relevant code, and write a fix that actually works.
This is fundamentally different from benchmarks like HumanEval or MBPP, which test whether a model can write a function from a description. SWE-Bench tests whether a model can debug and patch existing code—a much more realistic software engineering task.
How Scoring Works
Each SWE-Bench sample has two types of tests:
- FAIL_TO_PASS tests: Tests that fail before the fix and should pass after. These verify the issue is resolved.
- PASS_TO_PASS tests: Tests that pass both before and after. These verify the fix didn’t break anything else.
A solution is only correct if both sets pass. This prevents models from “solving” issues by breaking other functionality.
Score = percentage of tasks where all tests pass
SWE-Bench Variants
SWE-Bench Verified
The original SWE-Bench had quality issues—some tasks were underspecified or had unreliable tests. OpenAI collaborated with the SWE-Bench authors to create SWE-Bench Verified: a human-validated subset of 500 high-quality samples.
Key improvements:
- Professional software developers screened each sample
- Removed tasks with ambiguous descriptions
- Fixed unreliable test environments
- Added difficulty ratings
SWE-Bench Verified is now the standard benchmark for comparing AI coding capabilities.
SWE-Bench Pro
Released by Scale AI, SWE-Bench Pro is a harder variant designed to push frontier models. It addresses data contamination (models memorizing test cases) and includes more complex, multi-file changes.
The scores are much lower: top models like GPT-5 and Claude Opus 4.1 score around 23% on SWE-Bench Pro, compared to 70-80% on SWE-Bench Verified [Source: Scale AI SEAL Leaderboard].
Other Variants
- SWE-Bench Lite: A smaller subset for faster evaluation
- SWE-Bench Multilingual: Tests coding across multiple programming languages
- SWE-Bench Multimodal: Includes image-based problems
Official Leaderboards
The official SWE-Bench leaderboards are hosted at swebench.com. Key leaderboards include:
- SWE-Bench Verified Leaderboard — the primary benchmark
- SWE-Bench Pro Leaderboard — hosted by Scale AI
- SWE-Bench Multilingual
Approximate Score Ranges (2026)
| Benchmark | Top Score | Typical Frontier Model |
|---|---|---|
| SWE-Bench Verified | ~79% | 70-80% |
| SWE-Bench Pro | ~23% | 18-24% |
Scores vary by scaffold (the tooling around the model) and evaluation date. Always check the official leaderboards for current numbers.
Why SWE-Bench Matters
For Model Developers
SWE-Bench provides a realistic signal of coding capability. Unlike synthetic benchmarks, it tests:
- Code comprehension across large codebases
- Debugging and root cause analysis
- Generating minimal, targeted fixes
- Not breaking existing functionality
For Engineering Teams
If you’re choosing an AI coding assistant, SWE-Bench scores correlate with real-world usefulness. A model that scores 75% on Verified is more likely to:
- Correctly implement features from descriptions
- Find and fix bugs without extensive prompting
- Suggest changes that don’t break tests
For AI Safety
SWE-Bench is part of OpenAI’s Preparedness Framework. Autonomous software engineering capability is a key risk metric—models that can reliably fix complex bugs have implications for AI autonomy.
Limitations
SWE-Bench isn’t perfect:
- Python only: All repositories are Python, so scores may not generalize to other languages
- Limited scope: Tasks are bug fixes, not feature development or architecture decisions
- Test reliability: Even Verified has edge cases where tests are overly specific
- Scaffold dependency: Scores vary significantly based on the agent scaffold
The last point is important: raw model capability differs from scaffolded performance. A better tooling setup can boost scores by 10-20%.
How to Interpret Scores
| Score Range | Interpretation |
|---|---|
| <30% | Struggles with real engineering tasks. Good for completions, not debugging. |
| 30-50% | Can handle straightforward bug fixes. Needs supervision. |
| 50-70% | Solid for common issues, multi-file changes. Still makes mistakes. |
| 70-80% | Strong performance. Reliable for most engineering tasks. |
| >80% | Not yet achieved on Verified. Would indicate near-human performance. |
Primary Sources
- Official Website: swebench.com
- Original Paper: SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
- GitHub Repository: github.com/swe-bench/SWE-bench
- SWE-Bench Pro Leaderboard: Scale AI SEAL
The Bottom Line
SWE-Bench is the best available benchmark for evaluating AI coding capabilities on realistic software engineering tasks. Current frontier models score 70-80% on SWE-Bench Verified—impressive, but still meaning they fail on 1 out of 5 real issues.
For teams adopting AI coding tools, SWE-Bench scores are a useful signal, but should be validated against your own codebase and workflow. The benchmark tests bug-fixing ability, not all the skills that make a great software engineer.
Last updated: March 10, 2026
Related: