Skip to content

Real-SWE 1.1

Benchmarking frontier AI models on private, real-world, enterprise codebases.

Read the blog and analysis

Leaderboard

October 2026
  1. Rank 1
    Resolution rate: 56.80%, one standard error either side: 53.5% to 60.1%
  2. Rank 2
    Resolution rate: 56.00%, one standard error either side: 54.4% to 57.6%
  3. Rank 3
    Resolution rate: 55.20%, one standard error either side: 52.5% to 57.9%
  4. Rank 4
    Resolution rate: 51.20%, one standard error either side: 49.0% to 53.4%
  5. Rank 5
    Resolution rate: 47.20%, one standard error either side: 44.8% to 49.6%
  6. Rank 6
    Resolution rate: 45.60%, one standard error either side: 42.5% to 48.7%
  7. Rank 7
    Resolution rate: 42.40%, one standard error either side: 39.9% to 44.9%
  8. Tied for rank 8
    Resolution rate: 39.20%, one standard error either side: 35.9% to 42.5%
  9. Tied for rank 8
    Resolution rate: 39.20%, one standard error either side: 35.8% to 42.6%
  10. Tied for rank 10
    Resolution rate: 38.40%, one standard error either side: 35.9% to 40.9%
  11. Tied for rank 10
    Resolution rate: 38.40%, one standard error either side: 35.8% to 41.0%
  12. Rank 12
    Resolution rate: 33.60%, one standard error either side: 30.9% to 36.3%
  13. Rank 13
    Resolution rate: 32.00%, one standard error either side: 28.9% to 35.1%
  14. Rank 14
    Resolution rate: 28.00%, one standard error either side: 25.3% to 30.7%
  15. Rank 15
    Resolution rate: 26.40%, one standard error either side: 23.9% to 28.9%
  16. Rank 16
    Resolution rate: 24.00%, one standard error either side: 21.2% to 26.8%
  17. Rank 17
    Resolution rate: 10.40%, one standard error either side: 8.5% to 12.3%
  18. Rank 18
    Resolution rate: 8.00%, one standard error either side: 6.5% to 9.5%
  19. Rank 19
    Resolution rate: 5.60%, one standard error either side: 4.1% to 7.1%
  20. Rank 20
    Resolution rate: 4.80%, one standard error either side: 3.4% to 6.2%
RankModelResolution rate
1Claude Opus 5.5
56.80%, one standard error either side: 53.5% to 60.1%
2GPT 6 Astra
56.00%, one standard error either side: 54.4% to 57.6%
3Gemini 4 Carbon
55.20%, one standard error either side: 52.5% to 57.9%
4Claude Sonnet 5.5
51.20%, one standard error either side: 49.0% to 53.4%
5GPT 6.1 Sol
47.20%, one standard error either side: 44.8% to 49.6%
6Claude Fable 5.1
45.60%, one standard error either side: 42.5% to 48.7%
7Muse Wonder 1.0
42.40%, one standard error either side: 39.9% to 44.9%
Tied for 8Gemini 4 Argon
39.20%, one standard error either side: 35.9% to 42.5%
Tied for 8Kimi K3.1 Pro
39.20%, one standard error either side: 35.8% to 42.6%
Tied for 10Particle-1
38.40%, one standard error either side: 35.9% to 40.9%
Tied for 10DeepSeek V4.1 Flash
38.40%, one standard error either side: 35.8% to 41.0%
12Hy4 preview
33.60%, one standard error either side: 30.9% to 36.3%
13Inkling 2 Small
32.00%, one standard error either side: 28.9% to 35.1%
14Grok 4.7
28.00%, one standard error either side: 25.3% to 30.7%
15GLM-5.3
26.40%, one standard error either side: 23.9% to 28.9%
16Qwen3.8-Max prime
24.00%, one standard error either side: 21.2% to 26.8%
17GPT 6 Luna
10.40%, one standard error either side: 8.5% to 12.3%
18Inkling 1
8.00%, one standard error either side: 6.5% to 9.5%
19Seed 2.1 Turbo
5.60%, one standard error either side: 4.1% to 7.1%
20Nemotron 3 Ultra
4.80%, one standard error either side: 3.4% to 6.2%
Resolution rate is equivalent to pass@1, averaged over five independent runs per task.All runs use the highest effort possible for each model with its harness.

We use native harnesses to reflect how enterprise engineers work in practice, evaluating model-and-harness combinations rather than models in isolation.

Pareto curve

Resolution rate by estimated cost per rollout

Models in leaderboard order, with estimated cost per rollout
ModelResolution rateEstimated cost per rollout
1Opus 5.556.80%$13.95
2GPT 6 Astra56.00%$3.78
3Gemini 4 Carbon55.20%$7.16
4Sonnet 5.551.20%$10.09
5GPT 6.1 Sol47.20%$0.79
6Fable 5.145.60%$14.24
7Muse Wonder 1.042.40%$3.76
8Gemini 4 Argon39.20%$4.66
9Kimi K3.1 Pro39.20%$2.03
10Particle-138.40%$0.56
11DeepSeek V4.1 Flash38.40%$0.19
12Hy4 preview33.60%$1.13
13Inkling 2 Small32.00%$4.70
14Grok 4.728.00%$2.66
15GLM-5.326.40%$3.35
16Qwen3.8-Max prime24.00%$7.06
17GPT 6 Luna10.40%$0.08
18Inkling 18.00%$5.13
19Seed 2.1 Turbo5.60%$3.03
20Nemotron 3 Ultra4.80%$1.39
The dotted line connects the best observed trade-offs, not intermediate evaluations.

About the benchmark

Private codebases

Agents must navigate proprietary systems whose code and solutions aren’t available on the public internet.

Work with business consequences

Getting billing right, calculating taxes, migrating customers. Changes that affect how a business runs, often across multiple services.

Company-specific complexity

Every company has its own rules and ways of writing code. Agents have to understand those conventions and make changes that work with what’s already there.