Leaderboard
SWE-Together · 109 tasks · opencode harness · k = 2. Models sorted by pass@1 (shown to one decimal for models we ran; paper rows as published). On each bar, the dark number = pass@1 and the white number = pass² (both runs solve at judge ≥ 0.85); the hatched tail is unstable (pass@1 − pass²). Oracle is the gold-patch reference ceiling. The best value in each column is bold (judge ↑ higher is better; correction / tokens / min / $ ↓ lower is better). Tokens = output + reasoning per task. $ / task = average model API cost per task at the vendor’s public list price (uncached input, cached input, cache writes and output + reasoning billed at their respective rates; where the vendor bills long-context calls at a higher tier — OpenAI above 272K prompt tokens, xAI above 200K — each call is priced at its own tier); “—” where per-token usage was not recorded. Sandbox audit (Sep 23, 2026): all 2,616 trials behind the models we ran were audited for sandbox escapes; the 111 trials that had pulled upstream code past the GitHub block were re-run on the hardened sandbox (SWE-Together #16) and re-judged. GPT-6 Astra, Sol and Luna rows are from an Oct 1 re-run (same protocol, k = 2) after a judge-timeout fix. Paper rows were not run by us and stay as published.
Evaluation runs · provider · date (2026)
Get in touch
We'd love to hear from you
Have a question, a session you think would make a great task, or feedback on the benchmark? To contribute or collaborate, contact Yifan Wu, Shengzhi Li and Zhuokai Zhao, join the Discord, or open an issue or pull request on GitHub.
Citation
Cite SWE-Together
@article{wu2026swetogether,
title = {SWE-Together: Interactive Coding-Agent Evaluation},
author = {Wu, Yifan and Zhao, Zhuokai and Li, Songlin and Lee, Ho Hin and Zhu, Jiacheng and Wu, Shirley and Yu, Tianhe and Li, Serena and Zhang, Lizhu and Fan, Xiangjun and Li, Shengzhi},
year = {2026},
journal = {arXiv preprint arXiv:2606.29957},
url = {https://arxiv.org/pdf/2606.29957}
}