Multi-agent systems, where a lead agent delegates to several sub-agents, are a popular answer to hard tasks. Vals AI tested the idea in October 2026 on Vibe Code Bench, a benchmark in which a model builds a complete web app from a specification, using OpenAI's GPT 6 Sol and Anthropic's Claude Opus 5.5. The finding: teams cost 1.8x to 5.1x as much per app, and only one of four team-versus-single comparisons produced a statistically significant gain.
OpenAI
AnthropicHow the study was run
Each setup built 50 web apps, and each setup ran once. The score is the share of UI tests passed, averaged over the 50 apps. Vals compared two models, two reasoning-effort levels (medium and max) and two configurations (single agent and team), giving eight setups. Significance is from a paired t-test on the same 50 apps. Vals notes the two models used different harnesses, so cross-model gaps mix model and harness effects.
The results
Score and cost per app, eight setups
Vibe Code Bench score (%), with mean cost per app on the right. Green bars are multi-agent teams, grey are single agents. The axis starts at 70% to make gaps legible.
| Model | Effort | Single score / cost | Team score / cost | Difference | Significant? |
|---|---|---|---|---|---|
| GPT 6 Sol | Medium | 77.6% / $1.22 | 84.9% / $3.07 | +7.3 pts | Yes (p = 0.005) |
| GPT 6 Sol | Max | 89.0% / $4.82 | 90.4% / $8.54 | +1.4 pts | No |
| Claude Opus 5.5 | Medium | 91.5% / $4.08 | 91.2% / $9.10 | -0.3 pts | No |
| Claude Opus 5.5 | Max | 89.8% / $23.77 | 93.2% / $122 | +3.4 pts | No |
Analysis: cost per extra point
Dividing extra spend by extra score points (our arithmetic from Vals' published means, treating non-significant differences as indicative only) shows how uneven the returns are:
- Sol, medium, single to team: $1.85 more for +7.3 points, about $0.25 per point.
- Sol, single agent, medium to max effort: $3.60 more for +11.4 points (p below 0.001), about $0.32 per point. Raising effort bought more than adding a team at the same starting point.
- Sol, max, single to team: $3.72 more for +1.4 points, about $2.66 per point, not significant.
- Opus, max team vs medium single agent: $117.92 more for +1.7 points (93.2% vs 91.5%, p = 0.36), about $69 per point, at roughly 30 times the cost of the medium single agent.
The pattern is diminishing returns. Where a model starts lower, structure helps. Where it already scores near 90%, the ceiling and the noise dominate. Opus 5.5 at medium effort as a single agent (91.5%, $4.08) landed within the statistical noise of its own max-effort team (93.2%, $122).
Runtime and orchestration
Teams did not reliably save time either. For Sol, the team ran 11.9 minutes against 12.4 at medium effort, and 27.6 against 38.4 at max. For Opus, the team was slower: 26.6 minutes against 20.3 at medium, and 174.1 against 74.7 at max, about 2.3 times as long. At max effort, Vals reports the Opus lead agent used 6.8 sub-agents per app with a median sub-agent runtime of 46 minutes, and the Sol lead 5.1 on average. Vals also observed different orchestration styles: Sol split work along architectural lines and the lead tested the app itself, while Opus wrote a shared contract first and delegated in waves.
Opus 5.5 at medium effort as a single agent scored within noise of its $122 max-effort team.
Metir analysis of Vals AI data
Caveats
- One run per setup. Intervals and p-values reflect variation across the 50 apps, not repeated runs of the same app, so a single bad or lucky run could move a result.
- Harness differences. Model and tooling are not separated, so do not read the table as a clean model ranking.
- One task type. The authors say results may not transfer to other kinds of work, and the benchmark rewards passing UI tests, not code quality or maintenance cost.
- Not significant is not zero. A non-significant result means the study cannot rule out no effect, not that teams never help. Three of the four comparisons simply lacked enough evidence either way.
- Related evidence. Compare this with Epoch AI's study of AI on real office tasks and OpenAI's multi-agent long-horizon model.
What it means for choosing a setup
The study argues for measuring before scaling out. For a given task, the cheapest lever to test first is reasoning effort or model choice, since Vals found those moved Sol's score more per dollar than a team. A team earns its cost when the single-agent baseline is weak and the work splits cleanly. Because the best configuration differs by model and budget, tools that let you try several models on the same prompt, as Metir does, make this kind of comparison practical without committing to one provider's stack.
Sources:
- Multi-agent Vibe Code Bench - Vals AI
- Vals AI Finds Multi-Agent Coding Teams Cost 5x More but Rarely Score Better - AlphaSignal
- Can AI automate Epoch? - Metir analysis
Image credits
Header image: rows of servers in a data center, by BalticServers.com, Wikimedia Commons, CC BY-SA 3.0 (source); illustrative. In-body image: NVIDIA H100 modules, by geekerwan, Wikimedia Commons, CC BY 3.0 (source).