Research · January 2026 · Past experiment
Multi-Model Consultation on SWE-bench Verified: A January 2026 Experiment
In January 2026 we tested whether a small, fast model (Claude Haiku 4.5) solves more real GitHub issues when it can consult other models while it works. This post describes the setup and the results of that experiment.
Note (updated November 6, 2026)
This is a past experiment, run in January 2026. The results come from that experiment's setup (the models, prompts and harness listed under Reproducibility) and are not a claim about the current Polydev product, its default models or its performance. The 74.6% figure combines the baseline run and the consultation run (283 + 40 + 50 of 500 issues; see the overlap breakdown below).
- Venkata Subrahmanyam Ghanta
- ASU & Polydev AI
- Pujitha Sri Lakshmi Paladugu
- Microsoft
The Question
A common assumption is that the biggest, most expensive model gives the best results. We tested this on SWE-bench Verified, a benchmark of 500 real GitHub issues that AI systems must solve by writing actual code.
We ran Claude Haiku 4.5 on all 500 issues twice: once alone, and once with the option to consult two other models. The results below compare both runs with the public leaderboard at the time.
How We Compare
SWE-bench Verified leaderboard (December 2025)
| Rank | Model | % Solved |
|---|---|---|
| 1 | Claude Opus 4.6 | 74.4% |
| 2 | Gemini 3 Pro Preview | 74.2% |
| 3 | GPT-5.2 (high reasoning) | 71.8% |
| 4 | Claude Sonnet 4.5 | 70.6% |
| 5 | GPT-5.2 | 69.0% |
| – | Ours (Haiku 4.5 + Consultation) | 74.6% |
Our approach uses Claude Haiku 4.5 as the base model, with GPT 5.2 Codex and Gemini 3 Flash Preview as consultants.
How It Works
In the consultation run, the base agent could ask other models for their view when it faced a difficult decision, then decide for itself what to do.
- 01
The base agent works
Claude Haiku 4.5 works on a SWE-bench issue and reaches a hard decision.
- 02
It consults other models
It asks the two consultant models in this setup for their view.
consultation (experiment setup)
GPT 5.2 Codex
"Try approach A because..."
Gemini 3 Flash Preview
"Watch out for edge..."
- 03
The agent decides
The base agent weighs the answers and makes its own choice. In this experiment the combined result was 74.6%, against 64.6% for the baseline run alone.
Why Different Models Help Each Other
Different AI models are trained on different data and have different strengths. When we analyzed which problems each approach solved:
283
Solved by both
40
Only Haiku alone
50
Only with consultation
Key insight: 24% of our successes came from one approach solving problems the other couldn't. The models have genuinely different blind spots, so combining them covers more ground than using either alone.
When Does Consultation Help Most?
Consultation isn't always beneficial. Here's what we found:
Most helpful for
- Ambiguous requirements (85% helpful)
- Multi-file changes (78% helpful)
- Complex algorithms (81% helpful)
Less helpful for
- Simple one-line fixes (42% helpful)
- Clear, well-specified bugs (65% helpful)
Takeaway: Use multi-model consultation for hard problems. For simple fixes, a single model is often faster and just as effective.
The Cost Advantage
Same performance, much lower cost:
| Approach | % Solved | Cost per problem |
|---|---|---|
| Claude Opus 4.6 (frontier) | 74.4% | $0.97 |
| Ours (Haiku + Consultation) | 74.6% | $0.37 |
62% cost reduction while matching performance. This includes running both the baseline and consultation approaches.
Limits of This Experiment
- One benchmark (SWE-bench Verified), one base model and two consultant models, as listed below.
- The 74.6% figure combines two runs, the baseline and the consultation run, and the cost figures include both.
- Models and prices have changed since January 2026, so the numbers may not hold for other models or today's versions.
How This Relates to Polydev Today
Polydev puts other AI models inside the app you already use (Claude, ChatGPT, Claude Code, Codex, Cursor and others): you name a model and the answer comes back where you work. Asking two models the same question and comparing their answers, the kind of consultation tested here, is one use among several. Polydev makes no benchmark claim of its own. Setup is on the install page.
Reproducibility
All our code, predictions, and reasoning traces for 500 problems are available:
- Benchmark
- SWE-bench Verified (500 instances)
- Base model
- Claude Haiku 4.5
- Consultation
- GPT 5.2 Codex, Gemini 3 Flash
- Total cost
- $136.34 (both approaches)