How Polydev Users Feed the Model Leaderboard

If you use ask_model (older name get_perspectives) in any app (Claude, ChatGPT, Claude Code, Codex, Cursor, Windsurf, Cline or another MCP client) and ask several models at once, each call produces a little ranking data. There is nothing extra to set up.

Such a call does two things: (1) returns each model's answer to you, and (2) records anonymous comparisons for the leaderboard.

Level 1: Comparisons from each call

When you call ask_model with several models, they answer in parallel. The server compares their outputs using quality heuristics (content length, code blocks, structured formatting) and records which model produced the better response.

# This single call generates ranking data automatically
# No additional code needed — just use Polydev normally

polydev.ask_model(
  "How should I structure the state management
   in this React app?",
  models=["model-id-a", "model-id-b", "model-id-c"]
)

# Behind the scenes:
# 1. The models you named respond
# 2. Prompt classified as: react / architecture / moderate
# 3. C(N,2) pairwise comparisons recorded
# 4. Elo ratings updated for all models

Level 2: Your ranking (rank_perspectives)

After the answers come back, you can ask your app to rank them from best to worst. It records the ranking with rank_perspectives. A ranking carries 1.5x the weight of an automatic comparison because it reflects a judgment made after reading all the answers.

# After reading the answers, rank them
polydev.rank_perspectives(
  ranked_models=[
    "model-id-b",   # Best answer
    "model-id-a",   # Second
    "model-id-c",   # Third
  ],                # IDs as returned by ask_model
  base_model="<the app's own model>",
  feedback_text="B gave the most complete
    architecture with clear tradeoffs"
)

The leaderboard tracks not just which models are being compared, but also which AI app is doing the judging. This is important because different base models may have different biases when evaluating responses.

Detection happens through three layers, in priority order:

1. 01

   Self-identification

   The AI model passes its own identity via the base_model parameter in rank_perspectives, such as the model ID it runs on. This is the most accurate method.

2. 02

   MCP client info

   During the MCP handshake, IDEs send their identity (e.g., claude-code, cursor, windsurf). The local npm client passes this along as ide and ide_version.

3. 03

   User-agent fallback

   If neither of the above is available, the server inspects the HTTP user-agent header for known patterns like claude-code or cursor.

The leaderboard at /leaderboard is public and requires no login. Here's how to get the most out of it:

Filter for your stack

Use the filter dropdowns to narrow rankings to your specific technology stack. The rankings recalculate in real time based on comparisons that match your filters.

Compare two models directly

Switch to the "Head-to-Head" tab to see the direct matchup record between any two models. The breakdown by judge method and task category reveals where each model has an advantage.

Read the methodology

Click "How rankings work" at the bottom of the page for details on the Elo system, judge methods, classification approach, and cross-model judging.

The leaderboard data is available through a public API. You can use this to build custom dashboards, automated model selection, or integrate rankings into your CI/CD pipeline.

# Fetch overall rankings
curl https://www.polydev.ai/api/leaderboard

# Filter by language and task type
curl "https://www.polydev.ai/api/leaderboard?language=python&task_type=debugging_runtime"

# Head-to-head comparison
curl "https://www.polydev.ai/api/leaderboard/head-to-head?model_a=MODEL_A&model_b=MODEL_B"

All 6 filter dimensions are supported as query parameters: task_type, language, framework, complexity, domain, intent.

The leaderboard improves as more developers use Polydev. Here are the factors that increase ranking quality:

The leaderboard covers the models people ask through Polydev: GPT, Gemini, Grok, Claude and more. Models are added as providers release them. The live list is on the leaderboard page.

Add Polydev to the app you already use with one link and sign in once in the browser. For Claude Code:

claude mcp add --transport http polydev https://polydev.ai/mcp
# then run /mcp in Claude Code and choose Polydev to sign in

Steps for Claude, ChatGPT, Codex, Cursor, VS Code, Windsurf and other apps are on the install page. Polydev is free: 10 free requests to try, then your ChatGPT plan (local client) or your own API keys pay.