I Stopped Asking Which AI Model Is Best

TL;DR I built a model frontier dashboard to help me decide which model to use for different kinds of work. It goes beyond leaderboard scores to calculate the average cost of completing an entire task.

Building it surfaced something counterintuitive: a lower per-token price does not mean a lower cost to finish the job. If a model needs more steps and reads more data along the way, you can end up spending more. On DeepSWE’s long-horizon benchmark, Sonnet 5, which has the lowest per-token price in its family, averages $26.4 per task.

Models are being released faster and faster. It feels like a Cambrian explosion. New names, new leaderboards, and new claims keep arriving, and it’s hard to judge which model to pick on intuition alone. So I built a model frontier dashboard that uses charts to organize model capability, cost, and appropriate use cases.

The day after the dashboard went live, I was staring at the scatter plot I had just added and had to double-check: was Sonnet 5 really the most expensive?

Here’s how to read the chart. Each model is a dot. The horizontal axis is the average cost to complete one task; the vertical axis is task success rate. To put “cheap and accurate” models toward the upper right, I reversed the cost axis: further right is cheaper, further left is more expensive. Claude Sonnet 5 sat at the far left of the chart, clearly separated from everything else: $26.4 per task on average, the highest of any configuration in the dataset.

It’s the lowest-priced model in the Claude family: $2 per million tokens. I assumed I had drawn the chart wrong.

The answer was in two other columns of the official data file. On this set of long-horizon coding tasks, Sonnet 5 averages 268 operation steps per task. More steps means it repeatedly reads code, calls tools, and revises its answers, accumulating a large token count throughout. The more counterintuitive finding was this: some models with a considerably higher per-token price end up with a lower total cost per task.

At that moment I knew the tool was pointed in the right direction. This is exactly the kind of thing leaderboards don’t tell you.

This article covers three problems I ran into while building the dashboard: why leaderboards aren’t enough on their own; why the cheaper token price can ultimately cost you more; and how to build a model-selection approach that doesn’t expire the week after you finalize it.

Why Leaderboards Can’t Tell You Which Model to Use

Let’s start with what leaderboards actually are. A model leaderboard prepares a standard set of problems, has different models complete them, and compares correct-answer rates or task success rates. The catch is that every leaderboard uses different problem types, tool environments, and time horizons. A model ranking first on one board means it’s particularly good at that specific test, nothing more.

The first limitation: different benchmarks produce different winners. SWE-bench primarily tests whether a model can handle real software bugs in actual codebases. DeepSWE focuses more on whether a model can operate autonomously over long stretches: repeatedly reading code, calling tools, revising, and verifying results. Claude Fable 5 leads on the former (around 95%); the GPT-5.6 family performs better on the latter. The two boards don’t contradict each other. They’re asking different questions.

The second limitation: small gaps at the top don’t necessarily reflect real capability differences. DeepSWE’s top three scores are 72.7%, 69.9%, and 69.6%. On the surface the leader is ahead by two or three percentage points, but the benchmark only has 113 problems, and running it again could shift results slightly. The 95% confidence intervals of all three overlap, so this data alone isn’t enough to conclude there’s a meaningful capability difference between them. Reading “two points higher” as “clearly better” is the most common misreading in leaderboard culture.

When capability differences are too small to distinguish, cost becomes one of the most important factors. For Agent workflows running at volume, a few dollars’ difference per task adds up fast. Within the GPT-5.6 family, Sol costs $8.39 per task, Terra $4.95, Luna $3.03. Their success rates sit in a similar range; their costs differ by nearly three times.

The Number AI Agents Actually Need: Cost Per Task

We’re used to looking at unit price: dollars per million tokens. For chat, translation, or single-turn tasks, that’s usually a reasonable proxy, since token usage is roughly fixed. Agent tasks are different: the model decides how many steps to take and how much to read. A cheaper model that plans less efficiently, or needs more attempts and corrections, can burn through more steps and tokens and cancel out any price advantage entirely.

For AI Agents, the right thing to look at isn’t the cost of a single call. It’s three things together: task completion rate, how many steps it takes to get there, and the total token and tool costs accumulated along the way. This isn’t a strict formula, but it’s a workable decision framework.

Back to Sonnet 5. Its per-token price is low, but this benchmark has it averaging 268 steps per task. Fable 5 at its maximum setting takes 88 steps. More steps means more code reads, more tool calls, and more iterative revisions. At each step, the model may also re-read prior context, so token usage compounds.

Working from the official data file, Sonnet 5 averages over 70 million input tokens per task, the cumulative total across all steps in a task, averaged across the benchmark. Its per-task cost comes to $26.4, with a success rate of 53.8%.

Fable 5 low costs just $3.76 per task with a 59.6% success rate. Opus 4.8 max achieves 59.0% at $13.22 per task. Similar performance; less than a third of the cost. The table also includes Fable 5 max so you can compare the same model across effort settings directly.

(Source: DeepSWE v1.1 official public data file, 113 tasks, 2026-07-09)

On this benchmark, Sonnet 5’s token pricing is lower, but completing tasks requires so many more steps that its total cost is the highest of any configuration. Token unit price is what you pay per unit of material. Task cost is the final bill for the whole job. The dashboard’s scatter plot uses cost per task on its horizontal axis, not unit price, precisely to make that final bill visible at a glance.

A Capability × Cost Map for Choosing Models

Combining leaderboard scores with cost per task is what produces a map you can actually use to choose a model. The first section of the dashboard organizes five common scenarios: everyday work, complex reasoning, software development, high-volume execution, and unconstrained budget. Each cell gives a current recommended choice and explains why.

Below that is the scatter plot. Six primary models each have an “effort curve.” Think of effort as how much compute and reasoning resources the model is willing to invest. The chart connects each model’s five settings, from low to high, by cost and score; a flatter curve means performance drops less when you reduce effort to save money.

The gold line is the efficiency frontier. Think of it as the options currently worth considering: every point on the line has no alternative that simultaneously offers a higher score and a lower price. Models off the line aren’t necessarily bad: from this data, something already exists that is more accurate, cheaper, or both. In this round, all 13 points on the frontier fall within the GPT-5.6 family’s configurations. That doesn’t mean GPT-5.6 tops every individual metric, but on this benchmark, it provides the most complete combination of capability and cost.

The complete list of all 13 models is below, kept in sync with the live dashboard:

(Source: DeepSWE v1.1, 113 tasks, updated 2026-07-09. Numbers shift with each weekly update; see the live version at https://paulkuo.tw/ai/models/.)

Additional data: The same dashboard separately tracks “generalization ability” — whether a model can reason on the fly about rules it has never seen, rather than pattern-matching to similar problems from its training data. This is a different dimension from the software-development cost discussion above; included here for reference.

(Source: ARC Prize, arcprize.org/leaderboard, verified 2026-07-12. This leaderboard’s model roster isn’t identical to the one above — these are voluntary vendor submissions, and Claude Fable 5, Claude Sonnet 5, Kimi K2.7 Code, and xAI Grok 4.5 don’t have official results yet. Live version at https://paulkuo.tw/ai/models/.)

Can You Trust the Numbers on a Leaderboard?

The genuinely hard part of building something like this isn’t the charts. It’s the data. Three risks have to be addressed in the design itself.

The first is missing an entire model. The list is curated by hand, and the biggest problem isn’t a wrong number: it’s a newly released model that never gets added. A wrong number is easy to spot; a missing model is invisible. So the page explicitly states the list is not exhaustive, and the weekly update process includes a dedicated step: actively searching for recently released models not yet in the dataset.

The second is misreading the source. When I traced the data back to its origin, I found that the DeepSWE site is dynamically rendered: the raw HTML contains no numbers at all. The AI summarization pipeline I was using couldn’t reach those dynamic values, yet it still produced a table that looked complete: proper columns, reasonable formatting, and some numbers that were simply wrong. We eventually used the browser to find where the site was actually pulling its data: an API endpoint, an official public data file with 41 model-configuration pairs and full 95% confidence intervals. From that point, the entire update pipeline stopped reading rendered text and started reading the raw data directly. The discovery that Sonnet 5 averages 268 steps per task came from that first-party source. Secondary summaries typically report only the score; they don’t show you how many steps a model took or how costs accumulated.

The third risk is the simplest and the hardest: will these numbers still be correct next week?

Automatic Updates Need Guardrails

The dashboard’s data is updated every Monday by an AI scheduler pulling the latest rankings. But nothing is published immediately after an automatic update. It first has to pass a set of pre-deployment checks, what agent workflow design calls guardrails. This dashboard currently has six:

1. Data file syntax: if the update corrupts the data file, block it.
2. Honest language: conclusions drawn from the dashboard’s own calculations cannot be presented as “official findings,” and claims of “human review” that never happened are not allowed; any language that overstates the evidence gets blocked.
3. History tamper protection: published historical records cannot be rewritten by an automatic update; any content that differs from last week’s version stops deployment and goes to human review.
4. Four-language consistency: if the numbers don’t match across all four language versions, block it.
5. Chart layout: if the data update causes labels on the scatter plot to overlap, stack on top of each other, or go outside the chart boundary, block it.
6. Prose quality: if the update introduces formulaic AI phrasing, unusual punctuation, or writing that doesn’t match the site’s voice, block it.

The third check is the one I care about most. The biggest risk with automatic updates is rewriting history. A system that overwrites itself every week, one that can’t preserve what it recommended last week and why, produces recommendations that can never be audited.

I summarized this approach in another article: workflows expire; judgment criteria compound. The tools, models, and update pipelines will all change, but the checks that each failure forces you to add keep accumulating. All six checks above were demanded by real problems. The full version of that argument is in Treating Difficulty as a Constant and Harnesses Expire, Criteria Compound.

Rankings Cannot Replace Task Judgment

Leaderboards can help narrow the field, but they can’t answer the final question: is this model right for my task?

For AI Agent work, model selection requires looking at at least four things at once: whether it can complete the task, how many steps it takes, what each completion costs, and what it costs to recover from a failure. That’s why the last section of the dashboard includes four copyable question templates: paste them into whatever AI you already use, bring your own task, budget, and error tolerance, and run the comparison yourself. That final judgment isn’t something a tool should make for you. It’s not something I should make either.

The dashboard is at https://paulkuo.tw/ai/models/, updated every Monday, available in four languages. More on human-AI collaboration is in the Intelligence & Order topic section.

Next time someone tells you a model is the strongest, ask the follow-up: strongest at which kind of task? And what would it actually cost to finish my work?