Cost Per Task, Not Cost Per Token:
A Measurement Framework for the Real Economics of Claude, OpenAI and Grok

Abstract

Per-token pricing is the dominant unit of account for large language model (LLM) workloads. It is the wrong unit for agentic work. A token is an input to a task, not the task itself. When an agent reads a repository, calls tools, reasons across dozens of steps, and retries after failures, the quantity that matters to a business is the cost of one completed unit of work, not the price of a thousand tokens.

This whitepaper proposes the Cost Per Task (CPT) Measurement Framework, a replicable protocol for measuring cost per completed task in LLM and agentic workloads. The framework defines a formal cost model over token classes and trajectory steps, an expected-cost-per-solved-task estimator under retry policies, a risk-adjusted variant that prices the cost of undetected failures, and a variance-aware reporting standard using P90 cost and consistency metrics.

It specifies task corpus construction, a trajectory-level instrumentation schema aligned to OpenTelemetry GenAI semantic conventions, outcome adjudication with inter-rater agreement thresholds, and a paired experimental design with bootstrap confidence intervals. We apply the framework to a Claude versus OpenAI comparison as of August 2026, and extend the price and benchmark data to xAI's Grok 4.6, released August 12, 2026.

The conclusion inverts depending on the unit of account. On coding, GPT-5.6 Sol wins on cost per solved issue. On policy-sensitive tool use, Claude wins once cleanup cost is priced in. Grok 4.6 undercuts both on measured cost per task while its reliability remains unmeasured, which isolates the framework's open variable. We connect the method to FinOps unit economics and describe implementation in Cloud Intelligence™ and Attribute.

1 · Introduction and problem statement

Teams choose models by looking at two numbers. The first is a benchmark score. The second is a per-token price. Both are misleading for agentic workloads.

A per-token price answers the question "what does a thousand tokens cost." A business cares about a different question. "What does one completed task cost." These questions diverge sharply once a model stops answering single prompts and starts running tools in a loop.

The divergence has a measurable cause. Agentic tasks consume far more tokens than chat, and the consumption is highly variable. Bai et al. analyzed trajectories from eight frontier models on SWE-bench Verified and found that "agentic tasks are uniquely expensive, consuming 1000x more tokens than code reasoning and code chat, with input tokens rather than output tokens driving the overall cost," that "token usage is highly variable and inherently stochastic: runs on the same task can differ by up to 30x in total tokens," and that "higher token usage does not translate into higher accuracy."² The same study found that on average Kimi-K2 and Claude-Sonnet-4.5 consumed over 1.5 million more tokens than GPT-5 on the same task set, a model-specific behavior rather than a task-difficulty effect.

Three consequences follow. First, a cheaper per-token price can produce a higher cost per task if the cheaper model takes more steps or fails more often. Second, run-to-run variance means a single measurement is close to meaningless. Third, the visible output tokens are a small fraction of the bill. Input tokens, cache reads, cache writes, and reasoning tokens dominate.

FinOps practice already has the right frame. The FinOps Foundation defines unit economics as "the essential system for translating technology spend (in this case, cloud) into meaningful business language," and advises teams to "define a key unit metric for your business (e.g., cost per analyzed transaction, cost per customer, cost per ride)" in order "to either reduce the cost to produce or the cost to serve one unit of value."⁶ For agentic AI, the natural unit is the completed task. This whitepaper builds the measurement discipline for that unit.

Our contribution is a framework, not a leaderboard. We call it the Cost Per Task (CPT) Measurement Framework. It is designed so a team can measure the real economics of its own workload on its own traffic, report the result in a way another team can reproduce, and make a defensible model-selection decision.

2 · Related work and background

Cost per task in public benchmarking. Artificial Analysis introduced per-task cost reporting in its Intelligence Index v4.1, described as a shift toward agentic workloads. Its method, in its own words, is: "We take the total cost, total time, and total output tokens for a model to run the Intelligence Index and divide by the number of tasks across its evaluations."¹ Cost per task is computed from input, cache-hit, cache-write, reasoning, and answer token prices, weighted by each benchmark's weight in the Index. Reported examples include GPT-5.5 (xhigh) at 0.99 USD per task, Claude Opus 4.8 (max) at 1.78 USD per task, and DeepSeek V4 Pro (max) at 0.04 USD per task. The v4.1.1 Index incorporates nine evaluations: GDPval-AA v2, tau-cubed-Banking, Terminal-Bench v2.1, SciCode, Humanity's Last Exam, GPQA Diamond, CritPt, AA-Omniscience, and AA-LCR, with v4.1 weights of roughly 20 percent on GDPval-AA v2, 16 percent on Terminal-Bench 2.1, 14 percent on tau-cubed-Banking, and 12 percent on Humanity's Last Exam. This is close to what we propose, but it measures cost per attempt on a fixed public suite. It does not divide by success probability, price the cost of undetected failures, or measure a team's own traffic.

Tool-use reliability. Yao et al. proposed tau-bench, a benchmark of dynamic conversations between a simulated user and an agent that must call domain APIs and follow written policy.³ It grades the final database state against an annotated goal. It introduced $\text{pass}^k$, the probability that an agent solves the same task on every one of $k$ independent trials. For a per-trial success probability $p$, $\text{pass}^k \approx p^k$, so consistency falls as $k$ grows. Sierra reported that "the agent powered by GPT-4o drops to ~25% on pass^8 in tau-retail, which is a staggering 60% drop compared to its corresponding pass^1 score," where GPT-4o's retail pass@1 was 60.4 percent. This is the reliability collapse that per-attempt cost figures hide. The benchmark has since been extended to tau2-bench across retail, airline, and telecom domains, with a v1.0.1 grading update in July 2026.

Coding benchmarks. SWE-bench Verified is a 500-task human-validated subset of SWE-bench, released by OpenAI in August 2024 with the original authors, and validated by 93 contracted software developers.⁵ Each task is a real GitHub issue graded by unit tests in a Docker container, drawn from 12 open-source Python repositories. Terminal-Bench 2.0 is a curated benchmark of 89 command-line tasks across ten technical domains, each with a Dockerized environment and pytest verification.⁴ The authors report that "frontier models and agents score less than 65%" on it. Terminal-Bench v2.1 is a verified refresh with the same 89 tasks and patched environments.

Trajectory token consumption. Bai et al. is the primary empirical source for the token-variance findings above.² Its dataset of eight models on 500 SWE-bench Verified tasks with four runs each is, to the authors' knowledge, the most granular public corpus of agentic trajectories. The paper also found that models predict their own token usage only weakly, with a best correlation of about 0.39, which makes pre-execution cost estimation unreliable.

FinOps unit economics. The FinOps Foundation Unit Economics capability defines the discipline of tying spend to a unit metric.⁶ Related guidance appears in Microsoft's FinOps Framework documentation, which recommends building unit costs into business KPIs and expanding them to include external licensing and labor.

Caching. Anthropic prices cache reads at 10 percent of base input, and cache writes at 1.25 times base input for a 5-minute time to live or 2 times for 1 hour, per its platform documentation.⁷ OpenAI prices cache reads at a 90 percent discount.⁸ Caching materially changes the effective input rate, so any cost comparison that ignores cache configuration is not reproducible.

Tokenizer differences. Token counts are not comparable across model families for the same text. Anthropic documents that "Claude 4.7 and later models and Claude Mythos Preview use a newer tokenizer. The same input text produces approximately 30 percent more tokens than on earlier models."⁷ Anthropic's own Sonnet 5 guidance gives a range of roughly 1.0 to 1.35 times depending on content type, and independent testing (Playcode, reported by The Register) found up to 1.73 times more tokens than GPT-5.x on one 2,888-character TypeScript file. This means a per-token price cannot be compared across vendors without normalizing to a common unit of text.

Harness effects. Practitioners increasingly hold that the harness, meaning the tools, context management, and verification loop around the model, can matter as much as the model. Faros AI, Databricks, and Thoughtworks have all articulated versions of "Agent equals Model plus Harness." This is a threat to validity for any comparison that does not hold the harness fixed.

Statistics. We use the Wilson score interval for success probability, which holds coverage near the nominal level for small samples and proportions near 0 or 1, unlike the Wald interval.^{9 10} We use the nonparametric bootstrap for cost intervals because cost per task is heavy-tailed and non-normal.¹¹

Positioning. Public per-task leaderboards measure cost per attempt on shared suites. Our contribution is the estimator for cost per solved task on your own held-out traffic, the risk adjustment for undetected failures, the variance-aware reporting standard, and the disclosure checklist for reproducibility. We do not claim to have invented cost per task as a concept. We claim a replicable measurement discipline for it.

3 · Current prices and benchmark data as of August 2026

Table 1 lists API prices as of August 2026. All figures are USD per million tokens. Prices change frequently, so every reported CPT must record the prices used.

Table 1. Model list prices, August 2026 (USD per million tokens).

Cache writes on Claude cost 1.25 times base input for a 5-minute time to live and 2 times for 1 hour.⁷ Anthropic cache reads are 10 percent of base input. OpenAI cache reads carry a 90 percent discount.⁸ GPT-5.6 tiers bill 2 times input and 1.5 times output for requests above 272K input tokens.

Grok 4.6 prices are the standard tier below a 200,000-token prompt.¹⁵ At or above 200,000 tokens, every token in the request bills at 4.00 input, 1.00 cached, and 12.00 output; the higher rate applies to the whole request, not the overflow, and the 500K window means a request can enter the doubled band well before it fills the context. A fast variant costs twice the standard rate, and priority processing doubles all token types including reasoning tokens. Grok 4.6 is excluded from xAI's 20 percent batch discount, which covers only older Grok models. Server-side tools add per-call fees on top of tokens, for example 5.00 per 1,000 web or code-execution calls. These mechanics matter for the framework because they change $C_{\text{attempt}}$ discontinuously with context length, which a per-token comparison hides.

Note the trap the framework is designed to catch. Opus 5 and GPT-5.6 Sol share a 5.00 input price. Opus 5 output is cheaper at 25.00 against 30.00. On per-token price, Opus 5 looks like the better deal. But Claude's newer tokenizer produces roughly 30 percent more tokens for the same text, and success rates and step counts differ. Per-token price does not settle the question.

Table 2. Selected benchmark scores, 2026 (as reported; harnesses vary).

Benchmark scores are reported with different harnesses and effort settings. Treat one to two point cross-lab differences as noise. GPT-5.6 Sol does not appear to publish tau2 retail or airline numbers. Its emphasized agentic figure at launch was Terminal-Bench 2.1, reported at 88.8 percent at announcement. The tau2 rows shown are aggregator figures attributed to Opus 4.6, not Opus 5, and are included only to indicate order of magnitude. A separate peer-reviewed constraint-violation benchmark (ODCV-Bench) reported a stark divergence in agent safety profiles, with Claude-Opus-4.7 showing a 0.0 percent misalignment rate against 21.2 percent for GPT-5.5, which is directional evidence for the reliability gap the case study prices in Section 6.¹⁴

3.1 A third vendor: Grok 4.6

xAI released Grok 4.6 on August 12, 2026, one day before this version of the whitepaper. It is a useful third data point because it separates two claims the per-token debate usually conflates.

On raw price, Grok 4.6 lists at 2.00 input and 6.00 output per million tokens, well below Claude Opus 5 at 5.00/25.00 and GPT-5.6 Sol at 5.00/30.00. On measured cost per task, Artificial Analysis places Grok 4.6 (high) at 0.84 USD per Intelligence Index task at a score of 61, against 1.04 for GPT-5.6 Sol, 2.03 for Claude Opus 5, and 2.75 for Claude Fable 5.¹⁶ Grok 4.6 is cheaper on both axes at once. Table 3 lists the independently measured figures.

Table 3. Grok 4.6, independently measured (Artificial Analysis, first-party API, August 12, 2026).

The agreement between per-token and per-task cost is itself instructive, because the whitepaper's other examples show the two can diverge. The mechanism is measurable: Artificial Analysis reports Grok 4.6 completing Index tasks in roughly 53 turns and 0.5B input tokens on average, against roughly 103 turns and 2.0B input tokens for Claude Opus 5 at max effort. Turn efficiency compounds with a lower output price. The tokenizer reinforces the point in the opposite direction from Claude: independent measurement of the predecessor Grok 4.5 put it near parity with OpenAI's o200k encoding (about 1.03x on a blended English coding workload), so Grok's low per-token price is not eroded by tokenizer inflation the way Claude's is. Tokenizer continuity from Grok 4.5 to 4.6 is inferred, not confirmed.

xAI's own launch table also self-reports parity or better against GPT-5.6 Sol and Claude Fable 5 on several agentic suites. We exclude those figures from Table 3 because the competitor numbers were drawn from published system cards rather than rerun, the bolded wins sit inside Artificial Analysis's published confidence intervals, and the clearest self-reported results are losses (DeepSWE 65.9 against 73 for Sol; Terminal-Bench v3.0 26 against 34.6), which a vendor rarely overstates. No SWE-bench Verified or Coding Agent Index result exists for Grok 4.6 from any source.

The teaching point for this whitepaper: per-token price still fails to settle the question, but for Grok 4.6 it fails in the buyer's favor. The numerator of the CPT estimator is well measured and low. The denominator and the leak term are not independently measured at all. Grok 4.6 is strong and cheap on everything the API bill can see, and unproven on everything it cannot, which makes it the cleanest illustration in this paper of why the reliability variable, not the price sheet, is where the decision lives.

4 · Formal model

4.1 Notation

Consider a workload of tasks. A task is a unit of business work with a verifiable outcome, for example resolving one issue or completing one customer request correctly.

An agent runs a trajectory to attempt a task. A trajectory is a sequence of steps $s = 1, \ldots, S$. Each step is one model call plus any tool calls.

At step $s$, the model consumes tokens in classes. Let:

• $x_s$ = fresh input tokens (uncached prompt content)
• $r_s$ = cache-read tokens
• $w_s$ = cache-write tokens
• $t_s$ = reasoning or thinking output tokens
• $y_s$ = visible output tokens

Let the per-token rates be $p_x, p_r, p_w, p_t, p_y$. On current APIs, reasoning tokens and visible output tokens bill at the same output rate, so $p_t = p_y$ in practice, but we keep them separate for measurement because they carry different information.

4.2 Cost of one attempt

The cost of one trajectory (one attempt) is the sum over steps of token counts times rates:

Because token counts vary run to run, $C_{\text{attempt}}$ is a random variable. We denote its mean by $\mathbb{E}[C_{\text{attempt}}]$ and estimate it from repeated runs.

The formulation above prices tokens because tokens dominate API-served workloads, but nothing in the estimator requires it. $C_{\text{attempt}}$ generalizes to the sum of all metered resources a trajectory consumes: GPU-hours for self-hosted models, and the storage, network, and compute the agents and their tools ride on. AI cost does not live in a vacuum, and a cost-per-task figure that excludes the infrastructure under the tokens undercounts the task. Section 7 discusses attribution of these non-token components.

4.3 Expected cost per solved task

Let $p$ be the probability that one attempt solves the task. Under a retry-until-success policy with independent attempts, the number of attempts is geometric with mean $1/p$. Expected cost per solved task is:

This is the central estimator. It says the true cost of a solved task is the cost of one attempt inflated by the reciprocal of the success rate. A model with a 20 percent lower attempt cost but a materially lower success rate can be more expensive per solved task.

Note that $p$ is not a property of the model alone. It is a joint property of the model, the prompt, the harness, and the task mix. A prompt revision or a verification-loop change moves $\mathrm{CPT}_{\text{solved}}$ through the same equation as a model change. This has two consequences. When comparing models, prompts and harness must be held fixed (Section 5.4). When optimizing cost, prompt and harness work compete with model selection on equal terms, and sometimes wins.

4.4 Capped-retry variant

Unlimited retries are rare in production. Under a policy of at most $N$ attempts, the probability of eventual success is:

Expected attempts until success or exhaustion is $\frac{1 - (1-p)^N}{p}$. For the tasks that eventually succeed, cost per solved task remains $\mathbb{E}[C_{\text{attempt}}]/p$. The useful figure for budgeting, however, is cost per task attempted, including the tasks that exhaust retries without success. That is $\mathbb{E}[C_{\text{attempt}}]$ times expected attempts, divided by the number of tasks attempted. Reporting both prevents a model with a high failure rate from looking cheap because its failures are excluded from the denominator.

4.5 Risk-adjusted total cost of ownership

A passed automated verifier is not the same as a correct outcome. Some fraction of tasks marked "solved" contain undetected defects. In tool use, some tasks violate policy. Call this the leak rate $L$, the fraction of accepted outputs that are actually wrong. Each leaked failure carries a cleanup cost $K$, which captures human rework, incident response, refunds, or reputational cost.

Risk-adjusted cost per accepted task is:

This term is where reliability enters the economics. A model with a higher attempt cost but a lower leak rate can win once $K$ is nonzero.

4.6 Variance and P90 cost

Because $C_{\text{attempt}}$ is heavy-tailed, the mean understates budget risk. Report the 90th percentile of cost per solved task, denoted P90 CPT, alongside the mean. The 30 times run-to-run variance documented by Bai et al. means the tail is not a rounding error.² It is the part of the bill that causes overruns.

4.7 Confidence intervals on success rate

With $n$ tasks and $k$ successes, the point estimate is $\hat{p} = k/n$. For small $n$ or $p$ near 0 or 1, use the Wilson score interval:

with $z = 1.96$ for 95 percent confidence. The Wilson interval stays inside $[0, 1]$ and holds coverage near nominal for small samples, unlike the Wald interval.^{9 10} Propagate this interval through $\mathrm{CPT}_{\text{solved}}$ to get an interval on cost per solved task.

4.8 Decision boundary and break-even cleanup cost

Compare two models A and B. Model B is more reliable and more expensive. Prefer B when:

Solve for the break-even cleanup cost $K^{*}$ at which the ranking flips:

If $L_A \gt L_B$ (A leaks more) and B costs more per solved task, then $K^{*}$ is positive. For any cleanup cost $K \gt K^{*}$, the more reliable model B is cheaper in total. Below $K^{*}$, the cheaper-per-solved-task model A wins. This single equation is the core decision rule. It turns a vague "reliability matters" claim into a threshold a FinOps team can test against its actual rework cost.

4.9 From cost to value

CPT is a cost-efficiency metric, not a value metric, and the distinction bounds what it can decide. Let $v$ denote the business value of one correctly completed task within a task type. Then net value per task is $v - \mathrm{CPT}_{\text{risk}}$, and the ratio form is:

Every CPT comparison in this paper carries an implicit assumption: that $v$ is constant across the systems being compared. Within a task type and a fixed quality bar, enforced by the verifier and the leak-rate audit, that assumption is reasonable. It fails when a cheaper configuration also degrades the value of accepted outputs, for example a resolution that passes the verifier but satisfies the customer less. Lower task cost is not better economics unless the tasks being completed deliver equivalent value. When the assumption is in doubt, the comparison must move from CPT to net value per task, which requires joining cost telemetry to outcome and quality telemetry.

Measuring $v$ is out of scope for this framework and is workload-specific. The framework's contribution is the denominator: without a trustworthy cost per correct task, no AI ROI figure can be computed at all, which is why CPT is the foundational step rather than the end state.

5 · The Cost Per Task Measurement Framework

This section specifies the protocol. A team that follows it can measure CPT on its own workload and publish a result another team can reproduce. The workflow:

5.1 Task corpus construction

Sample a held-out task set from production. Do not reuse public benchmark tasks as the primary corpus, because frontier models may be contaminated by training exposure.

Define the task before sampling it. A task should be a unit the business already counts, such as a resolved ticket, a merged pull request, or a completed customer request, with two properties: a machine-verifiable end state and an unambiguous start event. Do not invent a new unit for the measurement; borrow one an operations team already reports on, so CPT lands in a number the organization recognizes. CPT figures are comparable within a task type, across models, prompt versions, and time. They are not comparable across task types: a dollar per merged pull request and a dollar per support ticket are different currencies, and averaging them produces a number with no referent.

• Stratify by task type (for example coding, retrieval, tool use), difficulty, and context size. Difficulty can follow the SWE-bench Verified convention of binning by expected human fix time.
• Set a minimum n per stratum. Aim for at least 30 tasks per stratum so the Wilson interval on success rate is usable, and more where per-stratum decisions matter.
• Avoid contamination. Prefer tasks created after the model knowledge cutoffs. Hold the corpus private. Rotate a fraction each quarter to detect nonstationarity.
• Label a gold outcome for each task, meaning the machine-checkable success condition and, where relevant, the policy constraints that must not be violated.

5.2 Instrumentation specification

Log at the trajectory step level. Use OpenTelemetry GenAI semantic conventions as the substrate.¹² As of 2026 these conventions are published but most attributes remain experimental, and the GenAI Special Interest Group, formed in April 2024, has extended scope to agent orchestration, MCP tool calling, and quality evaluation. Use the stable core attributes and extend for token classes that the spec does not yet fully standardize.

Minimum schema per step:

Record token counts from the provider API response, not from a local tokenizer estimate, because tokenizer differences make local estimates wrong across vendors. Emit token usage as OpenTelemetry metrics, which are cheap to keep at full fidelity, and keep full traces on a sample. Model name and token counts belong on spans and metrics, not on resource attributes, because a service can call more than one model.

5.3 Outcome adjudication

Use automated verifiers wherever the task allows, following the SWE-bench and tau-bench pattern of grading final state against a gold condition. Automated grading is objective but brittle. A correct output formatted slightly wrong can score zero.

To estimate leak rate $L$, draw a human adjudication sample from the automated "pass" set and independently review it. Use at least two raters. Measure inter-rater agreement with Cohen's kappa and require $\kappa \geq 0.6$ (substantial agreement) before trusting the leak-rate estimate. If kappa is below threshold, refine the rubric and re-rate. The leak rate is the fraction of automated passes that human review rejects, with a Wilson interval on that fraction.

5.4 Experimental design

• Paired assignment. Run every model on the same tasks. Paired comparison removes task-difficulty variance and is far more powerful than comparing on different task samples.
• Fixed harness and prompts. Hold the harness, tools, and prompts constant across models, per the fairness principle that the harness can matter as much as the model. Report the harness version. If you must use each vendor's native scaffold, say so and treat the result as a system comparison, not a model comparison.
• Control effort settings. Record and fix reasoning or effort levels. These change both cost and success rate.
• Repeat runs. Run each task at least $k$ times per model to measure run-to-run variance and to compute consistency. Given the documented 30 times token variance, a single run is not a measurement. Use $k \geq 4$, matching the four-runs-per-task design of Bai et al., and higher for tasks with high variance.²
• Randomize run order and any user-simulator seeds to avoid ordering effects.
• Statistical tests. Report bootstrap confidence intervals on cost per solved task by resampling tasks (and runs within tasks) at least 10,000 times.¹¹ Use paired bootstrap or a paired test for the difference between two models. Report Wilson intervals on success rate.

5.5 Metrics definitions

• CPT (cost per solved task): $\mathbb{E}[C_{\text{attempt}}]/p$, per Section 4.3.
• Risk-adjusted CPT: $\mathrm{CPT} + L \cdot K$, per Section 4.5.
• P90 CPT: the 90th percentile of the per-solved-task cost distribution.
• Consistency ($\text{pass}^k$): the fraction of tasks solved on all $k$ runs, per tau-bench.³ Report alongside pass@1.
• Cost efficiency frontier: plot capability (success rate or index) against CPT. Up and to the left is better value. Models off the frontier are dominated.

5.6 Reporting standard

Publish a minimal disclosure checklist with every CPT result, analogous to a model card or MLPerf reporting rules. Without it, results are not comparable across teams.

5.7 Pitfalls and threats to validity

• Contamination. Public benchmark tasks may be in training data. Prefer private, recent tasks.
• Price changes. Prices move. The Sonnet 5 introductory rate expires August 31, 2026, and GPT-5.6 Terra and Luna were cut on July 30, 2026. Record prices with dates and re-run when they change.
• Tokenizer differences. Claude 4.7-plus tokenizers produce about 30 percent more tokens than earlier Claude models for the same text, and cross-vendor counts differ by more still.⁷ Never compare per-token prices without normalizing to a common text unit.
• Harness and prompt confounding. The harness can dominate the model, and so can prompt quality. Benchmark success rates come from fixed, well-specified prompts and are an upper bound for most production traffic. Hold both fixed within a comparison, report their versions, and treat any published $p$ as specific to the prompts that produced it.
• Reward hacking. Automated verifiers can be gamed. Sample for human adjudication.
• Nonstationarity. Model endpoints change silently, and rankings built on live prices drift. The Artificial Analysis Coding Agent Index reversed its cost-per-task ordering between two frontier models within a single quarter of 2026 as prices and runs refreshed. Re-measure on a schedule and version every result with a date.

6 · Worked case study: Claude versus OpenAI

We apply the framework to two workloads. The numbers extend the two worked examples from the accompanying blog post. They are illustrative of the method. A team must reproduce them on its own corpus.

6.1 Coding task (SWE-bench-style)

Assume a coding workload where both models run in a fixed harness at comparable effort. Suppose measured attempt cost and success rate are:

• GPT-5.6 Sol: $\mathbb{E}[C_{\text{attempt}}] = 1.20$, $p = 0.77$
• Claude Opus 4.8: $\mathbb{E}[C_{\text{attempt}}] = 2.30$, $p = 0.75$

We use Opus 4.8 rather than Opus 5 because it has independently measured single-harness figures; Opus 5's published results mix harnesses.

Under a naive per-token view, Opus 4.8 looks competitive because its output rate is lower. Under per-attempt cost, Sol is already cheaper, consistent with the Bai et al. finding that Claude-family models consumed over 1.5 million more tokens than GPT-5 on the same coding tasks.² Under cost per solved task, Sol wins by roughly a factor of two, at 1.56 against 3.06 per solved issue. If the coding workload has a low leak rate and low cleanup cost, this conclusion holds after risk adjustment. This is a case where OpenAI wins, and the framework says so plainly.

6.2 Tool-use task (tau-bench-style)

Now assume a policy-sensitive customer-service workload. Suppose:

• Claude: $\mathbb{E}[C_{\text{attempt}}] = 0.94$, pass@1 = 0.64, policy violations = 4 per 100
• GPT-5.5: $\mathbb{E}[C_{\text{attempt}}] = 1.53$, pass@1 = 0.58, policy violations = 9 per 100

The measured tau-bench figures predate GPT-5.6 Sol, so the OpenAI side is one generation behind. No comparable Sol run has been published, and this comparison should be re-run when one exists.

The framework applies to Grok 4.6 as well, but the reliability side lacks independent measurement entirely. A single-attempt tau-cubed Banking score exists (50.7 percent, Artificial Analysis), but no pass^k consistency figure and no policy-violation rate has been published for Grok 4.6.¹⁶ The predecessor Grok 4.5 recorded a 9.4 percent policy-violation rate in the same Contra Collective test series, more than double Opus 4.8, but that test has not been repeated on 4.6. Until a comparable measurement exists, the cleanup term $L \cdot K$ cannot be estimated for Grok 4.6, and this case study assigns it none. At a 0.84 USD measured cost per Index task, the break-even arithmetic of Section 4.8 shows how much reliability Grok would need: its price advantage over Claude buys roughly one dollar of headroom per task, which a violation rate near its predecessor's would consume at cleanup costs as low as 11 USD.

Attempt costs and success rates give:

Claude already wins on cost per correct task, at 1.47 against 2.63. Now price the risk. Set cleanup cost $K = 25$ per violated task, reflecting a support escalation. Treat the policy-violation rate as the leak rate: $L_{\text{Claude}} = 0.04$, $L_{\text{GPT-5.5}} = 0.09$.

Claude wins decisively once cleanup is priced. The break-even cleanup cost from Section 4.8 is:

Here the more reliable model (Claude) is also the cheaper-per-solved-task model, so it wins at every nonnegative $K$. If instead Claude had been more expensive per solved task, $K^{*}$ would tell us the exact cleanup cost above which its lower violation rate justifies the premium.

6.3 Interpretation

The two examples show the central point. The winner depends on the unit of account and on the workload. Naive per-token comparison is uninformative. Per-attempt cost is better but still incomplete. Cost per solved task changes the coding conclusion. Risk-adjusted CPT changes the tool-use conclusion further. No single model wins everywhere. The framework makes the trade explicit and testable.

Consistency reinforces the tool-use result. tau-bench $\text{pass}^k$ falls toward $p^k$ as $k$ grows, and Sierra observed GPT-4o falling from 60.4 percent pass@1 to about 25 percent $\text{pass}^8$ on tau-retail.³ A model with a lower single-trial success rate degrades faster under repeated trials, so its production reliability gap is wider than pass@1 suggests. For an unattended agent that commits actions without a human in the loop, the $\text{pass}^k$ number and the policy-violation rate are the production numbers, and pass@1 is only the demo number.

7 · Implementation in practice

A FinOps team operationalizes CPT in four moves.

Define the unit and instrument it. Choose the completed task as the unit metric, consistent with FinOps Foundation unit-economics guidance.⁶ Instrument trajectories with the schema in Section 5.2 using OpenTelemetry GenAI conventions.¹²

Attribute spend to the unit. This is the hard part in production. Tokens have no tags. LLM gateways appear as the consumer of all traffic, so tag-based and SDK-based allocation miss most of the cost. Gartner data cited in DoiT's own material puts the share of cloud costs tracked at the unit level at only 43 percent, and AI makes this worse. DoiT Attribute, generally available since July 7, 2026, addresses this with a lightweight eBPF sensor that installs in about 15 minutes and observes traffic at the kernel level, mapping every token, model request, and GPU cycle back to the process, container, customer, feature, and agent that caused it, without SDKs, tags, or code changes.¹³ It separates cached, reasoning, input, and output tokens automatically and joins runtime data with provider billing from Anthropic, OpenAI, Google Gemini, and AWS Bedrock. This provides the numerator (cost) at the granularity the CPT estimator needs.

Compute and show back CPT. Divide attributed cost by verified completed tasks per stratum. Implemented in Cloud Intelligence™, this becomes a showback of cost per business outcome rather than cost per token. Track CPT and P90 CPT over time and alert on regressions.

Integrate billing exports. Reconcile measured token classes against token-level billing exports from Anthropic, OpenAI, Bedrock, and Vertex, so the CPT numerator ties to the invoice.

Join cost to value. CPT alone answers what a task costs, not whether the spend was efficient (Section 4.9). The completing step is joining cost telemetry to outcome and quality telemetry, so spend decomposes along the chain from model to feature to customer to task to outcome, and each level carries both a cost and a value signal. In FinOps terms, CPT is a usage and rate metric; it becomes unit economics when tied to the revenue or business outcome the tasks drive. Most organizations today are earlier in this progression, still asking what they spent on tokens last month. The sequence is the point: attribution first, cost per task second, value per task third, and each step makes the next one computable.

The framework itself is vendor-neutral. Any team can implement it with open telemetry and a spreadsheet. The DoiT products reduce the attribution and reporting effort, particularly for shared GPUs and gateway-fronted APIs where tagging fails.

8 · Limitations and future work

The estimator assumes independent retries. Real retries often condition on the prior failure, which changes both cost and success probability. A conditional model would be more accurate.

Leak-rate estimation depends on human adjudication, which is costly and imperfect. Cohen's kappa guards against poor rubrics but not against systematic rater bias.

Every reported $p$ is conditional on the prompts and harness that produced it. The framework controls this within a comparison but cannot make success rates portable across teams whose prompts differ, which limits the external validity of any single published CPT.

Cleanup cost $K$ is workload-specific and often uncertain. The break-even analysis handles this by reporting the threshold rather than a point estimate, but teams still need a defensible $K$.

Benchmark scores in Section 3 come from mixed harnesses and are not directly comparable, and the tau2 figures we cite are aggregator proxies for an earlier Claude generation. The case-study numbers are illustrative, not measured on a common corpus. Nonstationary endpoints mean any result decays. The right cadence for re-measurement is an open question.

Grok 4.6 data is thinner and more vendor-self-reported than the Claude and OpenAI data. Its per-token prices and Intelligence Index cost per task are independently sourced, but its coding and agentic launch table is xAI self-reported, it has no independent SWE-bench Verified or Coding Agent Index result, and it has no independent reliability or reward-hacking measurement. Any cost-per-completed-task estimate for Grok 4.6 therefore rests on a measured numerator and an unmeasured reliability denominator. Grok 4.6 was released one day before this version; the data gap is partly a function of age and should close.

Future work includes a standard open corpus format, a reference implementation of the instrumentation schema, and an inter-team reproducibility study. A further direction is cross-organization benchmarking from production telemetry: anonymized, aggregated runtime data could yield CPT distributions by workload type that no synthetic suite can produce, letting a team place its own cost per task against a population percentile. The open problem is a workload taxonomy that makes tasks comparable across organizations, together with the consent and anonymization design such aggregation requires.

9 · Conclusion

Per-token pricing is the wrong unit for agentic work. The right unit is the completed task, priced with success probability and the cost of undetected failure, and interpreted against the value the task delivers: CPT is the denominator of AI ROI, not the end state. The Cost Per Task Measurement Framework gives teams a replicable way to measure it: a formal cost model over token classes, an expected-cost-per-solved-task estimator, a risk adjustment for leaks, a variance-aware reporting standard, and a disclosure checklist for reproducibility. Applied to Claude and OpenAI as of August 2026, the framework shows that neither wins everywhere. GPT-5.6 Sol wins the coding example on cost per solved issue. Claude wins the policy-sensitive tool-use example once cleanup cost is priced. Grok 4.6 arrives cheaper than both on measured cost per task with its reliability unmeasured, which is the framework's open variable in its purest form. The discipline, not the verdict, is the contribution. Teams can implement it themselves, and Cloud Intelligence™ and Attribute reduce the attribution and reporting cost of running it in production.

References

1. Artificial Analysis. Intelligence Index v4.1 and v4.1.1, per-task cost methodology, and benchmarking methodology. artificialanalysis.ai.
2. Bai, L., Huang, Z., Wang, X., Sun, J., Mihalcea, R., Brynjolfsson, E., Pentland, A., Pei, J. (2026). How Do AI Agents Spend Your Money? Analyzing and Predicting Token Consumption in Agentic Coding Tasks. arXiv:2604.22750. DOI 10.48550/arXiv.2604.22750.
3. Yao, S., et al. (2024). tau-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains. arXiv:2406.12045. Sierra Research, sierra-research/tau2-bench.
4. Merrill, M. A., et al. (2026). Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces. arXiv:2601.11868. tbench.ai.
5. OpenAI (2024). Introducing SWE-bench Verified. openai.com. Jimenez, C., et al. SWE-bench, ICLR 2024.
6. FinOps Foundation. Introduction to Cloud Unit Economics and Unit Economics capability. finops.org. Also Microsoft Learn, FinOps Framework, Unit economics.
7. Anthropic. Prompt caching and token counting documentation. platform.claude.com.
8. OpenAI. API pricing documentation, GPT-5.6 family. developers.openai.com.
9. Wilson, E. B. (1927). Probable inference, the law of succession, and statistical inference. Journal of the American Statistical Association 22(158):209-212.
10. Agresti, A., Coull, B. A. (1998). Approximate is better than exact for interval estimation of binomial proportions. The American Statistician 52:119-126.
11. Efron, B. (1979). Bootstrap methods: another look at the jackknife. Annals of Statistics 7(1):1-26.
12. OpenTelemetry. GenAI semantic conventions and GenAI SIG (formed April 2024). opentelemetry.io.
13. DoiT. Introducing Attribute and AI Cost Attribution. doit.com; attrb.io.
14. ODCV-Bench (2026). A Benchmark for Evaluating Outcome-Driven Constraint Violations in Autonomous AI Agents. arXiv:2512.20798.
15. xAI (2026). Grok 4.6 announcement and pricing documentation. x.ai/news/grok-4-6; docs.x.ai. Announcement page inaccessible on direct fetch during research; figures reconstructed from xAI pricing documentation and corroborating secondary coverage (eesel AI, VentureBeat, August 12 to 13, 2026).
16. Artificial Analysis (2026). Grok 4.6 benchmarks and analysis, Intelligence Index v4.1.1. artificialanalysis.ai/articles/grok-4-6-benchmarks-and-analysis.