Cost Per Usable Output Benchmark: Methodology, Results, and Audit-Grade Evidence

Defining Usable Output for Specific Workloads

Usable output is defined by strict, domain-specific acceptance criteria that an AI response must satisfy before reaching a downstream system or a human user. An output is unusable if it fails syntactic parsing, violates regulatory compliance rules, or contains factual errors. No partial credit. That constraint is what makes the metric auditable.

Criteria vary across core financial and software engineering workloads:

One practical note from validation reviews: teams almost always write the happy-path rubric first and the failure taxonomy never. Then the benchmark cannot explain why a model failed, only that it did, which is useless when a supervisor asks for remediation logic.

Why Cost Per Correct Answer Changes Model Comparisons

Cost per correct answer reshuffles model efficiency rankings by proving that cheap models are often expensive in production. When low-cost models fail, the accumulated expense of automated retry loops and manual interventions quickly surpasses the cost of a single run on a higher-priced model. Because expected attempts equal 1 / R_m(p), total spend scales linearly with the failure rate.

In an internal validation exercise on a trade reconciliation workflow, a mid-tier model with a $0.50 per 1M token list price achieved a 52% first-pass accuracy rate (n = 240 reconciliation cases; 95% Wilson interval ±6.3 points). The engineering team added automated retry logic to handle failures. That escalated average run attempts to 1.92 per task, raising the effective cost per correct answer to $1.28. Switching to a $2.50 per 1M token frontier model yielded a 96% first-pass rate. The frontier model produced a lower cost per usable output benchmark result while also reducing processing latency.

Evidence note: the case above is an internal, non-public validation exercise built on a hypothetical portfolio. Treat the sample size, interval, and retry multiplier as indicative rather than externally reproducible until the harness and dataset schema are published. Independent research reports the same directional effect:

Published cost-per-correct-answer figures show how wide the spread becomes. One legal-technology technical report defines the metric as total run spend divided by credit-weighted correct answers, and records $0.68 falling to $0.36 with accuracy held nearly constant. A separate 2026 evaluation reports roughly $0.046 per correct perturbation answer for an open-weight model against roughly $0.324 for a frontier model on the same task family. That is an approximately sevenfold gap, and it is invisible in per-token price sheets.

Standardized Test Datasets, Sample Size Math, and Uniform Run Conditions

Objective benchmarking demands a static dataset that covers standard operational queries, boundary cases, and adversarial edge inputs. All candidate models must process identical prompts under identical system instructions.

To prevent evaluation drift, fix these parameters across every execution:

• Temperature and Top_P set temperature to 0.0 (or deterministic sampling limits) for reproducible factual and structured tasks. Temperature sweeps, where required, should be run as a controlled sweep rather than mixed inside a comparison set.
• System messages standardize corporate compliance framing, output constraints, and persona directives across all runs. Published evaluation work finds that the system-instruction layer has a statistically significant effect on factual accuracy, often larger than temperature, so instructions must be frozen and version-controlled.
• Decoding limits cap maximum generation tokens to prevent run-away decoding loops during model failure states.
• Repeats and seeds run each prompt at least three times per model-and-effort cell and record the seed. Report majority-vote pass rate alongside per-run variance.

Classic UX benchmarking practice reaches the same conclusion from the opposite direction. Detecting a 20-percentage-point difference in completion rates at 90% confidence and 80% power with independent groups requires roughly 75 participants per group, or 150 in total (Sauro, "How to Benchmark Website Usability", MeasuringU). The lesson transfers directly: the finer the difference you claim, the larger the harness you must fund. Report every usable rate with a Wilson score confidence interval, and state explicitly when two models are statistically indistinguishable, rather than declaring a winner on a two-point delta.

Tracking Inference Costs, Token Usage, Reasoning Effort, and Human Review

Accurate cost tracking requires separating input prefill tokens, cached prompt tokens, generated output tokens, and hidden thinking tokens. Advanced reasoning architectures (such as o1, o3, or DeepSeek-R1) generate internal reasoning chains billed at standard output token rates. DeepSeek-R1, for example, documents a maximum generation length of 32,768 tokens covering reasoning plus final answer.

Formula for total attempt cost ($C_m$):

Where $N$ represents token counts and $P$ represents the specific vendor price rate per token. $C_{\text{external}}$ captures third-party API or sandbox tool execution costs. $C_{\text{human}}$ captures the fully loaded analyst minutes consumed by review, correction, or escalation of that attempt. $C_{\text{idle}}$ captures reserved-but-unused dedicated GPU capacity plus datacentre Power Usage Effectiveness overhead.

Omitting $C_{\text{human}}$ is the single most common error in enterprise AI business cases. It systematically flatters low-accuracy, low-price models, because their residual risk is silently absorbed by the second line of defence. Silently, and then loudly, at audit.

Then the risk-adjusted business metric is:

For teams that also meter adjacent generative workloads, the same accounting discipline applies to non-text modalities. See how per-request billing and quota accounting are structured in the Google Veo implementation guide, which documents API access, per-second cost drivers, and rate limits for a metered generative endpoint.

Validating Quality and Determining Successful Outputs

Quality assessment relies on deterministic validation tools supplemented by calibrated LLM-as-a-judge scoring frameworks. The overall success metric is expressed as the usable rate:

For structured outputs, pass and fail state is determined by automated JSON schema validators. For code generation, passing unit test suites defines usability, reported as pass@k, the probability that at least one of k samples passes the tests. For narrative and analytical tasks, an LLM judge evaluates output against a multi-point rubric calibrated against human expert annotations. The MT-Bench work reports GPT-4-class judges reaching over 80% agreement with human preferences, which sets a defensible calibration floor rather than a licence to skip human review.

Report pass@1 and pass@k side by side. A large gap between them is the clearest available signal of agent brittleness. A workflow that succeeds only when given three attempts is not a 90% workflow; it is a 60% workflow with a retry subsidy.

E-E-A-T: methodological verification

Model Selection, Reasoning Effort, and Answer Quality

Tokens, Batch Size, Throughput, and Latency

Long prompt contexts increase Key-Value (KV) cache memory usage on serving GPUs, which reduces maximum system concurrency and lowers token throughput per dollar.

Vendor engineering guidance is consistent. Doubling input tokens can require up to four times the processing work, because attention FLOPs scale with sequence length while other FLOPs stay constant, and long-context KV cache grows linearly with both context length and batch size. Published serving analysis reports that a 32k-context workload can collapse to roughly a dozen concurrently useful sequences and about 300 tokens per second of node throughput, a sixteenfold reduction in tokens per second per dollar versus short-context traffic.

Large batch sizes maximize GPU arithmetic utilization and improve system throughput, but they increase Time-to-First-Token. Decode throughput scales with effective batch size only until DRAM bandwidth saturates; beyond that point, additional batching buys diminishing throughput and materially worse latency. Real-time interactive applications require smaller batch sizes to hold latency down, accepting lower compute efficiency and higher unit output costs. Conversely, moving from batch size 8 to batch size 256 on identical hardware can change cost per million tokens by 10 to 30 times. That is usually the highest-return optimisation available to a team before any model change is considered.

Infrastructure Pricing, GPU Hosting, Quantization, and Deployment Types

Hourly TCO of a dedicated instance:

Cost per million tokens (CPM) on owned or rented hardware:

Capacity sizing: minimum model instances = planned peak requests per second ÷ optimally achievable requests per second per instance, where "optimal" means the highest-throughput configuration that still satisfies the latency ceiling. Where instances use different GPU counts, normalise to requests per second per GPU before comparing. Then multiply by a reserve factor for failover, and route the reserved-but-idle share into $C_{\text{idle}}$ in the attempt-cost formula. Hourly rates alone reveal nothing: an accelerator at $2.90 per hour that delivers double the throughput of a $1.64 part is the cheaper machine per usable output.

Precision and quantization economics. Numerical precision is the most underused lever in enterprise unit economics. FP16 and BF16 require roughly 2 bytes per parameter, so a 70B model needs about 140 GB of VRAM for weights alone, before KV cache. Reducing precision reduces both the GPU count and the memory pressure that throttles concurrency. It also degrades usable rate, and the degradation is task-dependent.

FP8 is a Hopper-class-and-newer hardware feature. On Ampere the equivalent path is INT8 via TensorRT-LLM, which yields a smaller gain. INT4 delivers the lowest nominal CPM and is generally acceptable for conversational and summarisation traffic. For code generation, mathematical reasoning, and precise factual recall, though, that 2 to 5% accuracy loss can push cost per usable output above the FP8 configuration once retries are priced. Never adopt a quantization tier without re-running the usable-rate harness on that exact build.

Summary Results Table Across Models, Quality, and Costs

The table below outlines benchmark performance across standard enterprise workloads, highlighting the gap between published API list prices and actual cost per usable output.

Footnotes: (a) the long-horizon agentic figure assumes 50 or more tool-iteration sub-loops per task, including sandbox execution and retrieval calls; it is a per-task total, not a per-message cost. (b) Latency figures are P50 and P99 measured at declared concurrency; self-hosted numbers assume dedicated hardware with vLLM continuous batching and are not directly interchangeable with API-hosted measurements. (c) All figures are engineering approximations dated to the run window, not vendor-guaranteed peak specifications.

Read the last row twice. A self-hosted open-weight build at 79.4% quality lands near the cheapest cost per usable output in the table, which is exactly why procurement should hold it as a reference price.

Pareto Frontier: When Higher Quality Justifies Higher Unit Costs

The Pareto frontier is the set of non-dominated models where no alternative provides higher accuracy without increasing overall cost. A model sitting on this frontier defines optimal trade-off economics.

Accuracy (%)
  ^
100|                        * Frontier Model B (High Cost, High Acc)
   |           * Frontier Model A (Mid Cost, Mid Acc)
 80|
   |   * Low-Cost Tier (Budget)
 60|____________________________________________________>
     $0.01          $0.10          $1.00          $10.00   Cost per Usable Output ($)

Selecting a high-priced model on the frontier is mathematically justified when the cost of an undetected failure is extreme. In automated loan origination or regulatory filing extraction, paying $0.05 per call for 99% accuracy is far cheaper than paying $0.005 for 85% accuracy when human audit review costs $25.00 per corrected file. The general rule: a premium model earns its price when one correct pass replaces several cheaper attempts, when reruns are operationally prohibited, or when latency budgets forbid sequential retries.

Note that the cost axis is not standardised across the literature. Published frontiers use dollars, latency-adjusted utility, tokens per task, or FLOPs per query, and the frontier's shape changes with the axis. Declare your axis explicitly in the benchmark report. Frontier-selection research also reports that accounting for multi-run selection and routing can reduce error by up to 82% and match state-of-the-art accuracy at up to 85% lower cost. In other words, routing architecture, not model choice alone, determines where you sit on the curve.

Code, Math, and Analytic Reasoning Workloads

Code generation and mathematical reasoning require deterministic correctness. Automated test suites and compiler verification serve as objective success filters, which makes retry loops unusually effective here.

In an automated software refactoring pipeline, an engineering team deployed a lightweight model at $0.20 per attempt. The model achieved a 30% unit test pass rate, requiring an average of 3.33 attempts per usable function, or $0.66 per success. Upgrading to a specialized reasoning model priced at $1.50 per attempt yielded a 92% pass rate, or $1.63 per success. Higher, on the surface. Once the developer review time saved by avoiding broken builds entered the model, total process cost dropped by 40%.

Evidence note: this refactoring case is an internal engineering observation without a published harness, sample size, or verification methodology. Treat the 40% figure as a directional, illustrative estimate pending publication of the test set. Independent research supports the mechanism:

Public competition-grade data shows how sharply price and accuracy diverge at the top of the curve. On uncontaminated 2025 and 2026 mathematics competitions, a maximum-effort frontier model reached 91.3% average accuracy at an average run cost index of 4.826, while a fast reasoning variant reached 90.6% at 0.185. That is a 26-fold cost gap for 0.7 points. On code tasks in the same benchmark, the frontier model scored 55.0% at 21.578 cost against 47.5% at 2.225 for the cheaper variant, meaning the marginal 7.5 points cost roughly ten times more. Whether that is rational depends entirely on the downstream cost of a defect.

Document Processing, Long Context, and Structured Outputs

Document analysis relies heavily on system prompt efficiency and prompt caching. Caching lets models store large context prefixes (compliance manuals, SEC filings, credit policy documents) and serve subsequent queries at a substantial discount on input token rates. Published rates differ by vendor: cached input is billed at 50% of the base input rate on one major platform ($1.25 versus $2.50 per 1M for a flagship model), while another charges cache reads at 10% of base input with cache writes at 125% for a five-minute TTL. Azure's implementation states that caching alters only latency and cost, not output content, and that structured-output schemas are appended as a system-message prefix, which makes them cacheable.

For structured data extraction, schema compliance is non-negotiable. Using strict JSON mode guarantees structural validity, eliminating syntax parsing errors and lowering overall cost per usable output. Score structured tasks on four axes rather than one: parse success, schema compliance, field and type correctness, and content fidelity against the source document.

Metered billing units also deserve scrutiny before any contract is signed. Teams negotiating credit-denominated enterprise AI agreements can study how consumption, resets, and overage interact in the ai video pricing and credits breakdown, and in the wider comparison of AI video generators by pricing and credits. The accounting lesson transfers cleanly: any metered unit that is not the unit of business value will misprice the workload.

Customer-Facing Chat and Agentic Workflows

Customer-facing chat prioritizes real-time responsiveness, which means low Time-to-First-Token and tight latency bounds. Usable outputs in this domain are measured by resolution rates at defined Customer Satisfaction (CSAT) thresholds. Targets should be numeric and percentile-based (p50, p90, p99), not average-based, and set per phase so that phase budgets sum to the end-to-end objective.

Multi-step agentic workflows execute external tool calls and API requests. The cost of an agentic usable output must sum token costs, API subscription fees, and sandbox execution runtimes across every sub-task phase, including the phases that fail and are retried. Report cost as USD per completed workflow, not USD per model call, and publish mean workflow latency alongside it.

One governance boundary is worth stating plainly. An agent in a regulated workflow is a digital worker: it needs a named owner, an approved role, access limits, an escalation path, an audit trail, and a shutdown mechanism. No evidence, no autonomy.

Decision Matrix: Workload Mapping to Model and Reasoning Tiers

An enterprise decision matrix routes incoming production requests to the lowest-cost model tier capable of satisfying the task's accuracy bar.

Tier 1 (budget tier)form-filling, basic text formatting, routing, simple classification. Priority: latency and low token cost.

Tier 2 (standard tier)summarization, standard document drafting, conversational search. Priority: balanced throughput and quality.

Tier 3 (frontier reasoning tier)contract review, regulatory compliance, complex code generation, trade reconciliation. Priority: absolute accuracy and no unsupported assertions.

The three tiers describe capability bands. The matrix below assigns them, plus an explicit effort setting, to concrete enterprise workflows. Treat it as a starting policy and re-derive it against your own quality bar.

Open-weight economics deserve a standing line in the matrix. Independent testing places a leading open-weight model at high reasoning within 4 to 7 quality points of a closed frontier model at medium effort across a mixed suite, at roughly one twelfth of the cost. Where a 4 to 7 point gap is tolerable, self-hosted high-effort open weights set the procurement floor against which every vendor quote should be benchmarked.

Continuous Benchmarking and Adapting to Price Changes

Foundation model providers frequently drop API prices, update model weights, and adjust rate limits. A static cost per usable output benchmark becomes obsolete within months. Sometimes within weeks.

Continuous benchmarking pipelines run automated evaluation suites against updated model endpoints on a fixed cadence, and on every trigger event: new model revision, published price change, deprecation notice, or API contract change. Because benchmark cost is simply token counts multiplied by current provider prices, a vendor price cut changes your ranking without any change in model behaviour. It must be recomputed, not assumed. When a provider reduces input token costs or releases a lighter model variant, the pipeline should update internal routing thresholds so the saving lands immediately. Efficiency research also shows that well-designed benchmark subsetting can cut evaluation cost substantially with minimal reliability loss, which makes a weekly cadence affordable for material workloads.

For teams building the budget model that sits on top of this pipeline, ground the projection in metered-unit reality rather than list prices. The Google Veo API cost and limits documentation is a compact worked example of how quota tiers, per-request pricing, and rate limits combine into a forecastable monthly figure.

Enterprise benchmark execution checklist

Define usability rubricsdocument explicit pass and fail criteria, schema definitions, latency ceilings, and the cost of a false positive and a false negative for every target workload.

Assemble baseline test setscurate a representative set covering standard queries, complex edge cases, adversarial inputs, and compliance boundaries. Size it from the power formula in §[5], typically 200 to 500 prompts for tier-level comparisons, and 1,000 or more where you must separate two near-identical frontier candidates.

Deploy a pinned harnessbuild an automated runner that pins model revision hashes, harness commit SHA, decoding parameters, seeds, and system prompts.

Log token granularitycapture raw input, output, cached context, and hidden reasoning tokens alongside external API charges, sandbox runtimes, and reviewer minutes.

Execute validation rulesrun deterministic syntax validators, unit tests, and calibrated LLM judges to calculate usable rates with confidence intervals; report pass@1 and pass@k.

Price the human layerattach fully loaded review, correction, and escalation cost to each attempt class, so that $C_{\text{human}}$ appears in the unit economics rather than in an unallocated operations budget.

Construct Pareto matricesmap total execution costs against usable rates, declaring the cost axis, to identify non-dominated candidates.

Assess plausibility before actingsanity-check results against an external reference (HELM, Artificial Analysis, LMSYS) and investigate any outlier before it becomes decision-grade.

Implement dynamic routingupdate API gateways to route production workloads according to the verified Pareto decision matrix.

Report, register, repeatfile results with the second line, update the model inventory entry and decision matrix, and schedule the next cycle.

Required Components of Benchmark Reports and Data Tables

Complete evaluation reports must include four core operational artifacts:

Data tables must be reconstructible from code and data alone. Preserve raw and processed layers, match the published row order, column order, and rounding, and ship a codebook plus README so a reviewer can regenerate every reported number without contacting the author.

System configuration logscomplete records of model revision hashes, hardware configurations, precision and quantization build, temperature settings, seeds, and API software versions.

Raw generation dataunedited JSON response payloads containing all output tokens, reasoning steps, and HTTP timing metrics, plus the processed data files behind every published chart.

Validation rulesetsopen-source harness code, JSON schemas, automated unit tests, and LLM-as-a-judge prompt templates with their calibration set.

Itemized cost schedulesgranular breakdowns detailing input, output, cached, reasoning, tool execution, idle capacity, and human review expenditure per task attempt.

Identifying Flawed and Incomparable Benchmarks

Model risk auditors should look first for the methodological flaws that invalidate benchmark comparisons outright:

Incomparable benchmarks often evaluate one model with zero-shot prompts while granting a competing model multi-shot examples or external web search tools. Always verify that run conditions remain strictly identical across all tested candidates. The systems-performance literature names the recurring offences directly: selective benchmarking, improper baseline, misleading presentation, and missing platform specification. Public-sector benchmarking guidance adds that data should be validated and re-based to a common basis before any comparative figure is produced. Contamination control and construct validity are the two additional gates specific to LLM evaluation. A benchmark score is evidence only if the test data was not in training, and only if the task actually measures the capability you intend to deploy.

Regulatory Alignment: SR 11-7, OCC 2011-12, and NIST AI 800-2

For regulated financial institutions, a cost per usable output benchmark is not merely an engineering artefact. It is model validation evidence, and it should be structured to survive supervisory review.

• Conceptual soundness (SR 11-7 and OCC 2011-12, Section IV): document why the usable-output definition is the correct measurement of the model's intended business use, and why the acceptance thresholds correspond to the institution's stated risk appetite.
• Ongoing monitoring: the continuous benchmarking pipeline in §[21] is the monitoring control. Define thresholds at which a usable-rate decline triggers escalation, and specify who owns the decision to suspend automated routing.
• Outcomes analysis and benchmarking: cost per usable output, usable rate with confidence intervals, and pass@1 versus pass@k are outcomes-analysis artefacts. Retain them per model version in the model inventory.
• Effective challenge and independence: the second line must be able to re-run the harness independently from published code, seeds, and datasets. If it cannot, the benchmark is management assertion, not validation.
• Third-party and vendor models: where the model is a vendor API, the institution remains accountable for validation despite limited access to internals. The benchmark harness is often the only available conceptual-soundness evidence, which raises rather than lowers its documentation bar.
• Automated evaluation practice (NIST AI 800-2 draft, 2026): record benchmark size, model version, API and parameters, repeat count, and experiment date for every reported figure.
• Consumer-impact controls: where outputs influence credit, pricing, or account decisions, add adverse-action explainability and fair-lending testing to the usable-output definition. An output that is accurate but unexplainable is not usable in that context.
• Escalation: define, in advance, the residual-risk level at which a workload must revert from autonomous to human-in-the-loop, and register the trigger in the AI inventory or GRC platform alongside the model's owner and validation date.

E-E-A-T: primary sources and research repositories

HELM (Holistic Evaluation of Language Models)

Stanford CRFM benchmark repository providing standardized multi-metric model evaluations across core scenarios and prominent models (HELM framework).

LMSYS Chatbot Arena

crowdsourced human preference evaluation platform tracking real-world model utility, with a public leaderboard and published methodology.

Artificial Analysis

independent benchmark provider measuring LLM latency, throughput, token pricing, blended price, and cost-per-task metrics using fixed workloads, repeated tests, and P50 measurement windows (Artificial Analysis data).

NIST AI 800-2 (draft, 2026)

Practices for Automated Benchmark Evaluations of Language Models, documentation and reproducibility requirements.

IETF draft (2026)

Benchmarking Methodology for Large Language Model Serving, standardised serving and cost, latency, and throughput measurement.

Federal Reserve SR 11-7 and OCC Bulletin 2011-12

Supervisory Guidance on Model Risk Management, covering conceptual soundness, ongoing monitoring, outcomes analysis, and effective challenge.