What a prompt change actually costs

You shorten a prompt. Output tokens drop by a third. You read a dozen responses, they look fine, and you ship it.

That is the normal process, and it is not a measurement — it is a vibe with a cost saving attached. The question underneath it is perfectly precise and almost nobody answers it with evidence: did that prompt change save money without losing quality?

prompteval exists to answer exactly that. Most of the difficulty turned out to be statistical rather than architectural, and the interesting decisions are the ones about what counts as knowing.

Cost as an axis, not a chart#

Existing eval tooling — Braintrust, Langfuse, promptfoo, Phoenix — treats cost as a metric you graph. You can see it going up or down over time.

That answers what happened. It does not answer should I ship this, because the thing you need is not a cost line and a quality line on separate axes. It is one comparison where cost and quality are the same verdict:

                          v1        v2        Δ        95% CI          p
────────────────────────────────────────────────────────────────────────────
mentions_required_terms   0.84      0.79      −0.05    [−0.13, +0.03]  0.21
professional_tone         0.81      0.83      +0.02    [−0.05, +0.09]  0.58
────────────────────────────────────────────────────────────────────────────
total cost                $0.158    $0.099    −37%     [−42%, −32%]    <0.001
avg latency               1.4s      1.1s      −21%     [−28%, −14%]    <0.001
 
Quality verdict:  no significant regression
Cost verdict:     significant 37% reduction
Recommendation:   ship v2 — cost savings real, quality holds

The shape of that output is the whole design. Cost is a comparison axis sitting alongside quality, so the recommendation can be a single sentence rather than two dashboards and a judgement call.

Pairing, or you are measuring your sample#

The first thing that has to be right is which numbers get subtracted from which.

Run v1 on a hundred examples, run v2 on a hundred examples, compare the averages — and a chunk of the difference is the examples, not the prompts. Some inputs are simply longer, harder, or more expensive than others.

So comparison is paired: the same examples go through both prompts, matched on example.id, and only ids present in both runs are used. The delta is mean(score_b − score_a) across matched pairs, not the difference of two independent means. Every example is its own control.

Two consequences fall out of that:

• An example that errored in either run is excluded from both. Keeping it on the side where it succeeded would quietly bias the comparison toward the prompt that failed.
• Fewer than two paired examples raises, rather than returning a number. Statistics on one pair are undefined, and a tool that returns a confident figure there has produced a lie with a decimal point in it.

The confidence interval is the decision#

Every delta carries a 95% bootstrap confidence interval and a paired t-test p-value. The design note in the source is explicit about which one matters:

That ordering is deliberate. A p-value tells you whether a difference is distinguishable from zero. A confidence interval tells you how large the difference plausibly is, which is the thing a shipping decision actually turns on.

[−0.13, +0.03] on a quality scorer says the regression could be as bad as 13 points or could be an improvement — you do not know, and the honest move is more data, not a shrug. [−42%, −32%] on cost says the saving is real and sizeable at either end of the range. Same statistical machinery, completely different decisions, and neither is visible from a p-value alone.

The CIs use the percentile bootstrap with 10,000 resamples. Bootstrap rather than a parametric interval because it does not assume normality and works on any sample shape — including the 0/1 output of a deterministic scorer, which is about as non-normal as data gets. The RNG is seeded by default, so the same data produces the same interval on every run. An eval that returns slightly different bounds each time invites people to re-run until they like the answer.

Percent-change intervals need a different bootstrap#

This is the subtle one, and it is the reason there is a hand-written bootstrap loop next to the scipy calls.

A per-scorer delta is a mean, so scipy.stats.bootstrap over a one-dimensional array of paired differences is correct.

A cost delta is not a mean. It is a ratio of two sums: (total_b − total_a) / total_a. Resampling a one-dimensional array of per-example percent changes and averaging them answers a different question — the average example's percent change, which is not the percent change of the total, and diverges badly when examples differ in cost.

So percent-change CIs resample paired indices and re-sum both totals from the resampled set, preserving the pairing and rebuilding the ratio from scratch each time.

Getting that wrong would not throw. It would produce a plausible interval around the wrong quantity, which is the worst failure mode a statistics routine has.

Two ways to count tokens wrong#

The cost model is a pure function from a model name plus a usage record to a breakdown, and two details in it are the ones people get wrong:

Cached input is charged at the cached rate and the remainder at the input rate — not the whole prompt at the input rate plus the cached portion again. Reasoning tokens are surfaced as their own counter purely for transparency; they are already inside completion_tokens and billed at the output rate, so adding them separately double-counts.

Every component is reported separately, because "v2 is cheaper" is much less useful than knowing whether it got cheaper on input, output or cache hits — they have very different implications for what to change next.

A gate that fires on noise gets switched off#

prompteval runs in CI with a --fail-on spec — clauses like cost+10% or quality-5% that fail the build when crossed.

The rule that makes it usable: only statistically significant breaches count.

A noisy 12% cost wobble with a p-value of 0.3 does not trip cost+10%. That restraint is the gate's entire job — green when the signal is real, red when it isn't. A quality gate that goes red on sampling noise gets marked continue-on-error within two weeks, and then it is not a gate at all. The statistics are not there to be rigorous for its own sake; they are there so the gate stays worth obeying.

What it does not do yet#

The repo is early alpha, pre-v0.1, and says so. Three limitations are worth naming because they change how you would read a report:

• LLM-judge calls cost real money and are not in the cost comparison. They run inside scorer functions, separately from the prompt calls being compared. Treat them as eval-time overhead, not part of the v1-versus-v2 number. Separating "prompt cost" from "judge overhead" is planned.
• The p-value uses a paired t-test even for binary scorers, where Wilcoxon signed-rank would be more correct. It is deliberately deferred — the CI is the decision-driving signal, so the approximation sits on the secondary one.
• Costs are floats. Fine for cents across a golden set; wrong for exact-cent aggregation across millions of calls, where Decimal belongs.

The general advice the code gives is also the cheapest: prefer a deterministic scorer whenever you can write one. Exact match, a regex, a JSON-schema check — cheaper, faster, and free of judge noise. Reach for a model to grade a model only when quality genuinely cannot be checked any other way.

The reason any of this matters is that the alternative is not neutral. Shipping the shorter prompt because a dozen samples looked fine is a decision made on evidence too weak to detect the regression you are worried about — and the saving is easy to see while the quality loss shows up as support tickets a month later.

prompteval is open source and still early: github.com/mahAnuj/prompteval.