What an LLM Feature Really Costs

TL;DR - The per-token price is not the cost of your feature. The cost is per-token price run through your actual usage:

• The formula: monthly active users, times sessions each, times tokens per session (input and output priced separately), adjusted for retries and cache hits, times an agent multiplier (1x for a simple call, up to 10x for an agent that loops).
• What drives it: the input side is usually bigger than founders expect, because retrieval stuffs the prompt. Output is priced 5x to 6x higher per token, so long answers hurt.
• The levers: prompt caching cuts cached input by about 90%, batch cuts 50%, and using a cheaper model for the easy calls often beats every other trick.
• Self-hosting almost never wins until you are past roughly 100 million tokens a day. Below that, the API is cheaper once you count the engineers.

The number is knowable before you build. Estimate it, do not discover it on the invoice.

The sticker price is a trap

You are about to add an AI feature. You open the pricing page, see a few dollars per million tokens, multiply by “some tokens,” and conclude it is basically free. Then the first real invoice arrives and it is a line item you have to explain.

The gap is that the per-token price tells you almost nothing on its own. A token is roughly four characters, about three-quarters of a word. The cost of your feature is that price run through how many people use it, how often, how much text goes in and comes back, how often it retries, and whether the thing loops. Here is what the headline prices actually look like in mid-2026, per million tokens:

(Prices move. These are the list prices as of 7 July 2026, verified against each provider; check the provider before you commit a number to a spreadsheet. Claude Sonnet 5 is running an introductory $2 / $10 through the end of August 2026. The cost calculator carries the same figures and is kept current.) The first thing to notice: output is the expensive side. It costs five to six times more than input on the frontier models, and still two to four times more on the cheaper open-weight ones. A feature that reads a little and writes a lot is priced very differently from one that reads a lot and answers briefly.

The formula

The real cost of an LLM feature is this, and it fits on one line:

users x sessions x tokens-per-session x (retries) / (cache hits) x (agent multiplier)

In code, so it is unambiguous:

long inputTokens  = mau * sessionsPerUser * inputTokensPerSession;
long outputTokens = mau * sessionsPerUser * outputTokensPerSession;

double monthlyCost =
      (inputTokens  / 1_000_000.0) * inputPricePerMillion
    + (outputTokens / 1_000_000.0) * outputPricePerMillion;

monthlyCost *= (1 + retryRate);      // you pay for failed and retried calls too
monthlyCost *= (1 - cacheHitRate);   // cached input is billed at roughly 10%
monthlyCost *= agentMultiplier;      // 1x a single call, up to ~10x an agent loop

The last three lines are where estimates go wrong. Founders model the happy path, one clean call per action, and forget that things retry, that caching only helps if you designed for it, and that an “agent” is not one call, it is a loop that can make ten.

A worked example

Take an internal copilot, the kind of feature a vertical SaaS bolts onto its product. Say 3,000 monthly active users, each active about 18 days a month, four sessions a day. That is roughly 216,000 sessions a month. Each session sends 2,500 input tokens (the user’s question plus retrieved context) and gets back 400. Run it on Sonnet 5 at $3 in, $15 out:

• Input: 216,000 x 2,500 = 540M tokens x $3 = $1,620
• Output: 216,000 x 400 = 86.4M tokens x $15 = $1,296
• Add 8% for retries: +$233
• Subtract a 35% cache hit rate on input: -$1,021
• Multiply by a 1.5x agent factor: the whole thing lands near $3,200 a month.

(Illustrative, built on the list prices above and stated assumptions, not a real client bill.) The point is not the exact figure. It is that “a few dollars per million tokens” became a few thousand dollars a month, and every input to that was knowable in advance. Change the model to Haiku and it roughly thirds. Let the answer run to 1,500 tokens instead of 400 and the output line triples.

What actually drives the bill

• Retrieved context is the silent cost. The moment you add retrieval over your domain data, every request carries the retrieved chunks as input tokens. That is usually where the input number balloons, not the user’s actual question. Retrieve less, and retrieve more precisely.
• Output length. It is the expensive side. Capping response length and asking for structured, terse output is a direct cost lever, not just a UX choice.
• Retries and failures. You pay for calls that error out or get retried. Timeouts and aggressive retry policies quietly multiply spend.
• Agent loops. A single completion is one unit. An agent that reasons, calls a tool, reads the result, and calls again can be five or ten units for one user action. This is the multiplier that turns a cheap feature expensive.

The levers that actually cut it

• Prompt caching. If a large, stable chunk of your prompt repeats (a system prompt, a fixed knowledge base), providers bill cached input at roughly 10% of the normal rate, about a 90% cut on that portion. This is the highest-leverage optimization for most RAG features.
• Batch processing. For anything that does not need to be real time, batch APIs run about 50% cheaper. Stack it with caching and repeated work gets very cheap.
• Model tiering. The biggest lever is often the dumbest: do not send easy calls to your most expensive model. Route the simple classification or extraction to a Haiku or a nano tier, reserve the flagship for the calls that need it. Halving the model price beats most clever engineering.
• Trim context and cap output. The two levers you fully control on every request. Both directly reduce the token counts in the formula.

Cost and latency trade off

Cost is one axis; the other is speed, and they pull against each other. A bigger model reasons better and costs more and is usually slower. What users feel is time to first token: under a second reads as instant, past two seconds they notice. Streaming does not make the full answer finish sooner, but it puts the first words on screen fast, so the feature feels responsive even on a slow model. The cheap, fast tiers (Haiku, the nano models) sit under about 600ms to first token, which is why routing easy calls to them helps the bill and the experience at once.

When to self-host

Founders reliably overestimate the savings. Running an open model on your own GPUs looks cheaper per token in a spreadsheet, but the real cost is three to five times the raw GPU rental once you count idle time, DevOps, and the engineer-hours to keep it healthy (call it 10 to 20 hours a month of senior time). The break-even against a frontier API sits somewhere north of 100 million tokens a day. Below that, and that is almost everyone, the API is cheaper, and it is not close once the engineers are on the ledger. Self-host when data residency or volume forces it, not to save money at small scale.

The honest takeaway

An LLM feature’s cost is not a mystery you discover on the invoice. It is a calculation you can run before you write a line: your users, your sessions, your token sizes, your model. Plug your own numbers into the LLM cost calculator and you will have a defensible monthly figure in a minute, which is exactly what you want before you commit to building. And cost is only one input to the build decision, alongside latency, reliability, and whether the feature needs AI at all.

Summary

The per-token price is the sticker, not the cost. The cost is that price run through real usage: users, sessions, input and output tokens, retries, cache hits, and how many times an agent loops. Model it before you build, watch the output side and the retrieved context because that is where the money goes, pull the levers (caching, batch, cheaper models for easy calls), and skip self-hosting until your volume genuinely forces it. The number is knowable. Know it first.

Weighing an AI feature and want a straight answer on what it will cost and whether it is worth building? That is the first conversation in AI Enablement. Book a free 30-minute call and we will size it against your real numbers.