CODE IN TOKYO

What It Actually Costs to Run an LLM Feature in Production

Before an LLM-backed feature goes to production, someone always asks the same question. What will this cost per month?

Token prices are published, so the arithmetic is easy. The number you get from that arithmetic rarely matches the invoice.

Sometimes it is off by close to a factor of two. Here is what the arithmetic leaves out.

Start with the naive calculation

Take an internal support chat as the example.

One thousand requests per day. Per request, 8,000 input tokens and 500 output tokens.

Input dominates because RAG stuffs retrieved internal documents into the context. For most LLM features, input is where the volume is.

Claude Sonnet 5 lists at 3 USD per million input tokens and 15 USD per million output tokens.

About 32 USD per day. Roughly 950 USD per month.

Anyone can produce that figure. The problem is everything after it.

Prompt caching brings the input side down

Say 6,000 of those 8,000 input tokens are identical on every request — the system prompt and a fixed block of reference material.

That portion can be cached. Reading from cache costs roughly one tenth of the normal input rate.

So 6M tokens per day cost a few dollars instead of 24. The input side drops by more than half.

Writing to the cache costs more than a normal request, though: 1.25× for a five-minute lifetime, 2× for an hour. At five minutes, the second request breaks even. At an hour, you need three.

For a feature with sporadic traffic, the cache keeps expiring and you pay the write premium without ever collecting the read discount. This has to be measured, not assumed.

Then the costs that are not on the price sheet

Three things are missing from the naive calculation.

The first is retries. Timeouts, rate limits, and output that fails to parse as JSON. All of them mean running the request again. Budget a few percent.

The second is verification calls. Few applications can return the model’s answer as-is. Checking whether the response asserts something absent from the context usually means a second model call.

Adding that verification step doubles your call volume outright.

The third is evaluation. Every prompt change means re-running the full test set. This continues for as long as development does.

Evaluation does not need a live response, so it can run through the Batch API at half the token price. Always use it there.

Adding it up

Back to the example.

With caching applied, roughly 15 USD per day. The verification call multiplies that by about 1.7. Retries add a few percent.

Around 800 USD per month.

That lands close to the naive figure, but the composition is different. What caching saved, verification and retries consumed.

Without caching implemented at all, the same feature runs well past 1,300 USD.

The largest line item is not inference

This is the actual point.

Against 800 USD per month of inference, ask how many hours go into keeping the feature alive.

At twenty hours a month, back out an engineer’s rate and the labor cost exceeds the inference cost.

The cost of an LLM feature is, in practice, a labor cost. Time spent arguing about token prices will not change the order of magnitude.

What needs cutting is the design that requires human attention to operate.

How to build the estimate

So the estimate gets built this way.

Measure the tokens per request. Do not estimate them — run ten representative inputs through and count.

Project the request volume, separating peak from steady state.

Apply the multipliers: verification calls, retry rate, evaluation frequency.

Finally, estimate the operational hours. An estimate without that line is not a meaningful number.

Prices move

One last thing.

Model pricing changes. New models push older ones down, and introductory rates appear and expire.

So do not bake prices into the estimate as constants. Keep the price as configuration and the token counts as the asset.

With measured token counts in hand, a price change is a one-minute recalculation.