What AI Costs to Run: Token Economics for a Product That Ships
The feature works, usage is growing, and the inference bill is growing faster. This is the ordinary trajectory, and it happens because almost every cost decision in an AI product is made implicitly, by someone writing a prompt, long before anyone looks at a price.
Start with the asymmetry that governs everything: output tokens cost roughly three to five times what input tokens cost. That single ratio means response length is a bigger cost lever than prompt length, and most optimisation effort goes into the prompt.
The second structural fact is that you are billed for the whole context, every call. A conversation on its twentieth turn re-sends nineteen turns of history. Cost per conversation grows quadratically with length while the user experiences it as linear, which is why long sessions are disproportionately expensive and nobody notices until they are.
Four levers do nearly all the work. Caching repeated context, routing easy requests to cheaper models, moving asynchronous work to batch pricing, and simply making responses shorter. Published claims for each run from 40% to 85% savings, and those numbers come from vendor blogs rather than measurements, so treat them as directions rather than targets.
What you should not do is start optimising. Start measuring, because the usual outcome of instrumenting an AI product properly is the discovery that one feature nobody prioritised accounts for most of the bill.
Three questions decide where your money goes. What fraction of your requests genuinely need your best model? How much of your context is identical between calls? And does anything in your product actually need an answer within seconds?
Instrument before optimising. Log input tokens, output tokens, cached tokens, the model, the route, retry count, latency, and the feature and user each call belongs to. That turns a monthly invoice into an itemised bill, and it usually shows one feature taking most of the spend while serving a fraction of users.
The first move is instrumentation, and it is cheap relative to everything it tells you.
Log, for every call: input tokens, output tokens, cached tokens, which model, which route, retry count, latency, and the feature and user it belongs to. That is a handful of fields and it converts a monthly invoice into an itemised bill.
Attribute to features, not just to the application. The invoice says you spent a sum on a provider. What you need to know is that the document-summarisation feature is 70% of it while serving 4% of your users, which is a product decision rather than an engineering one.
Attribute to users as well. AI costs are long-tailed in the same way agent costs are: a small number of heavy users generate a disproportionate share. If your pricing is flat and your costs are not, that gap is your margin, and it is better discovered in a dashboard than in a board meeting.
Count retries separately. A failed call that is retried three times costs four times as much and shows up as one request in your own metrics. Retry storms during a provider incident are a common source of surprise bills.
Track cost per successful outcome, not cost per call. A cheap model that needs three attempts is not cheap, and only outcome-level accounting makes that visible.
Keep a simple unit alongside the raw numbers. Cost per conversation, per document, per generated report, whatever your product's natural unit is. Token counts are hard to reason about and a cost per unit of work is something a product manager can hold in their head and notice changing.
Do this before any optimisation. Teams that optimise first routinely spend a sprint improving something that was 6% of the bill.
Prompt caching lets a provider reuse the computation for a context prefix it has already processed, at a substantial discount. It is widely ranked the highest-value change available, with published claims of 60% to 80% reductions on chat and agent workloads.
It works when the start of your prompt is stable. A long system prompt, a fixed set of tool definitions, a document being discussed across several turns. All of that is identical call after call and is exactly what caching is for.
Which means prompt structure is now a cost decision. Put everything stable at the front and everything variable at the end. A prompt that interleaves a user's name into the middle of a long system message has broken the cache for the entire prefix, and that is a formatting choice costing real money.
Cache lifetime is short and usage-dependent, so the benefit concentrates in active sessions and bursts rather than in sparse traffic. A product with steady conversation traffic gains far more than one with occasional single-shot calls.
Semantic caching is a different thing and a narrower win. Serving a stored answer for a semantically similar question helps when queries genuinely repeat, such as a support assistant fielding the same twenty questions. For products where every request is unique it does not justify the infrastructure, and published guidance says so plainly.
Caching changes the economics of long context rather than removing them. A large stable document in the prefix becomes affordable to keep in every call, which is often a better design than retrieving fragments of it. The cheapest retrieval is sometimes no retrieval, and caching is what makes that true.
Measure the cache hit rate as a first-class metric. It is the number that tells you whether a prompt change helped or quietly destroyed your largest saving.
The common pattern in production systems is a frontier model handling every request, when published analysis suggests something like 70% to 80% of requests are routine. That figure is a vendor estimate rather than a measurement, but the shape is right for most products and you can verify it against your own traffic in an afternoon.
Classify first, then route. A small, cheap model decides whether a request is simple or hard, and only the hard ones reach the expensive model. The classifier costs a fraction of what it saves.
Escalate rather than guess. Where a cheap model's answer can be checked, try cheap first and escalate on failure. This is strictly better than routing on a prediction when verification is itself cheap.
Different tasks, different models, deliberately. Classification, extraction and formatting rarely need a frontier model. Reasoning, long-context synthesis and code often do. A product with one model for everything is almost always overpaying on one side and underperforming on the other.
Build the routing layer so models can be swapped without a deploy. Model pricing and capability both move on the provider's schedule, and a system where changing a model means a release will not keep up.
Provider diversity is a cost lever and a resilience one. Routing across more than one provider lets you move volume when pricing changes and keeps you serving when one has an incident. The cost is an abstraction layer and the discipline not to depend on any single provider's particular behaviour, which is worth paying on a product that cannot afford a day offline.
Quality-check the cheap path continuously. The risk of routing is silently serving worse answers to the majority of your users, and the only defence is sampling the cheap path's output and grading it on a schedule.
Providers offer substantial discounts for asynchronous work. OpenAI's batch tier is published at 50% off with completion within 24 hours, and the pattern is common across providers.
The constraint is latency tolerance, nothing else. The same model, the same quality, half the price, on a delay. Any workload where a person is not waiting qualifies.
More of your product qualifies than you think. Overnight enrichment, bulk classification, generating embeddings for a corpus, periodic summarisation, evaluation runs, backfills after a prompt change. Published guidance suggests a meaningful share of total token volume can shift to batch in a queue-based architecture, which is an estimate rather than a measurement but a plausible one.
Designing for it is a queue, not a rewrite. Separating the request from the response is ordinary engineering, and the benefit is permanent.
Your evaluation runs are the easiest candidate and are frequently the largest non-production cost in a mature AI product. Nobody is waiting for them.
Watch the completion window. A 24-hour guarantee means plan for 24 hours, not for the ten minutes it usually takes. Products that quietly depend on batch finishing quickly break on the day it does not.
Given output costs three to five times input, the cheapest available change is to stop generating text nobody reads.
Ask for what you need. A model asked an open question produces an essay. The same model asked for three bullet points produces three bullet points, at a fraction of the cost and often with better usability.
Structured output is cheaper than prose. JSON with four fields costs less than a paragraph conveying the same four facts, and it is easier to use downstream.
Set maximum token limits deliberately per call type, rather than leaving a provider default that permits a very long response nobody will ever read.
Watch for reasoning tokens. Models that think before answering bill for that thinking, and the cost can exceed the visible answer. Reasoning is worth paying for on genuinely hard tasks and is waste on classification, which makes it another routing decision.
Streaming does not reduce cost at all, only perceived latency. It is worth doing and it is not a saving, and conflating the two is common.
Summarise rather than truncate where the history matters. Replacing fifteen old turns with a short summary keeps the thread coherent at a fraction of the token cost, and it is one of the few changes that improves quality and cost together.
Trim conversation history. Since you pay for the whole context each turn, summarising or truncating old turns directly reduces cost on long sessions, which are the expensive ones.
The engineering levers matter less than one commercial decision: whether your price survives your heaviest users.
Flat pricing against metered cost is a bet on the average. It works while usage is evenly distributed and fails the moment it is not, and AI usage is never evenly distributed. The top few per cent of users routinely consume a multiple of the median, and under a flat plan they consume your margin.
Credits are the common answer and they change the product. Exposing consumption to users makes them ration, which protects margin and suppresses the engagement you wanted. It is a real trade rather than a neutral mechanism, and teams underestimate how much it changes behaviour.
Seat pricing with fair-use limits is the pragmatic middle for business products: predictable to buy, with a ceiling that only the genuine outliers notice.
Whatever you choose, instrument the worst case before launch. Model the margin for a user at the 99th percentile of usage, not the median. If that user is unprofitable, you have a pricing problem that no amount of caching fixes.
Know your cost per user per month as a tracked number. It is the figure that tells you whether an improvement in the product is also an improvement in the business, and most teams can quote their infrastructure spend and not this.
Revisit after every model change. Provider pricing moves, usually downward, and a plan priced against last year's rates may be leaving margin on the table or protecting against a risk that no longer exists.
Most cost surprises are not gradual growth. They are a defect that nobody noticed.
Retry loops during a provider incident. A degraded endpoint returns errors, your client retries, and the retries are billed. Cap retries, back off exponentially, and alert on retry rate rather than only on error rate.
A prompt change that triples context. Someone adds examples to improve quality, the prompt grows, and every call now costs more. Prompt size belongs in code review, with the token count visible in the diff.
Runaway agent loops. An agent that re-plans indefinitely burns tokens at machine speed. A hard step cap is the cheapest control in this entire post.
Abuse and scraping. An exposed AI endpoint without rate limiting is an expense anyone can increase. Rate-limit per user and per key from the first day it is public.
Forgotten background jobs. An enrichment process running over a growing table, a re-embedding job triggered more often than intended, an evaluation suite on every commit. These scale silently with your data rather than with your usage.
Set a billing alert at a threshold you would be unhappy to reach, with a human owner. It is the least sophisticated control available and it catches the cases that none of the sophisticated ones do.
At some point running your own inference becomes cheaper per token than paying an API, and the crossover is further away than enthusiasm suggests.
The serving stack is mature. Continuous batching, standard in the common inference servers, is credited with two to three times the throughput of static batching on typical decoder workloads, and that efficiency is what makes self-hosting viable at all.
Utilisation decides everything. A GPU costs the same whether it is busy or idle, so self-hosting wins with steady high volume and loses badly with spiky or low traffic. An API charges nothing between requests; your hardware does not.
The hidden cost is operations. Model serving, scaling, upgrades, monitoring, quality regression testing and someone on call. That is a role, not a line item, and it belongs in the comparison.
You also give up model improvements. Frontier models improve continuously at no cost to API users. A self-hosted open model is the model you deployed until you deploy another one.
The hybrid is the usual answer again. Self-host the high-volume routine work where a smaller open model is sufficient, and call an API for the hard minority. That captures most of the saving without taking on frontier-model operations.
Be sceptical of all published savings figures here, including the ones in this post. The ranges circulating are vendor-authored and unverified, and the only number that matters is the one from your own traffic.
Week one is instrumentation only. Per-request logging of tokens in, tokens out, cached tokens, model, route, retries and latency, attributed by feature and user. No optimisation.
Then read the distribution. Which feature, which users, which call types. The answer is reliably surprising and it determines everything after.
Then the cheap structural wins in order: restructure prompts so stable content is at the front and caching works, cap output lengths, move anything asynchronous to batch pricing.
Then routing, which is more work and more benefit, with sampled quality grading on the cheap path from day one.
Self-hosting last, and only with a utilisation number in hand.
Our rate band is published: $40 to $100 an hour by role, blending to $60 to $70 for a mixed team. The pattern we see most often is a team about to self-host that has never measured its cache hit rate, which is several weeks of infrastructure work standing in for an afternoon of prompt restructuring.
Three reasons compound. Output tokens cost roughly three to five times input tokens, and richer features generate longer responses. You are billed for the entire context on every call, so a conversation on its twentieth turn re-sends nineteen turns of history, making cost grow quadratically with session length while users experience it as linear. And retries are billed but usually invisible in your own request metrics.
Per request: input tokens, output tokens, cached tokens, which model, which route, retry count, latency, and the feature and user it belongs to. Attribute to features rather than only to the application, since the common discovery is that one low-priority feature dominates the bill. Attribute to users too, because costs are long-tailed and a flat price against long-tailed costs is a margin problem. Track cost per successful outcome, not per call.
Prompt caching lets a provider reuse computation for a context prefix it has already processed, at a discount. Published claims put savings at 60% to 80% on chat and agent workloads, though these are vendor figures rather than verified benchmarks. It only works when the beginning of your prompt is stable, which makes prompt structure a cost decision: keep stable content at the front and variable content at the end, because interleaving a variable into a long system prompt breaks the cache for the whole prefix.
Almost certainly. Published analysis suggests 70% to 80% of requests in a typical production system are routine enough not to need a frontier model, which you can verify against your own traffic. Use a small cheap model to classify requests and route only the hard ones to the expensive model, or try cheap first and escalate when the answer fails a check. Sample and grade the cheap path continuously, since the risk is silently serving worse answers to most users.
Providers discount asynchronous work substantially; OpenAI's batch tier is published at 50% off with completion within 24 hours. The only constraint is latency tolerance, so anything where a person is not waiting qualifies: overnight enrichment, bulk classification, embedding a corpus, periodic summarisation, backfills after a prompt change, and evaluation runs, which are often the largest non-production cost in a mature AI product. Plan for the full 24-hour window rather than the time it usually takes.
At steady, high utilisation, and later than most teams expect. A GPU costs the same idle or busy, so self-hosting wins with consistent volume and loses with spiky traffic, while an API charges nothing between requests. Continuous batching in the common inference servers is credited with two to three times the throughput of static batching, which is what makes it viable at all. Include operations — serving, scaling, upgrades, monitoring, on-call — as a role rather than a line item, and remember that self-hosted models stop improving until you redeploy.
No. Streaming improves perceived latency by showing tokens as they are generated, but you are billed for the same tokens either way. It is worth implementing for the experience and should never appear in a cost-reduction plan. The changes that genuinely reduce output cost are asking for shorter answers, using structured output instead of prose, setting explicit maximum token limits per call type, and avoiding reasoning tokens on tasks that do not need them.
Vikas has around fifteen years of experience building software and now builds generative AI systems at Zyneto. His work covers retrieval augmented generation, agentic AI, knowledge graphs, AI memory, and the evaluation and guardrails that decide whether any of it is safe to put in front of customers. He has shipped enterprise copilots, document AI, chatbots and predictive analytics for e-commerce, fintech and marketing teams, and works day to day in Python, JavaScript and SQL. He follows multimodal models, business process automation and enterprise AI security closely, and mentors engineers moving into AI. He writes about architecture, inference cost and the failure modes that only show up at production scale.
Share your details and we will talk soon.
Be the first to access expert strategies, actionable tips, and the trends actually shaping the digital world. No fluff - just practical insights delivered straight to your inbox.
Dive into our blog and stay ahead of the curve with expert perspectives, future-ready trends, and tech tips written for decision-makers and doers alike.
