AI training and inference are the two phases of a model's life, and they are billed very differently. Training is the one-time job of teaching a model from data: a massive, concentrated cost paid once (Sam Altman said GPT-4 cost "more than $100 million" to train). Inference is the recurring job of running the finished model to answer each request: a smaller per-query cost that repeats forever. At scale, inference dominates. Amazon Web Services states that inference "can represent up to 90% of overall operational costs for machine learning initiatives," and the wider industry consensus puts inference at roughly 80 to 90 percent of the lifetime compute cost of a deployed model. The simple analogy: training builds the factory, inference runs it, and running the factory around the clock is what actually costs the most.
If you follow AI at all, you have heard the eye-watering numbers about what it costs to train a frontier model. Those headlines are real, but they hide the more important truth about where the money in AI actually goes. The training bill is a one-time spike. The inference bill is a meter that never stops running. Once a product has real users, the cost of answering their questions, day after day, quietly grows past the cost of building the model in the first place. This is why AI companies obsess over efficiency, why API prices are shaped the way they are, and why the hottest race in hardware right now is not about training bigger models but about serving them cheaply. Below we define both phases in plain English, show why the "training is the expensive part" myth breaks at scale, and break down exactly what drives the inference bill and what you can do about it.
The two phases, defined simply
Every large AI model lives two separate lives. First it is trained. Then, for as long as anyone uses it, it runs inference. The two look similar from the outside because both involve expensive chips crunching numbers, but they are economically opposite: one is a capital project, the other is an operating expense.
Training: learning the world, once
Training is the process of teaching a model. You take a very large dataset, feed it through the model, and repeatedly adjust the model's internal parameters (its "weights") so that its predictions get better. This is done once per model version, over weeks or months, on enormous clusters of specialized chips running in parallel. The output of training is a finished set of weights, a file that captures everything the model learned. If you want to understand what those weights actually represent, our explainer on how large language models work in plain English walks through it step by step.
The defining feature of training is that it is a one-time, front-loaded cost. You pay it before the model earns a single dollar, and once it is done, you never have to repeat that exact run. Frontier training is genuinely expensive: Sam Altman put GPT-4's training cost at "more than $100 million," and research group Epoch AI has tracked the cost of leading training runs roughly tripling each year, with Anthropic's CEO predicting individual models costing more than $1 billion. Those are the numbers that make the news. But they describe a cost you pay once.
Inference: answering, every single time
Inference is the process of using the trained model. Every time you send a prompt and the model writes a reply, that is one inference. The model loads its weights, reads your input, and generates an output, consuming compute for that single request. Then it does it again for the next request, and the next, for millions of users, indefinitely.
The defining feature of inference is that it is a recurring, per-request cost that scales with usage. A popular product does not run inference once; it runs it billions of times. Each individual query is cheap, often a fraction of a cent, but multiply a tiny number by a gigantic volume repeated every day for years and the total balloons. Inference is not a project you finish. It is a utility bill that arrives forever, and it grows every time your product gets more popular.
The factory analogy that makes it click
The cleanest way to hold this in your head is a factory. Training is building the factory. Inference is running the factory.
Building a factory is a huge, one-time capital expense. You buy the land, pour the concrete, install the machines, and hire the engineers to get it working. It costs a fortune, and it makes headlines when a company announces it. But once the factory is built, that cost is behind you.
Running the factory is the recurring operating cost: the electricity, the raw materials, the workers on every shift, the maintenance, day after day. Any single unit coming off the line is cheap to produce. But the factory runs continuously, for years, producing millions of units. Add up all those small per-unit costs over the life of the plant and they dwarf the one-time cost of building it. Nobody evaluates a factory by construction cost alone, because the money is in the operating years. AI is the same: the training run is the construction, and inference is the operating decades. When people say training is the expensive part, they are staring at the construction invoice and ignoring the utility meter that runs for the rest of the model's life.
Why "training costs the most" is a myth at scale
The intuition that training is the expensive part is not wrong, it is just incomplete. It is true at the moment a model launches, when the training bill has been paid and almost no one is using the model yet. But that snapshot flips the instant the model has real traffic. From then on, every day of usage adds to the inference side of the ledger, while the training cost stays frozen at whatever it was.
The industry data is blunt about the outcome. AWS states that inference "can represent up to 90% of overall operational costs for machine learning initiatives," and analysts broadly estimate that inference makes up roughly 80 to 90 percent of the lifetime compute cost of a model in production, with the remaining 10 to 20 percent going to training. The reason is structural: you train a model a handful of times, but you serve it continuously, around the clock, for as long as the product exists. A model that is queried billions of times will always spend more on answering than it did on learning.
A worked example: one-time versus recurring
Numbers make this concrete, so here is a deliberately simple, illustrative scenario. The figures are round hypotheticals chosen to show the mechanic, not real accounts for any specific model.
Suppose training a model costs $100 million, paid once (the order of magnitude Altman cited for GPT-4). Now suppose the finished model becomes popular and answers 100 million requests per day, and that each answer costs, on average, one fifth of a cent to generate. The daily inference cost is 100 million multiplied by $0.002, which is $200,000 per day. Over a year, that is $200,000 times 365, which is about $73 million. So in roughly 16 to 17 months of serving, cumulative inference spending passes the one-time $100 million training cost. Keep the product running for a few years and inference is several times larger than training ever was.
| Cost | Type | Illustrative amount |
|---|---|---|
| Training the model | One-time | $100 million (paid once) |
| Inference, per day | Recurring | $200,000 per day |
| Inference, per year | Recurring | about $73 million per year |
| Inference, over 3 years | Recurring | about $219 million |
Notice how the numbers behave. Nothing here needs exotic assumptions: a single popular product at real-world scale burns through its training budget on inference in a bit over a year, and everything after that is inference stacking higher and higher. This is the whole reason the 80 to 90 percent figure holds. Training is a wall you climb once; inference is a road with no end.
What actually drives the inference bill
If inference is where the money goes, it pays to know what turns the meter faster. Four levers do most of the work.
Tokens, especially output tokens
Models read and write in tokens, small chunks of text of roughly four characters each. Inference cost scales with the number of tokens processed, and crucially, the tokens the model generates cost more than the tokens it reads, because text is produced one token at a time while the input is read in a single parallel pass. That is why providers publish separate input and output rates, with output commonly three to five times pricier. Reasoning models add a twist: the hidden "thinking" tokens they produce before answering are billed as output too. If you want the full mechanics, our guide to AI model pricing explained breaks down input, output, and cached tokens with a worked cost example. The short version: verbose outputs are where an inference budget quietly bleeds.
Batching and utilization
A single request rarely keeps an expensive accelerator busy. To be efficient, inference servers batch many users' requests together and process them at once, spreading the fixed cost of loading the model across many queries. High batch sizes and high utilization mean the cost per request falls; idle or under-filled chips mean you are paying for hardware that is not doing work. This is why serving infrastructure is a hard engineering problem in its own right, and why a company can be paying for enormous compute yet still bleed margin if its utilization is poor. Getting more useful answers out of each chip-second is the core game of inference economics.
Caching
A great deal of inference work is repeated. Many requests share the same long system prompt, the same reference documents, or the same conversation history. Caching lets the server store that repeated prefix and reuse it instead of recomputing it every time, and providers pass the saving on: cached input is often billed at around 10 percent of the normal input rate, roughly a 90 percent discount on those tokens. For agents with big fixed instructions, retrieval systems that share a knowledge base, and multi-turn chats, caching is one of the highest-return ways to cut the inference bill without changing the model at all.
Hardware and memory
Inference is often limited not by raw math but by memory. Generating each new token requires reading the model's weights and its running context out of high-bandwidth memory, so inference tends to be memory-bound: the speed and cost are gated by how fast data moves, not just how many calculations a chip can do. That is why the amount and speed of memory matters so much, and why running a model locally is a question of fitting its weights into memory in the first place, as our guide to how much VRAM you need to run an LLM explains. It is also why the memory suppliers have become quiet kingmakers of the AI boom: Micron's record results, which we covered in the AI memory tax analysis, and Samsung's push into high-bandwidth memory, covered in our HBM4E report, are downstream of exactly this inference-memory demand.
Why inference is the reason behind API prices
Once you see that inference is the dominant, recurring cost, the shape of AI pricing stops looking arbitrary. Providers charge per token because tokens are the unit of inference work, and they charge more for output than input because generating tokens is the expensive part. Prompt caching discounts exist because cached prefixes are genuinely cheaper to serve. Batch APIs offer roughly half off because non-urgent work can be packed for higher utilization. Every line on a pricing page is, in effect, a rule about how much a particular kind of inference costs to run. If you want to turn that understanding into lower bills, our practical guide to cutting your AI API costs collects the concrete tactics.
This is also why token economics have become a boardroom topic rather than a footnote. When a product's usage grows, its inference bill grows with it, and at large enough scale the cost of running an AI feature can rival the cost of the humans it was meant to help. We dug into that exact tension in our analysis of the token economics driving Microsoft's AI cost decisions, where the recurring price of inference, not the sunk cost of training, is what shapes the strategy.
The efficiency race: MoE, distillation, and quantization
Because inference is the bill that never stops, shaving even a little off the cost of each query compounds into enormous savings across billions of requests. That is why so much AI research is really inference-efficiency research in disguise. Three techniques dominate.
Mixture of experts
A mixture-of-experts (MoE) model splits its parameters into many specialized "experts" and activates only a small subset for any given token, instead of running the entire network every time. The model can be huge in total capacity while only a fraction of it does work on each request, which cuts the compute per token dramatically. This architecture is a major reason some very capable models are strikingly cheap to serve; the low published inference prices on models like DeepSeek V4 lean heavily on sparse, MoE-style designs. Efficiency at inference time, not just raw capability, is now a headline feature.
Distillation
Distillation trains a smaller "student" model to imitate a larger "teacher" model, transferring much of the capability into a model that is far cheaper to run. The result is a compact model that approximates the big one's behavior at a fraction of the inference cost. Distillation has become strategically sensitive precisely because it is so effective at capturing an expensive model's value cheaply, a tension we covered when Anthropic accused a rival of industrial-scale distillation. For anyone paying an inference bill, though, a well-distilled smaller model is often the single biggest lever available.
Quantization
Quantization stores and runs a model's weights at lower numerical precision, for example using 8-bit or 4-bit numbers instead of 16-bit ones. Lower precision means less memory and less data movement per token, which, because inference is memory-bound, translates directly into faster and cheaper serving with only a modest quality trade-off when done carefully. Quantization is often what makes it feasible to run a capable open-weight model on modest hardware at all, a key theme in our guide to self-hosting an open-weight model.
The inference-chip arms race
Follow the money and you arrive at silicon. If inference is 80 to 90 percent of the lifetime bill, then a chip that serves inference even slightly more cheaply is worth an enormous amount across a model's life. That economic gravity is reshaping the hardware industry. Training grabbed the early attention because it needed the biggest, most exotic clusters, but the durable, growing market is inference, and everyone wants a specialized chip for it.
This is why OpenAI moved to design its own inference silicon, which we examined in the story behind its first custom inference chip, and why a wave of startups is chasing the same prize, like the UK inference-chip maker in our Fractile funding analysis. The broader shift, from a training-centric hardware world to an inference-centric one, is the theme of our deeper look at why inference, not training, is becoming the real infrastructure war. When the recurring cost is the big cost, whoever serves a token most cheaply holds the advantage.
What this means for you
You never pay a model's training cost directly; it is baked into the price of every token. What you do pay, directly and repeatedly, is inference. That makes the inference levers the ones worth pulling, whether you are an individual using an API or a company building on top of one.
Right-size the model. The single biggest inference lever is choosing the smallest model that clears your quality bar. Frontier models cost many times more per token than smaller ones, and a great deal of real work, classification, extraction, routing, summarizing, runs perfectly well on a cheaper tier. Reserve the expensive model for the hard cases. If you are weighing hosted versus self-hosted options, our guide on choosing between closed and open-weight models lays out the trade-offs.
Cache what repeats. If your prompts share a fixed prefix, a system prompt, instructions, a reference document, prompt caching can cut the input cost of that portion by around 90 percent. It is the highest-return optimization for any workload with repetition, and it requires no change to the model.
Batch what can wait. Work that does not need an instant answer can go through a batch API at roughly half price, because it lets the provider fill chips more efficiently. Overnight report generation, bulk tagging, and offline analysis are ideal candidates.
Control output length. Because output tokens are the pricier ones, capping how much the model writes and asking for concise formats is often the fastest way to lower a bill. On reasoning models, check whether you can limit or disable extended thinking on simple tasks, since those hidden tokens are billed as output.
The bottom line
AI has two costs that behave in opposite ways. Training is a spectacular one-time expense, the price of teaching a model, and it is the number that makes headlines. Inference is the quiet, recurring cost of running that model to answer every request, and at any real scale it is where roughly 80 to 90 percent of the lifetime money goes, up to 90 percent of operational cost by AWS's own accounting. Training builds the factory; inference runs it, forever, and running it is what actually costs the most.
That single fact explains most of what looks confusing about the AI industry. It is why token-based API pricing is shaped the way it is, why companies pour research into mixture-of-experts, distillation, and quantization, and why the hardware war has shifted from training clusters to custom inference chips. And it is why, as a user or builder, your leverage lives entirely on the inference side: pick the right-size model, cache what repeats, batch what can wait, and keep outputs tight. You cannot rewrite the training bill, but the inference meter is yours to control.
Frequently Asked Questions
What is the difference between AI training and inference?
Training is the one-time process of teaching a model from data by adjusting its internal parameters, done once per model version on huge chip clusters over weeks or months. Inference is using the finished model to answer each request, a smaller cost that repeats every single time someone queries it. Training is a capital project; inference is an operating expense that runs for as long as the model is in use.
Does inference really cost more than training?
At scale, yes. Amazon Web Services states that inference "can represent up to 90% of overall operational costs for machine learning initiatives," and the industry broadly estimates inference at roughly 80 to 90 percent of a deployed model's lifetime compute cost. Training is larger at the moment of launch, but because a popular model is queried continuously for years, cumulative inference spending overtakes the one-time training cost within a year or so of real usage.
How much does it cost to train a large AI model?
Frontier training runs are expensive and rising. Sam Altman said GPT-4 cost "more than $100 million" to train, and research group Epoch AI has tracked the cost of leading training runs roughly tripling each year, with predictions of individual models exceeding $1 billion. These are one-time costs, however, paid before the model serves any users, unlike inference which recurs indefinitely.
Why is training a one-time cost but inference recurring?
You train a model version once to produce its finished weights, and you never have to repeat that exact run. But every user request requires the model to run again, consuming compute each time. A popular product runs inference billions of times over its life, so while any single query is cheap, the total recurring cost grows without end and eventually dwarfs the one-time training expense.
What is the factory analogy for training versus inference?
Training is like building a factory: a huge, one-time capital cost that makes headlines. Inference is like running the factory: the recurring cost of electricity, materials, and labor to produce each unit. Any single unit is cheap, but the factory runs continuously for years, so the operating costs dwarf the construction cost. In AI, the training run is the construction and inference is the operating decades.
What drives the cost of AI inference?
Four levers dominate: the number of tokens processed (output tokens cost more than input, commonly three to five times), how well requests are batched to keep chips busy, how much repeated context is cached (cached input is often about 90 percent cheaper), and the underlying hardware and memory, since inference is usually memory-bound. Controlling these levers is how teams keep inference bills down.
Why are output tokens more expensive than input tokens?
A model reads your whole prompt in one efficient parallel pass, but it generates its reply one token at a time, with each new token requiring another pass over the growing context. Producing text is more compute-intensive per token than reading it, so output typically costs three to five times more than input. Reasoning models also bill their hidden "thinking" tokens as output, which can multiply the charge.
How do MoE, distillation, and quantization reduce inference cost?
Mixture of experts activates only a small subset of a model's parameters per token, cutting compute per query. Distillation trains a smaller student model to imitate a larger one, capturing much of its ability at a fraction of the serving cost. Quantization runs the model at lower numerical precision (for example 8-bit or 4-bit), reducing memory and data movement. All three attack the recurring inference bill rather than the one-time training cost.
Why is inference the reason behind AI API prices?
API pricing mirrors the cost of running inference. Providers charge per token because tokens are the unit of inference work, charge more for output because generating it is pricier, discount cached prefixes because they are cheaper to serve, and offer roughly half-price batch modes because packed work uses chips more efficiently. Every line on a pricing page is a rule about how much a kind of inference costs to run.
Why is there a race to build custom inference chips?
Because inference is 80 to 90 percent of a model's lifetime compute cost, a chip that serves inference even slightly more cheaply is worth an enormous amount across billions of requests. That economics has shifted the hardware industry from a training focus to an inference focus, driving companies like OpenAI to design their own inference silicon and funding a wave of specialized inference-chip startups.
How can I reduce my own AI inference costs?
Pull the inference levers you control: pick the smallest model that meets your quality bar, since frontier models cost many times more per token; cache any repeated prompt prefix to cut that portion by around 90 percent; send non-urgent work through a batch API at roughly half price; and cap output length, since output tokens are the expensive ones. You cannot change the training cost, but these steps directly lower the recurring inference bill.
Is inference the same as running a model locally?
Running a model on your own machine is one form of inference, so the same economics apply. Because inference is memory-bound, the practical limit is usually fitting the model's weights into your memory, which is why quantization to lower precision matters so much for local use. The main difference is that you pay for local inference in hardware and electricity rather than per token, but it is still the recurring "run the model" phase, not training.
