AI Inference Costs Explained: Why Tokens Get Cheaper, Bills Don't

AI inference cost is what it costs to run a trained model each time someone uses it, usually measured per million tokens of input and output. For any fixed level of capability that cost has fallen about tenfold or more a year since 2021. But the cost of a task often rises anyway, because reasoning models write thousands of hidden tokens, agents re-send their growing context at every step, and customers keep moving to the newest, dearest model.

Both trends were visible by mid-2025. In late July 2025 Anthropic told subscribers to its $20, $100 and $200 Claude plans that from 28 August it would add weekly usage limits to Claude Code, because some users were running the agent "continuously in the background, 24/7" (company statement, reported). Six weeks earlier Cursor, which bought its intelligence from Anthropic and others, had moved heavy users on its $20 plan to usage billed at API rates, then apologized and refunded unexpected charges. Yet in the sixteen months to July 2024, the blended price of a GPT-4-class answer had fallen roughly 140 times, from $37.50 per million tokens to about $0.26.

What does an AI token cost to make?

A token, roughly three-quarters of a word, costs the rental of an accelerator for the fraction of a second it takes to produce it. The research reduces this to one identity: cost per million output tokens is roughly the all-in cost of an accelerator-hour divided by the millions of tokens it produces in that hour. The numerator includes the chip's depreciation or rental, its share of servers and networking, space, power, cooling and operations. The denominator depends on the model and how busy the chip is.

Serving a request has two phases. In prefill, the model reads the whole prompt in parallel, which uses the chip's arithmetic efficiently; this is billed as cheaper input tokens. In decode, it writes the answer one token at a time, and each token requires reading the model's weights from memory, so speed is limited by memory bandwidth. That is why output tokens typically cost four to eight times as much as input, and why high-bandwidth memory became one of the industry's chokepoints.

The most important lever on cost is batching: serving many users with each pass through the weights. If one read serves 64 users, its cost is shared 64 ways. A busy, well-batched cluster produces tokens far more cheaply than an idle one, which is why utilization matters as much as the chip's price. Other levers include quantization (storing weights in fewer bits), caching repeated input, speculative decoding, smaller distilled models and chips built for fast decode, which Groq and Cerebras sold as token services. None of these stays proprietary for long.

How fast is the cost of AI falling?

Very fast, for a fixed capability. OpenAI's GPT-4 cost $30 per million input tokens and $60 per million output at launch in March 2023. GPT-4o mini, which scored higher than the original GPT-4 on MMLU, cost $0.15 and $0.60 in July 2024 (company price lists). a16z estimated in November 2024 that the cost of a given capability had fallen about tenfold a year in 2021–24, and Epoch AI found in March 2025 that the price of reaching a fixed performance level fell between 9 and 900 times a year depending on the task, with a median of about 50 times.

Competition drove the biggest shocks. DeepSeek's R1 launched in January 2025 at $0.55 per million input tokens and $2.19 per million output, against $15 and $60 for OpenAI's o1 at list. In June 2025 OpenAI cut o3's price by 80%, to $2 and $8. Open-weight models served by many hosts put a ceiling on what any closed lab can charge for capability the open models have reached.

Why does the price of the best model fall more slowly?

The figures above describe the price of a capability that customers stop wanting once something better exists. The price of the best available model fell far more slowly.

In 2026 the main frontier models clustered roughly between $2 and $5 per million input tokens and $12 and $30 per million output, with premium tiers far higher (third-party trackers). On these figures the frontier's price per token fell perhaps three to ten times over three years, while a fixed capability fell a hundredfold or more. A business that always uses the best model lives on the slower curve.

List prices also overstate what sophisticated buyers pay. Anthropic's prompt caching (August 2024) reads repeated input back at 10% of the base price; OpenAI's automatic caching (October 2024) gives 50% off; both discount batch jobs by about 50%; and committed-use contracts through the clouds carry private discounts.

Why do AI bills go up when token prices go down?

Three forces turn falling token prices into rising bills.

Hidden reasoning tokens. A reasoning model writes a chain of thought before it answers, and those tokens are billed as output even when the user sees only a summary. A request with 2,000 input tokens and a 500-token answer costs about one cent at $2.50 and $10 per million; add 25,000 reasoning tokens and the same request costs about 26 cents at the same list price.

Compounding context. An agent works in steps, and at each step the whole conversation so far, including tool results, goes back to the model. Input grows with the square of the number of steps. A developer who launches long agent runs dozens of times a day can consume hundreds of millions of tokens a month on a plan priced for someone typing questions.

The Jevons effect. When a resource gets cheaper, total use can rise enough that spending increases, as William Stanley Jevons observed of coal in 1865. Microsoft's Satya Nadella invoked him after the DeepSeek shock. The effect concentrates in a few users: a chat user's monthly compute falls to cents while an agentic coder's rises by orders of magnitude.

What does one AI task cost?

The book follows a single document-review task through three years in a stylized calculation. The 2023 and 2024 prices are those models' launch list prices; the 2026 prices and all token counts are assumptions.

The task became about 170 times cheaper in a year on a fixed capability. Moving to a reasoning model gave back the entire gain. Handing it to an agent made it about thirty times dearer than in 2023, or nine times with caching. At 30 agent tasks a day for 22 working days, one analyst's monthly bill would be about $4,300 without caching and about $1,260 with it, many times any software seat price. The agent may be worth it if it replaces real work, but the price per token says almost nothing about the bill. What matters is tokens per task, tasks per user and the share of input that can be cached.

What gross margins do AI companies make on inference?

Gross margin in AI depends heavily on what counts as cost of revenue, so compare figures with care.

Training almost always sits outside gross margin, so a lab can serve paying customers profitably and still spend more on compute than it earns. Hosts of open-weight models, such as Together AI and Fireworks, sell identical products, so their prices converge on cost. Applications sit at the top and inherit everyone's choices: the Wall Street Journal reported in October 2023 that GitHub Copilot lost more than $20 per user a month on average early that year, at a $10 price (reported; Microsoft did not confirm).

The extreme case is coding. Cursor had a gross margin reported at about −23% in early 2025 (The Information, reported), because a flat $20 plan bought access to models whose cost rose with use. A stylized seat in the book shows why: at $40 a month on a frontier model billed at $9 per million blended tokens, a light user earns a 53% gross margin, a typical user at four million tokens a month earns −3%, and a heavy agent user at 25 million tokens costs more than five times the seat price.

How do AI companies reduce inference costs?

Applications defend their margins in five ways, described in the inference economics study:

1. Routing most traffic to cheaper models and reserving the frontier for hard requests.
2. Caching repeated input such as system prompts and documents.
3. Training their own models for high-volume tasks, as Cursor did with Composer in October 2025.
4. Pricing on usage or outcomes rather than flat seats, so heavy users pay for what they consume.
5. Waiting for prices to fall, which works only if users do not move to dearer models in the meantime.

Labs use the same tools at their own layer, plus usage limits. Anthropic's weekly caps on Claude Code and OpenAI's admission in January 2025 that it was losing money on the $200 ChatGPT Pro plan were the same discovery: the customers who choose unlimited plans are the ones who use them most. Our guide on how to price an AI product covers the pricing side in detail.

How should you read inference numbers?

The research offers five checks. Ask which price fell: a fixed capability's, or the model customers actually use. Count tokens per task, not price per token. Check what cost of revenue includes, since free users, partner revenue shares and training can sit inside or outside it. Find the heavy tail, because the top few percent of users, often running agents, set the margin of any flat price. And ask who passes on the savings when token costs fall: competitors and open-weight hosts decide whether the gain goes to the vendor or the customer.

In terms of The Intelligence Equation, inference turns on the second factor, unit economics that hold at real usage. A product can become cheaper to run per token and more expensive per customer in the same year, because the most capable models and the most enthusiastic users consume the most. The wider economics of each layer are in our guide to how AI companies make money.

The takeaway

AI inference cost per token has fallen faster than almost any price in history for a fixed capability, but the frontier has fallen far more slowly, and reasoning and agents multiply the tokens in each task by ten to a hundred times or more. That is why Anthropic capped Claude Code, Cursor repriced its plans and fast-growing AI apps average gross margins near 25%. The companies that benefit from falling prices are the ones that measure cost by the task, price by the task and keep their heaviest users inside the economics. Chapter 12 of the book, Inference Economics, works through the full arithmetic.