Training a model is a one-off capital project; inference, running the model every time someone asks it something, is the recurring cost of the AI business and, by 2025–26, the larger part of the industry's compute bill. Its unit is the token, and its price has fallen faster than almost any price in economic history: Epoch AI found that the price of reaching a fixed level of performance fell by between 9x and 900x a year depending on the task, with a median of ~50x (March 2025), and OpenAI's list price for a GPT-4-class answer fell more than a hundredfold between March 2023 and July 2024. Yet AI companies' compute bills kept rising, and many AI applications run at gross margins far below software norms. Three forces explain the paradox. First, the price of frontier capability falls much more slowly than the price of yesterday's capability, and customers keep moving to the frontier. Second, reasoning models and agents multiply the tokens per task, by ten to a hundred times or more, through hidden chains of thought and long multi-step loops whose context grows with every step. Third, cheaper tokens unlock new uses (the Jevons effect), so total spending rises as the unit price falls. This section explains what a token costs to produce (prefill and decode, memory bandwidth, batching, the KV cache, quantisation, caching and specialised chips), how and why prices have fallen, how reasoning and agentic workloads blow up cost per task, what margins labs and applications actually earn, and how to read the numbers. Two companion files give a worked example of price-decline curves and reasoning-token cost blow-ups, and a checklist for diligence on inference costs.
What the study covers
- 1Overview: why inference is the business
- 2What a token costs to make
- 3The levers that lower cost per token
- 4How fast prices have fallen
- 5Reasoning models and agents: the cost blow-up
- 6Margins at the labs
- 7Margins at AI applications
- 8On-device and edge inference
- 9How to read inference numbers
- 10Key numbers
- 11Landmark companies and cases
- 12Pitfalls
- 13How an expert thinks about it
- 14Glossary
Technical exhibits
Worked examples, checklists and case studies included with this study.
Price Decline and Reasoning Cost Worked Example
Stylized price-decline curves at 3x, 10x and 40x a year; a list-price history of real OpenAI and Anthropic models; per-task cost of a non-reasoning versus a reasoning answer; a 100-step agent loop with and without prompt caching; and why spend per user can rise while price per token falls (computed in Python)
Inference Cost Diligence Checklist
Questions to ask of a lab, inference provider or AI application about its cost per token, cost per task and gross margin