Inference Is the New Oil: Why AI Capital Is Moving From Training to Serving
Training a model is a check you write once. Inference is a meter that runs on every user action. Why capital now chases cheap inference, and who keeps margin.
Two funding stories landed a day apart last week. On October 5, TechCrunch reported that Etched, a startup that builds chips for running models rather than training them, was fielding offers at valuations between $40 billion and $50 billion, about double the $21 billion it was priced at in its last round. On October 6, Bloomberg reported that DeepSeek was close to raising at least 80 billion yuan, about $12 billion, with Tencent and CATL among the largest backers. Neither deal had closed. Etched declined to comment, and Reuters said it could not immediately verify the DeepSeek report.
So here is the honest question. Where does the money in AI end up living? For three years the answer was obvious: whoever trained the biggest model. We don’t know yet whether that is still true. What we can do is look at what a single user action costs to serve, and follow that number.
One support reply, priced twice
Picture a support lead at a mid-sized software company. Her team wires a model into the help desk so that every incoming ticket gets a drafted answer. The feature is one user action: a customer asks, the model reads the ticket and the relevant docs, and it writes a reply.
The two models are not the same model. GPT-4o mini is a smaller one that OpenAI aimed at exactly this kind of high-volume work. But if the smaller one is good enough for a support reply, and for many tickets it plausibly is, the bill shrinks by a factor of 160.
Notice what never appeared on that bill: the cost of training either model. The support lead never paid for a training run. She paid, and keeps paying, for every single reply. The training cost is the model maker’s problem, paid once. The reply cost is hers, paid forever, and it grows with every customer she adds.
The model: the check and the meter
Here is the framework the rest of this piece uses, in one sentence. Training is a check you write once; inference is a meter that runs every time a user does anything, and in a product people actually use, the meter eventually outspends the check.
Everything else follows from asking who owns the meter, who sets its rate, and who gets squeezed when the rate moves.
Finding 1: training is a capital event, inference is an operating cost
A training run is a big, lumpy purchase. You buy or rent a cluster, run it for weeks, and end up with a set of weights. Accountants would recognize it as something close to capital spending: paid up front, used for a long time.
Inference behaves like electricity in a factory. It is consumed per unit of output. Every chat message, every drafted email, every agent step burns a little more. A product with ten users barely notices it. A product with ten million users notices nothing else.
Where the crossover sits depends on the model and the product. For a frontier model that billions of requests hit every day, it can arrive quickly. For a niche model nobody uses, it never arrives. That is the whole point: the meter only matters if people show up, and once they do, it dominates.
Finding 2: the meter was already big before reasoning arrived
Nvidia’s finance chief, Colette Kress, gave the figure on the February 2024 earnings call, at a time when the public story was a training arms race. Yet the company selling the shovels was already reporting that two dollars in five were going to serving models rather than building them.
Then the models started thinking before they answer. A reasoning model burns extra tokens on every request before it writes a word you can see, which makes each user action heavier. At Nvidia’s GTC keynote in March 2025, Jensen Huang said the computation now required was “easily 100 times more” than the industry had expected a year earlier, and he pinned it on reasoning and agents. That is a CEO’s estimate, not a measurement, and he sells the chips. The direction is still hard to argue with. Every step toward agents moves more of the spend onto the meter.
Finding 3: the rate on the meter is collapsing, and whoever sets it keeps the margin
Guido Appenzeller at a16z gave this curve a name in November 2024: LLMflation. His estimate was that the cost of a fixed level of capability was dropping about tenfold every year, and about a thousandfold over three years.
Cheap tokens sound like good news for everyone. They are good news for whoever is buying. For whoever is selling, they are a price war. The application layer gets a gift every time the rate drops, but it does not control when the gift arrives, and its competitors get the same gift on the same day. The durable advantage sits lower in the stack, with whoever can serve a token for less than everyone else and still make money on it. That is the oil analogy in one line. The refinery with the lowest cost per barrel decides what everyone downstream can charge.
Finding 4: capital is following the meter
Etched is the cleanest example. Its chip, Sohu, is built for one job: running transformer models. It does not train them. In July the company said it had already booked about $1 billion in customer orders, and its reported valuation has roughly quadrupled since. Investors are paying for a lower cost per token, which is a bet on the meter and nothing else.
DeepSeek is a subtler case. It is a model lab, and labs train. But its reputation was built on doing more with less compute, and a lab that serves tokens cheaply is a lab that can price below everyone who serves them expensively. Whatever Tencent and CATL are buying, they are buying into a company whose edge shows up on the meter.
What this gets wrong
The first weakness is the biggest. Training has not stopped mattering; it decides who has something worth serving at all. A lab that falls a generation behind can run the cheapest meter in the world and still watch its traffic leave. The honest version of the claim is that training buys you a seat at the table and inference decides how much you keep once you are sitting there.
The second weakness is the oil analogy itself, which cuts both ways. Owning cheap supply is a great position until everyone has cheap supply. If serving costs keep falling tenfold a year across every provider, inference could turn into a commodity where the margin flows straight through to the application, or to the end user, and the people who funded the chips are left holding a price war.
The third is plain caution about the evidence. Every valuation in this piece is a report from people familiar with the matter. None of them had been confirmed by the companies at the time of writing. A pattern built on two reported deals is a hypothesis with good timing, not a result.
What to do with this
If you build products on top of models, stop treating the token bill as an IT line item. It is your cost of goods sold, and it moves every few months without asking you. Know your cost per user action to the fraction of a cent. Keep a second provider wired up, so that the next price cut is something you can take rather than something you hear about. And design features so that a cheaper model can slot in without a rewrite, because the cheaper model is coming.
If you fund or run infrastructure, the question to ask about any AI company is no longer “how big is the model.” It is “what does it cost them to serve one answer, and who sets that cost.”
