For years, the divide between training and inference was so clean it became gospel: Training was slow, sprawling, stochastic—a beast that required orchestration, orchestration that required hardware, and hardware that required real estate and megawatts.
Inference, by contrast, was the featherweight at the end of the fight. A quick jab. A flash of insight. You trained once, then served forever. And infrastructure—entire philosophies of scale, cost modeling, and system design—was built around that.
Ah, but in no time at all inference mutated, it grew teeth. What was once a reaction has become a process.
Inference used to mean pushing tokens through a frozen graph. Now it means sustaining a reasoning loop, balancing memory, navigating external data, juggling vector search and dynamic context resolution while hitting latency targets that no training run ever had to worry about.
I'll resist the urge to put this in all-caps: We are now watching the inversion of inference and training unfold.
The workloads are beginning to blur or maybe worse, they’re beginning to resemble each other in shape but not in temperament. The worst qualities of both are now present in inference: the scale of training, the brittleness of production, the interactivity of a user-facing service, and the utterly unforgiving demands of real-time latency.
Early on, you could serve a language model from a single 40GB GPU and hit sub-100 ms response times. You could fake it. You could compress. Now, inference workloads are spilling over into 72-GPU nodes, fused together with high-bandwidth interconnects just to keep up.
And these nodes aren’t just heavier—they’re hotter. One inference rack will likely draw over 600 kilowatts in the next few years. That’s more than some entire enterprise datacenters are provisioned to handle at the facilities level but that’s its own conversation.
And yet people still speak about inference as if it's small.
They think they’re scaling endpoints but they’re actually scaling the equivalent of training runs.
These inference paths are recursive, multi-modal, and multi-hop. One call becomes three. One token prediction depends on five external sources, a vector DB lookup, and a ranked re-ranking pass with a secondary model. It’s compute chasing compute.
And it doesn’t pause. Training can afford to be slow. It can fail and restart. It can overfit, underperform, be tuned and resumed. But no such luxury for our dear inference must simply work and work all the time and at speed. And it must do so under the illusion of effortlessness—no visible retries, no observable drift, no downtime.
This is not serving static intelligence. This is animating something that never sleeps.
We can no longer say with a straight face that inference is the lightweight cousin. It’s the same computational graph, just played in real time, with no room for failure.
The distinction used to be helpful. Now it's a liability. Now it’s the thing preventing us from architecting infrastructure that actually fits.
Because here’s the quiet disaster: most datacenters were never designed for this kind of inferencing. Not in terms of power. Not in terms of thermal design. Not in terms of memory capacity, scheduling logic, or distributed systems behavior.
Inference has become a power-sucking, latency-sensitive, GPU-scaled workload—and the systems we built for 30kW racks and REST APIs are not going to carry us forward.
For a nice long time folks will keep saying bah, inference is fine, it scales just fine–that you can run it at the edge, that you can run it in the cloud, that you can run it on their new half-size cards with smarter firmware.
But the reality—quietly admitted in strategy decks and whispered across colo procurement calls—is that inference is not getting easier. It’s getting bigger. It’s getting deeper. And it’s converging with the worst parts of training.
The result is a new category of infrastructure. Or rather, the death of categories altogether. Training and inference, batch and online, production and R&D—none of these distinctions hold. Not anymore. What we are left with is a unified compute reality where models never sleep, where serving is learning, and where every prompt is the beginning of a simulation that may fork a hundred times before it resolves.
So now we have to ask: what kind of machine can do all of this?
The uncomfortable answer is that we don’t know. Not really.
The racks are getting taller. The GPUs are getting denser. The networking is getting faster. But the architecture—the organizing logic of how we plan capacity, allocate power, and distinguish between dev and prod—that hasn’t caught up. And until it does, we will keep pretending that inference is still the endpoint of the story, instead of what it’s become: the story itself.
And when the power runs out—and it will—it won’t be training we have to choose between. It’ll be the illusion of simplicity we’ve clung to for far too long.
Here’s a final thought: maybe that’s what the next great infrastructure layer has to encompass...not a training system. Not an inference service. But a substrate built to erase the distinction. One that treats data like memory, inference like orchestration, and collapse like an assumption, not a crisis.
Not something retrofitted, not stitched together, but born from the reality that AI doesn’t split cleanly anymore. It just runs. Everywhere. All at once. And maybe ultimately what wins won’t be the one that optimizes for the past, but the one that absorbs the weight of what inference has become—and doesn’t break.