The AI industry is about to discover what infrastructure waste actually costs, and the bill will be measured in billions.
For most of the cloud era, it was relatively easy to buy your way out of inefficiencies because hardware was cheap, abundant, and replaceable enough that architectural waste rarely threatened the business itself. That model is breaking, especially among the largest consumers of AI compute.
That old cloud world model is hard to let go of because it was rooted in abundance. It made things easy. If a storage architecture wasted capacity through metadata overhead, rebuild inefficiency, weak reduction rates, or bloated reserve requirements, the answer was usually just more hardware. Yes, it was expensive and operationally annoying, but it was survivable.
At massive scale, those same inefficiencies compound, almost exponentially, into additional flash allocation, power contracts, cooling infrastructure and deployment delays. Eventually this turns into billions of dollars in infrastructure that never needed to exist in the first place.
In the old world, infrastructure buyers could afford to think about hardware and software as separate conversations. One team would negotiate servers/flash arrays, another negotiated licensing and such, and another worried about operations later. Somewhere in the middle a spreadsheet approximated total cost close enough for everyone to move on, because the penalties for inefficiency were survivable.
At hyperscale, software architecture stops being separate from infrastructure economics
What does that mean on the ground? Compression ratios affect how much flash must physically exist in the environment, metadata overhead affects usable capacity, rebuild behavior changes reserve requirements, data reduction changes procurement timelines and “little” things like erasure coding efficiency changes power draw, footprint expansion, and at the end of the day, how much infrastructure a company must buy before a single model generates a single token.
So that old enterprise habit of negotiating hardware over here and software over there starts to look kinda absurd once deployments move into the tens of exabytes. At 50 exabytes, a few percentage points of inefficiency can mean billions of dollars in additional infrastructure.
That is the part of the AI infrastructure market many buyers still do not fully appreciate. The software architecture underneath these systems increasingly determines the amount of physical infrastructure the world has to build to sustain AI growth. A few percentage points of inefficiency no longer disappears into enterprise budget math, it compounds into billions of dollars in more flash procurements, not to mention power, cooling, delays, and increasingly, additional exposure to supply chain constraints that are already tightening with no reprieve in sight.
And let’s address something now: a platform can still cost billions even if the software itself is free.
Inefficient architectures require so much additional infrastructure that licensing costs become almost secondary. A customer might save hundreds of millions on software and still spend billions more on the hardware needed to support it.
The real question you should be asking isn’t “how much does the platform cost?” it should be “how much infrastructure does this architecture force me to build?”.
Now Entering the Allocation Era
At scale we no longer can assume infrastructure will simply appear when it is needed.
Flash allocation is tightening, lead times are stretching, and the AI infrastructure ecosystem is starting to feel more like the energy industry or semiconductor manufacturing than IT. Buying capacity quarter to quarter is colliding with the reality that some of the most important infrastructure decisions now have to be made years in advance.
Unlike previous booms, this one’s happening alongside an explosion in persistent AI data: long-context inference, KV cache retention, vector DBs, synthetic training sets, real-time event streaming/processing/analysis, multimodal pipelines, and continuous analytics systems that are all competing for the same underlying flash infrastructure. At the same time hyperscalers and AI clouds are racing to secure future capacity before it disappears into someone else’s deployment pipeline.
Memory pricing and flash availability aren’t moving together like they used to. DRAM pricing will eventually cool and GPU availability may improve, even if unevenly. Flash constraints will not disappear, because AI systems are increasingly dependent on persistent context, historical data, vector stores, and long-lived inference infrastructure.
Companies that might have bought infrastructure reactively are now trying to secure allocation years in advance. Demand keeps climbing, and that changes buying behavior. AI cloud providers are packaging long-term contracts around future flash availability because they understand something the broader market is only beginning to absorb:
If you can’t secure storage infrastructure at scale, eventually the GPUs stop mattering too. Without persistent data infrastructure, it all pretty quickly turns into stranded capacity.
We’re entering a phase where infrastructure efficiency determines who gets to scale
That might sound dramatic but look at what is already happening across the market.
AI providers are already becoming far more selective about what data they retain, how long they retain it, and how aggressively they expand infrastructure footprints because future capacity is no longer a given. Some hyperscalers are buying years ahead simply to preserve optionality, while others are discovering that AI demand is scaling faster than the infrastructure supply chain beneath it.
Zoom out for a moment. If AI infrastructure requires far more hardware to achieve the same outcome (operationally speaking), then inefficiency moves on from being a “local” budgeting problem and is actually part of the global supply equation (more flash has to be made, more power and yet more datacenters need to come online, etc.).
The entire physical footprint of AI expands to compensate for architectural waste sitting higher in the stack. We’re talking about literal industrial-scale resource consumption during the largest infrastructure buildout the technology industry has attempted.
Those infrastructure constraints are exposing another reality the AI market has mostly avoided confronting so far:
GPUs alone are not enough to build a durable AI cloud business
The first generation of AI clouds won simply by getting access to compute faster than the hyperscalers could. If you had GPUs, networking, power, and enough infrastructure to stand up clusters quickly, customers arrived almost automatically. That worked when training was the golden goose. The new battleground is persistent inference, and that’s been a tough shift.
The moment customers need real-time event processing, vector search, analytics pipelines, metadata systems, object services, or inference infrastructure around the models, many of them end up drifting back toward AWS, Google Cloud, or Azure where the rest of the stack already exists.
A GPU cluster might have been an AI powerhouse when procured but it’s not an AI platform.
Eventually the models need systems around them that can continuously move, process, retain, and sling data. Event streams have to stay live, vector databases need to sit close to inference, metadata becomes infrastructure, and data lakes shift from archivalists to active feeders.
GPUs are expensive and under constant pricing pressure, but this changes the economics of the entire business. The surrounding data services are where durable margins and customer retention start to appear. Eventvent brokers, vector DBs, analytics, and inference-oriented data processing layers give customers reasons to stay inside the platform instead of pushing workloads back to the hyperscalers.
AWS, Google, and Azure didn’t become dominant piecemeal, they built interconnected service ecosystems that were really hard to leave. Databases connected to analytics, analytics connected to event systems, event systems connected to AI services and so on. And over time the platform itself became the moat.
AI clouds are beginning to move in the same direction now. The real long-term business for these AI cloud builders has moved beyond compute and centers on the continuous movement and processing of data around that compute.
Some of our internal modeling around AI cloud service catalogs starts to expose how different these economics become once providers move beyond raw compute.
Event Broker services approach ~97.5% gross margin
Vector Database services approach ~98.6%
High Performance Network File Storage approaches ~93.1%
Block Storage approaches ~91.9%
High Performance Object Storage approaches ~81.0%
Data Lake services land around ~65.4%
Real-Time Data Lake services land around ~62.9%
Standard Object Storage falls much lower at ~11.2%
Notice where the margins are accumulating?The closer infrastructure gets to real-time processing, the more economically valuable it becomes.
Event brokers, vector systems, metadata infrastructure, analytics engines, and inference-oriented processing layers increasingly sit directly in the operational path of AI systems. They are no longer peripheral services attached to the compute environment after the fact. They are becoming part of the inference environment itself.
Continuous inference requires continuous infrastructure
Event processing is a great example here because it’s clear that GPUs alone are useless without systems capable of continuously moving, analyzing, and actively using live data around the models.
What many AI clouds are starting to discover is that building a real data platform means solving a problem the industry historically split into separate systems.
Streaming and analytics systems evolved independently, so everyone still thinks of them that way: event systems handled ingestion and movement, analytics handled historical analysis later. Data pipelines reflected that, moving through stages (ingest, analyze, act). Enterprise apps could tolerate the lag between those steps. AI can’t, at least not at meaningful scale..
We’re moving into a world where inference has to be persistent and at the ready. There’s no room there for distance between ingest, analysis and action. Fraud systems detect patterns while transactions are happening. Telco systems need to reroute before outages spread. Recommendation engines need to react while users are still active.There are countless examples, but the point is the same there’s no waiting around for batch analytics pipelines hours later.
That changes the role of event infrastructure pretty dramatically. Kafka environments, event brokers, vector systems, metadata services, and real-time analytics engines are increasingly becoming part of the inference environment itself because the models depend on live awareness to remain useful.
Most event systems were built for moving data, not deeply analyzing it in real time, while most analytics systems were built for historical analysis instead of continuous operational reasoning. The result is fragmented infrastructure where streaming, analytics, warehouses, databases, and inference pipelines all operate independently, creating latency between ingestion, analysis, and action. Data moves between systems, analytics drift behind live conditions, and by the time processing finishes the environment that generated the event may already have changed.
Continuous inference depends on continuous analytics. Once those systems start operating continuously against live data, the infrastructure under that also changes in ways the industry might not fully grasp yet.
Here’s our view of the future based on what we’re seeing from our AI factory and AI cloud builders:
Long-context inference changes how much information systems must retain close to the models.
KV cache turns inference history into persistent operational infrastructure.
Vector search keeps growing because models increasingly depend on external context retrieval to remain useful.
Event systems continuously feed telemetry, operational signals, user behavior, and real-time updates into inference environments.
Metadata systems become active coordination layers. AI clouds evolve into always-on operational environments instead of isolated compute pools.
The takeaway? None of this behaves like short-lived scratch infrastructure anymore.
The infrastructure underneath AI is becoming increasingly persistent, interconnected, and continuous. Data is constantly moving between inference systems, event pipelines, vector environments, analytics systems, orchestration, and live apps. Context is retained longer. Awareness becomes part of the model lifecycle itself.
Which is why many of the assumptions that shaped earlier cloud infrastructure are starting to break down simultaneously. Here are a few more observations:
Procurement cycles break because infrastructure expansion can no longer happen instantly.
Traditional storage and analytics architectures struggle because latency is unacceptable.
GPU-centric cloud models become incomplete because the surrounding data systems increasingly determine whether inference environments remain useful at scale.
The real cost of architectural waste
Every unnecessary copy of data, every fragmented analytics pipeline, every oversized reserve requirement, every poorly utilized flash layer, every redundant infrastructure tier eventually expands into more hardware, more power, more cooling, more floor space, and more pressure on supply chains already struggling to keep up with AI demand.
The economics shift because the infrastructure supporting the models starts consuming enormous amounts of persistent flash, power, cooling, and physical footprint over long periods of time.
Building larger compute environments is out. Building continuous inference infrastructure at industrial scale is in.
The question worth taking into the next infrastructure review isn't how much capacity has been bought or how many GPUs are on order. It's how much hardware the underlying architecture is forcing the organization to buy in the first place. At enterprise scale that question is a footnote. At AI scale it's a budget. And at the scale the largest AI builders are now operating, it's the difference between a viable cloud business and a stranded capital problem.
The winners of this buildout won’t be determined by who has the most GPUs. They’ll be determined by who wastes the least infrastructure around them.
