By a broad estimate, over $100B was spent on AI inference in 2025, across capital expenditures and token budgets. How long before that number doubles? Let’s look at what is driving the cost of AI inference.
Inference becomes an economic problem when small costs repeat at enormous scale. A few milliseconds, an extra data movement, or capacity added in the wrong place may be negligible for one request. Across billions of requests, they shape the margin of the service.
Most of the industry’s attention has gone to the GPU, where even modest efficiency gains can increase the output of an entire fleet. But the GPU is only one part of the service.
Every inference request depends on a much broader data estate. Model weights and adapters have to be stored and delivered. Customer prompts, files, images, conversation histories, and tool definitions have to be accepted and governed. Retrieval systems supply documents, records, vectors, and prior context. Request events feed operations and billing. Logs, metrics, and traces explain what happened. Analytical systems connect technical behavior to customers, models, regions, and infrastructure.
All of this sits behind the price of a token.
The economic burden comes from operating these systems as separate infrastructure estates. Each one brings its own capacity, protection, scaling model, operational tooling, and engineering ownership.
The Cost of Building Every Data Service Separately
The point-solution architecture is easy to understand.
Object storage holds model assets and customer files. A shared filesystem supports applications that expect POSIX access. Kafka carries request and usage events. A database stores operational records. A vector database supports retrieval. Observability platforms retain logs, traces, and metrics. A warehouse or lakehouse supports historical analysis.
Each decision makes sense on its own. The problem appears when they all become permanent parts of the same inference platform.
Every stateful system needs growth headroom. Every system needs replication, recovery, monitoring, upgrades, and regional deployment. Capacity reserved for one platform cannot usually absorb growth in another. A system can be underused while the team is urgently expanding the one beside it.
The provider also inherits the work required to preserve meaning as data crosses those boundaries. The request ID in an event stream has to connect with the trace in an observability platform and the usage record in a billing system. A source document has to stay aligned with its vector representation. An object written through S3 may have to appear in a file-oriented workflow. Event history retained for replay may also need to be written into a different system for SQL analysis.
These transfers are automated. The cost lies in building, operating, and recovering the automation at hyperscale.
The Boundaries Become Part of the Platform
A boundary between two data systems is rarely handled by one simple connector. It becomes an operating layer with its own state, failure modes, and recovery procedures.
Model delivery is a good example. Model registries and build pipelines commonly publish weights, adapters, and configuration through S3. But for illustrative purposes, let’s say the inference runtimes needs to load those assets from a filesystem. The provider then needs a process that copies each approved model into the serving environment, verifies the checksum, records which version is complete, and makes it visible to workers only after every required object has arrived.
That workflow also needs rollback. If a model release is withdrawn, the serving fleet has to return to the previous version without loading a mixture of files from both releases. Regional deployments need the same controls, along with enough bandwidth to move large models and their quantized variants into each serving environment. Old versions have to be removed from the registry and the filesystem without deleting something that an active worker still needs.
The second copy exists because one part of the platform uses an object API and another expects a file path. It adds capacity, network traffic, deployment state, and another place where the model catalogue can become inconsistent. This kind of protocol mismatch exists all over AI pipelines.
Request events create a different boundary.
A provider may publish request starts, completions, token counts, routing decisions, errors, and cache outcomes into Kafka. Live services consume those records for rate limiting, operations, and billing. The same events are then exported into a warehouse or lakehouse so engineering, finance, and product teams can query them with SQL.
These boundaries all have the same economic shape. The provider stores connected data in separate systems, pays to move it between them, and operates the machinery that keeps the different representations trustworthy. This adds truckloads of engineering time building, and maintaining the boundaries, and exasperates all troubleshooting. Not to mention service interruption risk, which at the largest AI inference providers can put tens of thousands of dollars in revenue at risk every minute.
A Shared Foundation Changes the Expansion Model
VAST begins by changing what has to scale together.
The Disaggregated Shared Everything (DASE) architecture separates the CNodes that serve and process data from the persistent capacity in DNodes. Every CNode can reach the shared data and metadata in the cluster. Service performance can expand without adding storage in the same proportion, and capacity can expand without purchasing another fixed bundle of front-end processing.
This maps more closely to the uneven growth of an inference platform.
A larger model fleet may increase file and object traffic while adding relatively little new data. Longer retention may increase persistent capacity without creating the same increase in service demand. A rise in analytical activity may require more query resources. Each resource can be expanded in response to the pressure actually appearing in the system.
The shared architecture also gives multiple data services access to a common capacity foundation. File, object, database, event, and persistent KV workloads can use their required interfaces while drawing from the same underlying pool.
That does not make their performance requirements identical. Model loading must remain protected from a large analytical scan. A burst of event ingestion cannot be allowed to interfere with latency-sensitive operations. Quotas, tenant boundaries, and quality-of-service policies remain essential.
The economic difference is that unused capacity can remain available to the platform as a whole. It is no longer permanently assigned to one protocol or one data service simply because those workloads were deployed as separate systems.
Industry research puts the near-term savings from simplifying data architecture and reducing infrastructure waste at 5% to 15% of annual data spending. For a hyperscale AI inference provider spending tens of billions of dollars on data infrastructure, that translates into potential savings measured in billions each year.
Remove Boundaries That Add No New Information
Some copies exist because the data has changed. A quantized model, a derived billing table, and an aggregated performance report are all new assets with their own purpose.
Other copies exist only because the next application uses a different interface.
The VAST Element Store addresses the second case. It stores data in a protocol-independent form that can be presented through S3, NFS, or SMB. The same element can be written as an object and read as a file from a shared namespace, with both interfaces operating on the same underlying data.
For model delivery, this removes the need to maintain one durable copy in the registry and another in the serving filesystem. The build or publication workflow can use S3. The inference runtime can reach the same approved model through NFS.
There is no transfer window during which the object is current and the file copy is not. Promotion does not depend on synchronizing two repositories. Rollback points to the previous model rather than rebuilding another copy of it. Deletion applies to one underlying asset.
This becomes increasingly important as the catalog grows. A provider may retain original weights, several quantizations, long-context variants, draft models for speculative decoding, adapters, and customer-specific versions. Eliminating one redundant representation of each asset changes both capacity consumption and the work required to distribute the catalog across the fleet.
VAST applies a similar principle to vector data.
VAST DataBase stores embeddings as vector columns alongside structured records and metadata. The query engine can combine nearest-neighbour search with SQL filters through the same query path. The vector, the fields used to filter it, and the record returned to the application remain in one table rather than being synchronized across separate persistence systems.
For an inference provider offering retrieval or persistent context, the operational change is concrete. A tenant identifier, model version, expiration time, or access field can be stored beside the embedding and evaluated as part of the query. Updating or deleting the record also updates the data being searched. The provider still computes embeddings and maintains a vector index, but it does not need a separate database and replication path simply to make those vectors searchable.
File and object unification removes a protocol boundary. Native vector search removes a data-model boundary. In both cases, the application keeps the interface it needs while the provider removes an independently operated copy of connected data.
Preserve the Most Expensive Data the Service Creates
Model assets, events, vectors, and telemetry support the inference operation. KV cache is produced by the operation itself.
During prefill, the model processes the input context and generates the key-value state needed for attention during token generation. That state represents GPU work already completed.
The fastest place to keep it is GPU memory. HBM is also limited, expensive, and local to the worker. Host memory and node-local storage extend the cache hierarchy, but the state remains tied to a relatively small failure and scheduling domain.
When the cache is evicted or the next request lands on another worker, the new GPU may have to process the same prefix again.
At hyperscale, repeated prefixes appear in long system prompts, agent instructions, shared context, code bases, conversation histories, and common prompt templates. The provider may spend GPU cycles rebuilding state that existed elsewhere in the fleet only moments earlier.
A shared persistent tier gives that state a longer life and a wider reach.
Cache managers such as NVIDIA Dynamo, LMCache, and SGLang HiCache decide which KV blocks should be retained and recovered. VAST provides persistent capacity that can be reached across the cluster. Current integrations can move cache data through NFS over RDMA and use GPUDirect Storage to transfer it between VAST and GPU memory without staging the data through host memory.
A later request can recover existing KV state instead of repeating the entire prefill. A conversation can resume on another worker. A scheduler can move work across the fleet without automatically discarding all of the context already computed.
Put Engineering Capacity Back Into the Inference Service
A fragmented data estate creates continuing work around model staging, broker exports, vector synchronization, schema translation, backfills, and independent capacity expansions. Teams maintain dashboards that measure whether these paths are keeping up and tools that repair them when they are not.
A shared data foundation removes some of those paths and shortens others and the improvement can be measured in practical terms:
How much capacity is reserved but unused across separate platforms? How many copies exist because applications use different protocols? How much request history is written twice? How many pipelines exist only to keep connected data consistent? How much GPU time is spent rebuilding context that the fleet has already computed?
Those figures expose the portion of inference economics controlled by the data architecture.
Improve the Economics Around Every Token
GPU hardware will remain the largest factor in inference economics, but the data plane determines how much supporting infrastructure and repeated work must accompany that fleet.
When every access pattern becomes a separate data estate, the provider pays for several capacity reserves, several protection models, duplicate representations of connected data, and the automation required to keep those representations aligned.
VAST Data’s AI Operating System changes that structure.
VAST’s AI OS lets service performance and persistent capacity grow independently. It allows file and object applications to use the same underlying model data and it keeps vectors beside the records and metadata used to query them. The AI OS supports live consumption and historical query access from the same retained event and it provides a shared persistent storage that lets selected KV state remain available across the GPU fleet.
These architectural changes leave the specialized inference stack intact. They remove infrastructure and operations that accumulate around it.
At hyperscale, that work sits behind every token. Improving it changes the economics of the service.
