AI infrastructure has always been a moving target, but the pace of change today is challenging even the most experienced system architects. Models are evolving, inference workloads are becoming more complex, and the traditional approach of adding more compute is running into practical limits around memory, bandwidth, and data movement.
Meanwhile, enterprises are making infrastructure investments that need to remain relevant for years in an environment where requirements can change in months.
In this episode of Shared Everything, I chat with Anat Heilper, Director of AI Architecture at VAST Data, about the changing definition of AI optimization and what it takes to build infrastructure that can adapt.
With several previous years at Intel working across computer vision, AI accelerators, and large-scale training and inference systems, Anat brings a perspective that spans several generations of AI architecture.
The Changing Definition of AI Optimization
For much of AI’s recent history, optimization centered on accelerating computation. Hardware vendors focused on increasing processing capabilities, while software developers worked to extract more performance from increasingly powerful accelerators.
As Anat explains, adding compute capacity is often easier than increasing memory bandwidth or capacity, where physical limitations become more pronounced. At the same time, the emergence of specialized accelerators and increasingly diverse model architectures means there is no longer a single approach to optimization that works across every workload.
The distinction between prefill and decode in large language model inference illustrates this shift. Each phase has different computational characteristics and presents different opportunities for optimization. Rather than treating inference as one monolithic workload, architects are increasingly looking at how individual components can be accelerated and supported more efficiently.
KV Cache and the Economics of Inference
One of the most significant opportunities involves KV cache, particularly as AI applications move toward more complex, agentic workflows.
By preserving and reusing previously computed attention states, KV caching can eliminate redundant prefill computation, reducing GPU utilization, latency, and power consumption. For workloads that repeatedly access similar contexts or involve multiple interacting agents, the implications for infrastructure efficiency can be substantial.
Anat points to recent developments from DeepSeek as an example of how quickly thinking around KV cache is evolving. Even as model innovations make individual caches more compact, the potential value of retaining and reusing enormous volumes of cached context continues to grow.
This creates an important role for high-performance data infrastructure. Optimization is no longer confined to what happens inside the GPU. It increasingly depends on how efficiently data and intermediate computation can be stored, accessed, and reused across the broader system.
Inference Is More Than an LLM API Call
Another important theme is the growing complexity of the inference application itself. Enterprise AI applications involve far more than sending prompts to a model and receiving responses. Retrieval-augmented generation, vector search, data synchronization, application logic, and increasingly autonomous agents all contribute to the end-to-end inference pipeline.
Anat explains how the VAST AI Operating System brings these capabilities together through a unified data platform, including the VAST DataEngine and VAST DataSync. Rather than assembling and maintaining a fragmented collection of independent tools, organizations can build and manage more of the inference application around a common infrastructure foundation.
This becomes increasingly important as enterprises move beyond experimentation and begin deploying AI applications at scale, where operational complexity, security, and reliability matter alongside model performance.
When Agents Become the Workload
The rise of agentic AI introduces another architectural challenge. Instead of supporting relatively discrete inference requests, infrastructure must increasingly accommodate fleets of agents, multiple models, persistent context, and potentially enormous numbers of concurrent activities.
For Anat, two requirements stand out: scale and flexibility. Scale is essential as the number of agents and the volume of associated data grow. Flexibility is equally important because the underlying workloads are still changing. Infrastructure designed around one particular model, framework, or optimization technique may become restrictive as new approaches emerge.
VAST’s shared-everything architecture addresses this challenge by supporting diverse workloads on a common platform, from model training and inference to KV cache and other data-intensive AI operations.
The discussion also extends to physical AI, where robotics and autonomous systems introduce additional requirements for collecting, processing, and acting on data from the physical world.
Future-Proofing Infrastructure in an Unpredictable AI Landscape
Perhaps the most difficult question facing AI infrastructure architects is how to plan for the next three to five years when the technology itself may look dramatically different in six months.
Anat’s answer centers on avoiding unnecessary architectural commitments.
That means prioritizing flexible hardware configurations, high-performance networking, open-source technologies, and data infrastructure capable of supporting multiple workloads without requiring specialized, isolated systems for each new application.
It also means designing for growth without assuming today’s models, frameworks, or deployment patterns will remain dominant.
The objective isn’t to predict exactly what AI infrastructure will look like in five years. It’s to create a foundation that can evolve as the requirements become clearer.
Putting Inference Optimization into Practice
Throughout the conversation, Anat emphasizes that effective optimization requires continuous experimentation and validation as inference frameworks and model architectures evolve.
The VAST engineering team regularly tests configurations across open-source inference technologies, including LMCache, SGLang, and NVIDIA Dynamo, and publishes practical guidance for deploying these frameworks with VAST.
For architects looking to explore these implementations, the VAST Data Inference Knowledge Base provides configuration guidance, validated integrations, and regularly updated best practices for building high-performance inference environments.
As AI moves toward more autonomous, distributed, and data-intensive applications, the definition of optimization will continue to expand. The challenge is no longer simply making individual models run faster, but building the infrastructure that allows the entire AI system to operate efficiently today while remaining adaptable for whatever comes next.
