How Inference Is Forcing a Rethink of AI Infrastructure
When it comes to large-scale systems for AI, the engineering focus has shifted from training to inference where the performance bottlenecks look very different and where infrastructure decisions increasingly determine everything from user experience and economics.
That shift is at the heart of the most recent Shared Everything podcast episode with Stephen Bates, AMD Fellow in the AI Group, and Alon Horev, Co-founder and CTO of VAST Data.
Using AMD and VAST's expanded partnership as a jumping-off point, the conversation explores how inference workloads, particularly agentic AI, are changing assumptions about storage, networking, software, and system architecture.
Bates describes the current AI market as a shift happening inside an even bigger shift. The industry is moving from building models to running them, but even the way we run models is changing quickly. Early chatbots usually worked with relatively short prompts. Today's AI agents are expected to read entire documents, search code repositories, pull information from databases, and combine all of that before producing an answer. That means models are working with vastly more context than they were even a year ago, creating new performance bottlenecks that simply didn't exist before.
The result is that inference is no longer just about having the fastest GPU. It's about how every part of the system works together. Bates breaks inference into two stages.
As prompts become much larger, that first stage takes significantly longer, increasing the time it takes before users see the first word of an answer. Infrastructure that worked well only months ago is already struggling because today's workloads are so different.
That change is forcing people to rethink the role of storage. During training, storage mostly feeds data into GPUs and saves checkpoints in the background. During inference, storage suddenly becomes part of the live request itself. Horev explains that KV cache now sits directly in the path between a user's prompt and the model's response, which means storage has to be treated as part of the AI system rather than simply somewhere data lives.
Designing AI infrastructure now means thinking about compute, storage, networking, and software as one architecture instead of separate pieces.
The industry is also changing how it thinks about KV cache. Instead of leaving that information inside the memory of a single GPU, the goal is to make it available across many GPUs without slowing anything down. That sounds straightforward, but it creates a difficult engineering challenge. The more systems that need to share the data, the more it has to travel across the network. At the same time, inference demands extremely low latency. Balancing those two requirements is becoming one of the biggest design problems in AI infrastructure.
Networking therefore becomes just as important as storage. Moving KV cache outside GPU memory only works if information can move between storage and the GPUs fast enough that users never notice the extra trip. That requires fast storage, high-performance networking, and software that minimizes unnecessary data movement.
Bates also points out that if more of this data can move directly between storage and GPUs, organizations may be able to reduce how much expensive server memory they need, lowering the cost of very large AI clusters.
Horev says this also changes how AI clusters are built. It's no longer enough to plug storage into an existing network. Organizations have to design networks around the workloads they expect to run, how many users they'll support, and how data will move across thousands of GPUs. The network is becoming an active part of AI performance instead of simply connecting the hardware together.
The discussion also touches on the growing role of open software. AMD is contributing to projects such as ROCm, vLLM, and LMCache because optimizing inference is becoming too complex for any one company to solve alone. Working in the open allows hardware vendors, software developers, and customers to improve the same code, adopt new techniques more quickly, and keep pace with how fast AI infrastructure is evolving.
The larger message from the conversation is that AI performance is no longer determined by GPUs alone. Storage, networking, software, and memory all play a direct role in how quickly models respond and how efficiently expensive hardware is used.
As AI agents handle larger amounts of information and longer conversations, building better AI systems will increasingly depend on designing the entire infrastructure stack to work together.
