As enterprise AI transitions from simple chat interfaces to complex, long-context agentic workflows, infrastructure requirements for LLM inference have fundamentally shifted. To drive autonomous agents capable of processing entire codebases or thousands of pages of financial data, systems must manage massive amounts of session state, known as the key-value cache (KV cache).
Today, VAST Data and AMD are proud to announce an expanded strategic collaboration that transforms inference efficiency. By integrating the VAST AI Operating System with 6th Generation AMD EPYC™ CPUs, AMD Instinct™ GPUs, and the AMD Helios rack-scale infrastructure, we are enabling persistent, high-performance context management at global scale.
In this blog, we show how the VAST AI OS with a high performance network can achieve 9X speedup in TTFT and 9.7x greater token throughput, for large-scale inference on an 800 Gbps network, using NFSoRDMA with 120K token context length. As context length continuously increases to 1 million context length, the potential gain and speedup from KV cache will be much higher.
The Core Challenge: Avoiding Context Recomputation
In modern inference pipelines, the “Time-to-First-Token” (TTFT) is the primary indicator of responsiveness. During a multi-turn conversation, agent context windows can grow to increasingly large sizes, reaching 128,000 tokens and beyond, which causes the KV cache to grow to between 20-50GB for a single user, depending on model and data type. For a production environment serving hundreds of concurrent agents, the aggregate memory requirement quickly surpasses the local High Bandwidth Memory (HBM) available on even the most advanced GPU clusters. For modern production inference pipelines, it is critical that both TTFT and high concurrency are optimized for efficient and productive agentic computing.
Without an efficient offloading strategy, the system must choose between:
Limited Context: Shortening the agent’s memory, which degrades the quality of the output.
Constant Recomputation: Forcing GPUs to re-process context for every prompt, which spikes TTFT and wastes massive amounts of compute capacity.
Reduced number of active sessions: The number of active sessions that the system can serve with reasonable latency is dependent on the amount of HBM capacity on the GPU.
Solution, Testing, and Setup
To eliminate wasteful compute processing, VAST and AMD engineering teams tested and documented standard recomputation against VAST-optimized KV cache offloading at a 128K token context length to determine the benefits of avoiding recomputation.
The testing was done in a VAST Data lab. Our setup was built with the following:
GPUs: The testing environment utilizes an AMD MI355X node.
Network:
The host is equipped with AMD Infinity Fabric 800Gb NICs which connects to a Mellanox MSN4700 switch.
The network configuration for VAST storage is NFSv3 over RDMA
Storage: The team tested two primary configurations for KV cache offloading:
Local host: offloading to local host RAM
Remote VAST Data storage: Offloading to a VAST storage partition via NFSv3 over RDMA, with NFS multi path
Software:
vLLM: Version 0.22.1.with LMCache
AMD ROCm: open foundation with programming models, compilers, libraries, runtimes and deployment tools
Model: GPT-oss 120B, TP-1
We experimented with a tp=1 setup, across various concurrency levels from 1 to 300, tuning system configuration to increase performance, following VAST guidelines, VAST Quick NFS Read Ahead Tuning, to increase performance:
nconnect: Set to 32
Read-ahead increased to 8192
LMCache configuration: LMCache .yaml file:
chunk_size: 8192
save_decode_cache: false
local_cpu: false
Enable_async_loading: true
max_local_cpu_size: 300
remote_storage_plugins: ["fs"]
extra_config:
remote_storage_plugin.fs.base_path: "/shared/data/KV/gpu0"
remote_storage_plugin.fs.use_odirect: true
remote_storage_plugin.fs.read_ahead_size: 16777216
save_chunk_meta: false
Results
Time to First Token (TTFT) was tested with and without KV Cache offload to VAST remote storage (using LMCache).
As seen in the diagram above, a system with an AMD Instinct MI355X GPU and the VAST AI OS, on an 800 Gbps NFS/RDMA network demonstrated 9X speedup in time-to-first-token (TTFT)
and 9.7X greater token throughput for high concurrency agentic AI workloads. Results were measured by comparing the scenario where (a) VAST-optimized KV cache offloading was used, and (b) no KV cache offloading was used, requiring recomputation at the start of each decode turn. The host platform was an AMD MI355X.
These accelerated and high concurrency results are critical to deliver low latency, lower cost and energy efficient inference deployments.
Note on baselines and results:
Because speedups are always relative to the underlying hardware, running these workloads on a GPU baseline with lower compute capacity may yield a higher relative speedup compared to the latency of reading directly from storage.
All performance and/or cost savings claims are provided by VAST and have been reviewed, but not independently verified by AMD. Performance and cost benefits are impacted by a variety of variables. Results herein are specific to VAST and may not be typical.
Why This Matters for Production AI
Offloading KV cache to the VAST data platform does more than just deliver faster TTFT and accelerate agentic computing, it maximizes GPU efficiency, ensuring organizations can maximize the return on their compute investments. By combining VAST’s enterprise data services with AMD Instinct™ accelerators, organizations can break the limitations of localized GPU memory to run production-scale inference efficiently, reliably, and at scale. This powerful integration ensures that massive AI services run smoothly and securely, without the need for complex third-party software or disruptive maintenance windows.
Key benefits for production inference include:
Predictable scaling: As the number of concurrent sessions increase to overwhelming sizes with agentic workflows, VAST ensures that inference latency remains perfectly stable, eliminating the exponential performance degradation typically caused by context recomputation.
Maximized GPU ROI: Freeing up precious High Bandwidth Memory (HBM) allows for much larger batch sizes and drives significantly higher utilization of your underlying AMD Instinct hardware.
Unified context space: Leveraging VAST’s shared global storage tier enables seamless load balancing across massive GPU factories, allowing any compute node to instantly pull from a unified context pool.
Streamlined AI pipelines: Engineered-in data services enable organizations to build, deploy, and scale end-to-end AI pipelines natively, eliminating the complexity of bolted-on third-party software.
Continuous operations and zero-trust security: A comprehensive zero-trust security framework — paired with the ability to dynamically load and manage new, larger models without downtime — ensures your AI services remain secure, compliant, and always-on.
Looking Ahead
This collaboration is just the beginning. We are excited to showcase these results at the AMD Advancing AI 2026 Conference in San Francisco this week. Join us to see these results in action and learn how we are building the blueprint for the next generation of AI factories.
