The Infrastructure Challenge of the Agentic Era
As the use of LLMs evolves from simple prompts into complex, multi-turn "agentic" workflows, the underlying infrastructure must adapt. AI-native organizations and neoclouds now face a daunting task: supporting context windows that extend to millions of tokens while maintaining strict SLAs, as well as provide important data services including secure multi-tenancy, data reduction, and encryption both at rest and on transit.The primary barrier to this scale is the GPU HBM (high bandwidth memory), which is expensive and limited to holding the large, dynamic KV cache required for long context agentic processing. Every new turn, tool-call, and promptrequires the model to re-process the entire conversation history, wasting critical GPU cycles on "prefill" for data it has already seen. This recomputation slows Time-to-First-Token (TTFT), degrading user experience and impacting revenue streams.
Why the KV Cache is Critical
Building the KV cache is compute-heavy, and it gets heavier as context grows - prefill scales with input length, and attention cost grows faster still. Being able to cache that work once and reuse it is enormously valuable: across the turns of a single conversation, across an agent’s tool-calling loop, or across users who share a common system prompt or retrieved document.
The catch is where it lives. GPU HBM is fast but small and expensive, so it can’t hold long histories for many concurrent sessions. When a session pauses, the cache is typically evicted and the next request pays a full cold-start prefill. To support long-context workloads like AI-assisted coding or large-document analysis, you need a way to park that memory cheaply and retrieve it at speed.
One point worth keeping in mind: KV reuse only ever helps the prefill stage (and therefore TTFT). Prefill processes the input tokens, which can recur across requests, so their KV can be computed once, stored, and retrieved later. Decode is fundamentally different: it generates new tokens one at a time, and each token’s K/V has never existed before this request. It is new data, so there is nothing in any cache to retrieve - and decode is bound by memory bandwidth, not compute.
Benefits of KV cache offload are therefore about reaching the first token faster and reclaiming prefill compute, not about speeding up the token-by-token generation that follows.
Solving this requires a coordinated strategy across three pillars:
Software frameworks - KV cache managers like NVIDIA Dynamo KVBM, LMCache, or SGLang (with HiCache) that decide what to keep, when to load it, and how to match it.
Hardware architecture - a fast fabric (RDMA / RoCE, GPUDirect Storage) so any GPU can reach any KV block at line rate, plus the right single-node vs. multi-node and disaggregated-serving topology.
A persistent, shared data platform - the tier that actually stores the cache so it survives evictions and is reachable across the whole cluster. This is the pillar most teams under-plan, and it’s where VAST fits.
Increasingly, these layers are becoming more modular: cache managers such as LMCache or Dynamo KVBM decide what KV state should be stored or retrieved, while data transport frameworks such as NVIDIA NIXL optimize how that data moves between GPUs, memory tiers, and shared storage.
Deciding on Software Frameworks
The right framework depends on various considerations, including:
Existing and projected hardware - including heterogeneous fleets: mixing vendors or new and old GPUs across prefill/decode stages)
Scale and topology - single vs. multi-node and aggregated vs. disaggregated serving
Workload pattern - are inputs/outputs long and reuse-heavy? Do sessions resume hours or days later?
Model-coverage needs - Do you adopt new models the day they drop, or prioritize validated, stable workloads?
All of these frameworks converge on the same memory hierarchy and NVIDIA developed names for each layer: GPU HBM at the top (G1 layer), then host DRAM (G2), then local SSD (G3), then networked storage (G4). The top tiers are private to an instance and the G4 tier is shared across the entire cluster. At the G1 layer, GPU memory is the most expensive and offers the highest in performance. At the G4 layer, shared storage is the least expensive and relatively slowest in performance. In this storage hierarchy diagram, note a new G3.5 layer has been added. More on that in the section on NVIDIA CMX.
Hardware Architecture Options
Key variables include bandwidth and scale. What GPU-to-GPU and GPU-to-storage bandwidth do you have? VAST testing has found that most KV offload operations are network-limited, not storage-limited - so the fabric (RDMA/RoCE, GPUDirect Storage, and accelerators like NVIDIA Spectrum-X) matters more than raw disk speed. For mass-scale deployments, dedicated context-memory infrastructure (e.g., NVIDIA’s upcoming CMX reference design) becomes relevant.
Which infrastructure do you actually need? Walk this from the top - each “no” is the lightest setup that works for that case, and following the “yes” path adds what cross-node reuse and scale demand:
All the lower outcomes run on VAST. Loading KV from shared storage beats recompute even over NFS-over-TCP, which can saturate a GPU’s I/O on most networks; RDMA/RoCE with GPUDirect Storage is an optimization that lowers latency, frees host CPU, and loads KV straight into GPU memory; mass scale then layers on a dedicated context tier, KV-aware routing, and disaggregated prefill/decode.
Challenges of Sizing of KV$
Conventional storage sizing methodologies rely on a predictable workload as the primary input. In the absence of this data, the entire framework collapses, and attempting to force a precise calculation often leads to incorrect outcomes.This is not a failure of planning, but rather an intrinsic challenge of deploying AI infrastructure before specific workloads are fully understood. Acknowledging this uncertainty is the critical first step toward building a defensible infrastructure strategy that can scale with evolving demands. This is important in order to support various challenges and requirements such as:
A fundamental requirement for modern AI infrastructure is the selection of a flexible architecture capable of scaling alongside evolving workloads. Relying on rigid, single-purpose solutions presents significant risks throughout the operational lifespan of a GPU investment. Constructing a defensible infrastructure strategy allows organizations to effectively address key operational challenges, including:
Seamless Scalability: Facilitating mid-life expansions and upgrades without incurring performance degradation or system downtime.
Heterogeneous Support: Enabling the integration of diverse server types, storage media, and multiple technology generations within a unified cluster.
Operational Resilience: Ensuring non-disruptive firmware and software updates, alongside the ability to perform network infrastructure upgrades without service interruption.
The Data Platform: VAST as the Shared KV Cache Tier
This is the third pillar. VAST does not replace your inference framework; it sits underneath all of them as the same persistent, shared bottom tier they offload to. The framework owns the cache logic; VAST owns persistence, capacity, cluster-wide sharing, and the data-platform services a raw cache silo can’t provide.
A production-grade KV tier has to deliver:
Expandable capacity at scale. KV cache is gigabytes per prompt; across thousands of users and resumable sessions it reaches petabytes. The tier must grow elastically without re-architecting.
Line-rate bandwidth, low latency. Because offload is network-limited, the tier must saturate the GPU’s I/O. Over RDMA/RoCE with GPUDirect Storage, VAST loads KV blocks directly into GPU memory. In testing, saturating a single H100 at ~25 GB/s, confirming storage was not the bottleneck.
Truly shared, global access. Any GPU or node reads any KV block, so cache computed on one prefill worker can be reused by a decode worker or a different replica entirely. This is the difference between a private per-node tier (L2) and a shared cluster tier (L3/G4).
Persistence. The cache survives session pauses, evictions, and server restarts - eliminating cold starts across hours or days, not just within one process.
Enterprise data services. Multi-tenancy and isolation, encryption at rest and in flight, audit and governance, and data reduction (deduplicating shared system prompts and common context). And it’s the same platform that already holds your training data, RAG corpora, and checkpoints with no separate context silo.
Framework independence. VAST attaches under KVBM (as a G4 backend via NVIDIA NIXL and the GPUDirect Storage plugin), under SGLang HiCache (as an L3 backend), and under LMCache (as its remote backend). You’re not locked into one inference stack.
Where NVIDIA CMX fits
NVIDIA CMX, based on NVIDIA STX foundation for AI-native storage, establishes a new “G3.5” tier in the KV hierarchy: a pod-level, Ethernet-attached flash tier for ephemeral, latency-sensitive KV cache, driven by the BlueField-4 storage processor and orchestrated by Dynamo, NIXL, and DOCA over Spectrum-X. It sits between node-local SSD (G3) and durable shared storage (G4), filling the KV capacity that G1–G3 can’t hold, so a pod’s agents share and reuse evolving context at near-memory latency without exhausting HBM.
NVIDIA’s DOCA interfaces let storage partners integrate as the CMX layer directly. VAST is built to be integrated as the G3.5 tier itself, not just the G4 layer beneath it. VAST support for the G3.5 tier is landing in upcoming Dynamo/VAST releases. Integration with BlueField-4 DPUs enables unencumbered performance with zero-copy KV transfer from storage straight into GPU memory.
The VAST AI OS also provides security, multi-tenancy, and lifecycle management into the context tier, while reducing the overall compute overhead by utilizing the BF4 compute resources. Because the same platform also spans G3 and G4, VAST can serve the hot, pod-level context tier and the durable, cross-pod persistence layer (inactive multi-turn KV state, query history, logs) as one system - with Dynamo and NIXL prestaging blocks up the hierarchy ahead of decode - rather than VAST being relegated to slow storage below CMX.
VAST Data: The Unified Platform for AI Data AND KV Cache
VAST Data has partnered closely with NVIDIA on NVIDIA Dynamo to transform KV Cache from a temporary byproduct into a long-term enterprise asset. By leveraging the VAST DASE architecture, VAST enables GPUs to offload context memory to a shared pool via NFS over RDMA (and soon S3 over RDMA). The VAST Advantage Includes:
90% Reduction in GPU Compute Time: By retrieving stored KV blocks, GPUs bypass the prefill stage entirely.
20x Faster TTFT: Context is made available at network line rate, ensuring near-instant response times even for multi-million token windows.
Infrastructure Efficiency: VAST tested and achieved 1.4x data reduction on KV Cache data, which allows providers to support more concurrent users and longer sessions with the same storage footprint.
Unmatched Security: Unlike fragmented cache silos, VAST provides at-rest/on-the-wire encryption, multi-tenancy, and audit trails for regulated environments.
Sizing Your Inference Pipeline
Determining the appropriate KV cache size for inference deployment requires a careful evaluation of several key variables:
Model Architecture and Specification: The specific models being served dictate GPU selection, weight-related memory demands, required context length, and the KV bytes needed per token. The size of KV blocks is directly related to model architecture.
Workload Distribution: This is the relative percentage of sessions assigned to different use cases (translation, coding, agentic workflows, etc.), each of which carries unique context length demands. Simple text chat workloads generate less KV cache than multimodal agentic workloads.
Concurrency Requirements: This is the total volume of active sessions the infrastructure must support during peak load periods. More active sessions will require more KV cache offloads and reloads from a fixed pool of GPUs.
Retention Strategies: Policies defining the duration a session remains available for rapid retrieval, which helps minimize prefill penalties and improve GPU utilization.
Anticipated Outcomes: Precise sizing enables organizations to project critical metrics, including TTFT acceleration, GPU-hour savings via minimized recomputation, expected cache hit rates, and the total storage needed for target retention periods.
To streamline production-ready capacity planning, VAST Data experts can walk you through a KV$ sizing analysis, designed to convert these operational variables into specific infrastructure recommendations:
Core Inputs: The analysis evaluates model architecture, anticipated workload mix (e.g., document Q&A or coding assistants), and GPU counts to determine L1 tier (HBM) availability and offloading requirements.
Primary Outputs: The analysis generates projections for average TTFT gains over standard recomputation, recovered GPU cycles, hit rate percentages for token retrieval, and the storage capacity essential for meeting retention objectives.
Simplify the Complex with VAST
Scaling inference does not have to be a source of architectural complexity. VAST provides a single, unified platform that handles every stage of the AI pipeline without requiring separate silos for each. Because all data lives in one place VAST ensures your model context is persistent across sessions and users across your GPU estate without compromising on security and governance.
Stop wasting GPU cycles on recomputation. Reach out to the experts at VAST to learn how to scale your inference workloads with the performance and efficiency demanded by the next generation of AI.
Learn more
Watch our VAST Forward 2026 breakout session, Breaking the GPU Memory Wall, with NVIDIA’s Dr. Vikram Sharma Mailthody and VAST’s Anat Heilper.
