Rethinking KV Cache Offloading: The Architectural Imperative of Shared vs. Local NVMe Storage

In modern scale-out LLM inference architectures, managing KV cache capacity is a primary driver of throughput and overall cost efficiency, with DeepInfer publishing that KV cache optimizations helped them achieve an overall revenue of $38 per GPU-hour. Modern high-bandwidth networks (400 Gbps today, scaling rapidly to 800 Gbps) and reference architectures like NVIDIA Cloud Partner architecture have eliminated network bandwidth as a primary bottleneck for externalized KV cache traffic. This enables inference architectures to evaluate storage placement based on performance, endurance, operational scale, and cluster efficiency.

While utilizing local Direct-Attached Storage (DAS) SSDs on compute nodes is often considered for the local tier, scale-out production deployments benefit from evaluating disaggregated shared storage over high-speed East-West fabrics.

This blog examines the architectural tradeoffs of Local Flash Offloading versus Disaggregated Shared Storage. We demonstrate how local KV cache offloading creates workload contention, accelerates flash endurance depletion (drive write wear), makes operations brittle and introduces operational fragility through duplicate host writes. Conversely, we show how disaggregated shared-everything storage maximizes flash durability through global wear-leveling, single-write cluster accessibility, and pooled flash endurance, while preserving local node storage for I/O-intensive, non-delegable host tasks such as system logging, stateful container volumes, and crash dumps.

The Anatomy of Tiered KV Cache Offloading

In modern scale-out LLM inference architectures, utilizing KV cache is a primary determinant of cost and throughput. The KV cache hierarchy spans three distinct memory and storage tiers, each serving a critical role in context retention:

• Tier 1: GPU High-Bandwidth Memory (HBM) – The primary execution layer providing ultra-high bandwidth (TB/s) required for real-time token generation. However, HBM is severely capacity-constrained (80–288 GB per GPU) and represents the most expensive per-gigabyte resource in the cluster.
• Tier 2: Host Dynamic RAM (DRAM) – The primary overflow buffer, offering higher capacity per node (typically 512 GB–4 TB) with lower cost per GB (compared to HBM), accessible via PCIe or NVIDIA NVLink-C2C at nanosecond/microsecond latencies for rapid swapping.
• Tier 3: Solid-State Drives (NVMe SSDs) – The persistent/warm spillover tier, offering 10’s of terabyte-scale capacity per node or petabyte-scale capacity in aggregate at a fraction of the cost of DRAM and HBM. Flash storage enables extended context retention across session pauses and prompt prefix reuse.

Intuitively, utilizing local DAS SSDs on compute nodes seems like the ideal offloading target. Local NVMe drives are often perceived as "free" or sunk-cost hardware already present on the server chassis. Furthermore, offloading the KV cache to SSDs serves two essential production goals: reducing latency by avoiding prompt re-computation, and saving money as reusing KV cache tensors saves expensive GPU compute cycles and power, directly reducing the cost per token.

However, in scale-out production inference environments, this local DAS architecture reveals severe operational liabilities. When tens or hundreds of GPU nodes independently write, evict, and duplicate KV cache context prefixes across their local SSDs, the data center suffers from accelerated physical drive wear, and stranded capacity.

Capacity Multipliers and Session Sizing Across the Tiered Hierarchy

To see why flash matters, consider an 8× NVIDIA Blackwell B200 node (1,440 GB HBM3e) serving GLM-5.3-Flash (KV$ in BF16, ~16 KiB compressed KV per token):

• HBM Budget (~800 GB allocated for KV cache per 8× NVIDIA B200 node): Supports ~50.0 million active tokens total, equivalent to ~195 concurrent 256,000-token (256K) context sessions.
• Host DRAM Budget (~1 TB allocated for KV cache): Expands capacity to ~62.5 million tokens (~244 concurrent 256K context sessions).
• NVMe SSD Tier (~30 TB allocated for KV cache): Expands capacity by ~30× over DRAM and ~37.5× over HBM, scaling retention to ~1.95 billion tokens (~7,324 concurrent 256K context sessions).

Agentic workloads routinely push contexts toward 1M tokens. At ~16 GB of KV per 1M-token GLM-5.3 session, an 8× B200 NVIDIA HBM budget (~800 GB for KV) holds only ~50 concurrent agent sessions. Flash is what keeps those histories warm without forcing full prefills.

DeepSeek V4.1 Flash shows extreme compression is possible (on the order of ~890 B/token), but most production models still need large KV footprints. Even DeepInfer retains durable block KV for days at a published ~14.4 TB logical per GPU, a volume that, once adjusted for physical replication and cross-node cache sharing, makes local DAS impractical.  

While flash storage is indispensable for providing the order-of-magnitude capacity expansion required for large-scale session retention, landing this massive KV cache write stream directly onto local compute-node SSDs creates a destructive cycle of drive wear - rapidly consuming a drive's rated Drive Writes Per Day (DWPD), which measures how many times its total capacity can be written daily over its warranty lifetime - and local node instability.

The Pitfalls of Local Storage KV Cache Offloading

Cache Hits Stranded on One Node

A KV cache on local flash only helps if a session's next request lands on the node that holds its prefix. That forces affinity (sticky) routing, which works against load balancing; NVIDIA Dynamo router documentation describes this trade-off, where favoring locality (and cache reuse) improves TTFT, but favoring shared cache (and load balancing) improves inter-token latency (ITL). Shared prefixes such as system prompts and common documents also end up duplicated on every node, wasting KV cache capacity that could otherwise be used to improve latency on more requests. 

These drawbacks are leading many leading AI companies to adopt distributed KV caches so any prefill engine can hit any prefix with no affinity routing at all. NVIDIA, DeepSeek, and Alibaba Cloud have all documented the benefits of shared KV cache at scale.

Storage Wear and Write Amplification Factor (WAF)

Flash has finite Program/Erase cycles. KV offload writes fixed-size serialized tensor blocks continuously as new prefixes arrive and old pages are evicted. That churn, plus unaligned updates and SSD garbage collection, drives a high effective Write Amplification Factor: multiple flash writes for every application byte.

Light-use drives (about 0.8–1.5 DWPD) can burn out in months under this pattern rather than years. Research by the University of Chicago, Samsung, and TensorMesh reaches the same conclusion: estimating that even a 30 TB enterprise SSD can exhaust its rated endurance in under a year. In one inference trace with Kimi-K3, (see links above for references), a naive write-back policy wrote ~42 TB to a 1 TB SSD in two hours and projected only ~3.5 days of device life versus a five-year rating.

Endurance Tied to the Wrong Lifecycle

GPU nodes typically provide five to six years of useful compute. Local SSDs under continuous KV writeback can wear out much sooner, coupling long-lived accelerators to short-lived drives, and when a drive fails, maintenance lands on the GPU server itself. Because local drives are usually striped into a single RAID0 volume, one worn drive takes the entire KV cache with it. If one uses RAID-10, which performs well, that has the unfortunate drawback of reducing usable capacity by a factor of 2, exacerbating the capacity problem further. When affinity-based routing is used to direct queries to the same servers, busy nodes will wear out first while quiet nodes strand unused endurance. The bad choices are swapping drives on GPU servers early, throttling retention, or accepting storage-driven GPU downtime.

Shared storage breaks that coupling: wear is leveled in a pool sized for the write intensity, and GPU refresh stays independent of flash refresh.

No Simple Path to Scale-Out

Local offload also has no clean scale-out path. The sizing math in Section 3.2 puts a busy rack near ~0.25 PB/day and a modest fleet in multi-petabyte territory. Growing local flash means touching every compute node: more drives, more SKUs, more stranded capacity. New model architectures can shift KV footprint per token in either direction, but local drives are fixed at purchase and cannot flex to meet the rapid changes at the frontier of AI. 

Shared storage grows independently: adding capacity to the storage pool happens while serving nodes keep inferencing. Local flash looks simple on day one but compounds in complexity as demands change; multi-day fleet reuse is where shared scale-out stays boring at tens or hundreds of petabytes.

Architectural Solution: Disaggregated Shared Storage

Transitioning KV cache offload target architectures from local drives to disaggregated shared-everything storage using NVMe-over-Fabrics (NVMe-oF) resolves the endurance, duplication, and isolation challenges inherent to local flash.

In practice, a shared tier does three jobs that local DAS cannot deliver on:

• Level write wear across a managed enterprise flash pool sized for the retention window, rather than unevenly burning isolated drives on individual GPU hosts with sticky routing.
• Keep KV state and flash endurance independent of GPU refresh or local drive failure, so compute and storage lifecycles stay decoupled and can flex independently as model requirements change.
• Leave local NVMe free for staged model weights, container images, compile and JIT caches, and other host working state whose access patterns and lifetimes are more modest. 
• Eliminate Prefix affinity routing, which significantly simplifies scheduling policies and routing decisions

The comparison table below summarizes the tradeoffs.

Technical Comparison

How Much Shared KV Do You Actually Need?

Local NVMe in a GPU rack is usually only tens of terabytes (roughly 0.06 to 0.2 PB). This is fine for short spillover, but not enough for multi-hour or multi-day fleet retention.

A practical estimate of needed KV cache capacity uses the same mental model as VAST’s internal KV cache sizer (used with customers) and as DeepInfer in production. This model is governed by two questions: How fast does the rack generate new KV, and how long do you keep it?

To answer these questions, take the same GLM-5.3-Flash workload already sized for HBM in Section 1.1, and put it on a full rack of 8× NVIDIA HGX B200 nodes (64× GPUs) at a conservative fleet-average decode rate of 250 output tokens/s/GPU. Calculating how quickly the rack generates new KV and how much capacity is required is simple:

1. Decode throughput: 64 GPUs × 250 tokens/s/GPU = 16,000 output tokens/s. This is about 32 requests/s at 500 output tokens/requests.

2. KV written per request: With a 95% hit rate on a 100k-token prompt, each request writes the uncached 5% (~5,000 prompt tokens) plus 500 output tokens. This totals ~5,500 KV tokens to store. Across the rack, that is about 176,000 KV tokens/s (32 req/s × 5,500).

3. Capacity required: The capacity needed to store these KV tokens is the rate of KV tokens written × bytes/token × retention duration. Continuing the example, ~176,000 tokens/s × 16 KiB ≈ 2.9 GB/s write rate. Retaining all cached KV over 24 hours requires ~0.25 PB per rack (~3.9 TB/GPU). This capacity scales linearly with both retention time and rack count. For example, retaining KV cache for 3× longer (72 hours needs 3× more capacity (~0.75 PB or ~12 TB/GPU). Scaling to 16 racks (1,024 GPUs) gives ~4 PB for 24 hours, while 100 racks require ~25 PB for the same time.

DeepInfer uses this same ingest × retention model, and their stack lands near ~14.4 TB logical KV per GPU over 72 hours due to higher bytes per token of DeepSeek V4.1-Flash. Note that physical capacity needed is larger to account for erasure coding and over-provisioning.

A Better Architecture for KV Cache

Offloading KV cache to flash is the right strategy for the large contexts that coding agents and other long-context applications generate, but offloading to local host SSDs carries limitations and high operational costs and complexity. Local offload traps each cached prefix on the node that wrote it, wears out drives inside GPU servers faster than those servers are retired, and leaves KV capacity fixed at whatever each node shipped with. Shared storage, as provided by the VAST AI Operating System, makes every cached prefix reachable from any node, spreads wear across a pool sized for the cluster’s aggregate write rate, and lets KV capacity grow independently of the GPU fleet as model needs change. It also leaves local flash free for model weights and other runtime state that suit its locality and endurance better.

While it is tempting to consider building some sort of clustered file system using the local NVMe drives across all the GPU nodes, instead of just using them locally (that would solve the Cache Reachability constraint, for example), the overall issue of endurance still persists. The aggregate endurance gain by increasing the size of the pool comes with an equal increase of the number of GPU nodes using it, thus giving minimal gain for DWDP exhaustion. Similarly,  external storage should also deliver very significant gains in endurance as well, without over provisioning excessively, or using high endurance SLC/MLC drives which are significantly more expensive. VAST AI OS has a unique implementation of a Flash Translation Layer that significantly improves the endurance for the media by an order of magnitude or more by shaping the writes, and managing small writes effectively so as to not reach the limits of the drives. 

This post compared offloading KV cache to local NVMe with offloading it to shared storage, but the VAST AI OS does more than store KV cache. It can serve as a single data platform across an organization’s entire AI pipeline, including training, RAG, inference, analytics, and databases. Running KV cache on the same platform as the rest of the pipeline means it doesn’t become another storage system to buy, secure, and operate, and the same pooling argument that applies to KV cache extends across the pipeline: capacity can shift between KV cache and other workloads as needs change.

Learn more about the VAST AI OS for KV Cache here.

For related technical articles, visit VAST’s Documentation portal on Inference here.