Anyone building an AI platform on Kubernetes might ask why they should bother with remote storage when local NVMe is sitting right there in the node. I's a fair question. Local disk is fast and it's cheap. For a single pod running once, it might even be enough. However, the real question isn't whether local storage is fast.
The real question is whether fast is enough once you're running training jobs that outlive any one node; inference caches shared across a fleet; vector databases that need to survive a failure; and a growing list of teams who all need a slice of the same infrastructure without stepping on each other. That's where remote storage earns its keep, and it's the question this post is answering: Why remote and, specifically, why VAST?
Five questions typically come up when platform teams ask that second question. This post will cover all of them at the platform level, but let’s begin with VAST’s answers to them:
VAST supports storage at a scale that matters once you're talking about more than a handful of volumes - up to millions of block devices on one system.
VAST gives you the choice to share data through NFS or keep it strictly isolated through block, rather than forcing one model on every workload.
VAST makes snapshots and clones cheap enough to use constantly, not just for quarterly backups.
VAST lets a platform team hand a Kubernetes cluster owner a tenant, with its own quotas and its own access, instead of standing up a whole dedicated cluster just to keep two teams apart.
And VAST replicates the way mission-critical, stateful applications actually need it to.
A follow-up post next week will go deep on the sharpest edge case of all this - AI agent sandboxes - where every one of these trends shows up at once, compressed into workloads that live for minutes.
Millions of Volumes, Not a Handful
For a long time, block storage was the quiet cousin in cloud-native circles. File and object got the attention: training pipelines read from enormous shared datasets, and checkpoints piled up in object stores. NFS handled the “many pods, one dataset” pattern well enough that block felt like something reserved for a database that wanted its own disk.
AI workloads changed that math from several directions at once:
Training jobs want throughput that makes yesterday's “big” volumes look small.
Vector databases and feature stores want fast, low-latency access to small chunks of data over and over.
Inference services need to scale horizontally without every replica fighting over the same disk.
Although none of these individually demand millions of volumes, a platform running all of them, and for more than one team, starts approaching that number fast. Scale like that has real operational consequences, not just theoretical ones. Dynamic, ephemeral node pools - autoscaling infrastructure that adds and removes nodes constantly - create host and network state that has to be cleaned up somewhere. And the storage underneath has to be built for it, not stretched to reach it.
Figure 1. Local NVMe is fast and cheap, but it doesn't survive a lost node, doesn't share across pods, and doesn't scale past the box it's sitting in.
We built VAST's Block CSI Driver as general-purpose NVMe/TCP block storage for exactly this kind of scale, supporting up to millions of block devices on one system. We addressed this directly in the block Helm chart: an automatic host-pruning option removes unused VAST Host entries, the NQNs that identify NVMe/TCP initiators, once no PVC on a node references them anymore. The block driver also supports both TCP and RDMA transport depending on what the network underneath can offer, and optional client-side encryption for workloads that need it.
Isolation is the default, but it isn't absolute. VAST's block driver defaults to RWO; one volume, one writer. But it also offers a read-only, multi-node access mode so several pods on different nodes can mount the same block volume read-only. It’s narrower than what NFS offers, but enough that a workload reading a shared dataset doesn't automatically need a different storage system.
Share If You Need To, Don't If You Don't
The next question platform teams hit is about sharing. Some workloads genuinely need it, such as several services reading a common dataset, or a pipeline handing output from one stage to the next. However, many workloads don't, and they pay a real coordination tax when a platform forces file-style sharing semantics onto something that only ever has one reader.
VAST's answer is structural rather than a workaround. Block and file show up as two separate CSI provisioners under one platform, each with its own Helm chart and StorageClass parameters; they’re genuinely separate drivers provisioning against the same underlying VAST cluster.
That separation matters more than it sounds like it should. A platform that only offers block has to bolt on a second system the day someone needs to share a dataset; add a different vendor or a different set of credentials; or has a different failure mode to reason about. A platform that only offers file pays a locking and consistency tax on every workload that never needed to share in the first place. Offering both from the same cluster, with the same admin tooling and the same quotas underneath, means the choice is a StorageClass parameter rather an architecture decision you have to get right on day one and live with for the life of the platform.
Cheap Snapshots and Clones, Used Constantly
Snapshots and clones matter here too, and they're worth calling out on their own because cheap ones change how you use them. A pre-baked environment template can be cloned into a running volume almost instantly instead of cold-initializing from scratch. A shared dataset can be distributed to many consumers by cloning a snapshot rather than copying gigabytes repeatedly. A database backup becomes a point-in-time recovery option instead of a maintenance window.
Figure 2. Once a snapshot is nearly free to take, it stops being a quarterly backup task and becomes a building block for how workloads actually get provisioned.
VAST has supported snapshot creation and listing since our earliest CSI releases; added volume cloning, including cloning directly from a snapshot in read-write mode; and added a setting that lets you choose whether a clone waits for the underlying snapshot to fully complete before you start writing to it. That guarantee matters once clones aren't a rare operation anymore, but are something you're running constantly. What changes at AI platform scale isn't the primitive itself, snapshots and clones are standard, it's the frequency: hundreds or thousands of operations a day, sitting directly in the hot path of how workloads get provisioned rather than tucked into a nightly job.
One Cluster, Many Tenants
Managed Kubernetes — via EKS, GKE, AKS, or OpenShift — has become the default way enterprises run anything serious. That shift changes who is the storage platform's actual customer: it’s no longer just the developer asking for a PVC, but the platform team that owns the cluster and has to hand capacity to a dozen tenants without becoming a bottleneck. You don't want to hand a Kubernetes cluster owner a whole dedicated storage cluster just to keep two teams apart. You want to hand them a tenant.
VAST's answer isn't a slide with the word “multi-tenancy” and a checkmark next to it. We built formal multi-tenancy through StorageClasses, and tenant-scoped authentication that lets a StorageClass authenticate as a tenant administrator instead of a cluster-wide one. Setting up a tenant means creating a Tenant object; a Virtual IP Pool scoped to it; a Tenant Admin role scoped to it; and an API token, which becomes the secret a dedicated StorageClass points to. Once that scaffolding is in place, a QoS policy parameter on the StorageClass ties a named policy to every volume a tenant provisions, and capacity limits come from the same View and quota system VAST uses outside Kubernetes. So a tenant's ceiling is enforced by the storage system itself, not approximated on the Kubernetes side.
Figure 3. In VAST's documented workflow, a cluster admin provisions each tenant once; the tenant's own team then self-serves against its dedicated StorageClass from that point on.
Replication Built for Workloads That Can't Go Down
The fifth question shows up in a meeting when somebody asks what happens if the primary VAST cluster goes down. This matters most for the applications a business actually can't afford to lose: databases, message queues, and anything running inside Kubernetes that qualifies as mission-critical rather than disposable.
This is also where storage choice shapes how database operators in Kubernetes get built. A good operator is stateless: it holds no state of its own, so it can restart or fail over without losing track of anything. Everything that matters lives in the backing volume rather than in the operator's own memory or local disk. A block backend that behaves like local storage, survives a site failure, and replicates on its own is what makes that stateless design actually safe to build on, instead of just deferring the risk to whenever the volume underneath finally lets the operator down.
VAST's CSI Helm charts, block and NFS alike, plug into the CSI-Addons standard, with VAST's own replication CRDs for declaring which StorageClass pairs replicate to which peer and on what schedule. We've also added pod-remap automation within a single Kubernetes cluster stretched across two sites, not two independent clusters, restarting or redirecting affected pods to the new primary automatically the moment a role change happens. It's opt-in, asynchronous, and sized for real scale: the replication configuration caps at ten thousand PVCs per group.
Figure 4. VAST's extensions controller wires the block CSI driver into Kubernetes-native replication CRDs, and can remap running pods to the new primary during a role change.
Why VAST for AI Workloads on Kubernetes
Put these five together and the answer to the original question gets concrete. Local disk is fast and cheap right up until you need it to:
Survive a lost node
Be shared across a team
Hand off to a tenant you don't fully trust with the whole cluster
Clone itself a thousand times a day
Fail over under a workload that can't go down
That's a different bar than the one CSI drivers were originally built to clear, and VAST's own block driver shows how fast it's moving to meet it: host-sprawl protection, QoS-aware storage classes, and pod-remap replication have all landed within a handful of point releases of each other. This is because there's one architecture underneath it, what VAST calls DASE (Disaggregated and Shared Everything), built once and stretched in the direction the workloads were already pulling it.
These five trends show up everywhere on an AI platform, but nowhere more intensely than in agent sandboxes, where volumes get created and destroyed by the thousand, isolation is the default, and none of it can afford to be slow. That's the subject of the next post.
