Native Compute for DataEngine
Modern data platforms increasingly need to process data continuously, not just store it. The VAST DataEngine is a serverless compute framework for running containerized processing directly against data stored in VAST. Serverless functions can be triggered automatically by events, file arrivals, or scheduled workflows.
The problem is that traditional AI pipelines are slowed by the cumulative cost of moving information between fragmented services like S3, Kafka, data warehouses, databases, and vector stores. Every hop adds latency, operational overhead, and more things that can break.
DataEngine brings files, tables, event streams, and vector data into the same execution environment. This reduces data movement and allows real-time processing against live operational data.
With VAST 5.5, we are taking the next step. We are introducing VAST Native Compute, a managed compute layer built directly into the DataEngine, so compute and data operate under the same management layer.
How DataEngine Works
DataEngine runs containerized functions directly against data stored in VAST. When new data arrives, events occur, or batch jobs complete, DataEngine can automatically process and respond without requiring separate Kubernetes infrastructure or orchestration tools. The architectural difference is that VAST brings together data services that most organizations still run as separate systems:
Files and objects for unstructured data
Tables and metadata for structured data
Event streams for real-time, event-driven processing
Vector embeddings for AI and semantic search workloads
Because compute runs alongside all of these services, data does not need to move between external systems, reducing latency and operational overhead.
Common DataEngine Workloads
DataEngine is already being used across industries to automate workloads that previously required bespoke infrastructure. Here is a snapshot of what is possible:
Use Case | Description |
|---|---|
Video Search & Summarization | Automatically indexes video content to generate highlights and searchable timestamps, making large media libraries instantly navigable. |
Auto Classification & Labeling | Uses AI to tag images and documents at scale, eliminating the bottleneck of manual data entry across high-volume pipelines. |
PII Detection | Scans files for sensitive information including SSNs and payment data to enforce privacy compliance automatically on ingest. |
RAG (AI Chatbots) | Connects LLMs directly to your private data so responses are grounded in your company's actual knowledge, not general training data. |
Data Preprocessing | Cleans, transforms, and prepares raw data for machine learning models. |
Automatically pulls in and organizes logs, Parquet, and CSV files, making them immediately available for analysis. | |
Medical Image Analysis | Identifies patterns or anomalies in X-rays and MRIs to assist clinical teams with faster, more consistent diagnostic support. |
Fraud Detection | Monitors transactions in real time to flag suspicious or anomalous behavior before it becomes a costly incident. |
Genomic RAG | Applies AI-powered reasoning to multi-terabyte biological datasets, enabling queries that were previously impractical at that scale. |
Deployment Options: BYOC or VAST Native Compute
Running DataEngine in production means making a choice about how compute is managed.
For organizations with existing hardware investments, Bring Your Own Compute (BYOC) lets you leverage that infrastructure directly while using the VAST control plane for data management and pipeline orchestration. Organizations retain control of the underlying infrastructure while VAST manages the orchestration and data services layer.
For organizations that want a more hands-off experience, VAST Native Compute removes the need to provision or maintain a separate Kubernetes cluster entirely.
Some organizations want full control over their compute environment, while others want a managed experience. VAST supports both approaches.
Why Teams Choose Managed Compute
That said, self-managed Kubernetes carries well-documented operational complexity. Most teams feel it in three ways: infrastructure silos that require separate teams to manage storage and compute with no unified view; complex networking that demands specialist knowledge to optimize throughput between the data and compute planes; and a shortage of Kubernetes expertise that leaves that burden concentrated on a small number of people.
Most teams do not want to operate Kubernetes clusters. They are trying to deploy data and inference workflows faster. VAST 5.5 is built around that priority.
The Operational Burden: Day 1 vs. Day 2
The Kubernetes "Day" framework describes the lifecycle of infrastructure from initial design through long-term operations. Day 0 covers governance and infrastructure strategy. Day 1 covers the initial deployment: networking, security, pod configuration, and the deployment of operators. Day 2 is everything that follows, and it never really ends.
Day 2 is where the complexity compounds. Manual Kubernetes upgrades, persistent storage backups, fragmented monitoring across separate systems, certificate rotations, access control drift. These are not edge cases. They are the standard operational reality for any team running self-managed Kubernetes at scale.
Many organizations grapple with three specific pressures as a result:
Infrastructure silos, where storage and compute are managed separately.
Complex networking between compute and data planes.
Limited in-house Kubernetes expertise.
The result is a shift in where engineering time actually goes. Instead of accelerating AI pipelines and data initiatives, teams end up managing infrastructure. VAST 5.5 is designed to change that.
VAST's Solution: VAST Native Compute
To address these challenges, VAST 5.5 introduces VAST Native Compute as a core component of the VAST DataEngine.
VAST Native Compute reduces the operational burden of running and securing separate Kubernetes clusters for DataEngine functions. By bringing compute, storage, streaming, and database services into the same operational layer, VAST enables enterprises to build and run production AI pipelines faster, with fewer moving parts, stronger governance, and simpler lifecycle management than fragmented "data here, compute there" architectures.
Self-Managed vs. VAST Native Compute
Control Plane | You manage etcd membership, API servers, and schedulers across at least 3 nodes for HA. | Fully offloaded. VAST handles the control plane. You focus on consuming the API, not keeping it alive. |
Networking | Manual CNI configuration (Calico/Cilium), Load Balancer setup, and IPAM management. | Unified VIP Pools and high-bandwidth connectivity between the compute and data plane, out of the box. |
Upgrades | Manual version jumps (e.g., 1.29 to 1.30) and OS patching carry high breakage risk. | Automated rolling updates for both the Kubernetes version and the underlying VAST OS data services. |
Disaster Recovery | Requires external tooling and manual testing of etcd restores and volume snapshots. | Built-in, using VAST as the persistent layer for native backups and high-speed recovery. |
Security | Manual RBAC, TLS certificate rotation, and OS-level security patching on your team. | Security is built-in, and aligned to STIG and CIS standards, alongside automated certificate rotation and continuous security patching to keep your operations secure and compliant. Tenant-level isolation, unified audit logs. |
Serverless Functions | You deploy, upgrade, and monitor operators, functions, and pipelines yourself. | Pre-deployed and continuously monitored by the VAST control plane. |
VAST Native Compute handles Kubernetes lifecycle management behind the DataEngine control plane. It provides a unified management layer that handles the lifecycle of multiple clusters automatically, so teams can focus on their pipelines rather than their infrastructure.
Deployment Topologies: Edge to Core
Edge and Core Deployments
VAST Native Compute is designed to operate across a range of deployment environments, from distributed edge locations to large-scale core data centers.
Edge: Processing data where it is generated, at IoT hubs, remote labs, or distributed facilities, reduces backhaul costs and keeps latency low for time-sensitive workloads.
Core: Massive-scale aggregation and heavy processing in the primary data center, where dedicated compute profiles unlock the full performance of AI and analytics pipelines.
Multi-Tenancy Models
VAST Native Compute supports two approaches to multi-tenancy, depending on your security and isolation requirements:
Single Cluster (Namespace Isolation): Best for internal teams sharing resources; efficient but shared.
Multiple Clusters (Physical Isolation): For high-security environments or distinct external tenants where network and resource "noisy neighbor" effects must be zero.
The VAST control plane manages multiple tenants independently. Each tenant can receive its own Kubernetes cluster and its own set of security permissions, with lifecycle management handled centrally.
Sizing and Scalability
As the cluster scales from dozens to hundreds of workers, the control plane must adapt to handle the increased scheduling and API pressure.
The architecture scales from small deployments to large multi-cluster environments, starting with a baseline for stability and scaling into a high-availability configuration as the node count increases.
Minimum Baseline: 3 CNodes (combined Master and Worker roles). For lightweight workloads the CNodes can be shared and colocated with the VAST Datastore.
High Availability (HA) Threshold: Clusters exceeding 5 nodes will transition to a 5-master+worker configuration to ensure K8s control plane stability.
Scaling Flexibility: The architecture supports bi-directional scaling. You can add or remove nodes as needed, including scaling a cluster back to the 5-node threshold if resource demands decrease.
Scaling Strategies
Strategy | Description | Best For |
|---|---|---|
Colocated CNodes | Compute and storage share the same physical hardware. | Resource efficiency and small-to-medium footprints. |
Dedicated CNodes | High-performance, compute-only nodes. | Large AI/ML workloads requiring maximum CPU and RAM throughput. |
Selecting the correct CNode profile is critical for performance tuning, especially for data-heavy operations.
For production-grade environments, separation of concerns between compute and storage is strongly recommended to prevent resource contention.
Edge and POC deployments: Use the Low Compute profile as the standard entry point.
Heavy processing workloads: For DataEngine, analytics, and RAG pipelines, use Dedicated CNodes with the Pure Compute or Memory Heavy profile.
Protocol management: Dedicated CNodes should also be used for protocol workloads, to isolate communication overhead from heavy processing logic.
Proile | Compute/Storage Ratio | Primary Use Case |
|---|---|---|
Low Compute | 25% | Edge deployments, POCs, and lightweight services. |
Balanced | 50% | General-purpose workloads requiring steady I/O and CPU. |
Memory Heavy | High RAM | Data Analytics and RAG pipelines (min. 2 CNodes per 8 DBoxes). |
High Compute | High CPU | DataEngine and classification tasks (min. 2 CNodes per 8 DBoxes). |
For Memory Heavy and Pure Compute profiles, maintain a minimum ratio of 2 environment-only CNodes for every 8 DBoxes. This ensures processing overhead does not compete with the storage I/O that feeds it.
The shift to 5 masters at the 5-node threshold ensures the cluster remains resilient even if multiple master nodes experience issues during high-scale operations.
Ecosystem & DevOps Integration
Automation and Infrastructure as Code
VAST integrates with Terraform to automate the provisioning of VAST Native Compute clusters and DataEngine as infrastructure as code, enabling consistent, repeatable deployments across all environments.
Observability
Native Prometheus integration provides real-time visibility into function performance and cluster health across both the compute and data plane, without requiring a separate monitoring stack.
Programmability
A REST API gives developers and DevOps engineers the ability to trigger deployments or scale resources programmatically, making DataEngine easy to integrate into existing CI/CD workflows.
Roadmap: What Comes After 5.5
VAST Native Compute is the foundation. The capabilities available on top of it in upcoming releases include:
SyncEngine: Moving data from external sources and across geographic regions with the same containerized simplicity as local processing. (Available now!)
AgentEngine: Autonomous processing where agents observe data patterns and take action in real time.
GPU Nodes (Technical Preview): Hardware-accelerated CNode-X worker nodes that will unlock generative AI and deep learning workloads directly within the VAST ecosystem, eliminating data movement to an external GPU cluster.
Why Native Compute Matters
DataEngine collapses the separation between where data is stored and where it is processed.
With VAST 5.5 and VAST Native Compute, storage, database, vector search, event streaming, and managed compute all operate under a single model. The result is fewer infrastructure layers to manage and a simpler operational model for large-scale data and inference workloads.
If you want to see what this looks like for your environment, reach out to your VAST account team or visit vastdata.com to request a demo. You also can join our VAST AI OS 5.5 webinar next week to dig in deeper.
