Rethinking Event Streaming TCO with VAST on Cloud

Real-time data streaming architectures have historically operated as isolated infrastructure silos within data center and cloud environments. Event brokers, stream-processing engines, and lakehouses are maintained as separate distributed systems. As streaming workloads scale to gigabytes per second (GB/s), retention extends across multiple days, and multi-consumer fanout increases read and egress volumes, costs rise and complexity increases.

Managed streaming platforms, such as Confluent Cloud, reduce cluster management overhead but do not eliminate core architectural and cost tradeoffs. These include multi-availability zones event replication, separate streaming and analytical infrastructure, and usage-based pricing tied to throughput, retention, data retrieval, and multi-consumer fanout.

VAST on Cloud (VoC) presents an alternative by incorporating the VAST Event Broker into the VAST AI Operating System running inside storage-optimized virtual machines as virtual EBoxes (Everything Boxes).

Figure 1. VAST on Cloud

Since the VAST Event Broker uses the Kafka protocol, it works with client libraries, APIs, and tools already in place. Message topics are stored natively as tables within VAST DataBase, replacing broker-managed log replication with VAST’s data protection mechanisms, including erasure coding and Cloud-HA.

Additionally, VAST enables blob expansion, JSON payloads are turned into real, queryable columns as they land (Read more on how Kafka messages arrive already queryable). Analytical query engines, including Trino, Presto, and Apache Spark, can query these topic tables directly using native VAST connectors without requiring ETL pipelines or duplicating data into a separate analytical lakehouse.

Architectural Foundations: Shared-Nothing Streaming vs. Disaggregated Shared-Everything

Before we dive into a detailed cost analysis, it is important to understand the architectural foundation that these solutions are built upon.

Shared-Nothing Streaming

In standard Apache Kafka clusters, individual brokers coordinate partition leadership, client requests, and local log state. When a producer writes records to a partition, the partition leader broker receives the payload, writes it to local append-only log segments on attached storage, and coordinates network replication to follower brokers residing in distinct AZs to satisfy a replication factor of three (RF 3). Follower brokers continuously issue fetch requests across the internal network, consuming broker memory, CPU cycles, and east-west network bandwidth.

This stateful design adds complexity to cluster management. When cluster nodes fail, or when ingestion rates expand beyond broker thresholds, partitions must be systematically redistributed across the cluster. This partition reassignment requires copying historical and active log segments across the network, generating inter-broker traffic that contends with producer writes and client reads.

Managed cloud-native Kafka services have reduced some of these constraints. Confluent’s Kora engine uses local NVMe storage for active log segments and replicates active data across AZs before asynchronously offloading closed segments to cloud object storage (such as Amazon S3) for long-term retention. Simultaneously, it automates active partition reads, writes, and leadership.

However, it retains Kafka's partition-centric execution model, making partition counts, broker throughput, replication, and workload distribution important considerations as streaming workloads scale.

Disaggregated Shared-Everything Architecture

VAST on Cloud uses VAST’s Disaggregated Shared-Everything architecture. In the virtual EBox deployment model, VAST combines the CNode data services and the DNode storage functions into a single virtual machine instance.

Each virtual EBox runs three VAST containers (Figure 2). The CNode container manages cluster metadata and serves incoming data requests across the environment. Working alongside it, the two DNode containers make the VM’s locally attached SSDs available across the NVMe fabric. Because VAST uses a shared-everything architecture, every CNode container can directly access the cluster’s shared backend data.

Figure 2. Virtual EBox

Unlike traditional Kafka brokers, VAST Event Broker does not persist separate partition-log replicas or topic segments on individual broker nodes. Producer requests enter through a dedicated pool of virtual IP addresses (VIPs) distributed across the EBox VMs. Instead of appending records to broker-local log files, VAST inserts event messages as rows within the topic’s VAST DataBase table. Each record’s Kafka offset serves as its row ID, so per-partition ordering is guaranteed by construction, and the message key, value, headers, and timestamp are all preserved as columns.

To scale broker capacity as demand increases, you simply add more EBox VMs to the VAST cluster. Because VAST Event Broker decouples partition storage from individual CNode containers, adding cluster resources does not require Kafka-style migration of partition logs or redistribution of replicated topic data between brokers. Partition leaders can move between CNode containers while the underlying topic data remains accessible through VAST’s shared data architecture.

Data Protection: Multi-Copy Replication vs. VAST on Cloud

Kafka-based streaming platforms frequently protect event data through replication, where partitions are duplicated among nodes across AZs. Those designs lean on complex replication and consensus mechanisms, which generate heavy network traffic between nodes, and use a replication factor of three, resulting in 200% data capacity overhead.

Figure 3. Replication Factor 3

VAST on Cloud takes a different approach. VAST AI OS protects data stored on local NVMe drives through erasure coding, rather than maintaining three full copies of each event log. Data is striped across EBoxes in the cluster, providing continuous data availability in the event of a disk or node failure. This requires between 2.7% and 20% parity overhead depending on the number of VMs deployed and resiliency type, compared to 200% for three-way replication.

For deployments in AWS and Microsoft Azure, Cloud-HA adds a persistent cloud-storage layer beyond the local NVMe infrastructure. Metadata and write buffers are synchronously persisted to attachable block storage, while newly written data is continuously streamed to object storage.

For multi-drive or multi-node failures where the cluster remains online, data availability is fully preserved, though users may experience elevated read latency for affected stripes. In the unlikely event of a full AZ failure, block volumes are reattached to newly provisioned VM instances (taking 3 to 15 minutes, depending on cluster size). Metadata becomes available nearly instantaneously, allowing immediate cluster access while data is lazy-loaded from object storage.

Figure 4. VAST Cloud-HA (AWS and Azure)

Additionally, VAST applies similarity-based data reduction prior to committing data chunks to persistent flash. By evaluating chunk resemblance across varying offsets using high-performance hashing algorithms, VAST compresses and deduplicates streaming payloads globally.

TCO Analysis: Where VAST on Cloud Delivers Economic Advantage

The baseline workload parameters:

Table 1. Baseline

TCO Cost Breakdown

The table below (Table 2) summarizes the component expenses across both platforms for a sustained 4 GB/s ingestion rate and a five-day retention window. Under these parameters, VAST on Cloud has a monthly cost of $238,387 ($23.78 per terabyte ingested). By comparison, Confluent Cloud Enterprise requires $1,106,530 per month ($110.38 per terabyte ingested), driven by data egress across multiple consumers and post-replication storage charges. Capacity scaling for Confluent Cloud is metered in elastic Confluent Units for Kafka (eCKUs) for elastic tiers and CKUs for Dedicated clusters.

An Enterprise cluster is capped at 32 eCKUs (1,920 MB/s ingress), so this workload requires at least three Enterprise clusters; the analysis prices them as a single pool, which makes the Confluent figure a floor.

Table 2. Cost Breakdown

Confluent Cloud figures use published list pricing as of October 2026 (low end of the regional range)

Multi-Consumer Fanout Economics

In traditional streaming architectures, core telemetry streams are often consumed simultaneously by multiple applications: for example, real-time security indexing, operational monitoring, and offline model training.

On Confluent Cloud, each additional consumer reading the full 10,025 TB monthly stream generates additional data read volume. At the $0.020/GB rate used in this analysis, each full-stream consumer adds approximately $205,313 per month in data egress charges. Expanding consumption from a 1:1 to a 4:1 fanout ratio adds more than $615,000 (3 extra consumers x $205,313). Note that at 4:1 the cluster becomes egress-bound and grows to 89 eCKUs, another ~$28,000/month.

VAST on Cloud takes a different approach. Event Broker consumers access event data through the virtual EBox cluster rather than independently replicated broker logs. Additional consumer reads do not introduce a separate per-gigabyte Event Broker charge.

Additionally, while historical data replays on Confluent Cloud can cause broker contention and slow down real-time consumers, VAST avoids this entirely by serving historical reads from shared flash across any node.

Ingress Throughput and Retention Duration TCO

Multi-consumer fanout is only one factor that shapes streaming costs. Cost differences also become evident as ingestion throughput increases and retention windows expand.

Ingress Throughput TCO

VAST on Cloud is consistently less expensive than Confluent Cloud across all throughput levels. In our first example, we keep retention constant at 5 days and consumer fanout at 2:1. As throughput grows, the usage-based fees for Confluent Cloud accrue quickly. In contrast, VAST on Cloud operates on a provisioned infrastructure model, enabling greater cost efficiency as the ingestion volume increases.

Figure 5. TCO Based on Throughput

Beyond 4,000 MB/s, VAST on Cloud's cost advantage becomes most evident, reaching more than $5 million in monthly savings compared with Confluent Cloud at 24 GB/s.

Retention Duration TCO

For the next example, we maintain a constant ingestion throughput of 4,000 MB/s and a consumer fanout ratio of 2:1, with retention windows covering 1 day to 365 days.

Figure 6. TCO Based on Retention

Because Confluent Cloud bills storage post-replication, retaining data is nearly twice as expensive compared to VAST on Cloud, even out to a full year ($30 million vs. $17 million per month). However, we need to consider that long-term retention on VAST on Cloud requires scaling out virtual EBoxes with high-performance flash and can become more costly than cloud-native offerings that leverage archive storage. VAST on Cloud provides the most value when deployed for retention windows from 1 to 14 days.

This analysis prices Confluent Cloud’s Enterprise tier, which provides private networking and low-latency delivery comparable to VAST on Cloud. Confluent also offers a Freight tier for lag-tolerant logging workloads, billed at $0.03/GB-month pre-replication. Under the Table 1 baseline (4 GB/s, 2:1 fanout, five-day retention), Freight costs approximately $592,000 per month ($59 per TB ingested), still roughly 2.5x VAST on Cloud.

From Event Streaming to a Unified AI Data Foundation

For customers operating sustained, high-volume pipelines, VAST Event Broker offers an alternative to shared-nothing event streaming platforms in the cloud. VAST Event Broker unifies streaming, analytics, and AI on a single platform, so event data is usable the instant it lands. Producers and consumers keep using familiar Kafka client libraries and APIs, and existing data pipelines keep working. This reduces infrastructure costs and makes real-time enterprise knowledge available to AI applications.

• Rapid cloud-based deployment

• Low TCO for ingress throughput and short-term retention

• Zero data triplication

• No ETL pipeline complexity

• Unified management VAST on Cloud is much more than a simple cloud storage offering. It is a unified AI data foundation in the cloud. Best of all, if you are a VAST on Cloud customer, Event Broker is included in the license.