VAST 5.4 saw the initial release of the VAST Event Broker, a Kafka implementation built on our Disaggregated Shared-Everything (DASE) architecture with a predilection for high performance and absolute resilience. VAST 5.5 greatly extends performance on an already fast implementation while diving right into advanced features, which come in the form of unique capabilities and the addition of enterprise security features. I’m going to keep this short, so here we go.
Before we can start discussing the Event Broker, we need to cover the VAST DataBase (and more generally, the VAST AI Operating System), since it is closely linked as the fundamental building block for Event Broker topics. So let’s do that.
The VAST DataBase is a flexible and scalable presentation layer for structured and vector data. At its core, it’s a columnar table format with advanced data optimization features (partitioning, sorting, vector clustering, auto-defragmentation) and an API that allows supported processing engines and Python ecosystems to access it directly. It is the “data half” of the Native Query Engine, VAST’s SQL environment, and it is also where the Event Broker stores Kafka topics.
While traditional Kafka implementations build on a shared-nothing design - where topic data is written to log segments on local storage and partition replicas are used for resilience and HA - VAST commits messages to the DataBase. This has a lot of advantages outlined in a previous article, but in summary, VAST DataBase tables incur very little write amplification, facilitate full HA, make scaling easy (no rebalancing), and deliver enterprise features with near-absolute availability and resiliency.
Moreover, messages committed to the DataBase in this way can be interrogated directly by the VAST Native Query Engine, Trino, Spark, and Python—in addition to being consumable by Kafka protocols. SQL and dataframe access allows you to process data by partition, offsets, timestamps, message headers, and the messages themselves. But the cake is a new feature called “blob expansion."
Blob expansion, or Topics to Tables (you can read Chris Snow’s excellent blog post for details), is a feature that parses JSON messages that enter a topic, “flattens” them if they contain any nesting, and writes them as structured data to a DataBase table in parallel with the topic itself.
This capability is synchronous, meaning that the ACK to the producer doesn’t happen until the message is committed to both tables. The main significance of this capability is that you can use arbitrary data optimizations on the message expansion table according to the message schema. It can be partitioned and/or sorted, and the fields selected for inclusion can be a subset of those present in the message. Or the expansion table can be a superset of the message schema to be filled out in merge or update operations later. All of this, put simply, means that your streaming JSON data is immediately and practicably queryable once produced to the brokers. And the system can handle these message expansions at high throughput.
Did someone say “throughput?” Let’s talk about performance.
While the initial 5.4 release of the Event Broker sported some pretty impressive performance figures, a focus of the 5.5 release was on performance yet again (we’re never bored with the subject). Let’s get to the point and show some numbers:
The above chart indicates read/write throughput per cluster node, where we rate the maximum sustained throughput of a topic with a read-amplification factor of 1 (each message read once). This is done using 1KiB messages and results in the above single-rate figure that can be assumed to be 50% read and 50% write (2GB/s = 1GB/s read + 1GB/s write).
Not only is VAST faster than any other broker available, it represents a major performance improvement over VAST 5.4 (already at the top of the list) where the performance figure was 2.2 million messages per second per broker.
What does this mean in practice? First, it means you get more out of your hardware investment, which is pretty simple. And you can sustain more consumer groups on a single topic. It also means you have more operational headroom: No more running in the margins where outside interruptions can mean long catch-up times and expanding a topic involves dangerous partition reassignments (balances) on an already stressed system.
On VAST, expanding a cluster to handle increased traffic is as simple as adding nodes to the cluster. Once expanded, all of your partitions are immediately available without any rebalancing since DASE obviates the very concept of a node “owning” data.
For many customers, security is the more pertinent subject. On that front, VAST has added mTLS authentication to Event Broker 5.5 so that clients can prove their identity to the broker. mTLS is popular in hardened zero-trust environments, IoT ecosystems and microservice architectures.
The last major feature addition in this release is an increased maximum message size to 950MB to ensure interoperability with existing Apache environments. This in addition to many internal updates that manifest improved latency and monitoring of broker topics.
To learn more about how VAST Event Broker can accelerate your event-driven workflows, contact us today or schedule a demo with one of our technical experts.
