Thought Leadership

The Infrastructure Around Kafka Is Ready for Reinvention

August 19, 2026
Authored by
Nicole Hemsoth Prickett
Head of Industry Relations
The Infrastructure Around Kafka Is Ready for Reinvention

For long enough that’s become legendary, Kafka was the hard part especially in areas like financial services.

Moving millions of events every second between applications, databases, and services was a difficult engineering problem, and Kafka solved it. That's why it became foundational infrastructure across industries, from beyond trading desks to manufacturing, inside telcos, retail, and cloud computing.

But today, Kafka is rarely the bottleneck, instead it is everything that happens after Kafka.
On this week’s episode of the Shared Everything podcast, guest Matt Mohr from VAST breaks the past, present and future down and more important, shows a path forward for organizations still mired in the Kafka of days gone by.

In a typical enterprise, events flow into Kafka before being written into object storage, Parquet files, Iceberg tables, Spark jobs, feature stores, data warehouses, vector databases, and AI platforms. Every one of those technologies solves a real problem. Together, they create an architecture where data is constantly being copied, transformed, cataloged, and rewritten before anyone can actually use it.

That model made sense when analytics was mostly batch processing. Reports ran overnight. Dashboards refreshed every morning and so waiting an hour or so for new data wasn't a problem but AI turns that timetable over.

AI systems, whether recommendation engines, fraud detection platforms, and quantitative trading models all expect fresh data immediately. They don't just need the newest event from Kafka. They need that event combined with years of historical data, continuously updated features, and analytical contextall without waiting for another batch job to finish.

That's where the architecture starts to break down because hile Kafka can deliver an event in milliseconds. Making that event useful can still take minutes or hours because the surrounding infrastructure was designed for a different era.

The next generation of data platforms has to solve that problem because instead of treating streaming, storage, analytics, databases, and AI as separate systems connected by increasingly complicated pipelines, companies like VAST are collapsing those layers into a single architecture. Streaming data lands once, becomes immediately queryable, and is available simultaneously for analytics, feature engineering, and AI without repeated transformations or endless movement between systems.

We cover all of this and far more. Thanks for listening and subscribing to the Shared Everything podcast.

Share this

More from this topic

How Inference Is Forcing a Rethink of AI Infrastructure

How Inference Is Forcing a Rethink of AI Infrastructure

When it comes to large-scale systems for AI, the engineering focus has shifted from training to inference where the performance bottlenecks look very different and where infrastructure...
November 11, 2026
The New Rules of AI Inference Optimization and Future-Proofing

The New Rules of AI Inference Optimization and Future-Proofing

AI infrastructure has always been a moving target, but the pace of change today is challenging even the most experienced system architects. Models are evolving, inference workloads are becoming more…
November 4, 2026
CoreWeave on Building Resilient AI Infrastructure at Massive Scale

CoreWeave on Building Resilient AI Infrastructure at Massive Scale

At CoreWeave’s scale, the definition of a computer is getting bigger. A rack full of GPUs can now behave as a single logical system, and increasingly the same thinking has to extend across rows and…
October 28, 2026

Let’s talk about your data

Whether you want to speak with sales, see a personalized demo, or work through your architecture with a systems engineer, we’re here to help. Choose how you’d like to connect.