The Infrastructure Around Kafka Is Ready for Reinvention For long enough that’s become legendary, Kafka was the hard part especially in areas like financial services.
Moving millions of events every second between applications, databases, and services was a difficult engineering problem, and Kafka solved it. That’s why it became foundational infrastructure across industries, from beyond trading desks to manufacturing, inside telcos, retail, and cloud computing.
But today, Kafka is rarely the bottleneck, instead it is everything that happens after Kafka. On this week’s episode of the Shared Everything podcast, guest Matt Mohr from VAST breaks the past, present and future down and more important, shows a path forward for organizations still mired in the Kafka of days gone by.
In a typical enterprise, events flow into Kafka before being written into object storage, Parquet files, Iceberg tables, Spark jobs, feature stores, data warehouses, vector databases, and AI platforms. Every one of those technologies solves a real problem. Together, they create an architecture where data is constantly being copied, transformed, cataloged, and rewritten before anyone can actually use it.
That model made sense when analytics was mostly batch processing. Reports ran overnight. Dashboards refreshed every morning and so waiting an hour or so for new data wasn’t a problem but AI turns that timetable over.
AI systems, whether recommendation engines, fraud detection platforms, and quantitative trading models all expect fresh data immediately. They don’t just need the newest event from Kafka. They need that event combined with years of historical data, continuously updated features, and analytical contextall without waiting for another batch job to finish.
That’s where the architecture starts to break down because hile Kafka can deliver an event in milliseconds. Making that event useful can still take minutes or hours because the surrounding infrastructure was designed for a different era.
The next generation of data platforms has to solve that problem because instead of treating streaming, storage, analytics, databases, and AI as separate systems connected by increasingly complicated pipelines, companies like VAST are collapsing those layers into a single architecture. Streaming data lands once, becomes immediately queryable, and is available simultaneously for analytics, feature engineering, and AI without repeated transformations or endless movement between systems.
We cover all of this and far more. Thanks for listening and subscribing to the Shared Everything podcast.
