The Grand Unification Theory of AI Infrastructure: Part II

What’s in a name?

In 2023, I laid out the case for VAST to consolidate the entirety of the data stack… and established that a new architecture makes it possible to embark upon this grand consolidation all without introducing compromise to AI and traditional applications. In fact, a new architecture can do more than maintain status quo - this new architecture could solve for bottlenecks created by legacy point solutions and unlock new levels of scale, parallelism, efficiency and resilience all while also enabling mass consolidation. The name… The VAST Data Platform.

2023 was the year of training… when VAST just started going big with AI builders. VAST customers have deployed 10s of exabytes of infrastructure in support of building the largest frontier models and now store massive amounts of unstructured data in the VAST DataStore as well as massive amounts of context as metadata in the VAST DataBase.

Everything is accelerating. The rate of AI progress is constantly increasing as model builders build on each other’s discoveries and push the boundaries ever farther. While we’ve been talking about thinking machines since early 2022, the advent of reasoning models in the last 12 months means that the era of thinking machines is actually now upon us.

The rate of change is not exclusive to AI builders. Today, every progressive customer we talk with is in a race to transform the core logic of its business to be agentic. This is not hyperbole - we see investment banks racing to get a 4x improvement in employee efficiency. Companies are talking about deploying millions to 100 million AI agents to assist and accelerate their workforce.

So now you have these new applications, we now have radically powerful AI GPU hardware to drive them… the question is, how can we make this simple and scalable for the masses so that every organization can take the journey toward agentic computing?

On the other hand, you have VAST Data. Since the introduction of the VAST Data Platform in 2023, the term has felt to us somewhat limiting. Why?

1. Data Platforms were born in the BI space, and never really solved for unstructured data or deep learning in the way that VAST has optimized for. As testament to this… GPU vendors don’t prescribe BI tools for training or inference, but they do prescribe VAST.

2. Data Platforms also historically have been deployed in public clouds, so they forfeit all of the opportunity to find critical infrastructure-level efficiency that VAST has discovered by managing hardware in new and different ways. Legacy Data Platforms, in a sense, just peanut butter legacy concepts that cloud vendors impose on them based upon the core infrastructure decisions they made 20 years ago. On the other hand, VAST started architecting only in 2016 - with a blank sheet of paper and an eye to new underlying hardware technologies - and has built a modern Disaggregated, Shared Everything (DASE) architecture for the AI era. DASE is now the basis for how most GPU clouds compute on data - with the scale and efficiency to happily feed billion-dollar machines.

3. Data Platforms have also not been thought of as real-time systems that you can run a business off of. This, to us, is a deal-breaker as agentic workflows get pushed more and more toward the ‘front’ of the business. Even as the term ‘AI Data Platform’ settles into the market, this spin on Data Platforms is essentially a set of batch metadata services run on top of enterprise data storage that makes storage addressable by AI pipelines… yet AI Data Platforms were not envisioned to be the infrastructure of core business logic themselves. Why?... because legacy architectures weren’t designed for scalable real-time transactions, and the methods in which data is captured, contextualized and analyzed has always been approached with a batch computing mindset. Customers don’t even envision a case where large quantities of data can be synthesized with AI in real-time. Why is this? Architecture. Until now, everything scalable has been built from partitioning data sets across large clusters. Partitions slow down transactions at scale because clusters have to ensure consistency, and as they grow there’s too much east-west traffic. In short… Legacy storage and database systems simply weren’t designed for this synthesis… so AI suffers.

So - then the question is, if the term data platform leads people down the wrong mental path… how can we pull this all together?… and how can we create an environment for agents to exist and collaborate, to discover and action in real time? And how can we make it simple for every organization to leverage this new application paradigm and make simple use of the revolution in computing hardware that’s also happening in front of us?

Let’s talk about Operating Systems.

For decades, it’s been operating systems that make it easy for developers and IT users to get value out of underlying hardware. From the desktop era (PC) to the mobile era (smartphone) to the cloud era (cloud data center) each computing revolution was defined by a new application stack (office productivity, mobility, big data) that is made possible by new device platforms.

What’s an operating system, and how do we see requirements in the agentic era?

• a set of device drivers that manage hardware

• a storage environment that stores data, with indexes that help you find data quickly

• a runtime with a scheduler to run applications

• a messaging system that allows applications to talk to each other via common APIs

As we considered what VAST was making… we considered our “thinking machine” north star that we defined for the company in 2016. We then realized that an operating system best defines the characteristics of what VAST is offering for the AI era. Let’s look at it: 

• we build from the HW layer, up… scaling to exabytes with unlimited parallel processing scale

• the VAST DataSpace extends this HW environment to federate storage and compute globally

• the VAST DataStore is the destination for rich data that comes from the natural world

• the VAST DataBase is the catalog where big metadata lives, with real-time support for writing to and querying tables and vectors at massive scale

• the VAST DataEngine is the compute framework that triggers and orchestrates AI workloads

• the VAST InsightEngine is the data refinery working within the DataEngine that transforms raw unstructured data into context using AI embedding models, and makes them available as RAG tools

In essence, you’ve got all of the core components of an Operating System, each designed for the level of scale, performance and efficiency needed to run even the most scalable real-time AI pipelines. We’ve built a singular, monolithic software stack that dramatically reduces infrastructure complexity and solution sprawl that covers everything but agents… until today.

Today, we take the covers off of a new AI agent deployment and orchestration system that runs natively within the VAST DataEngine called the VAST AgentEngine. AgentEngine is a low-code agent deployment and orchestration system that is equipped with all of the MCP-compatible tools needed to equip agents as well as a rich observability platform that helps developers monitor, manage and govern AI pipelines. This solution will be available in the second half of 2025 and is designed to make it effortless to define agents, pair them with reasoning models and autoscale them across the VAST platform.

AgentEngine is the final piece of the puzzle that rounds out the core services we believe are needed to run agentic applications comprehensively on AI hardware. With this, we graduate to the next phase of the product’s evolution - and introduce to you the VAST AI Operating System.

Not only are we introducing a full stack operating system for AI agents, but we’re also bringing example agents to the table as open source references that customers can build on to accelerate their journey to AI. We think of it as an analog to Minesweeper when Microsoft released Windows. Minesweeper helped people appreciate the new uses and power of the new computing platform… so we’re going to introduce our own Minesweepers. Every month, we will release a new open-source AI agent that will offer up a demonstration of the power of AI agents. Video editors, sales strategists, scientific researchers, data engineers, financial analysts, compliance managers… each of these will either have a specific industry specialization or will be simple assistants that help IT organizations manage their operations.

Applications + Runtime + Infrastructure. It’s an Operating System… albeit a very capable one.

“Every act of creation is first an act of destruction.” — Pablo Picasso

When’s the last time you bought a RAID controller for your storage array? 15 years ago? Time moves on, and as products become more capable and encompassing, point solutions fade into the background just as RAID controllers or MIDI device drivers are no longer things that the average person has to buy.

In 2016, we released a video using the phrase “reduce in order to expand.” We at VAST believe that the path to the greatest potential gain is to simplify and reduce the overall fundamental challenges that need to be resolved. If we can build a simple approach to encompass nearly all of the infrastructure layers needed for AI, without compromise… customers supremely benefit.

For decades, customers have been accustomed to integrating countless infrastructure solutions and services to build their application estate. Take a standard AWS blueprint and you’ll see 10-30 services stitched together to build a computing pipeline. Why? No system has ever been built to provide the best capabilities in all possible dimensions... until VAST. Not only does our architecture allow us to aggressively compete in a market across so many different use cases, but we also bring to the table a simple and synthesized solution, fit for purpose in the age of deep learning, that customers now understand is the key to ensuring AI readiness.

In many cases, the innovation VAST brings to the table not only enables mass infrastructure consolidation and simplification, it also renders many legacy approaches to computing irrelevant. I rarely call out competition in these blogs… in part because our product has no equivalent, but as we move into the abstract (operating systems) I thought it important to call product categories and associated products who will be disrupted by the adoption of our AI Operating System:

DASE is our unfair advantage that unleashes the AI Operating System and makes it competitive in each category. The diverse capability of the system further separates us from other solutions by unifying data management and security and bringing multi-modal classic and AI computing directly to the data.

In total, there’s $100s of Billions of infrastructure waiting to be disrupted and modernized. As we progress our vision into the market, we envision that the VAST architecture and style of computing will result in customers not needing whole technology categories in the future:

• Who needs tiered storage when flash is nearing the cost of tape? No one

• Who needs independent block and file storage? No one

• Who needs an event bus when data warehouses can be transactional? No one

• Who needs a separate vectorDB when vectors can be embedded into raw data? No one

These are examples of why we think of VAST as a killer of categories, not a category killer. One of the true superpowers of VAST is a lack of concern for market categorization. 20 years from now, we will look back on today’s technology and wonder why everything was so complicated. We just work to do the right thing for the customer and solve the problem at its most basic level, and then invention flows.

So, welcome to the new era of VAST. The AI Operating System era.

One computing platform | all your AI workloads One system | edge to AI factory One unified data environment | unstructured and structured One tier | real-time access to all data