With the agentic AI era kicking into high gear, data management is more important than ever. AI agents need more than just data, no matter how well it’s annotated or curated. What AI agents need is data that’s organized, contextual, and easy to parse. They need to know how data from different sources (both structured and unstructured) relate to each other and the broader world through a shared ontology.
In short: AI agents need semantic data. Inside large organizations, they need enterprise search that not only works, but also is supercharged with the ability to choose the best sources to serve and then to reason over them.
Working with NVIDIA Agent Knowledge Foundation, including libraries like NVIDIA Auto-Ontology, we’ve developed a framework for implementing these capabilities without the heavy manual effort it previously would have required. By combining the VAST AI OS with NVIDIA’s growing collection of open models, accelerated libraries, and data-processing technologies, we’re able to automate an entire semantic workflow from data ingestion through 10x more accurate predictive results. All human experts need to do is ask their questions.
Example: Putting AI agents to work in health care
Rather than presenting a generic list of the components involved, we want to show how such a workflow operates in the real-world. Healthcare provides a great setting because of how much data hospitals need to analyze and the wide variety of formats it comes in. Specifically, we’ll look at the use case of a radiologist needing more information about a patient to provide context for a recently completed CT scan.
Data ingestion and processing
Ingestion runs continuously as new structured and unstructured data—like medical records, physician notes, or imaging files—is entered into the hospital’s system:
VAST SyncEngine delivers each new item to the VAST DataStore.
Every ingestion triggers the VAST DataEngine to execute a processing pipeline.
Unstructured documents run through a VAST InsightEngine RAG workflow, which first sends them to NVIDIA Nemotron Retriever open models for parsing and embedding, and then exports them to the VAST DataBase as vectors.
The NVIDIA cuVS library uses GPU acceleration for high-speed nearest-neighbor (or vector similarity) indexing.
Structured records are written to tables in the VAST DataBase. The NVIDIA cuDF library flattens certain document-header records into rows.
All new tabular data is mapped by NVIDIA Auto-Ontology into a semantic layer.
If a hospital stopped here, it would already have significantly improved its data management posture. Rather than managing huge collections of raw records spread across various systems, everything is now stored within a single system; semantically indexed and linked; and ready to be queried using either natural language or SQL.
However, ingesting and processing data is only half the battle. We still need to empower doctors and other hospital staff to make the most of this information without learning how to query a database or navigate yet another piece of medical-industry software.
Running the agentic reasoning workflow
The agentic reasoning workflow can run automatically or when instigated with a prompt. Either way, VAST AgentEngine acts as the linchpin for executing the agentic workflow by managing the core AI agent’s access to data and other systems.
In an automated approach, an agent might get to work as soon as a certain event occurs—for example, once CT scan images have been ingested, processed, and loaded into the appropriate datastores. The next time a radiologist opens her inbox, she might see a full patient brief: comparable prior scans with their measurements; passages from an earlier report that describe the nodule; whether any pathology is linked to the finding; the relevant history from clinic notes; and the probability that this patient will miss the recommended follow-up.
Using a prompt-based approach, a radiologist might be seeing a new patient referral, so she asks the agent for a full profile on the patient and his previous medical history. Once she enters the prompt to the agent:
Any requested information that’s stored as structured data is identified and sent to NVIDIA Auto-Ontology, which performs a text-to-SQL transformation and runs a customized semantic query over the VAST DataBase.
Unstructured requests (e.g., “What did the patient’s previous doctor say about this condition?”) are routed to VAST InsightEngine, which runs a hybrid search over unstructured clinical documents. The question is embedded by an NVIDIA Embed NIM microservice, then searched against the vector index in VAST DataBase, where the hybrid search runs on the VAST cluster itself and is scoped to the patient’s record ID so only that patient's documents are candidates. (, part of NVIDIA AI Enterprise, is a set of easy-to-use microservices designed for secure, reliable deployment of high-performance AI model inferencing across clouds, data centers, and workstations.)
An NVIDIA Nemotron Rerank NIM microservice orders the candidates by relevance, and an LLM NIM generates a grounded answer with citations back to the source documents, which InsightEngine returns to the doctor. Access to each patient’s documents is enforced by InsightEngine's tenant and identity controls, independent of the query.
The agent calls the NVIDIA Kumo Relational foundation model to help predict this patient’s risk level, and NVIDIA cuGRAPH to analyze similar cases. Before reading data and performing their computations, each tool asks Auto-Ontology which tables and columns hold the concepts it needs and how they join.
AgentEngine then compiles everything into a full, grounded brief, complete with citations, and presents it to the doctor, who acts on the information accordingly.
The following video demonstration walks through this example, illustrating a sample user experience, as well as highlighting some of the backend processes using the VAST Management System.
Semantic data for enterprise AI agents in any industry
Although our example use case focuses on health care, this pairing of the NVIDIA Agent Knowledge Foundation and the VAST AI OS can run in any industry, on any data. And by enabling enterprise intelligence based on both unstructured data and the structured data on which most organizations still run, it’s an important step toward opening the floodgates of enterprise AI innovation.
Check out our Github VAST Research Assistant Agent with Auto-Ontology.
For more on how VAST is working with NVIDIA on the Agent Knowledge Foundation, read NVIDIA’s announcement.
To see how VAST helps AI agents run safely and effectively against valuable enterprise data, read our VAST Agent Sandboxes announcement.
