Solutions

FAIR Data Platform: Curating Research Data Where It Lands

November 16, 2026
Authored by
Chris Snow
Senior Systems Engineer
James McClung
Technical Account Manager
FAIR Data Platform: Curating Research Data Where It Lands

Most Research Data Can't Be Found, Even by the People Who Made It

Every research computing centre has the same problem. Huge amounts of data sit on its storage, and most of it is described nowhere: a folder name, a file called export_4.csv, and a researcher who has since moved on. The research data exists, but nobody can find it, trust it or use it again. A FAIR data platform can help address this challenge by making research data discoverable, understandable and reusable from the moment it lands.

Studies show how much is lost. When a team asked for the data behind 516 published papers, the odds that a dataset still existed fell by 17% for every year since publication (Vines et al., 2014). Of 1,792 authors whose papers said their data was available on request, about 7% actually sent it (Gabelica et al., 2022). A study for the European Commission put the cost to Europe of research data not being FAIR (findable, accessible, interoperable and reusable) at a minimum of €10.2bn a year (PwC, 2018).

Sources: Vines et al., 2014; Gabelica et al., 2022; PwC for the European Commission, 2018; State of Open Data, 2025.

Funders now require it. NIH's data sharing policy, Horizon Europe and UKRI all expect research data to be FAIR. In practice that means a standard, machine-readable description of every dataset, a lasting identifier, and a licence, so that people and software can find it and use it again. The same description is what makes data AI-ready: an ML pipeline can't train on data it can't find or doesn't understand.

Describing data by hand doesn't scale. Data stewards curate datasets one at a time, while instruments and simulations write new ones every day. And researchers get little help or credit for doing it themselves: in the 2025 State of Open Data survey, 69% said they receive too little credit for sharing data.

How a FAIR Data Platform Curates Research Data Where It Lands

We built a working demonstration that does this automatically. Instead of asking researchers to fill in forms, or copying data into a separate catalog system, the storage platform does the first pass itself, the moment data arrives.

A researcher, or an HPC job, writes a folder of results to VAST over S3 or NFS. Within minutes that folder has a draft description built from the files themselves, a FAIR score, and a page in a review portal. A researcher checks the draft, fills in what only a person can know, and updates. The dataset then has scientific domain metadata written next to it, an identifier, a landing page, a preserved snapshot of exactly what was published, and it can be found by meaning.

Each published dataset also gets a Croissant file, the format machine learning tools use to describe datasets, so the data can be loaded straight into training and analysis.

Each site also shares the records it has published, and only those, through open feeds, so another catalog can offer one search across institutions.

The data stays where it was written, and its description is stored beside it.

How It Works on the VAST AI OS

The whole pipeline runs on services already in the VAST AI OS, providing the foundation for a FAIR data platform that curates, indexes and preserves research data without moving it between systems.

The demo pipeline. Dashed boxes are options that aren't built in this demo.

Everything inside the box runs on the VAST cluster. The language model only proposes descriptive fields, and nothing is published until a person approves it.

  • Events where data lands. VAST DataEngine element triggers fire as each file arrives, over S3 or NFSv4, and put an event on a Kafka topic on the VAST Event Broker.
  • DataEngine runs a curation function for each event. It profiles each file as it lands (format, header, variables, coverage, checksum). When the folder is complete it drafts a record; when a person approves, it publishes.
  • VAST DataBase maintains every version of every record, along with the underlying files, their checksums and the associated search indexes. These include VAST Vector Search, which enables search by meaning, and a structured index of every variable to answer precise queries such as “every dataset with sea surface temperature.”
  • VAST DataBase maintains every version of each record, It also holds the published data itself, as tables: the portal queries them in place through the VAST DB Python SDK, and Trino, which VAST runs on the cluster’s own CNodes, can answer SQL over them (this demo doesn't deploy Trino).
  • VAST Catalog finds the folders nobody has described, across the whole namespace.
  • Snapshots preserve exactly what was published, so an identifier never points at data that has since changed.
  • Sharing with other catalogs. The demo shows VAST working with the OAI-PMH sharing (from the Open Archives Initiative) that university repositories and research aggregators already use. The portal harvests partner repositories over OAI-PMH, whether or not they run on VAST, so one search spans every site, and any OAI-PMH harvester can collect its records in turn. Between VAST sites there's a further option. Whenever a dataset is published, updated or withdrawn, the site could post an event, with the record attached, to its own VAST Event Broker. Partner catalogs subscribe, so changes, withdrawals included, reach every site as they happen.
Search by meaning, answered by VAST Vector Search: "ocean temperature from satellites" puts the NOAA dataset first, well clear of the rest (0.63 against 0.39 for the next).

Nothing to Stitch Together

Every job in the pipeline is done by a feature of the platform (VAST AI OS) that already holds the data.

Each job the pipeline needs, and the VAST feature that does it.

Separate systems are easy to set up. Keeping them connected is the hard part. Each connection needs its own credentials, retries, monitoring and upgrade plan. Copies of the data go stale as soon as they're made, and a second permission model can leave gaps. Here there are no connections to maintain: the event, the function, the tables and the search index sit on the same platform, under one tenant and one identity policy.

The pipeline is about 3,500 lines of Python, and almost all of it is about metadata. There is no integration code here that:

  • Polls folders to notice new files
  • Copies data somewhere else to process it, and copies the results back
  • Keeps a separate search index in step with the files, or rebuilds it when it drifts
  • Crawls the namespace to find old data
  • Takes a second copy of the data to preserve what was published
  • Defines permissions twice, once for the files and once for the database

Each of these is usually a separate project.

Each of these is a service on the cluster, not software to install. A team provisions it in the VAST web UI, with the CLI or through the REST API, so all of it fits infrastructure as code. The demo's entire environment, from its tenant and buckets to its event topics, function, triggers and pipeline, is created by one re-runnable script and removed by another command. Tenants keep each team's services, data and permissions separate, each with its own administrator.

‍

Trust by Design: The Model Never Invents a Fact

The obvious way to build this, though asking a language model to describe each dataset, would produce confident nonsense: invented authors, wrong licences, plausible-sounding instruments. A FAIR record that's wrong is worse than none. So the design puts facts first and limits what the model is allowed to do.

  • Facts come first, taken as they are. File headers already say a great deal: NetCDF climate conventions, FITS keywords from telescopes, VCF headers from genomics, ROOT files from physics. So do the researcher's own README, CITATION.cff and LICENSE files.
  • The model only proposes descriptive fields, such as a description, keywords, the domain and what each variable means. Its confidence is capped, and every proposal carries the evidence it came from.
  • It never supplies creators, licence, publisher or identifiers. If nothing in the folder states them, the record says so, and publishing waits for a person to add them.
  • A person approves every publication. The review screen shows, field by field, whether a value came from the file, the researcher's documents or the model, so the reviewer knows what to check.
  • Every decision is kept. Each curation and each review is a new version in VAST DataBase, with who made it.
A documented dataset after publishing: every field says where it came from (the file's own header, or the researcher's CITATION.cff), and the score rose from 52 to 92. The pipeline also noticed that the same bytes exist in another folder.
An undocumented file. The model describes the physics well from the variable names, but publishing is blocked: the licence and creators are never guessed.

‍

Measured, Not Claimed

We tested the pipeline on six public datasets from six fields, chosen because their licenses allow reuse: NOAA sea surface temperature (climate), a TESS light curve (astronomy), CMS open data (particle physics), gnomAD (genomics), BBBC039 microscopy images (bioimaging) and UK road traffic counts (public sector). Four also had a "dark" copy: the same bytes, renamed, with no documentation.

  • Accuracy: 62 of 64 fields right. We graded the pipeline's own drafts - before any human edits -  against the data providers' published metadata in two runs on different clusters. The other two were left blank for a person in one run; in the other, one value was wrong: for a renamed physics file that can't show it's a simulation, and therefore the model guessed "collision-based event reconstruction,” at a confidence of 0.7, below its confident fields. That's the kind of mistake the review step is designed to catch.
  • Zero invented facts. In all seven cases where nothing in the folder stated the licence or the creators, the pipeline left them blank and held the dataset for review.
  • FAIR scores rise when a dataset is published. A documented dataset scores 24 to 52 out of 100 before publishing and 72 to 92 after. An undocumented file, named export_4.csv, goes from 8 to 76 once a person adds its licence and creator.
  • An independent assessor agrees. F-UJI, the open-source FAIR assessment tool from the FAIRsFAIR project, scored every published dataset between 83 and 88.
FAIR scores for the six published datasets: F-UJI 3.5.0 in blue, the demo's built-in checks in grey.

The two scores sit within a few points of each other. F-UJI's ceiling here is set by the demo's test identifiers: they aren't registered with DataCite, so "findable" can't score full marks until a service mints real DOIs.

The Data You Already Have

New data is the simpler case. Most of a centre's data is already on its storage, and nobody is going to upload it again.

VAST Catalog keeps a regularly refreshed index of every file in the namespace. Finding undescribed folders is a query against it, not a crawl of the file system: on a namespace with billions of files, that's the difference between a dashboard and days of crawling. 

The portal's Backlog view queries the index and lists every folder that holds data but has no FAIR record: its size, its file formats, whether it has a README, a citation file or a licence, and how old it is. One click curates a folder in place: the same DataEngine function profiles and describes where it sits, and a person reviews it as before.

The backlog, straight from VAST Catalog: folders with data and no record, what's in them, and whether anyone left documentation.

It also catches duplicates. VAST DataBase compares checksums, so when the same bytes already exist in another folder, the review screen says so before a second copy is published.

Open Standards, Open Source, No Lock-In

Research institutions are right to be wary of putting their metadata into one vendor's system. A FAIR data platform should make research data more accessible and reusable without tying its metadata to a proprietary system. That's why everything here uses open standards.

  • The records are open standards, written next to the data. Each published dataset gets RO-Crate 1.1 files, from the Research Object community, and MLCommons' Croissant 1.0 files in its own folder, with schema.org metadata and a provenance trail (PROV-O). They validate with the standard tools, and they stay readable and valid if the data ever moves elsewhere.
  • Croissant is what ML tools load. It's the MLCommons format for describing datasets to machine learning frameworks, so a published dataset is ready for AI work without another conversion.
  • The FAIR score is checked independently. Alongside the built-in score, which is keyed to the Research Data Alliance's FAIR maturity indicators, every dataset is assessed by F-UJI, an open-source tool the research community already uses.
  • The code is open source under the Apache 2.0 licence. The curation logic is plain Python and runs anywhere, even on a laptop with no cluster. What VAST adds is the platform around it: processing where the data lands, search at scale, the view of all existing data, and preservation.
What a published dataset gets: a landing page for people, RO-Crate and MLCommons' Croissant for machines, a checksum for every file, and the snapshot that preserves exactly what was published.

Use the Data without Downloading It

Being found is only the start. A researcher who finds a dataset usually still has to download all of it, work out the format and write a reader before asking a first question. So publishing does one more thing: it loads the data itself into VAST DataBase, one table per part. A CSV becomes rows, a NetCDF grid one row per cell, a light curve one row per observation, a VCF one row per variant, and a particle-physics file one row per event, with a table for each kind of particle.

Each column keeps what curation established: its units, its standard name and the meaning a person approved. Each table names the published file it came from and that file’s checksum, so every row traces back to the citable dataset.

Published once, used three ways. A question in plain language becomes filters a person can check, and is answered from 4,798 of the dataset’s 691,150 grid cells, without downloading the file.
  • Answers, not downloads. The average sea-surface temperature in a box of the North Atlantic is one filter on the published NOAA dataset. Only the 4,798 matching grid cells of 691,150 come back, already in degrees Celsius.
  • Any tool can ask. The same question runs in the portal, over plain HTTP as CSV, JSON or Parquet, from Python with the VAST DB SDK, and as SQL on Trino, which VAST runs as a managed application on the cluster’s own CNodes, so there is no separate query environment to set up and run. This demo doesn't deploy Trino itself; its queries go through the SDK.
  • Or just ask. “Average traffic on the M4 by year” works in plain language. A model reads the question against each column’s reviewed units and meaning and proposes the filters; they are checked against the table and shown as ordinary controls, so a person sees exactly how the question was read and can change it. The model sees only the column descriptions and a few example values, and never writes code.
  • Same platform, same rules. The tables sit beside the files, in the same tenant and under the same permissions, loaded by the same pipeline at publication. Nothing is exported to another system and left to drift.
  • FAIR in practice. Each table is listed in the dataset’s RO-Crate, so harvesters and assessors can see that the data itself, not only its description, can be read through an open protocol, with a shared meaning for every column.

From Findable to Answerable

Curation is half of making data AI-ready. The other half is being able to ask it questions. VAST’s research assistant (github.com/vast-data/vast-research-assistant-agent) does that half: VAST InsightEngine chunks and embeds documents as they arrive, and an agent answers questions about what they say. The two fit together. This pipeline describes the data and a person approves the description; every published record’s landing text is also written to a bucket InsightEngine can index, so the assistant can answer from descriptions a curator has checked, with an identifier to cite. Both run on the same DataEngine triggers, VAST DB and Kubernetes.

From Demonstration to Service

This is a demonstration, built from shipped VAST features. Turning it into a production service takes more work, by your own team or a partner. Here are some of the things it would need:

  • Real identifiers. The demo mints test identifiers; a service would register DataCite DOIs and serve landing pages from a stable public address.
  • Researchers' text treated as untrusted input. Every README and file header is someone else's text sent to a model, and should be handled the way a web application handles form input.
  • The right model for the data. Open data can use any model; embargoed, personal or export-controlled data needs an on-site model and classification before anything is sent.
  • Confidence calibrated per field and per community, on a larger labelled set, before anything is accepted automatically.
  • Sign-in and roles for reviewers, such as institutional single sign-on or ORCID, so every decision is tied to a person.
  • Locked snapshots and a retention policy, so preserved data can't be removed by mistake.
  • Agents in production. When agents move from answering questions to acting on the data, VAST AgentEngine provides the runtime, a shared registry of MCP tools, and the observability to run them reliably, on the same platform as the curated data.

Try It

The best way to judge it is to run it on your own data.

  • Get the code: github.com/vast-data/fair-demo, with a setup guide for any VAST cluster.
  • See it on your data. Share a sample folder, or point us at datasets your institution has already published, and your VAST team will curate them on a VAST cluster and walk you through the results. Nothing to install, including data that’s been sitting undescribed for years.
  • Work on it with us. We’d like to hear from groups already working on FAIR curation, or who’d like to collaborate on it in a non-commercial setting. The code is open source, and contributions are welcome.
  • Talk to us about building a FAIR data platform that turns research data into discoverable, reusable and AI-ready datasets, right where the data already lives.
Share this

More from this topic

From Batch Jobs to Self-Service HPC: The New Shape of HPC Infrastructure

From Batch Jobs to Self-Service HPC: The New Shape of HPC Infrastructure

November 16, 2026
Rethinking KV Cache Offloading: The Architectural Imperative of Shared vs. Local NVMe Storage

Rethinking KV Cache Offloading: The Architectural Imperative of Shared vs. Local NVMe Storage

In modern scale-out LLM inference architectures, managing KV cache capacity is a primary driver of throughput and overall cost efficiency, with DeepInfer publishing that KV cache optimizations helped...
November 11, 2026
Deploying Relational and Multimodal AI Models for Financial Services

Deploying Relational and Multimodal AI Models for Financial Services

Everyone in banking is talking about artificial intelligence and, more specifically, large language models. However, the more interesting AI shift happening underneath that conversation is the…
November 4, 2026

Let’s talk about your data

Whether you want to speak with sales, see a personalized demo, or work through your architecture with a systems engineer, we’re here to help. Choose how you’d like to connect.