From Batch Jobs to Self-Service HPC: The New Shape of HPC Infrastructure
_VAST-Data.webp)
For decades, high-performance computing was built around a relatively stable model. Expert users worked within centrally managed systems, submitted jobs into schedulers and waited for infrastructure to allocate the resources they needed. That model remains exceptionally effective for large-scale scientific computing, but the users arriving in HPC environments today increasingly expect something else as well: infrastructure they can interact with directly, software environments they can control themselves and resources that can be provisioned without waiting for every component to be configured manually.
Self-service HPC is an approach to high-performance computing that lets users provision and manage compute, networking and storage resources through software-driven interfaces, while infrastructure operators retain control over the underlying systems. It brings cloud-like flexibility to HPC environments without requiring organizations to give up the performance, security or control of infrastructure they operate themselves.
For John Taylor, co-founder and CEO of StackHPC, this is part of a much longer evolution in supercomputing. Taylor began his career programming a Cray-1 during his PhD in theoretical physics before working through several generations of high-performance systems and the eventual rise of large-scale Linux clusters.
StackHPC was founded around a decade ago to help organizations build and operate cloud-like infrastructure for HPC and other large-scale computing environments. Rather than moving those workloads wholesale into public cloud, the company uses technologies including OpenStack and Kubernetes to give research organizations, enterprises and cloud providers more flexible ways to provision and consume their own infrastructure.
“AI has really acted as a catalyst for the convergence of HPC and cloud,” Taylor says.
From Fixed Infrastructure to Self-Service HPC
Traditional HPC and cloud infrastructure emerged with very different assumptions. A conventional supercomputing environment is optimized around centrally managed systems and scheduled jobs, while cloud infrastructure gives users much greater freedom to provision resources, bring their own software environments and interact with infrastructure through APIs.
Taylor says modern environments increasingly need both. Traditional batch computing is not disappearing, but it now sits alongside persistent services and applications that expect a much more dynamic relationship with the underlying infrastructure.
That is where technologies such as OpenStack continue to play an important role. OpenStack provides infrastructure software for exposing compute, networking, identity, storage and bare metal resources programmatically, giving administrators a common control plane while allowing different levels of self-service for users.
“The real reason we chose OpenStack was the ability to ground infrastructure orchestration in well-defined APIs,” Taylor says.
That cloud model also brings multi-tenancy into the infrastructure by design. Different users or organizations can share the same physical environment while remaining securely isolated from one another, provided the underlying systems being orchestrated offer equivalent isolation themselves. For storage, that means the data platform must also understand and enforce tenant boundaries.
A self-service HPC environment depends on three layers working together: compute resources that can be provisioned on demand, networking that connects users and workloads to those resources, and storage that can be allocated and accessed through the same orchestration framework. If any one of these remains dependent on separate manual processes, the self-service experience breaks down. Storage, in particular, must be treated as an integral part of infrastructure orchestration rather than a separate service to be arranged after compute is provisioned.
For StackHPC’s customers, the result is infrastructure that can be shared more flexibly without requiring every user or organization to operate on a completely separate physical system.
Orchestration Cannot Stop at Compute
Making compute easier to consume only solves part of the problem. A cloud-like environment stops feeling self-service very quickly if users can provision compute and networking dynamically, but still depend on separate manual processes to get access to the storage they need.
“We need to orchestrate storage in the same way we orchestrate network and compute. They should never be separable,” Taylor says.
OpenStack services such as Cinder for block storage and Manila for shared file systems allow storage infrastructure to be managed through the same API-driven environment. StackHPC, which helps organizations design and operate OpenStack-based cloud infrastructure, has worked with VAST to integrate its data platform with OpenStack services including Cinder and Manila.
The significance is not the API call itself. It is the ability to expose more of the underlying infrastructure as a service, reducing the amount of manual coordination required to give users the environments they need.
CINECA Brings the Two Models Together
Italy’s CINECA provides a useful example of why these models increasingly need to coexist. Alongside its traditional supercomputing infrastructure, CINECA operates GAIA, an OpenStack-based cloud environment that gives users a different way of consuming computing resources. GAIA itself spans 500 servers, more than 130,000 CPU cores and over 250 GPUs, and brings together several Italian research and innovation programmes supported by the National Recovery and Resilience Plan (NRRP) and the National Complementary Plan (NCP), with the backing of the Ministry of University and Research (MUR). The contributing programmes include ICSC, TeRABIT, MEET, GRINS, NBFC and D34Health.
In a traditional CINECA HPC environment, users work within centrally managed systems and submit jobs through schedulers. GAIA allows users to provision virtual machines and bring their own operating systems, libraries and software stacks, making it better suited to complex software environments, interactive applications and services that need to remain running for longer periods.
The model also allows CINECA to give researchers from different institutions access to the same physical GAIA infrastructure while maintaining isolation between their compute environments and data.
“The objective is not to replace HPC with cloud because they solve different problems,” said Mirko Cestari, Head of AI-HPC Infrastructure and Services, HPC Supercomputing Directorate at CINECA. “What we increasingly need is the ability to offer users different ways of consuming infrastructure. For some workloads the traditional HPC model remains exactly right, while others benefit from greater control over their own software environment and the ability to provision resources more dynamically.”
CINECA’s data architecture supports that flexibility. Its VAST system, known as Garda, provides file, object and block services across different parts of the wider computing environment, while also extending the tenant isolation required by the OpenStack model into the data layer. That allows different users or organizations to share the same underlying infrastructure without their data being exposed across tenant boundaries.
The same architecture also reduces the need to create a separate storage silo for every new compute environment. As new clusters, GPU systems and application platforms are introduced, the data can remain on a common platform rather than being repeatedly moved or copied between dedicated systems.
How Can HPC Enable Greater Accessibility Without Giving Up Control?
There is also a sovereignty dimension to this model, but it is a consequence rather than the starting point. Organizations can provide self-service, APIs and modern application environments while continuing to control the infrastructure and data underneath them.
Taylor describes sovereignty in three forms: data sovereignty, technical sovereignty and operational sovereignty. Open infrastructure does not remove every dependency, but it can give organizations more agency over how those dependencies are assembled and operated.
The next phase of HPC is not about choosing between centralized supercomputing and cloud-like flexibility. It is about giving users the right way to consume infrastructure for each workload, while preserving the performance, security and operational control that research and enterprise environments demand. That requires treating data services as part of the infrastructure orchestration model, not as a separate layer that users must navigate on their own.
For HPC operators, the opportunity is to make increasingly complex infrastructure simpler to consume without making it harder to control. Explore how VAST integrates with OpenStack to bring data services into a self-service HPC infrastructure model.


