The batch schedulers traditionally used across modeling have all followed a similar approach to mapping user jobs to cluster resources for decades: jobs enter a queue, the scheduler finds an optimal packing of all the jobs in that queue across all cluster nodes (perhaps with other constraints, such as topology), and then new jobs are executed in whatever crevices are available. Then the process repeats, over and over.
There is a problem with orchestrators designed around this model: they assume that humans are the entities submitting jobs, and humans are slow enough that the orchestrator can take a lot of time to solve the hard bin-packing problem of job placement without causing a back-up of jobs. This is no secret, either; just about every HPC center's documentation contains cautions against wrapping job submission commands in automated loops:
“Users submitting large numbers (> 100s) of jobs using loops will be contacted” -- University of Arizona HPC Documentation Site
“If you plan to submit many small jobs in an array that require the allocation of more than 10 jobs per minute, please use GNU parallel” -- ULHPC Technical Documentation
“In short: whenever you are tempted to write a shell loop around sbatchlike” -- the HPC Wiki
Even Slurm's documentation implicitly acknowledges the limitation:
It can accept 1,000 job submissions per second and fully execute 500 simple jobs per second
A thousand jobs per second is a lot on a campus cluster serving hundreds of researchers. But if you consider a future where beamlines and telescopes are generating dozens of images per second, accompanied by agentic workloads that dispatch tasks as fast as a loop can iterate, you can imagine this ceiling becoming a serious limit. The fact that batch schedulers cannot scale as autonomous data generation facilities and agent-driven workflows become commonplace is a fundamental limit to productivity.
This inability to scale is the result of a couple of assumptions that batch schedulers have historically made sense:
Maximizing cluster utilization is top priority. If a HPC cluster is 100% utilized all the time, then centers get more money to buy another supercomputer even if jobs have to wait a while in queue. Conversely, if jobs execute quickly but the cluster utilization is low, it looks like the cluster is not being used productively.
Humans take breaks. A researcher may submit 100 jobs at once, but they then go for coffee. Schedulers can get a little sluggish as they to pack those jobs into the queue, because the backlog will probably work its way through by the time the next researcher comes back from a coffee break and submits more jobs.
Humans vs. Machines in Practice
We can see this behavior in job submission data. This data is from a large collection of job submission data from the Fugaku system at R-CCS.
You can see that there are intense bursts of job submission rates, followed by quiet periods where researchers have gone home for the day, gone on holiday, or just stepped away. Job submission rate here is limited by individual humans' ability to decide how to compose an HPC job, write the job script for it, submit it, analyze the results, and repeat.
But as we move towards the world of AI-assisted discovery and autonomous laboratories, two factors make this type of task orchestration completely break down.
First, AI agent reasoning. This is an application, some GPUs, and an API replacing the human and a keyboard, and it never goes on coffee breaks or sleeps. It is always thinking about the next job to submit. If we take the Fugaku trace and squeeze out the times when each user went home to sleep, the job submission rate goes way up--by a factor of 8x.
Second, autonomous laboratories require real-time feedback so that agents can respond to changes in physical experiments. This adds an additional baseline amount of tasks that reflect, for example, turning a raw video feed into context that an agent can reason over. This task rate is a function of how fast an instrument can generate data and how much latency you can tolerate in reacting to visual data.
For example, there's an observatory in California that scans the night sky every night looking for new “stars” that appear, because they could be supernovae. Called the Zwicky Transient Facility, it takes 600 megapixel images in tiles every 30 seconds all night, and those images are continually compared with past images to see what has changed. Those images stream in and must be processed through an image processing and neural network to determine whether other telescopes should be told to look at the supernova as it is happening.
When you add all of these up, you get a task generation rate that goes far past what a traditional batch scheduler can sustain in a multi-user environment:
And there are no coffee breaks, so once the scheduler starts to fall behind, task throughput starts falling.
Fundamental Scalability of Centralized Orchestration
If you take this behavior to the extreme, the eventual behavior is that scheduling, not compute resources, becomes the bottleneck.
If an autonomous system generates lightweight tasks non-stop at a rate that causes Slurm to take more time to dispatch tasks than the GPUs take to process those tasks, you've reached a tipping point. Tasks take longer to schedule, fewer tasks can be dispatched per second, and cluster utilization starts dropping. This is where today's HPC workload managers will fail as they try to take on agentic systems, as described in NERSC's documentation on scheduling algorithms:
To be clear, I'm not picking on Slurm here. Slurm is just an example of a fundamental architectural limit of centralized orchestration: it couples task ingestion to task scheduling in the same serial process. Mapping jobs on to a cluster is fundamentally NP-hard, which means this centralized scheduling takes longer as tasks pile up.
At VAST, we hit this exact problem when we were developing a framework for automatically processing visual data in the context of public safety. In those cases, we needed to be able to process tens, hundreds, or thousands of tasks per second where these centralized orchestration schemes don't work. So, we applied the architectural principles that we used to solve for scale in the VAST DataStore to the compute side and took an event-based approach instead.
In this approach to orchestrating, we decoupled the part of the workflow that ingests tasks from the part that dispatches tasks and allow one to scale independently from the other.
Instead of a single task queue that all incoming tasks load into, we developed a durable event broker that scales linearly. It exposes a Kafka API so you can think of it as Kafka-like, but it is actually built on the same bones as the VAST DataBase, our transactional data warehouse. When a new task is published to this event bus, it also appears as a row insert into a per-topic table.
The NP-hard problem of scheduling is also broken up and distributed as O(1) operations on the workers themselves. Instead of tasks being assigned to nodes based on the entire state of the cluster, nodes pull tasks out of the event broker.
Together, this allows us to scale up the task queue (red nodes) and the task dispatchers (purple nodes) independently to keep up with increasing task generation rates.
So far though, this is just a lightweight task queue, not a workflow orchestrator. To close the loop on that, we encapsulated this scalable approach in an event-based runtime that puts a little structure around what a task has to look like.
Specifically, a task can only be launched in response to an event, and when it completes, it generates an event. But events can be any number of things:
It can be human-initiated, just like a batch job is submitted by a human.
But it can also be scheduled, if you want to run a nightly task.
It can also act in response to changes in data. Since the VAST Event Broker is an integral part of the VAST cluster, we get events like “a file has been created” for free.
Completion of tasks also triggers an event, allowing tasks to launch in response to other tasks.
Tasks themselves are just arbitrary code and are meant to be short-lived. For complex tasks like inferencing or running an MPI job, a task might call out to an external service and immediately return.
And when a task completes, it generates an event from which another task can trigger. If the task changes any data or creates an output file as its result, those also generate events that can trigger subsequent tasks.
Mapping Event-Driven Workflows to Visual Reasoning
Earlier we discussed the ZTF workflow, where a telescope generates a set of images that must be processed in near-real time. That workflow is pretty common to a lot of vision-based tasks in both science and industry, and it boils down to four stages:
Capture: Visual data, like a set of telescope images, is captured. It lands on some storage system somewhere. ZTF uses NFS for this.
Prepare: The data is prepared for analysis. Features might be extracted and preserved as metadata using a statistical model, and these metadata are preserved and indexed. ZTF uses Postgres to save cutouts that may be supernovae.
Reason: The data and metadata are combined with external context and a model to determine if there's anything actionable in that visual data. ZTF can run their prepared data (diffs) through a neural network-based classifier.
Act: Action is taken based on the outcomes of the reasoning step. ZTF will notify other telescopes to look at transients that are likely supernovae.
However, this same four-step process maps just as easily to public safety, where instead of a telescope, we have a CCTV camera.
However, a big difference in this commercial use case is that there is never just one camera;
A large shopping mall might have hundreds of cameras.
A large international airport may have thousands of cameras.
Metropolitan areas may have tens of thousands of cameras.
As these camera networks begin to adopt AI for public safety, the number of copies of this workflow becomes immense. A shopping mall might generate 100 events/sec; an airport 1,000/sec, and a city could be 10,000/sec.
But let's just look at one camera running this workflow using an event-driven system:
A video stream lands on some unstructured storage system, triggering an object creation event.
A data preparation task triggers off this object creation, and it segments the video, runs it through a VLM to extract objects and actions from each segment. A textual description of each clip might be embedded and put into a semantic store like a vector database. Structured metadata, like the camera's location and timestamp, go into an analytical database. Once this preparation completes, the task emits a completion event.
A reasoning agent triggers off of a data preparation completion event. This agent pulls in the segments and all of its metadata to determine if there's anything actionable in it. Did someone trip and fall? Did someone spill something? If so, an action must be taken.
The agent invokes an action. It could hit a REST API that makes a phone call, or it could call an MCP tool to create a service ticket for non-urgent actions.
And because this is event-driven, you can add more sensors, generate more video streams, and the orchestration of these tasks are not impacted. You still need to make sure you have enough physical infrastructure to store these videos and run the models that analyze them, but the task-based orchestrator won't be the bottleneck.
Events are Data, So Provenance is Simpler
One very powerful side effect of using event-driven orchestration is the fact that events--the triggers that connect tasks together--are themselves structured data. An event has an owner, it has permissions, and it can contain a payload of input parameters, delegated authorizations, and everything else a task needs to execute. Data can also be structured and indexed to make it queryable, and events are no exception. This simplifies observability, because a single query can answer questions like, "how many times did this task run as that user?" This is true of other types of data generated during orchestration and execution, including logs and traces.
Finally, because events and tasks are both stored as data, they form a durable task graph every time they execute, with nodes describing actions that modified data, and edges describing the inputs and outputs of each step. After an event-driven pipeline completes, its task graph can be walked backwards to understand exactly how that pipeline's outputs were generated.
This moves the burden of recording data provenance from users or pipeline developers into the orchestration infrastructure itself. When combined with a scalable, structured database, these task graphs (and the events that are generated when objects or files are changed) can also be indexed and queried. This allows the full lineage of a file or object to be discovered through standard queries rather than interpreting pipeline-specific logs.
Compare this to how provenance is often captured in batch-scheduled systems:
Tasks are encoded in bash scripts with embedded resource directives (#SBATCH --constraint=...). Whether these scripts are saved at all, and where they are saved, is entirely up to the user.
Dependencies are simplistic and specified on the command line (sbatch --dependency=afterok:...). Records of these are stored in an unstructured SubmitLine field, accessible via the sacct command.
The files or objects that are input and output by individual tasks are completely unknown to the batch scheduler and rely on users to document this themselves.
The result of workflows run on such systems is often data whose provenance might be spread across bash scripts, a recording of the Slurm command line used to submit the job, and README files scattered across the file system. In contrast, a well-designed event-driven system captures and indexes all of this information as an integral function of executing the pipeline.
Event-Driven Orchestration is an Ideal Match for Autonomous Science
This isn't to say that batch-scheduled orchestrators are bad; it's more a reflection that autonomous systems
generate many small, latency-sensitive tasks that don't require from the complex scheduling that batch systems were designed to handle, but
demand scalable task ingest and dispatch rates that expose the fundamentally serial nature of batch systems' scheduling loops.
Batch has its place in multi-node jobs where topology locality is critical, and tasks can call out to something like Slurm when these cases arise. And when these batch jobs complete, they would then trigger an event which kicks off the next stage of the workflow, providing a seamless integration of both MPI-style and autonomously driven tasks.
However, the underlying workload of autonomous systems--high throughput, low latency tasks--require an orchestrator that is fundamentally scalable, making a strong case for autonomous systems to adopt an event-driven approach first.
₁ For example, the Yorkdale Shopping Centre in Toronto runs “over 750 cameras.”
₂ The Los Angeles International Airport (LAX) has “approximately 3,000” CCTV cameras. ₃ The Singapore Police Force operates “over 90,000” cameras across Singapore. ₄ Assuming that each camera generates clips of around 5-10 seconds each. The size of these clips governs the minimum time between an event happening and an agent responding, so shrinking the clip sizes (and generating more clips per minute) is desirable as long as the video intelligence can keep pace.
