At CoreWeave’s scale, the definition of a computer is getting bigger. A rack full of GPUs can now behave as a single logical system, and increasingly the same thinking has to extend across rows and eventually the entire datacenter.
That was one of the central points in my conversation with Jacob Yundt, Senior Director of Compute Architecture at CoreWeave on the most recent episode of VAST’s Shared Everything podcast.
Yundt has spent much of his career building what he calls “hardware clouds,” from IBM big iron through scale-out CPU infrastructure and now massive GPU environments. What has changed most is how tightly all of the pieces have to be designed together.
With large GPU systems, compute can’t be considered separately from networking, storage, power or cooling. A rack-scale system has to be managed as a unit, including how power is delivered, how it is cooled and how workloads are scheduled onto it. As density increases, that coordination expands beyond the rack.
He describes the trajectory as moving from the server as the computer, to the rack as the computer, and eventually to the datacenter as the computer. At that point, power itself becomes a resource that has to be planned and potentially moved around dynamically alongside compute capacity.
Inference makes that problem more interesting because it introduces much more hardware diversity. Training tends to favor the largest, newest clusters available, but inference can make productive use of a much wider range of GPUs. Different model sizes and workloads can be matched to different generations and classes of hardware.
That puts more responsibility on the software layer. CoreWeave has to give customers the ability to place workloads on the right hardware while balancing performance, cost and available capacity. Spot instances are one example, allowing otherwise unused GPU capacity to support workloads that can tolerate interruption. Power allocation can also become part of the optimization as demand shifts across the datacenter.
Reliability becomes equally important when jobs span thousands or tens of thousands of GPUs. Yundt notes that something as small as a link flap can take down a large distributed training job. CoreWeave has built extensive hardware validation, automation and observability around preventing those failures and identifying problems quickly when they happen. Mission Control is central to that approach, validating infrastructure and monitoring cluster health to keep as many GPUs available and productive as possible. The same principles apply as CoreWeave expands inference services because the economics still depend heavily on useful GPU runtime.
That also explains why observability has become part of the architecture rather than something added after deployment. When CoreWeave introduces new infrastructure, Yundt says one of the first considerations is what telemetry, logging and metrics can be collected and integrated into its observability systems. Much of the same visibility CoreWeave uses internally is exposed to customers through its bare-metal cloud.
The lines between hardware, firmware and software are blurring as well. Firmware that might once have been updated occasionally is now managed much more like software and integrated with Kubernetes, schedulers and telemetry platforms. Cooling conditions, power availability and hardware health can all become information the software stack needs to understand.
That approach extends to storage. Yundt points to CoreWeave’s work with VAST as an example of designing resiliency throughout the system. At this scale, failure somewhere in the infrastructure is inevitable, so the goal is to keep a hardware failure from becoming a workload failure.
The physical limits of current AI infrastructure may ultimately force even bigger changes. GPU racks continue getting denser while the power and cooling infrastructure required to support them expands dramatically. Yundt expects that within five years, today’s approaches to power delivery, cooling, cabling and connecting GPUs to CPUs and storage could look surprisingly primitive.
The industry has largely evolved existing datacenter designs to accommodate each new generation of hardware. The scale and density now coming into AI infrastructure may require something more fundamental.
For CoreWeave, that work is already pushing system design well beyond the individual GPU or server. The datacenter is increasingly the unit that has to be engineered, observed and managed as a whole.
