Organisations rarely choose between Slurm and Kubernetes. They end up with both, usually without deciding to.
The pattern repeats often enough to be predictable. A research or data science group has run Slurm for years on hardware predating the current platform team. The platform team standardised on Kubernetes for everything else. Then AI workloads arrived, which sit cleanly in neither camp, and now training runs on Slurm while inference serves from Kubernetes, and nobody can answer a simple question about total cost across the two.
That question is worth answering, because the waste patterns are different in each and the fixes do not transfer.
Why both exist, and why neither is wrong
Slurm and Kubernetes were built for different workload shapes, and the differences are structural rather than a matter of maturity.
Slurm is a batch scheduler. Work arrives as jobs with defined resource requirements and finite duration, waits in a queue, runs to completion and releases its resources. It assumes a fixed pool of hardware and its job is to allocate that pool fairly and efficiently over time.
Kubernetes is a service orchestrator. Work is long-running, expected to stay up, and scaled horizontally in response to demand. It assumes elastic capacity and its job is to keep declared state true.
The clearest difference is how each handles a distributed job that needs several nodes at once.
Slurm allocates the whole set together. Kubernetes, by default, does not. The Kubernetes project’s own enhancement proposal for gang scheduling states the position directly: since version 1.0, the scheduler has treated the pod as the unit of scheduling, and each pod is an independent decision.
The practical consequence is the one that costs money. Submit a four-node distributed training job to Kubernetes and you can end up with three pods running and a fourth pending because no node has free GPUs. The three that started sit blocked at a synchronisation barrier, holding accelerators and doing nothing, until either the fourth arrives or somebody notices.
That single difference explains much of why serious distributed training still runs on Slurm.
Two clarifications worth having, because the terminology causes genuine confusion.
First, add-on schedulers do provide all-or-nothing semantics on Kubernetes. Volcano, Kueue, the upstream scheduler-plugins coscheduling plugin and NVIDIA’s KAI Scheduler all address this. A native in-tree API is also arriving, introduced in Kubernetes v1.36, so this is an actively moving area rather than a settled limitation.
Second, the phrase “gang scheduling” means different things in the two communities. In Kubernetes it means all-or-nothing scheduling of a set of pods. Slurm uses the same phrase for a separate feature involving time-slicing between suspended jobs. The contrast drawn here is between Slurm’s all-or-nothing allocation and Kubernetes default per-pod scheduling, which is a real difference regardless of what either feature is called.
Two schedulers, two different waste patterns
How Kubernetes wastes GPUs
Waste in Kubernetes is continuous and quiet. A pod declares resource requests, gets scheduled, and holds those resources for as long as it runs, which for a service is indefinitely. Nothing triggers a review. There is no completion event, no job record, no natural moment at which anyone compares requested against used.
Inference services are the common offender. A model server sized for peak traffic holds a full accelerator through the overnight trough. Development namespaces are worse, since notebook pods hold GPUs for the length of a working day while executing code for a few minutes of it.
How Slurm wastes GPUs
Waste in Slurm is episodic and, in one important respect, easier to find. Every job produces an accounting record with a requested allocation and an actual runtime. The evidence is there, whether or not anyone looks.
The pattern is over-requesting driven by asymmetric failure. Under-request and the job dies losing days of work. Over-request and nothing visible happens. Add time-limit padding, which degrades backfill efficiency across the whole queue, and idle periods inside long allocations during checkpoint writes and barrier waits.
Slurm’s documentation is explicit that backfill scheduling depends on reasonably accurate time limits, because the expected start time of pending jobs is calculated from when running jobs are expected to complete. Inflated limits therefore cost throughput for everyone, not just the padded job.
The consequence for tooling
Kubernetes cost tooling assumes continuous telemetry against long-lived objects and an elastic underlying pool. Slurm analysis is job-oriented and assumes fixed capacity where the meaningful currency is queue time rather than money.
A tool built for one produces confused output on the other. Which is why organisations running both typically run two sets of tools and reconcile nothing.
Making cost comparable across both
The unifying question is the same in both environments even though the mechanics differ. What was allocated, what was consumed, and what is the gap?
Four things to establish.
A single unit of account. GPU hours and node hours work across both. Currency does not, because in a mixed estate the Kubernetes side is often cloud-billed while the Slurm side runs on capitalised hardware. Comparing a monthly invoice against depreciated capital expenditure produces a meaningless number. Compare capacity first, then apply cost models separately if you need currency.
Consistent utilisation semantics. This is where most attempts fail. Kubernetes reports requests against usage per container. Slurm reports allocation against runtime per job. Neither says what the accelerator actually did. Device telemetry is the common denominator that makes them comparable, because it measures hardware behaviour independent of which scheduler asked for it.
Workload classification, not scheduler classification. The useful split is batch against service, not Slurm against Kubernetes. A batch training job has the same economics whether it runs under Slurm or as a Kubernetes Job. A long-running inference service has the same economics either way. Grouping by workload shape makes the numbers mean something.
Honest boundary accounting. Data movement between the two environments has real cost, especially where the Kubernetes side is in cloud and the Slurm side is on premise. Egress is easy to overlook and occasionally exceeds the compute savings from any placement decision.
Where workloads should actually run
Once you can see both, placement becomes a question you can answer rather than an accident of history.
Runs better on Slurm: multi-node distributed training needing all-or-nothing allocation, long-running simulation, anything requiring tightly coupled interconnect performance, and workloads with an established queue and fair-share policy that users understand.
Runs better on Kubernetes: inference serving with variable demand, anything needing autoscaling, workloads that must integrate with service meshes or standard CI and CD, and mixed workloads where GPU work is one component of a larger application.
Genuinely ambiguous: batch inference, model evaluation pipelines, and hyperparameter sweeps. These run acceptably on either, and the deciding factors are usually where the data already sits and which team owns the pipeline rather than anything technical.
One caution worth stating plainly. Consolidating onto a single scheduler is a large migration with an uncertain payoff, and it is frequently proposed for reasons of tidiness rather than economics. Running both is a legitimate steady state.
What to do with a mixed estate
Instrument the hardware, not the scheduler. Device-level telemetry gives you one truth about what accelerators did. Scheduler records tell you who asked for what. Join them on both sides and you have comparable numbers for the first time.
Report in capacity terms. GPU hours allocated, GPU hours consumed, and the gap between them, split by workload class. This single view usually reveals that one side of the estate is materially worse than the other, and it is not always the side people expect.
Fix the largest absolute waste first, wherever it lives. Not the worst percentage. A training job at 60% efficiency on sixty-four GPUs for three days wastes far more than a notebook at 5% on one card. Percentages consistently point at the wrong target.
Keep the safety model consistent. Whatever changes get made, apply them the same way in both environments. Recommendations reviewed by a human, changes scoped and reversible, and an audit trail in systems you already run. A cost programme that causes one incident does not get corrected. It gets cancelled.
The underlying point
Slurm and Kubernetes look like different worlds and are usually owned by different teams with different vocabularies. Underneath, the economics are identical.
Someone declares what a workload needs before it runs. The declaration is padded because under-requesting fails loudly and over-requesting fails silently. Nobody revisits the declaration because nothing prompts them to. The gap between declared and consumed is where the money and the capacity go.
That is one problem wearing two sets of clothes. Recognising it as one problem is what makes it possible to measure the whole estate with a single question, rather than running two programmes that never reconcile.
About OptOps. OptOps is the optimization layer for enterprise infrastructure, built to work across substrates rather than within one. The same engine reads real workload behaviour on Kubernetes in the cloud and on HPC and GPU compute on premise, producing sized recommendations with cost and confidence attached. Read-only by default, so it deploys where write-access tools cannot.
References
- Kubernetes Enhancement Proposal 4671. Gang Scheduling.
- SchedMD. Slurm Scheduling Configuration Guide, backfill scheduling.
- SchedMD. Slurm sbatch documentation, node allocation behaviour.
- SchedMD. Slurm Gang Scheduling documentation (time-slicing feature).