Every HPC centre has a version of the same conversation. Someone in operations notices that jobs are reserving far more than they use. They send a message asking people to be more accurate with their requests. Nothing changes. Six months later they send it again.
The message fails for a reason worth understanding, because it explains why most attempts at this go nowhere.
Over-requesting is not a knowledge problem. Most researchers know roughly what their jobs need. They pad anyway, and given how the incentives are arranged, they are right to.
The asymmetry that drives everything
Consider the two ways a resource request can be wrong.
Request too little, and the job dies. Out of memory nine hours into a twelve-hour run, or terminated at the time limit just before the final checkpoint writes. Days of work gone. The researcher resubmits, waits in the queue again, and explains the delay to a supervisor or a funder.
Request too much, and nothing happens. The job runs. The results arrive. Nobody sends a message. The cost lands on a shared resource that no individual is accountable for, spread thinly across everyone else as slightly longer queue times.
Anyone who has looked at container resource requests in Kubernetes will recognise this exactly. Same mechanism, different vocabulary.
What it costs, beyond the obvious
Two effects. The second is worse than most people assume.
Hardware held and unused. A job holding eight accelerators while genuinely using two occupies six devices that could have run other work. Multiply that across a queue and the effective capacity of the cluster drops well below what was purchased.
Degraded scheduling for everyone else. This is the expensive one and it is almost never discussed.
Batch schedulers fill gaps by starting smaller jobs when they can finish before a larger reserved job is due to begin. That calculation depends on how long each job says it will run. Slurm’s own documentation states the point plainly: because the expected start time of pending jobs depends on when running jobs are expected to complete, reasonably accurate time limits are important for backfill scheduling to work well.
So a job that reserves twelve hours and finishes in four does not merely waste eight hours of hardware. It caused the scheduler to make eight hours of decisions based on information that was wrong, keeping other jobs out of gaps they would have fitted into. Time padding degrades throughput across the entire queue, and it does so invisibly.
Why nobody notices
The instrumentation most centres run answers a different question from the one that matters.
Schedulers record what was allocated. They know a job asked for eight accelerators and held them for twelve hours, and they report that as utilisation. What they cannot see is what the hardware did inside that window. A device reserved and idle looks identical to a device reserved and saturated.
Getting the real picture means combining scheduler records with hardware telemetry, which is a deliberate exercise most centres have never set up. Neither data source answers the question alone.
NVIDIA’s engineering teams did exactly this on their internal research clusters, defining a specific measure of GPU idle waste and tracking it rather than relying on general dashboards. They reported bringing GPU waste down from roughly 5.5% to about 1%.
That is one organisation’s internal result, not an industry benchmark. What travels is the approach. They measured a clear baseline, ranked workloads by how much waste each one caused, and worked directly with the teams responsible for the largest.
Five things that actually change behaviour
Since asking people to be more accurate does not work, the useful interventions change either what happens by default or what people can see without effort.
Show researchers their own numbers
Most users have never seen a comparison between what they requested and what they used. Not because they are uninterested, but because producing it takes effort and nobody has done it for them.
A monthly summary sent to each user, showing requested against actual consumption for their jobs, shifts behaviour more reliably than any policy document. People are not defending their padding. They have simply never been shown it.
Set defaults from observed data
If typical memory consumption on a partition sits well below the default allocation, lower the default. Researchers who genuinely need more will ask for it. The ones who were accepting whatever the default happened to be stop consuming capacity they never needed.
Defaults do more work than policies, because they require nobody to do anything.
Match hardware to workload
In a mixed cluster, a request that does not specify what kind of accelerator it needs can land a small inference task on your most capable hardware, while a large training job queues behind it.
Configuring the scheduler so jobs can request appropriate hardware, and steering light workloads away from premium devices by default, recovers meaningful capacity in any centre running more than one generation of hardware.
Separate short jobs from long ones
Short and long jobs interact badly in a shared queue. Separating them, with tighter time limits on the short queue, gives the scheduler far more room to fill gaps efficiently.
It also reduces the incentive to pad, because a job that fits the short queue gets scheduled sooner. That rewards accurate estimation rather than penalising inaccuracy, which is the right way round.
Partition devices where workloads are genuinely small
Modern accelerators can be divided into smaller isolated instances, each with a dedicated slice of memory and compute. For inference, development sessions, notebooks and data preparation, this turns one device serving one small job into several.
It suits clusters with a clear population of small jobs, and adds unhelpful complexity to clusters that mostly run large training. Look at the actual distribution of job sizes before deciding.
What good sizing looks like
Sizing well means working from observed behaviour rather than from peaks, averages, or the memory of one bad experience.
Memory should be set from the observed high-water mark across recent runs of the same workload, plus a deliberate margin. Not the largest value ever recorded, and not the mean.
Accelerator count should reflect what the job can actually keep busy. A four-device job where one is saturated and three sit idle usually means the workload is not parallelising the way its author assumed. That is worth knowing regardless of cost.
Time limits should come from observed runtime plus a margin, and are best handled through partition defaults rather than left to individual judgement. Padding here is cheap for one person and expensive for everyone.
The goal is not minimum allocation. A cluster pushed toward zero headroom fails jobs, and losing user trust costs far more than any efficiency gain is worth. The goal is removing padding that exists because nobody measured, while keeping the headroom workloads genuinely need.
Where this usually ends up
Centres that run this exercise tend to find the same shape of answer. A minority of workloads account for most of the over-requesting, and they are usually large recurring jobs whose resource specification was written once, years ago, and copied ever since.
That is good news. It means this is tractable without asking everybody to change how they work. Revisit the twenty largest recurring workloads and you will have recovered most of the available capacity.
And because the hardware is already bought and already drawing power, recovered capacity does not reduce a bill. It becomes additional research output on the same installed base.
About OptOps. OptOps is the optimization layer for enterprise infrastructure. The same engine works across Kubernetes in the cloud and HPC and GPU compute on premise, learning how each workload actually behaves and producing sized recommendations rather than applying static rules. Read-only by default.
References
- NVIDIA Technical Blog. “Making GPU Clusters More Efficient with NVIDIA Data Center Monitoring Tools.” 25 November 2025.
- SchedMD. Slurm Scheduling Configuration Guide, backfill scheduling.
- SchedMD. Slurm Frequently Asked Questions, time limits and backfill.