Ask a supercomputing centre how well its machines are used and you will get a number in the eighties or nineties.
India’s government reports exactly that for the National Supercomputing Mission. In December 2025 it stated that 37 supercomputers totalling 40 petaflops had been deployed, and that most run at over 81% capacity, with few exceeding 95%.
There is something worth noticing in that sentence. Four months earlier, in August 2025, the government reported the same 37 systems and the same 40 petaflops, but described utilisation as over 85%, with many systems exceeding 95%. Identical hardware. A lower number and a quieter adjective.
More importantly, no official source anywhere defines what the percentage measures.
That matters more than the figure itself, because in high performance computing there are two very different questions hiding behind the word utilisation, and they produce very different answers.
What HPC utilization figures actually measure
Batch schedulers were built to answer one question well. Is the machine busy?
A job requests four nodes and eight GPUs for six hours. The scheduler grants it, marks those resources occupied, and reports them as utilised for the full window.
If the job finishes its real work in ninety minutes and spends the remaining four and a half hours writing checkpoints to a slow filesystem, the scheduler still reports six hours. If it requested eight GPUs and only ever saturated two, that is still eight GPUs reported busy.
This is allocation. It tells you the hardware was handed out and did not sit unclaimed in the queue. It says nothing about what happened inside the allocation once the job started.
Consumption is the other question, and answering it requires different instrumentation entirely.
Researchers from Boston University and Lawrence Berkeley National Laboratory studied this directly on Perlmutter, the supercomputer at NERSC. Analysing 118,276 jobs from July 2024 by combining GPU telemetry with scheduler job records, they found that compute utilisation was generally well balanced across most jobs, while memory utilisation stayed consistently low in over 50% of workloads.
None of that is a scheduling failure. The jobs ran, the allocations were honoured, and the utilisation dashboard looked healthy. The hardware was still doing less than it could.
Four places the waste actually sits
Over-requested allocations
The person submitting a job declares resources in advance, before knowing exactly how the run will behave. Under-request, and the job dies at hour nine of a twelve-hour run, losing everything. Over-request, and nothing bad happens to them at all.
Anyone who has lost three days of work to an out-of-memory kill requests generously forever afterwards. That is rational, and no amount of internal messaging changes it, because the person bearing the cost of the failure is not the person bearing the cost of the padding.
Idle time inside a live allocation
Long jobs are not uniformly busy. There are data staging phases, checkpoint writes, barrier waits where the fastest ranks sit blocked until the slowest arrives, and post-processing steps that use one core out of a hundred and twenty-eight.
NVIDIA’s engineering teams treated this as a measurable quantity rather than an accepted cost. On their internal research clusters they built a dedicated GPU idle waste metric, joining device telemetry with scheduler job metadata, and reported reducing GPU waste from roughly 5.5% to about 1%.
That is one organisation’s internal result rather than an industry benchmark. What travels is the method: define the metric, set a baseline, rank workloads by how much waste each one causes, then go and talk to the teams responsible for the largest.
Memory reserved and never touched
The Perlmutter finding again, and the largest recoverable category in most GPU clusters.
Memory is the resource users are most conservative about, because memory failures are abrupt and unrecoverable. So it gets padded harder than anything else. It is also frequently the constraint that determines how many jobs can share a node, which means over-reservation here blocks more work than over-reservation anywhere else.
Wall-clock padding
Users request twelve hours for a job that takes four, because a job killed at the time limit loses everything.
Schedulers use requested duration for backfill decisions. Slurm’s own documentation is explicit that reasonably accurate time limits matter for backfill scheduling to work well, because the expected start time of pending jobs depends on when running jobs are expected to finish.
So inflated time limits degrade packing efficiency across the entire queue. One user’s caution slows everyone else down, and nothing in a utilisation report shows it.
Why HPC waste is not cloud waste with different vocabulary
This is where the analogy most people reach for breaks down.
In the cloud, waste has a legible price. Idle capacity appears on a monthly invoice. Somebody in finance can see it. The feedback loop is slow and indirect, but it exists.
On-premise HPC has no such loop. The hardware is capital expenditure, already purchased and already depreciating. Whether it runs at 40% or 90% effective utilisation, this month costs the same. Nothing appears anywhere to indicate that anything is wrong.
The cost surfaces in three other currencies.
Queue time. The real price is paid by researchers waiting for allocation. When jobs occupy more hardware than they need for longer than they need it, the queue lengthens and work that would have run this week runs next month. That is a research output cost, and it is invisible in any financial system.
Deferred capability. A centre running efficiently serves more science on the same hardware. A centre carrying substantial invisible waste needs its next procurement earlier, which in a national programme means competing for capital that has other claims on it.
Power and cooling. Occupied nodes draw power regardless of whether the accelerators inside them are doing useful work. For large systems this is a significant operating line and, increasingly, a reported sustainability metric.
There is a fourth consequence that is harder to quantify and matters more. When a centre reports high utilisation and requests expansion funding, and a reviewer later discovers that effective hardware usage was materially lower, the next funding conversation gets harder. Not through hostility. Through arithmetic.
The measurement problem underneath all of this
This persists not because HPC operators are careless, but because the standard instrumentation answers the wrong question and the tooling that answers the right one has to be assembled deliberately.
Both the NVIDIA work and the Perlmutter analysis had to join the two, and that join is not something most centres have set up.
There is a second trap waiting for teams that do start measuring. The utilization percentage most people quote is defined by NVIDIA’s own management library as the share of time over the sample period during which one or more kernels was executing. It is a presence signal, not a capacity measure. A single small kernel produces the same reading as a job saturating the device.
Build an efficiency programme on that number and you will reach confident conclusions that are wrong.
What to measure instead
Five metrics that describe reality rather than bookkeeping.
Requested against consumed, per job. For CPU, memory and GPU separately. The ratio is the finding.
GPU idle time within allocations. The share of allocated GPU seconds during which the device did no meaningful work. This requires device telemetry joined to job records, and it is the single most valuable number a centre can start producing.
Memory high-water mark against memory requested. Given that memory is the most padded resource and often the binding constraint on node sharing, this ratio usually reveals the largest recoverable capacity.
Wall-clock requested against wall-clock used. Aggregate across the queue rather than per job. It tells you how much backfill efficiency is being lost to caution.
Spatial balance within multi-GPU jobs. Whether all devices in an allocation are doing comparable work. The Perlmutter analysis found imbalance both within and across nodes, and reported that utilisation imbalance was more pronounced than memory imbalance.
None of these require changing how anybody works. They require joining two data sources most centres already generate, and then looking at the result honestly.
Where to start
If you operate an HPC or GPU cluster and want to know whether this applies to you, the fastest useful exercise takes an afternoon.
Take the twenty largest jobs from last month by node hours. For each, compare requested resources against actual consumption using whatever telemetry you already collect, then compare requested wall clock against actual runtime.
If requests sit close to consumption, your centre is genuinely efficient and you should spend your attention elsewhere. If there is a consistent multiple between them, you have found capacity you already own and are already paying to power.
The second outcome is far more common, and it does not mean anybody did anything wrong. It means the incentive structure produced exactly what it rewards, and nobody has been asked to look.
About OptOps. OptOps is the optimization layer for enterprise infrastructure. The same engine works across Kubernetes in the cloud and HPC and GPU compute on premise. It is read-only by default, which is why it deploys in environments that write-access tools cannot enter.
References
- Press Information Bureau, Government of India. National Supercomputing Mission deployment and utilisation, Lok Sabha reply, 10 December 2025.
- Press Information Bureau, Government of India. NSM status, Lok Sabha reply, 20 August 2025.
- Sencan, E., Kulkarni, D., Coskun, A., Konate, K. “Analyzing GPU Utilization in HPC Workloads: Insights from Large-Scale Systems.” PEARC ’25, July 2025. DOI 10.1145/3708035.3736010.
- NVIDIA Technical Blog. “Making GPU Clusters More Efficient with NVIDIA Data Center Monitoring Tools.” 25 November 2025.
- SchedMD. Slurm Scheduling Configuration Guide.
- NVIDIA. DCGM API field documentation.