Most teams running GPU infrastructure track utilization. Far fewer are tracking what they think they are tracking.
The number almost everyone starts with comes from nvidia-smi, and it is genuinely useful for what it was designed to do. It is also probably the most misread metric in accelerated computing, and building an efficiency programme on it produces confident conclusions that are wrong.
What the utilization number actually means
When nvidia-smi reports GPU utilization at 100%, most people read it as the device working at full capacity.
NVIDIA’s own management library defines it differently. The figure is the percentage of time over the past sample period during which one or more kernels was executing on the GPU.
That is the whole definition. It is a duty cycle, a presence signal averaged over time. It says a kernel was running. It says nothing about how much of the hardware that kernel was using.
This is why teams reach a strange impasse. Utilization dashboards look healthy. Queue times are long. Researchers complain they cannot get allocation. Everything appears busy and nothing appears to be the problem.
What to measure instead
NVIDIA Data Center GPU Manager exposes device telemetry at a granularity that answers the actual question. Three profiling fields carry most of the value.
SM activity and SM occupancy
DCGM_FI_PROF_SM_ACTIVE reports the ratio of cycles during which an SM has at least one warp assigned. DCGM_FI_PROF_SM_OCCUPANCY reports the ratio of warps resident on an SM against the maximum it supports.
Together these describe how much of the GPU’s compute capability was engaged rather than whether anything at all was running. A job with high busy-time but low SM activity is exactly the profile the basic utilization number hides. This distinction is the single most useful thing DCGM gives you.
Memory bandwidth activity
DCGM_FI_PROF_DRAM_ACTIVE reports the ratio of cycles during which the device memory interface is actively sending or receiving data.
Many workloads are bound by memory bandwidth rather than compute. If bandwidth is saturated while SM activity is modest, the job is behaving normally and there is no waste to recover. Without this metric you would misdiagnose it as inefficiency and go looking for a problem that does not exist.
Memory used against memory allocated
Consistently the largest source of recoverable capacity in GPU clusters.
Researchers from Boston University and Lawrence Berkeley National Laboratory analysed 118,276 jobs on Perlmutter at NERSC, combining DCGM telemetry sampled every ten seconds with scheduler job records. Mean peak GPU utilisation across those jobs was 71.77%. Mean peak GPU memory utilisation was 28.64%. More than a third of jobs, 37.12%, never exceeded 15% memory utilisation at any point.
Memory is padded harder than anything else because memory failures are abrupt and unrecoverable. It is also frequently the constraint determining how many jobs can share a device, which means over-reservation here blocks more work than over-reservation anywhere else.
Per-device balance within a multi-GPU job
A four-GPU job where one device works and three idle is common and rarely noticed, because aggregate metrics average it away. The Perlmutter study found spatial imbalance both within nodes and across nodes in the same job, and reported that utilisation imbalance was more pronounced than memory imbalance.
This is usually worth surfacing to the researcher rather than to operations. It often means the workload is not parallelising the way its author believed, which is a correctness and performance issue as much as a cost one.
Joining telemetry to jobs, which is where it gets useful
Device metrics on their own tell you the fleet is inefficient. They cannot tell you which job, which user or which workload pattern is responsible, and without that you cannot act.
The join is the hard part and the part worth building. DCGM Exporter supports associating GPU metrics with scheduler job identifiers, which is what makes per-job idle waste calculable at all.
NVIDIA’s own engineering teams took exactly this approach on their internal research clusters. They aligned real-time DCGM telemetry with Slurm job metadata to build a unified view of how workloads actually consumed GPU resources. Slurm provided data at five-minute granularity, coarser than the DCGM fields, but sufficient for the join. On top of that they defined a dedicated GPU idle waste metric to establish a baseline and measure movement, then worked directly with the teams whose workflows drove the highest idle impact. The reported result was a reduction in GPU waste from roughly 5.5% to about 1%.
That is one organisation’s internal result rather than an industry benchmark. Three things about the programme are worth copying regardless of your environment.
They defined a specific metric rather than watching general dashboards, which meant improvement could be measured rather than asserted. They went to the users whose jobs caused the most waste instead of broadcasting policy to everyone. And they built self-serve tooling so improvements scaled beyond the people running the analysis.
NVIDIA's primary indicator in that work was the device-level utilization field, with a stated intention to move toward graphics engine activity as a more precise signal. If you are building this now, consider starting where they are heading rather than where they began.
MIG, and when partitioning genuinely helps
Multi-Instance GPU splits a supported physical card into isolated instances, each with dedicated compute and memory resources. The isolation is enforced in hardware: NVIDIA’s documentation describes each partition’s processors as having separate and isolated paths through the entire memory system, including crossbar ports, L2 cache banks, memory controllers and DRAM address buses.
That hardware isolation is the meaningful difference from time-slicing approaches.
Where MIG earns its keep is straightforward: workloads too small to justify a whole device. Inference serving, development and debugging sessions, notebook environments, preprocessing and data preparation, and CI pipelines for model code. Each routinely occupies a full high-end accelerator while using a fraction of it.
Where it does not help is equally clear. Large training runs need the full device. Any workload needing peak memory bandwidth suffers, since a partition receives a proportional slice. Jobs with widely varying resource needs do not fit fixed partition profiles well.
There is also a real operational cost. Profiles are chosen from a predefined set and configured ahead of time, so the split has to be planned against your workload mix. Reconfiguring requires draining the device first, since instances cannot be deleted while workloads are running on them. Scheduler integration adds complexity, because your scheduler must understand instance types as distinct resources.
The honest summary is that MIG is powerful in a cluster with a clear population of small jobs, and unhelpful overhead in a cluster that mostly runs large training. Look at the job size distribution before deciding.
MPS is the lighter alternative worth knowing about. It allows concurrent kernels from multiple processes without hardware isolation. NVIDIA’s documentation is explicit that MPS does not provide error isolation between clients, and that scheduling hardware, memory bandwidth, caches and capacity are all shared. That suits cooperative workloads from a single team but does not give the fault isolation a shared multi-tenant environment needs.
A practical starting sequence
If you are building this from nothing, the order that produces useful answers fastest:
Deploy DCGM Exporter with job mapping enabled. Without job context you get fleet-level numbers you cannot act on.
Define one idle waste metric and record a baseline. One clear metric everybody agrees on beats a dashboard of twenty nobody reads. You cannot demonstrate improvement without a baseline set before you started.
Rank workloads by absolute waste, not by percentage. A job running at 20% efficiency on four GPUs for an hour matters far less than one running at 60% on sixty-four GPUs for three days. Percentages mislead about where the recoverable capacity actually sits.
Talk to the top five. Waste concentrates. A small number of large recurring workloads usually account for most of it, and their owners are generally glad to know, because inefficiency in their job is also slowing their own research.
Then look at partitioning. MIG is a good answer to a question you should have measured first. Deciding on it before understanding your job size distribution is guessing with extra steps.
The point of all this
None of these measurements make anything faster on their own. They tell you where hardware you already own is doing less than it could.
In a cloud environment that translates into a smaller invoice. On owned infrastructure it translates into more jobs served, shorter queues, and a procurement cycle you can defer. Both are worth having. Only one of them shows up in a finance system, which is precisely why the other goes unmeasured for years.
About OptOps. OptOps is the optimization layer for enterprise infrastructure. The engine reads real telemetry across Kubernetes, HPC and GPU compute, learns how each workload actually behaves, and produces sized recommendations with cost and confidence attached. Read-only by default. OptOps is a member of the NVIDIA Inception programme.
References
- NVIDIA. DCGM API field identifiers documentation.
- NVIDIA. DCGM feature overview.
- NVIDIA. Multi-Instance GPU User Guide.
- NVIDIA Technical Blog. “Making GPU Clusters More Efficient with NVIDIA Data Center Monitoring Tools.” 25 November 2025.
- Sencan, E., Kulkarni, D., Coskun, A., Konate, K. “Analyzing GPU Utilization in HPC Workloads: Insights from Large-Scale Systems.” PEARC ’25, July 2025. DOI 10.1145/3708035.3736010.