The cloud number, and what it leaves out
Renting an accelerator has a published hourly rate, which makes it feel like the easy case.
It is not, for three reasons.
The instance price covers the whole machine, not just the accelerator. A multi-GPU instance bundles CPU, memory, local storage and network alongside the devices, and if your workload uses one accelerator at a time on an eight-device instance, your effective cost per used accelerator hour is eight times the naive figure.
Storage and data movement are separate and frequently large. Training and inference both move substantial data, and in a hybrid estate the egress between environments can exceed the compute difference that motivated the placement decision in the first place.
And the rate you actually pay depends on the commitment instrument sitting underneath it. A quoted on-demand price is rarely the price on the invoice, which means any comparison built on list rates is comparing something nobody pays.
The on-premise number, and what it takes to build
Owned hardware has no hourly rate at all. It has a set of components you have to assemble into one.
Acquisition cost, amortized over the useful life you actually get rather than the accounting life on the depreciation schedule. Power, at your real tariff rather than a national average, and accelerators draw meaningfully more than the general-purpose fleet the facility was probably sized for. Cooling, usually expressed as a facility efficiency multiplier applied to power draw. Facility cost, whether that is owned space, colocation or an internal allocation. Support and maintenance contracts. Networking fabric, which for tightly coupled multi-node work is a substantial line rather than a rounding error. And the personnel cost of operating the cluster, which is omitted far more often than it should be.
Sum those, divide by hours, and you have a rate. The mechanics are covered in more depth in why the cloud playbook does not transfer on premise.
Divide by which hours, though. That is the question this whole article turns on.
Three denominators, three very different numbers
Here is the part that produces the order-of-magnitude spread.
Available hours. The hours the hardware was powered on and could in principle have run something. This produces the smallest cost per hour and it is the number most often quoted in a business case, because it makes the investment look best.
Allocated hours. The hours a scheduler had assigned the device to a job. Larger than available in cost terms, because idle time between jobs is now excluded from the denominator and its cost redistributed onto the rest.
Busy hours. The hours the device was actually executing work, as measured at the hardware rather than inferred from the scheduler.
The gap between allocated and busy is where the money is, and it is usually enormous. A cluster can report allocation in the nineties while a large share of its accelerators sit at low actual occupancy, because a cluster reporting high utilization can still be substantially idle. The scheduler is reporting honestly. It is reporting the wrong thing.
There is a fourth denominator that nobody can currently measure well and that is the one that matters most: hours doing useful work. A device pinned at high occupancy inside a distributed job that is blocked at a synchronization barrier is busy and producing nothing. So is a device spinning through a training run that will be discarded because a hyperparameter was wrong.
Why the measurement is harder than it looks
The utilization figure most teams track is the one that is easiest to obtain and the most misread in accelerated computing. It answers whether any work was scheduled on the device during a sampling interval. It does not answer how much of the device was used.
A workload occupying a small fraction of an accelerator's compute capacity can report high utilization by that measure, which means the number that looks reassuring is frequently the number hiding the problem. The richer device-level telemetry that answers the real question exists, and what it tells you that the simple number does not is the difference between an occupancy figure and a work figure.
Two structural behaviors make the gap wider than intuition suggests.
People over-request accelerators, rationally. Under-request and a job dies losing days of work. Over-request and nothing visible happens. This asymmetry produces systematic over-allocation that no policy reminder fixes, because the incentive that creates it is untouched by the reminder.
Distributed jobs hold devices while waiting. Work that needs several devices at once can end up with some running and others pending, and the ones that started sit at a barrier holding accelerators and doing nothing. The scheduling behavior that causes this differs by platform and it is a real cost, not an edge case.
Cloud versus owning, done properly
The comparison people want is whether to rent or buy. It is answerable, but only after the above.
The break-even is a function of sustained utilization, not of list prices. Owned hardware carries its full cost whether or not anything runs on it. Rented hardware carries cost only while allocated. So the crossover point is determined by how much of the year your accelerators genuinely do work, which is exactly the number most organizations cannot produce.
Which means the sequence is fixed. Establish your actual busy-hour figure first. Then compute your owned cost against that denominator rather than against available hours. Then compare, including data movement between environments, which is the item most often left out and usually the one that decides the answer.
Comparing a cloud invoice directly against depreciated capital expenditure without doing this produces whatever conclusion the person doing the comparison started with.
What to instrument
Four things, in order, and none of them requires changing a running workload.
Device-level telemetry, continuously. Occupancy, memory footprint, power draw, per device. This is read-only and it is the foundation everything else sits on.
Scheduler records joined to device telemetry. The scheduler knows who asked and for how long. The device knows what happened. Neither is sufficient alone, and joining them is what turns utilization into accountability.
Allocated against busy, by workload class. Group by batch versus service rather than by which scheduler is involved, because the economics follow the workload shape and not the platform.
A written cost model, agreed with finance before publication. The same discipline applies here as applies to cloud savings. A rate finance did not agree to in advance is not a rate, it is a slide.
What we are building, and where
We are building a GPU optimization MVP together with C-DAC, the Centre for Development of Advanced Computing, India's national organization for high performance computing.
The reason that engagement is relevant to this article rather than just a credential is that it is the hardest possible version of the measurement problem. Sovereign infrastructure, air-gapped, no cloud invoice anywhere in the picture, accelerators acquired as capital rather than rented by the hour, and an environment where write access to production is not a negotiation but a prohibition. If you can produce a defensible cost per busy accelerator hour there, the cloud version is comparatively straightforward.
Two honest qualifications, because this is a topic where vendors overclaim. This is an MVP under active development rather than a shipped, generally available capability. And we have an obvious commercial interest in GPU visibility being ranked the top unmet need in the market, so read the survey rather than taking our word for it. It is published openly under an open license and the underlying data library is free.
The underlying point
Every serious conversation about AI infrastructure economics eventually reduces to one question, and it is not about price.
It is whether the accelerator did work.
Price per hour is knowable, comparable and largely commoditized. The variance that actually determines your cost per unit of output is not in the rate. It is in the fraction of purchased or rented hours during which the hardware was doing something worth doing, and that fraction ranges enormously between organizations running identical hardware at identical prices.
Which is why the first move is not a procurement exercise or a provider comparison. It is instrumentation. Until you can state your busy-hour fraction with a source attached, every cost per GPU hour in your business case is an assumption wearing a decimal point.
About OptOps. OptOps is the optimization layer for enterprise infrastructure, built to work across substrates rather than within one. The same engine reads real workload behavior on Kubernetes in the cloud and on HPC and GPU compute on premise, reporting allocated against consumed in comparable units. Read-only by default, so it deploys where write-access tools cannot.
References
- FinOps Foundation. State of FinOps 2026. Sixth annual practitioner survey, 1,192 respondents representing more than US$83 billion in annual cloud spend. Published under Attribution 4.0 International.
- FinOps Foundation. FinOps Open Cost and Usage Specification (FOCUS).
- NVIDIA. Data Center GPU Manager documentation, for device-level telemetry fields.
- SchedMD. Slurm accounting and scheduling configuration documentation.