Two numbers that shouldn’t exist in the same quarter
Five percent.
That’s the average GPU utilization across tens of thousands of production Kubernetes clusters, according to Cast AI’s 2026 State of Kubernetes Optimization Report. This is not a survey or estimate, this is telemetry from the clusters themselves.
Here’s the second number. In the same quarter that report landed, cloud vendors raised prices on a flagship data center GPU by roughly 15% — breaking a 20-year trend of compute getting cheaper every year.
So the most expensive hardware in the enterprise is doing nothing 95% of the time, and the price of doing nothing just went up. That would be a great magic trick if it wasn’t so sad.
Cast AI’s co-founder Laurent Gil put it plainly: a GPU sitting idle costs dollars per hour; a CPU sitting idle costs cents. For context, the same telemetry puts CPU utilization at 8% and memory at 20%. The industry has been quietly wasting cents for a decade. Now it’s wasting dollars, at scale, on the one line item every board is watching.
I have never seen a waste number like this survive a single budget cycle in any other department. In AI infrastructure, it’s been called “preparedness.”
How rational people got here
Nobody set out to buy hardware and leave it dark. If we look at the procurement journey we can see how we got here.
Two years ago, access was the problem. GPUs were allocated, not purchased. You got on a waitlist. Your account team called with an allocation window that closed in days. The terms were a multi-year commitment, and if you didn’t sign, the capacity went to the next name on the list. You could hear the desperation on calls with customers who NEEDED GPUs and were trying all kinds of angles to get access.
So you signed. Reserving capacity you couldn’t yet use was the responsible move, the alternative was being the infrastructure leader who told the CEO the AI roadmap was blocked on a purchase order.
Then the projects slipped. Data wasn’t ready. Governance wasn’t ready. The pilot that justified the reservation stalled in review. The GPUs sat, depreciating on a three-to-five-year schedule that treats them as a fixed cost whether they generate one token or a billion.
Scarcity-era behavior with an efficiency-era bill.
This is a GREAT outcome for the vendor. Capacity that’s reserved, paid for, and idle is the best revenue there is. It never generates a support ticket. A vendor’s incentive is to lock the commitment, not to help you use it. And in this case stickiness was not the worry because the FOMO that drove the buying was real and still driving behavior.
Why Kubernetes didn’t fix it
The obvious question: we have the most sophisticated orchestration layer ever built running on these clusters. Why isn’t it packing the work?
Because the orchestration layer was designed for a world where compute was cheap, fungible, and abundant. It models a GPU as a CPU-shaped thing you count. It has no native sense of topology, interconnect, memory placement, or queue state. Rightsizing runs once, at deployment, and Cast AI’s own finding is that one-time configuration is the core misconception holding efficiency back. The workload changes, the traffic pattern shifts, and the config that was right six months ago is wrong today.
Every one of those gaps gets filled the same way: a plugin, a scheduler extension, a device operator, a FinOps dashboard, and a team to run them. That’s not a platform solving the problem. That’s a platform outsourcing the problem to your headcount.
The reality is that this is an orchestration-maturity problem, not a silicon-scarcity problem. The chips are here. The layer that decides what runs on them, and when, and whether they should be powered at all, was never built for this.
The CFO math
Every percentage point of utilization is worth more today than it was a year ago, because the per-hour price rose and the depreciation clock didn’t stop.
At 5% utilization, 95 cents of every GPU dollar buys nothing. Gartner puts new AI infrastructure spending in 2026 at $401 billion. You don’t need to believe the waste rate is uniform to see what that implies about the amount of capital sitting dark inside enterprise data centers and reserved cloud commitments right now.
The buyers have noticed. VentureBeat’s Q1 2026 infrastructure tracker shows “cost per inference / TCO” jumping from 34% to 41% as a top procurement priority in a single quarter, overtaking raw performance. “Access to GPUs” fell from primary concern to secondary in 90 days. The panic is over. Now the audit will begin.
And the audit will land on a simple question: why does a platform this expensive require a review cycle, a dashboard, and a specialist to figure out whether the hardware is being used?
What it looks like when the platform does the packing
This is why we built ContextOS the way we did.
In ContextOS, hardware is inventoried in real time — every node, every GPU, every gigabyte of memory, as a live map the platform maintains itself. Workloads run inside Contexts, and the platform packs those Contexts onto that map dynamically: placing by actual topology, moving work as load changes, filling the gaps a static scheduler leaves open. When demand drops, the platform scales the workload down and releases the hardware. Not at the end of the month when the FinOps report comes out. Continuously, because that’s the platform’s job.
Nobody writes a placement policy. Nobody files a rightsizing ticket. Nobody hires a team to keep the GPUs busy. The intelligence that decides where work runs lives in the platform, not in a spreadsheet next to it.
We run our own inference this way. Our internal models run on ContextOS in a dedicated Context, packed against the same hardware as everything else, scaled by load. We didn’t build a separate GPU cluster with its own operator and its own utilization problem. We built the platform so that problem doesn’t exist.
That’s the distinction that matters for the VP of Infrastructure reading this. The industry is selling you tools to measure how idle your GPUs are. Measurement is not the missing piece. What’s missing is a platform that treats a dark GPU as a bug and fixes it without asking.
Ninety-five cents of every dollar is a big number. It’s also the most fixable problem in your budget — if the fix is in the platform instead of on your team’s to-do list.
Curious what your utilization would look like on ContextOS? Bring us your cluster numbers and we’ll show you. ContactUs
Sources
- Cast AI, 2026 State of Kubernetes Optimization Report, press release April 21, 2026 — 5% GPU / 8% CPU / 20% memory utilization; Laurent Gil quote; ~15% H200 price increase; “one-time configuration” finding
- Gartner, worldwide AI spending forecast, January 2026 — $401B new AI infrastructure spending in 2026 (as reported by VentureBeat, May 8, 2026)
- VentureBeat, Q1 2026 AI Infrastructure & Compute Market Tracker, May 8, 2026 — TCO priority 34%→41%; GPU access 20.8%→15.4%; 3–5 year depreciation cycles