One model per accelerator

§1 · THE OBSERVATION

The observation

Published measurements put most AI clusters at 30 to 50 per cent utilisation, and inference silicon idle for more than half its life.◆◆ The figures themselves are not much disputed. What they leave open is the cause, and in our experience treating underutilisation as one problem leads to single remedies that recover only a fraction of it.

Below we separate the observation into four mechanisms. They occupy different layers of the stack, they answer to different owners, and each is individually defensible, a combination that goes a long way towards explaining why the aggregate survives.

def effective_utilisation(sustained, overhead):
    # Net of the orchestration that produced it.
    return sustained * (1 - overhead)

effective_utilisation(0.923, 0.000)  # 0.923  E4 low
effective_utilisation(0.923, 0.035)  # 0.891  E4 high
effective_utilisation(0.900, 0.200)  # 0.720  busy, not
                                     #        productive

§2 · THE CONVENTION

The convention

Underneath the four mechanisms below there is a single habit, which is that the unit of allocation is a whole accelerator and the unit of work is one model. Almost nothing in the stack requires this. It is a convention, and like most conventions it survives because at the point it was adopted it was correct: when a model filled the device it ran on, one model to one device was simply a description of the facts.

It stopped being a description some time ago. Quantisation moved a useful class of model to a fraction of a device, and the frontier moved a different class beyond any single device. The convention now fails at both ends at once, stranding memory underneath small models and refusing to place large ones, and the four mechanisms below are what that failure looks like from four different layers of the stack.

§3 · FOUR MECHANISMS

Four mechanisms

3.1Allocation granularity

In most stacks the unit of allocation is a whole accelerator, and the mismatch this creates can be large: a 7B parameter model quantised to four bits carries roughly 4 GB of weights, on a device that holds 96.◆ Nothing above the allocator can recover the difference, because the allocator has already granted the whole device.

Of the four mechanisms this is the most tractable, and the tooling for it already exists: Multi-Instance GPU[2], MPS and several serving frameworks[1] can all place more than one workload on a device. Whatever prevents their use at scale, it is not their absence.¹

3.2The memory ceiling

The opposite failure occurs when a model is too large for any single device and has to be split across several. Tensor parallel splits generally assume the devices are identical in memory, bandwidth and clocks, and where they are not, most stacks simply decline the placement.

Estates, however, are rarely uniform, because they accumulate over procurement cycles; a cluster assembled over three years will typically hold two or three generations of silicon. Capacity that could hold the model in aggregate then holds nothing while the job waits for matching hardware. What an uneven split actually requires is the subject of a companion note.

3.3The kernel gap

Between a model and the silicon sit drivers, firmware, a runtime and kernels, and at each of these the throughput actually achieved can fall below what the device is rated for, without any of the shortfall being visible to a scheduler.

A generic kernel is generic by construction: it is written for a class of device, not for a particular device in a particular thermal and memory state. The loss it produces is not a scheduling problem, and no amount of clever placement recovers it, which is presumably why it so rarely appears in utilisation discussions that begin at the orchestrator.

3.4Thermal headroom

An accelerator that cannot reject heat reduces clocks. In dense air-cooled deployments that ceiling is reached before the electrical budget is, so the sustained capacity of a device is a function of its thermal environment and not only of its specification.

We treat the measurement problem this creates in the companion note on thermal density. For the decomposition the point is narrower: a scheduler reading a nameplate figure may be reading a number the machine’s environment will never permit.

§4 · WHY THE MECHANISMS PERSIST

Why the mechanisms persist

Each mechanism has a natural owner, and no owner has all four. Allocation belongs to the scheduler, sharding to the runtime, kernels to whoever writes them, which in most stacks means the silicon vendor, and the thermal envelope to whoever designed the chassis. Each party is behaving quite reasonably on its own terms; it is the aggregate that is unreasonable, and nobody in the chain is positioned to see it.

Closing all four therefore means owning the layers they occupy. think engineers five of the six layers between a model and the silicon, and leaves the serving and training frameworks open, so that vLLM, SGLang, llama.cpp, Unsloth, HuggingFace and LlamaFactory run unmodified.◆ Four patents pending cover parts of what sits underneath.◆

§5 · THE MEMORY CEILING, IN DETAIL

The memory ceiling, in detail

Of the four mechanisms the memory ceiling is the one where the remedy is most often refused outright rather than merely missed, so it is worth following in full. The other three cost capacity. This one costs placements: the work does not run slowly, it does not run.

5.1The uniformity assumption

Splitting a model across several accelerators is routine when the accelerators match. The standard tensor parallel formulation[3] divides a layer into equal parts on the assumption that each device has the same memory, the same bandwidth and the same clocks. Where that assumption holds it is a good one, and where it does not most stacks decline the placement rather than degrade.

Declining is a defensible engineering position, but it is an expensive one, because estates do not stay uniform. They accumulate across procurement cycles, and the practical consequence is that the older half of a cluster gets written off for any model that will not fit on one of its devices.

5.2What an uneven split requires

Three conditions have to hold simultaneously. Each is straightforward on its own and the combination is not common.

1. Real device state, not nameplate state. The placement layer needs free memory, current thermal headroom and what is already resident on each device. A specification sheet does not carry any of the three, and a device operating near its thermal ceiling does not deliver its rated figure.
2. Deliberately unequal shards. Equal shards across unequal devices means the smallest device sets the shard size and the surplus on every larger device is wasted. The split has to be sized per device, which makes the partition a function of measured state rather than of device count.
3. A runtime that spans vendors. A heterogeneous estate is usually heterogeneous across vendors as well as generations. ILM runs across NVIDIA and Intel today, and the accelerators still to come, because the kernel and driver layer is engineered here rather than inherited.◆

5.3What it makes available

What the three conditions buy, once they hold, is pooling. Four nodes of quite different sizes, a MicroNode, a SuperNode, an UltraNode and a RackNode, present 2.0 TB of pooled VRAM and 21.0 PFLOPS of dense FP8 as a single system under Constellation, though none of them could run a large model alone.◆◆

def shard_sizes(free_gb):
    # Uneven split. Equal shards would let the smallest
    # device size every shard, wasting the rest.
    total = sum(free_gb)
    return [round(f / total, 3) for f in free_gb]

shard_sizes([96, 96, 48, 48])  # [.333, .333, .167, .167]
shard_sizes([96, 96, 96, 96])  # [.25,  .25,  .25,  .25 ]

# free_gb is measured device state, not nameplate: a
# device near its thermal ceiling misses its rating.

The same argument applies inside a single machine. A RackNode pools 768 GB of GDDR7 across its eight accelerators as one memory space◆, which is what brings a trillion parameter model within range for adapter fine-tuning when no individual device comes close; we work through that calculation in a companion note.

All compute figures here are dense FP8, not sparse.

5.4The cost of the split

Sharding is never free: every forward pass across a partition pays for communication that a resident model does not. The useful question is not whether the cost exists but what the alternative is, and we find it clarifying to distinguish three regimes.

• The model fits on one device. Do not split it. The communication is pure loss.
• The model fits on no single device but fits on the pool. Split it. The comparison is against the model not running at all, and almost any overhead wins that comparison.
• The model fits on one device, but throughput is bound by memory bandwidth rather than capacity. This is the interesting case, the answer depends on the interconnect and the layer shapes, and we do not have a general result to offer. We would treat any general claim here with suspicion.

Most of the disagreement in this area lives in the third regime, and it is precisely there that a placement decision has to rest on measurement rather than on a rule.

§6 · RESULT ON A SINGLE NODE

Result on a single node

On one think AI node running a mixed serving and training workload we measure 92.3 per cent sustained utilisation, at 0 to 3.5 per cent orchestration overhead.◆◆

Placement decisions weigh memory fit, thermal headroom and affinity together: models co-locate on a device when they fit, models too large for any one device shard across unequal ones, and training runs alongside serving on the same node rather than on a second estate.

The effect is easiest to see in the memory itself. Before, four small models hold one device each and most of every device sits empty; after, the same four share the node.

§7 · SENSITIVITY: WHAT THE OVERHEAD COSTS

Sensitivity: what the overhead costs

Since the overhead is reported as a range, the effective figure is a range as well: at the bottom of it, it equals the sustained figure, and at the top it comes to 89.1 per cent.

§8 · WHAT THE OVERHEAD FIGURE IS FOR

What the overhead figure is for

A utilisation figure on its own does not establish very much, because utilisation can always be raised by adding work that never finishes. The overhead figure is what bounds how much of the gain is real, being the fraction of device time consumed by the orchestration itself.

A scheme that lifted utilisation to 90 per cent while spending a fifth of device time on scheduling would have moved a number without delivering throughput. That is why we publish the two figures together, and why we would suggest asking for both from anyone making comparable claims.

§9 · WHAT THE CONVENTION COSTS

What the convention costs

Everything to this point is a capacity argument, and capacity arguments are easy to agree with and easy to defer. The question that decides anything is what the convention costs, so the rest of this note prices the same workload three ways, on published on-demand rates, with the method stated in enough detail to be checked.

The figures below are compute only on the hyperscaler side, which understates their totals rather than ours.

9.1The workload

In our experience a production system is rarely one model. The common shape is a reasoning model, two or three task models, an embedding model and a reranker, which across a stack comes to roughly a dozen models resident at once.

The mix priced below is one we have measured rather than one constructed for the comparison: a 32B reasoning model, a 7B, a 4B and a 1.1B, co-located at three per accelerator.◆ Three is the conservative end of the three to four we observe on think hardware.◆

9.2Pricing method

Rates are on-demand list prices from each vendor, retrieved on 13 August 2026 from the AWS EC2 price list, the Azure Retail Prices API and the Google Cloud SKU list, and cross-checked against each vendor public calculator. A month is 730 hours.◆ The silicon class is the same throughout.

Hyperscaler figures are compute only. Storage, egress and support plans are billed separately and are excluded. Spot, committed use, reserved and Dev/Test rates are excluded.²

9.3Results

The per-model column carries the comparison, since the workload is identical in every row; whatever spread remains is a question of how much of each accelerator is actually doing something.

from math import ceil

RATE   = 15_000  # E18  think Grid SuperNode, all inclusive
ACCEL  = 4       # accelerators in a SuperNode
MODELS = 12      # E20  a measured production mix

def monthly(models, per_accelerator):
    nodes = ceil(models / (ACCEL * per_accelerator))
    return nodes, nodes * RATE, nodes * RATE / models

for r in (1, 2, 3, 4):      # E19  measured 3 to 4
    print(r, *monthly(MODELS, r))

# 1  3  45000  3750.0
# 2  2  30000  2500.0
# 3  1  15000  1250.0  <- priced here, conservative
# 4  1  15000  1250.0

Per gigabyte is the cleaner comparison, because it prices the memory rather than the arrangement. On the same silicon, in every region these providers offer it:

9.4Sensitivity: how much co-location is doing

The twelve-model result depends on the co-location ratio, and it is worth seeing what happens as that moves. A SuperNode carries four accelerators, so at ratio r a node holds 4r models.

9.5Why the default survives

The default does not survive for technical reasons. Multi-Instance GPU[2], MPS and several serving frameworks can all place more than one model on a device, and they are available to everyone.

Our explanation is the incentive. When silicon is rented by the hour the accelerator is the product, and capacity stranded inside an instance is billed to the tenant while costing the vendor nothing, so there is no commercial reason to build the layer that reclaims it. Such a layer would be expensive to engineer, invisible in a specification comparison, and its main effect would be to reduce the number of instances a customer needs.

think sells the workload rather than the device, which puts the stranded capacity on our side of the ledger, and that, more than any technical insight, is why the orchestration layer exists. The engineering followed the commercial model rather than the other way round.

Stranded capacity is billed to the tenant and costs the vendor nothing. There is no commercial reason to reclaim it.

9.6What co-location costs

Co-location is easy to do badly. Two workloads on one accelerator can interfere with each other through memory pressure, kernel contention, or the latency coupling in which one job slows down because its neighbour got busy, and a scheme that ignores these effects produces a good utilisation figure attached to a bad service.

Doing it well means weighing memory fit, thermal headroom and affinity before two things share a device, and being able to move them apart afterwards. On think hardware the measured cost of that machinery is 0 to 3.5 per cent of device time, against 92.3 per cent sustained utilisation.◆ We would encourage asking for both figures together, from us or from anyone else.

§10 · LIMITATIONS

Limitations

• The 92.3 per cent figure is for a mixed serving and training workload on a single node. It is not a claim about arbitrary workloads, it is not a fleet figure, and a stack of uniformly large models will not co-locate as densely.
• Published cluster utilisation figures in the 30 to 50 per cent range come from different fleets using different definitions of utilisation. They establish that the problem is large. They do not establish a precise baseline against which our figure is a delta.
• The four mechanisms are not independent. The decomposition assigns ownership. It is not an additive model and the four contributions should not be summed.
• Orchestration overhead is reported as a range because it varies with the workload mix. It is not a single measured constant.
• Pooled figures are aggregate capacity. They are not a claim that an arbitrary model achieves linear scaling across the pool, and they should not be read as throughput.
• An unequal split places a larger share of the model on larger devices, so the largest device becomes the critical path. Where the largest device is also the hottest, that constraint and the thermal one interact.
• The price comparison is compute only for the hyperscalers, which understates their totals. Storage, egress and support are real costs and they are excluded. A full accounting widens the gap rather than narrowing it, and we have not attempted to quantify by how much.
• Three models per accelerator is specific to this model mix. A stack of uniformly large models co-locates less densely and the advantage narrows.
• Term and committed-use rates are excluded on both sides. think Grid term rates reduce our figure as well, so the exclusion cuts both ways.
• Regions are not interchangeable. Latency, data residency and availability differ, and the table prices compute rather than suitability.
• List prices change. The retrieval date is stated for that reason and the figures should be re-checked before they are used in a decision.

We will send the workings behind any figure in this note on request.

§11 · CONCLUSION

Conclusion

The finding we are prepared to defend is the decomposition rather than either headline number. A cluster at 40 per cent is not failing at one thing. It is losing capacity at four points that no single team owns, and a remedy applied at one of them leaves the other three intact. That is why the aggregate has barely moved while every individual layer has improved, and it is why we would treat any single-cause explanation of the utilisation gap, including a flattering one about orchestration, with suspicion.

The common thread across all four is that placement decisions are made against what a machine reports as logically available rather than against what it is physically doing. A scheduler that grants a whole device to a four gigabyte model, or reads a clock speed the thermal environment will never permit, is not making a mistake in its own terms. It is answering the question it was given. Closing the gap means changing the question, which means owning the layers where the real answer lives.

The memory ceiling is the mechanism where this is least about scheduling and most about refusal. The standard tensor parallel formulation assumes matching devices, and where the assumption fails most stacks decline the placement rather than degrade. Estates do not stay uniform, so the practical effect is that the older half of a cluster is written off for any model that will not fit on one of its devices. Pooling unequal nodes is not an optimisation in that case. It is the difference between the work running and not running.

What the money adds is a reason to think this will not fix itself. The layer that reclaims stranded capacity would be expensive to build, invisible in a specification comparison, and its main commercial effect would be to reduce the number of instances a customer needs. When silicon is rented by the hour, capacity stranded inside an instance is billed to the tenant and costs the vendor nothing. The incentive to close the gap sits with whoever sells the workload rather than the device, and almost nobody sells the workload.

That is the testable form of our claim, and it is the one worth arguing with. If the utilisation gap were a technical limit we would expect it to have narrowed as the tooling matured, because the tooling has matured: Multi-Instance GPU, MPS and several serving frameworks have been able to place more than one model on a device for years, and they are available to everyone. It has not narrowed. An explanation that turns on incentives rather than capability fits that better than one that does not.

Our own measurement is a single node under one workload, not a fleet, and it should be read that way. What it establishes is an existence proof: when the four mechanisms are addressed by one team at once, the shortfall is not a law of the technology. What it does not establish is that the same figure survives at rack or estate scale, which we have not measured and do not claim. The price comparison is compute only on the hyperscaler side, which understates their totals rather than ours, and list prices change.

For anyone running an estate today the useful version of this note is a question rather than a number. Ask which of the four mechanisms your utilisation is being lost to, and who owns each of them. Then ask any vendor quoting a utilisation figure, ourselves included, what their orchestration overhead is, because utilisation without an overhead figure beside it is unfalsifiable: it can always be raised by adding work that never finishes.