Research

Memory budgets for adapter fine-tuning at trillion-parameter scale

Abstract

We work through a sizing exercise: whether a trillion parameter model, quantised to four bits and fine-tuned through adapters on a frozen base, fits inside the pooled memory of a single eight-accelerator RackNode. At the conservative end of the published footprint range it does, with 18 GB to spare, a margin of 2.3 per cent. Because that margin is smaller than the uncertainty in the multiplier it depends on, we present the result as a boundary case rather than a comfortable fit, and show how it moves with both the multiplier and the quantisation.

Findings

A trillion parameter base at four bit is 500 GB of weights before anything is added for the work of fine-tuning.
Published ranges put the total footprint of adapter fine-tuning over a frozen quantised base at 1.2 to 1.5 times the base weights. Sizing at the top of that range gives 750 GB.
A RackNode pools 768 GB across eight accelerators, so the budget closes at 750 against 768. We call that tight rather than headroom, because tight is the accurate word.
The multiplier is the only free term. The budget stops closing above k = 1.536, which is inside the published range rather than beyond it.

Think AI Research

Think AI, Riyadh

§1 · THE CALCULATION

The calculation

The natural place to start is the weights. A model of 10¹² parameters quantised to four bits occupies half a byte per parameter, which fixes the first term of the budget.

M_{weights} = N \cdot b = 10^{12} \cdot 0.5 B = 500 GB

Base weights: parameter count times bytes per parameter.

Adapter fine-tuning[1] on a frozen quantised base[2] does not carry optimiser state for the base, which is what makes the exercise tractable at all. The additional memory is adapter parameters, optimiser state for those parameters, activations retained for the backward pass, and working space.

§2 · THE MULTIPLIER

The multiplier

Published ranges for the total footprint of adapter fine-tuning over a frozen quantised base run from roughly 1.2 to 1.5 times the base weights. The spread comes from adapter rank, sequence length, batch size, and how aggressively activations are recomputed rather than retained.

We size at 1.5, the top of the range, on the view that a sizing figure which assumes the favourable end of a published range has stopped being a sizing figure.

M_{total} = k \cdot M_{weights} = 1.5 \cdot 500 GB = 750 GB

Total footprint at the conservative end of the published range.

§3 · THE RESULT AND ITS MARGIN

The result and its margin

A RackNode pools 768 GB of GDDR7 ECC across eight accelerators as a single memory space.

Select Play for automatic playback, or Step to advance one beat at a time. Space plays and pauses, the arrow keys step, R resets.

The ceiling: 768 GB pooled across eight accelerators.
Base weights land first, at 500 GB. Frozen and quantised, so they do not carry optimiser state.
Adapters, their optimiser state, retained activations and working space: 250 GB at the 1.5 multiplier.
The margin is 18 GB. That is 2.3 per cent, which is inside the error of the multiplier it depends on.

Figure 1. The budget against the ceiling, built one term at a time.

We describe 750 against 768 as tight rather than as headroom, because tight is the accurate word. At adapter ranks or sequence lengths toward the upper end of the range, or with less aggressive recomputation of activations, the budget stops closing altogether. The claim we are prepared to defend is that the configuration is in range, not that it is comfortable there.

\begin{array}{l} Δ = C - k M_{weights} = 768 - 500 k \\ Δ \geq 0 ⟺ k \leq \frac{768}{500} = 1.536 \end{array}

The whole result as one expression. C is the pooled ceiling, k the multiplier, and the budget closes at k = 1.536.

pythonthe arithmetic of this note, runnable

PARAMS     = 1e12   # E1  the case being sized
BYTES      = 0.5    # E2  four-bit, assumed not evaluated
CEILING_GB = 768    # E6  RackNode, eight accelerators

weights_gb = PARAMS * BYTES / 1e9        # E3  -> 500.0

for k in (1.2, 1.3, 1.4, 1.5, 1.55):  # E4  published
    total  = k * weights_gb              # E5
    margin = CEILING_GB - total          # E7
    print(f"{k:.2f}  {total:7.1f}  {margin:7.1f}")

# 1.20    600.0    168.0
# 1.30    650.0    118.0
# 1.40    700.0     68.0
# 1.50    750.0     18.0  <- sized here, 2.3 per cent
# 1.55    775.0     -7.0      does not fit

Because the multiplier is the only free term, the sensitivity is one line.

Figure 2. Sized footprint against the multiplier, with the 768 GB ceiling. The budget closes at k = 1.536. We size at 1.5, which leaves 2.3 per cent.

Table 1. The budget across the published multiplier range. Derived from E3, E4 and E6.

Multiplier k	Footprint	Margin	Margin %
1.20	600 GB	168 GB	21.9%
1.30	650 GB	118 GB	15.4%
1.40	700 GB	68 GB	8.9%
1.50 (sized)	750 GB	18 GB	2.3%
1.55	775 GB	−7 GB	does not fit

Register row E8. The published range stops at 1.5 and the budget closes at 1.536, so the configuration is inside the range by 2.4 per cent of the multiplier. That is the honest statement of how much room there is.

Quantisation is the other term that could move, and it moves the result much further than the multiplier does.

Table 2. Base weights for 10¹² parameters at three precisions. Derived from E1.

Precision	Bytes per parameter	Base weights	At k = 1.5	Fits 768 GB
4 bit	0.5	500 GB	750 GB	yes, at 2.3% margin
8 bit	1.0	1,000 GB	1,500 GB	no
16 bit	2.0	2,000 GB	3,000 GB	no

Four-bit quantisation is not an optimisation here, it is the condition of the result. At eight bit the model needs two RackNodes pooled under Constellation and the single-node claim does not hold.

§4 · WHAT IS OUT OF SCOPE

What is out of scope

Two neighbouring questions are deliberately out of scope. The first is pretraining: training a trillion parameter model from scratch is a different order of magnitude in both memory and time, belongs on a cluster, and any single-node claim at that scale is measuring something else.

The second is full fine-tuning. Unfreezing the base would require optimiser state proportional to the trainable parameter count, which moves the requirement by one to two orders of magnitude depending on the optimiser. We would also resist the framing of adapters as a cheaper route to the same outcome; they produce a different outcome, and for most enterprise work the one actually wanted, namely a model that knows a particular body of work rather than one with greater general capability.

§5 · WHY A SINGLE NODE MATTERS

Why a single node matters

To our mind the number itself is less interesting than the estate it removes. Fine-tuning on a separate cluster means moving data out of the environment that holds it, standing up a second set of machines, and moving the weights back afterwards. Because ILM runs the training on the same node that serves the model, there is no second estate and no transfer: the data never leaves the perimeter, and the tuned weights belong to whoever ran the job.

§6 · LIMITATIONS

Limitations

The 1.5 multiplier is a published range applied to a specific case. It is not a measurement of this configuration, and it is the assumption on which the entire result rests.
Pooled memory is not free memory. Pooling across eight devices carries its own overhead, which has not been deducted here. The calculation gives a capacity ceiling rather than an achieved footprint, and the achieved footprint will be lower.
Four-bit quantisation of a trillion parameter model is assumed rather than evaluated. Output quality under that quantisation is a separate question and this note does not address it.
A 2.3 per cent margin is inside the error of the multiplier it depends on. The correct reading is that the configuration is at the boundary, and a specific workload should be sized rather than assumed.

§7 · CONCLUSION

Conclusion

The result is that adapter fine-tuning at trillion parameter scale comes within range of a single machine, and it comes within range by 18 GB. That is the whole finding, and the margin is the most important part of it: at adapter ranks or sequence lengths toward the upper end of the published range, or with less aggressive recomputation of activations, the budget does not close at all.

Quantisation moves the answer much further than the multiplier does. At eight bit the base weights alone are 1 TB and no single machine in this class holds them. Anyone reading this as a capability claim should read the four bit assumption first, because it is doing more work than any other line in the calculation.

One further caveat belongs at the front rather than the back, because it cuts against the whole calculation. Everything above is a static sizing: a single worst-case number, fixed before the work starts, of the kind you would use to decide whether to buy a machine. The systems we build do not allocate that way. Memory is budgeted while the work is running, which means the 750 GB figure is a ceiling that a running workload should sit under rather than a level it occupies. That is the honest reading of the 18 GB margin: it is the margin on the worst case, not the margin in practice, and the two are not the same number.

What the note does not establish is throughput. A budget that closes says the work fits in memory, not that it finishes in a useful time, and we have deliberately not made a claim about the second thing. Nor does it establish that fine-tuning and serving can share a node, which is a separate claim, separately filed, and not evidenced here.

Evidence register

Every figure quoted in this note, with its source and its classification. Select any row, or any marker in the text, to open it.

Key	What	Value	Source	Class
E1	Parameter count	10^12	think AI	assumption
E2	Quantisation	Four bit, 0.5 bytes per parameter	think AI	assumption
E3	Base weights	500 GB	think AI	derived
E4	Adapter fine-tuning multiplier	1.2 to 1.5x base weights; sized at 1.5	Published ranges for adapter fine-tuning over a frozen quantised base	published
E5	Total sized footprint	750 GB	think AI	derived
E6	RackNode pooled ceiling	768 GB GDDR7 ECC across eight accelerators	think AI	specification
E7	Margin	18 GB, 2.3 per cent	think AI	derived
E8	Budget across the multiplier range	Closes at k = 1.536; 168 GB margin at k = 1.2	think AI	derived

References

External literature, cited for the technique. Our own figures are in the register above, not here.

[1]Hu, E. J., Shen, Y., Wallis, P., et al. “LoRA: Low-Rank Adaptation of Large Language Models.” 2021. arxiv.org/abs/2106.09685 (opens in a new tab)
[2]Dettmers, T., Pagnoni, A., Holtzman, A., Zettlemoyer, L. “QLoRA: Efficient Finetuning of Quantized LLMs.” NeurIPS, 2023. arxiv.org/abs/2305.14314 (opens in a new tab)

Citation

Think AI Research, “Memory budgets for adapter fine-tuning at trillion-parameter scale”, Think AI, 22 August 2026.

@misc{research2026memory,
  author       = {{Think AI Research}},
  title        = {Memory budgets for adapter fine-tuning at trillion-parameter scale},
  howpublished = {Think AI},
  year         = {2026},
  month        = {aug},
  url          = {https://think-ai.com/memory-budgets-for-adapter-fine-tuning}
}