The number on the slide and the number you get
Accelerator launches are quoted in FLOPS. Which FLOPS is in the footnote, and the difference is usually a factor of two.
Ahmed AlSharif Co-founder and CTO 2 min read
Every accelerator launch comes with a headline compute figure, and the figure is almost always the largest defensible one. That is not dishonesty. It is marketing, and everybody does it. It does mean you have to read the footnote.
Sparse and dense
The large number is normally quoted with structured sparsity assumed: the hardware skips a fixed fraction of the weights because they have been pruned to a pattern it recognises. When the workload is built that way, the number is real. Most production models are not built that way. They run dense, and the figure that describes them is the dense one, which is usually about half.
So two systems can be quoted at the same headline number and differ by a factor of two on the work you are actually going to run. Two vendors can quote the same silicon and disagree, because one of them printed the sparse figure and one did not.
A spec sheet is not wrong. It is answering a question you did not ask.
What we do about it
Every performance figure on this site is dense. Not because we are unusually virtuous, but because a number that inflates when you are not looking is worse than useless to us: we would have to deflate it again before we could design anything against it, and so would you.
It does make our figures look smaller in a side by side comparison, and we have had that conversation more than once. It is still the right call. If you are going to build a thermal budget, a memory plan and a serving schedule on top of a number, that number has to be the one you will actually get.
The practical version
- Check whether the figure is sparse or dense before comparing anything.
- Check the precision it is quoted at. FP8 and FP16 numbers are not interchangeable and get printed side by side constantly.
- Check whether it is per device or per package, because two dies in one package is a popular way to double a chart.
- If a figure has no footnote at all, treat it as sparse until told otherwise.
None of this is a secret and none of it is new. It is just tedious, which is why it does not get done, which is why the headline number keeps working.