Small models are having a moment
The interesting movement in deployment is not at the frontier. It is at seven billion parameters and below, and it changes what a well used accelerator looks like.
Ahmed AlSharif Co-founder and CTO 2 min read
Almost every deployment conversation we have had this year has involved more than one model, and most of those models have been small.
A tuned model for the support queue. A second one for internal search. Something small doing classification in front of both of them so the expensive calls only happen when they need to. None of these are frontier models and none of them need to be.
What that does to a device
One model per accelerator was already a wasteful default when the model was large. With a small model it stops being a default and starts being an obvious mistake. You have taken a device whose main asset is a large pool of fast memory and filled a fraction of it, and then you have done that again on the next device, and the next.
The part people miss is that the waste is not only memory. A small model rarely saturates the compute either, so you are idling in two dimensions at once, and the second one does not show up on any dashboard that reports memory used.
The question stops being how big a device you need and becomes how much of one you are actually using.
The awkward implication
If your workload is a handful of small models, then buying more accelerators is close to the least effective thing you can do about capacity. The capacity is already in the building. It is sitting behind an allocation model that hands out whole devices because that was the only safe thing to do when everyone was running one large model and nobody wanted to explain a noisy neighbour.
That constraint was real. It is less real now, and it is worth checking whether your deployment still assumes it. We wrote up how far a single node goes once several models share it, and what the one-model-per-accelerator habit costs, in one model per accelerator.
Small models are not a downgrade. For most of what companies actually deploy they are the right answer. The infrastructure just has not caught up with them yet.