LI
LLM inference batch size
Topic
What experts have said about LLM inference batch size
3 statements · 3 neutral
The batch size should exceed roughly 300 times the model sparsity ratio.
“batch size needs to be bigger than approximately 300 times sparsity”
Open the episode · Reiner Pope – The math behind how LLMs are trained and servedListen at 19:17
Practical batch sizes should be roughly two to three times the theoretical balance point.
“take this and maybe double it or triple it”
Open the episode · Reiner Pope – The math behind how LLMs are trained and servedListen at 19:51
The balance-point batch size depends on sparsity rather than overall model scale.
“beyond that it only depends on sparsity, not on scale”
Open the episode · Reiner Pope – The math behind how LLMs are trained and servedListen at 26:04
Statements are attributed to the speaker as said on the episode and reflect their view at the time, not PodLume's. They are not advice.