LI
LLM inference batching
Topic
What experts have said about LLM inference batching
3 statements · 1 positive · 2 negative
Batch size is the main driver of inference latency and cost tradeoffs.
“The big effect is batch size.”
Open the episode · Reiner Pope – The math behind how LLMs are trained and servedListen at 1:49
Not batching users can make inference economics roughly 1,000 times worse.
“the cost and the economics you get can be like a thousand times worse than if you do batch many two users together”
Open the episode · Reiner Pope – The math behind how LLMs are trained and servedListen at 4:25
Small inference batches are expensive because weight-fetch costs are poorly amortized.
“The cost initially starts very high at batch size of one. Actually, it almost goes to infinity. It's because we've got so many weight fetches which are not amortized over a large batch size.”
Open the episode · Reiner Pope – The math behind how LLMs are trained and servedListen at 12:35
Statements are attributed to the speaker as said on the episode and reflect their view at the time, not PodLume's. They are not advice.