KC
KV cache batching
Topic
What experts have said about KV cache batching
1 statement · 1 negative
Additional KV-cache memory traffic requires larger batches to amortize weight loading.
“If I add in more memory bandwidth, like something that consumes more memory bandwidth, then I have less available for the weight loads. And so I need to grow the memory bandwidth more and therefore the batch size more.”
Open the episode · Reiner Pope – The math behind how LLMs are trained and servedListen at 20:15
Statements are attributed to the speaker as said on the episode and reflect their view at the time, not PodLume's. They are not advice.