KC
KV cache pipeline sharding
Topic
What experts have said about KV cache pipeline sharding
1 statement · 1 negative
Pipeline stages cannot effectively shard KV caches to reduce per-GPU memory.
“you also can't shard it across pipeline stages”
Open the episode · Reiner Pope – The math behind how LLMs are trained and servedListen at 1:12:51
Statements are attributed to the speaker as said on the episode and reflect their view at the time, not PodLume's. They are not advice.