IP
Inference pipeline parallelism
Topic
What experts have said about Inference pipeline parallelism
2 statements · 2 neutral
Inference pipelining does not materially reduce memory time or compute time.
“in inference. What are we saving on? Are we saving on memory time or compute time? Not really.”
Open the episode · Reiner Pope – The math behind how LLMs are trained and servedListen at 55:35
Inference pipelining is neutral for batch size and latency.
“inference, actually the effect of pipelining on anything you care about like batch size or latency actually is neutral”
Open the episode · Reiner Pope – The math behind how LLMs are trained and servedListen at 1:00:59
Statements are attributed to the speaker as said on the episode and reflect their view at the time, not PodLume's. They are not advice.