01
Batching is a scheduling problem
Once retrieval works, the next wall is latency — and most of the latency you can actually control turns out to be a question of scheduling, not hardware.
1 min
Topic
Tagged Serving
1 essay on Serving.
1 essay on Serving.
Once retrieval works, the next wall is latency — and most of the latency you can actually control turns out to be a question of scheduling, not hardware.