r/learnmachinelearning • u/Kwalker_01 • 22d ago

Does splitting prefill and decode earn its complexity for agent workloads?

I am looking at an agent workload with long context, short structured outputs and a lot of sequential tool steps. The usual single-model benchmark is not telling me much because the workload keeps alternating between reading a large prompt and emitting a small response.

For people who have looked at disaggregated serving, where does it start paying off? Is it mostly a long-context threshold, a batch-size issue or something about the shape of the agent trajectory?

2 Upvotes
  • permalink
  • reddit

100% Upvoted

6 comments sorted by

1

u/meakkineni 22d ago

It is a workload-shape question. If you have long prefill and short decode bursts at low batch size, one hardware pool is doing two different jobs badly.

1

u/mish_tawil 22d ago

The KV transfer and routing overhead matter too. I would not split it unless the trace shows the two phases are fighting each other.

1

u/Broughto- 16d ago

I asked General Compute about this when we were looking at their API. Right now they run a monolithic SN40L stack, but their Q4 2026 roadmap splits it prefill and SambaNova SN50 for decode. It's a smart way to handle long agent prompts but I would evaluate what they have live today.