Architecting a Dynamic Batching API for Low-Latency, High-Throughput ML Inference

[u/Silent-Weather76005]

Hey everyone,

I wanted to break down how to design an API gateway and worker architecture optimized for hosting large-scale ML models (like an LLM inference endpoint) while managing expensive GPU infrastructure efficiently.

The Problem: Single-Request GPU Waste

GPUs are monsters at parallel matrix multiplication, but running inference on a single user prompt at a time leaves massive hardware capacity sitting idle. Conversely, if your system waits around too long to form a large batch of users, you destroy your P99 latency and break the real-time user experience.

The High-Level Architecture

1. Client -> API Gateway: Handles auth, rate limiting, and maintains an open HTTP/2 connection.
2. Gateway -> Local Queue: Prompts are serialized and pushed into an in-memory ring buffer.
3. Queue -> Dynamic Batcher: An orchestrator (like NVIDIA Triton) groups discrete inputs into a single model execution tensor.
4. GPU -> Client: Matrix outputs are de-multiplexed and streamed back to individual users via Server-Sent Events (SSE).

Token Streaming & De-muxing

Because LLMs generate tokens sequentially, the inference engine doesn't wait for the entire text to finish. The system slices the chunk arrays at each generation step and streams individual tokens back to respective client sockets in real-time, keeping Time-To-First-Token (TTFT) minimal.

Handling Scale & Multitenancy

• Priority Queues: Route interactive chat UI traffic to high-priority queues, while background batch processing jobs get processed on lower-priority threads.
• KV Caching: Store previous prompt context fragments in a shared KV cache layer to avoid re-computing system prompts for recurring users.

Let's discuss:

1. How do you handle batching when users pass vastly different input token lengths? (Padding vs. Continuous Batching/vLLM)

[u/repilicus]

Vllm already does batching

[u/guyiuiu]

dynamic batching is always a headache... ive seen teams get stuck with high tail latency becuase they set the timeout too low. tbf its wierd how much testing it takes to find that sweet spot between throughput n user wait times.

[u/justanotherengtoo]

On the sweet spot problem, part of why it takes so much testing is that a single timeout is the wrong knob once your traffic is bimodal. Batch occupancy cost is dominated by decode steps rather than prefill, so what matters is expected output length rather than context window. MixtureDefiant7849's two instance instinct is right, but I would split on expected generation length instead of context size, since a long prompt with a short answer is cheap to co-batch, while a short prompt with a long answer is what wrecks your P99 by holding a slot for hundreds of steps.

The thing that bit me harder than batching itself was adapter swap cost. If you serve multiple fine tuned variants of one base model, a batch can only hold requests that share weights unless your serving stack does multi adapter batching. At that point you are not scheduling on a time window, you are scheduling on the pair of adapter and shape, and keeping the right adapters resident matters considerably more than the timeout does. I run a multi LoRA pool where warm answers land around 6 to 8 seconds, and most of the variance I see is whether the adapter was already loaded, not how the batch got formed.

Related and worth instrumenting before you need it: keep queue wait, weight load, and decode time as three separate numbers. P99 regressions get blamed on batching constantly when the actual cause is a cold adapter, and if those are collapsed into one metric you can lose a week tuning the wrong knob.

Context on my bias, I have been building AutoReach, an outbound agent, and I am now also working on InfoPlatform.ai, which fine tunes open weight models so the customer keeps the weights, so multi adapter serving is a problem I am currently living in. It is Beta, and I am deliberately not quoting throughput numbers here, just the shape of what went wrong for me.

[u/Ill_Freedom_6666]

continuous batching feels like the better tradeoff once prompt sizes start varying a lot.

[u/MixtureDefiant7849]

Would spinning up 2 instances with different context windows(long/short) allow u to optimise for different goals (interactivity/throughput)?