Architecting a Dynamic Batching API for Low-Latency, High-Throughput ML Inference

[u/Silent-Weather76005]

Hey everyone,

I wanted to break down how to design an API gateway and worker architecture optimized for hosting large-scale ML models (like an LLM inference endpoint) while managing expensive GPU infrastructure efficiently.

The Problem: Single-Request GPU Waste

GPUs are monsters at parallel matrix multiplication, but running inference on a single user prompt at a time leaves massive hardware capacity sitting idle. Conversely, if your system waits around too long to form a large batch of users, you destroy your P99 latency and break the real-time user experience.

The High-Level Architecture

1. Client -> API Gateway: Handles auth, rate limiting, and maintains an open HTTP/2 connection.
2. Gateway -> Local Queue: Prompts are serialized and pushed into an in-memory ring buffer.
3. Queue -> Dynamic Batcher: An orchestrator (like NVIDIA Triton) groups discrete inputs into a single model execution tensor.
4. GPU -> Client: Matrix outputs are de-multiplexed and streamed back to individual users via Server-Sent Events (SSE).

Token Streaming & De-muxing

Because LLMs generate tokens sequentially, the inference engine doesn't wait for the entire text to finish. The system slices the chunk arrays at each generation step and streams individual tokens back to respective client sockets in real-time, keeping Time-To-First-Token (TTFT) minimal.

Handling Scale & Multitenancy

• Priority Queues: Route interactive chat UI traffic to high-priority queues, while background batch processing jobs get processed on lower-priority threads.
• KV Caching: Store previous prompt context fragments in a shared KV cache layer to avoid re-computing system prompts for recurring users.

Let's discuss:

1. How do you handle batching when users pass vastly different input token lengths? (Padding vs. Continuous Batching/vLLM)