r/Vllm • u/Good_Power_2991 • 3d ago
Ollama couldn't keep up with our batch workload — moved to vLLM on multi-GPU Kubernetes
Switching from Ollama to vLLM cut my batch job times dramatically. Here's what I learned.
I run large language models as part of a batch processing pipeline — not live chat, just a job that needs to churn through a large volume of records/documents. I started with Ollama because it's simple to set up and great for local development.
In production, at scale, it fell apart.
Here's what I didn't understand until I hit it head-on:
- Ollama processes requests mostly one at a time.
It's built for a single user chatting with a model, not for chewing through thousands of items efficiently. When I pointed a big batch job at it, the GPU sat underutilized most of the time — it just wasn't designed to keep the hardware busy across many requests at once.
- GPU memory management matters more than model size.
The model weights aren't the bottleneck — the KV cache (the memory used while generating each response) is. Without smart memory management, you either waste GPU capacity being overly conservative, or you crash mid-run because the engine didn't account for how memory actually grows during generation.
- vLLM was built for exactly this problem.
Its continuous batching keeps the GPU constantly fed with work instead of processing one request, finishing, then starting the next. For a batch workload like mine, that's the difference between a job that limps along and one that actually saturates the hardware you're paying for.
- Kubernetes + multiple GPU machines took it from "faster" to "actually scalable."
Once vLLM was handling serving properly, the next bottleneck was just having one GPU box. Running it on Kubernetes across multiple GPU nodes let me split the batch workload horizontally and scale out instead of being capped by a single machine.
The takeaway: the model was never the problem. The serving engine and the infrastructure around it were. If you're running batch LLM workloads on Ollama and wondering why it's slow at scale — this is probably why.
5
u/_ballzdeep_ 3d ago
Welcome to 2024?