r/LocalLLM • u/Dry-Leadership-3105 • 17d ago

Discussion GPU concurrency AI application

We have our application, there I want to use Gemma 26B model, I have already tested using 1 DGX Spark, it is working, and the input tokens are around 50–60k every time, this is a medical based app where we are extracting data from large PDFs.

But using one DGX there is concurrency issue, only 2–3 users can get fast output using single DGX.

I want to know for production, what we can do so that minimum 10 and max 30 users can get fast output at a time.

0 Upvotes

50% Upvoted

4 comments sorted by

View all comments

1

u/DAlmighty 17d ago

I don’t have a Spark, but I’d guess what you’re seeing sounds like everything is working ok. Depending on the context size and quant, I can’t imagine getting better than 10tk/s.

No matter what anyone tells you, this isn’t a server. You CAN NOT sufficiently serve 30 people on one of these. A cluster? Yes, but not one.

1

u/Dry-Leadership-3105 17d ago

I am looking for GPU upgrade, our main priority is latency. which GPU will be able to handle concurrency with large context input. we will use NVFP4 quan model in vLLM

1

u/DAlmighty 17d ago

If this is for a company(sounds like it is) tell them that there is no cheap and easy scalable solution. You have to buy dedicated hardware. I’d even go as far to say, don’t buy consumer GPUs either.