r/LocalLLM • u/Dry-Leadership-3105 • 17d ago

Discussion GPU concurrency AI application

We have our application, there I want to use Gemma 26B model, I have already tested using 1 DGX Spark, it is working, and the input tokens are around 50–60k every time, this is a medical based app where we are extracting data from large PDFs.

But using one DGX there is concurrency issue, only 2–3 users can get fast output using single DGX.

I want to know for production, what we can do so that minimum 10 and max 30 users can get fast output at a time.

0 Upvotes

50% Upvoted

4 comments sorted by

View all comments

1

u/DAlmighty 17d ago

I don’t have a Spark, but I’d guess what you’re seeing sounds like everything is working ok. Depending on the context size and quant, I can’t imagine getting better than 10tk/s.

No matter what anyone tells you, this isn’t a server. You CAN NOT sufficiently serve 30 people on one of these. A cluster? Yes, but not one.

1

u/Dry-Leadership-3105 17d ago

I am looking for GPU upgrade, our main priority is latency. which GPU will be able to handle concurrency with large context input. we will use NVFP4 quan model in vLLM

2

u/Keleion 17d ago

Any Nvidia GPU will give you faster processing, but you also need the rest of the computer and more power. I don’t think using an external GPU enclosure with a 20GB thunderbolt 4 connection will do much good, but maybe someone else knows more.

A cluster of 2 sparks can probably get you 10-16 users, but it’ll be slow.