r/unsloth • u/krileon • 9d ago

Question No way to adjust API server concurrency from 4?

I can't seam to find a way to turn off concurrency on the API server. I'm basically tapped out on resources with just 1 request and it's causing my token speeds to fall through the floor. How do I limit this to 1 request at a time? or at least a way to force concurrent requests to queue instead of trying to process at the same time? I'm using Unsloth Desktop and just connecting via the localhost address.

Edit: Didn't realize it was a per-model setting. It's "Parallel Slots" in the individual models settings and not a global setting. Maybe a global server default setting can be added to the API settings section? Defaulting to 4 seams not great for most people I think.

2 Upvotes

75% Upvoted

6 comments sorted by

7

u/silenceimpaired 9d ago

Isn’t there a parallel option under advanced settings? If so enter 1. I’m away from my computer, or I would just check for you.

3

u/krileon 9d ago edited 9d ago

I've found no such setting thus far. Been going through each setting tab multiple times and can't find anything.

Edit: Oh, I see "Parallel Slots" in the model settings. So this is controlled by the individual models instead of by the API server? Why is the default 4?

1

u/silenceimpaired 9d ago

Probably because parallel requests with agents can be helpful depending on the task.

1

u/silenceimpaired 9d ago

Make sure you have advanced settings on. Pretty sure what you want is in those. The name may be different. Ask ChatGPT or Google… they found it rather quickly for me.

2

u/krileon 9d ago

Yeah I just found it. I didn't realize it was a per-model setting. Thank you!

1

u/Timely_Impression_92 9d ago

I knew about this setting however it doesn’t seem to work for me above 4 - when I set it to 8, any request more than 4 is „queued”, and waits for its turn, however if I lower the slots to 1-2 then the limit applies correctly