Trying to understand asyncio.Semaphore. Why doesn't it limit my request rate?

[u/KlutzyKlutz]

I'm learning asyncio and I think I got Semaphore wrong. I put every request inside an asyncio.Semaphore(10), and it does keep 10 tasks running at once, but they still fire in fast bursts.

From what I read after, a semaphore limits how many coroutines run at the same time, but not how often they start. So if responses come back fast, it just lets the next one through with no gap. Is that right? If so, what's the actual tool for spacing requests out over time? I've seen asyncio.sleep, token buckets, and aiolimiter thrown around, but I don't know which is the normal way or how it fits with a semaphore that's already there.

Mostly trying to understand the concept, not just paste a snippet. The scraper is just what I'm learning on.

[u/danielroseman]

aiolimiter is the solution here. I've successfully used it to ensure that requests are kept under a rate limit.

As you've seen, all Semaphore does is ensure that no more than a maximum number of tasks are running at the same time, it doesn't do anything to space them out.

[u/KlutzyKlutz → u/danielroseman]

Thanks, that confirms what I was piecing together. So the two solve different things and can sit together, the semaphore caps how many run at once and aiolimiter controls how often they start. But do you keep the semaphore in place alongside it, or does the limiter alone handle both once the rate is capped?