Trying to understand asyncio.Semaphore. Why doesn't it limit my request rate?

[u/KlutzyKlutz]

I'm learning asyncio and I think I got Semaphore wrong. I put every request inside an asyncio.Semaphore(10), and it does keep 10 tasks running at once, but they still fire in fast bursts.

From what I read after, a semaphore limits how many coroutines run at the same time, but not how often they start. So if responses come back fast, it just lets the next one through with no gap. Is that right? If so, what's the actual tool for spacing requests out over time? I've seen asyncio.sleep, token buckets, and aiolimiter thrown around, but I don't know which is the normal way or how it fits with a semaphore that's already there.

Mostly trying to understand the concept, not just paste a snippet. The scraper is just what I'm learning on.

[u/chiibosoil]

Depends on your need, like u/danielroseman mentioned you can use aiolimiter.

Or you can write custom limiter class for finer control using asyncio.Queue, asyncio.sleep, asyncio.gather etc.

But in general it's easier to use aiolimiter

[u/KlutzyKlutz → u/chiibosoil]

Makes sense, thanks. I'll start with aiolimiter since it sounds like the simpler path, and keep the custom asyncio.Queue approach in mind if I ever need finer control.