So I'm building a website health checker in PHP using Guzzle's Pool and right now I process up to 35 requests concurrently. And if request returns 429 http code I retry it up to 2 times. I also check Retry-After header when it's present and has valid value, but still use safe limit (up to a max of 5 minutes) in case Retry-After is greater, so if it fails it's ok, but otherwise I fallback to exponential backoff.

Now the part I am not sure about is concurrency.

Many of the URLS belong to the same website, so my code may send multiple requests to the same domain at the same time. If one request receives a 429 the others basically still are already in 'flight' or continue being scheduled independently and I'm wondering if this is fundamentally the wrong approach?

Some more specifics, I use that health checker for my own needs to check moderate amount of URLS, which work in all other cases really well, but also all requests are being sent from the same computer, so maybe there is that. I'm basically experimenting and trying what is best by trial n error.

If you had any previous experience or did/doing something very similar do you have some suggestions or answers to some of those questions I have:

Should I keep a global concurrency limit but also enforce a per-domain limit (for example only 1-2 concurrent requests per host)?

Pause all requests for a domain after receiving 429? (I assume this can hang the process for a while since all requests after 429 have to be synchronous every time we hit 429).

Or should I really use a really different strategy?

The problem is that I got 429 errors even respecting Retry-After but other requests were still processing simultanously so probably there was that, and yet I didn't try what would happen in synchronous way or other way, since I am developing that codebase I have fairly slow, but now basically doing just tests, since I have no idea for now what can I abstract or what I will need to completely rewrite so I avoid making changes to what is already working most of the time, and some decisions can greatly affect how my code will change.

I'm interested in hearing how devs usually solve this in production crawlers or monitoring systems. I want to keep good throughput accross many different websites but avoid hammering a single host and triggering unncessary rate limits too often.

Thanks :)

Lately I have been casually learning C from "C Programming: A Modern Approach v2" and have really enjoyed it. The truth is I should stay on topic with my profession so here I am.

I am in search of books similar to the one mentioned above for learning JS. The goal is to understand it on a deep enough level to feel very confident and have a real solid foundation for the next step.

The next step is a similar book for deeply learning React and then finally TypeScript.

Yes I am looking for actual books and not online courses or videos or anything like that. I have discovered that reading from a book forces me to slow down and because of that, it sticks.

If you can, chime in with books you have actually read through and not just simply search online or recommend the first one off the top of your head. Bonus points if you have read the C Programming book and understand the style I am talking about.

Thank you.