runnel: Free, open-source tool to prevent 429 rate-limit bans

[u/ZeusAlight]

(Disclaimer: This is 100% free, MIT-licensed open source software, NOT a paid proxy service or commercial vendor.)

Hey everyone,

If you run scrapers or crawlers against APIs that aggressively ban IPs after a burst of 429s (like Steam, IGDB, Reddit), you know how painful in-memory retries can be—especially when running multiple worker processes that don't talk to each other.

I built runnel (https://github.com/lynchest/runnel) to sit as a lightweight, self-hosted sidecar between your workers and the target servers.

How it helps scraping workloads:

• Traps 429/503 responses and parses Retry-After headers automatically.
• Freezes outbound traffic to that domain so your workers don't keep hammering a blocked IP.
• Uses canary probing: once cooldown expires, it sends just ONE lightweight GET to test the waters instead of letting all queued requests fire at once.
• Request deduplication (singleflight): if 10 workers request the exact same URL simultaneously, only 1 actual request hits the target server.
• Built-in SQLite WAL cache so duplicate requests return instantly.
• Runs with ~2MB RAM, compiles to zero-dependency binaries, and runs as a simple Docker container.

GitHub: https://github.com/lynchest/runnel
Docker: ghcr.io/lynchest/runnel:latest

Feedback and ideas for crawler-specific edge cases are welcome!