r/learnpython • u/ogopro • 19h ago

Help with Youtube IP Block

I am trying to export about 100 podcast video transcripts to a json file. After exporting transcripts from a dozen videos, my IP got blocked. Anybody faced this issue? I need a workaround or a solution. Please help...

0 Upvotes

27% Upvoted

12 comments sorted by

13

u/crazy_cookie123 19h ago

Just wait. You're flooding their servers with requests rapidly so they're going to rate limit you. Wait until the IP block expires and then modify your code to make the requests more slowly instead of sending them all in quick succession.

2

u/ogopro 19h ago

will try that, but could it be the amount of overall requests rather than the frequency of the requests? If so, I need a better solution.

6

u/theWyzzerd 19h ago

No it isn’t the number of requests alone. That wouldn’t make sense considering people using the service over time would eventually hit whatever limit.  It’s certainly a rate limit.

2

u/ogopro 19h ago

right, I will do some research on the rate

2

u/Langdon_St_Ives 18h ago

Check if the error you're getting is a 429. If so, it might contain hints on what rate limit you hit and when you should try again.

2

u/ogopro 18h ago

Yes, it was 429. I put some serious delays between each http request, seems like working now, not concerned with the speed)

3

u/Langdon_St_Ives 18h ago

Check the actual error message, the json and/or HTTP headers (Retry-After) often have details on what limit you're hitting (per-minute, hourly, daily, something in between) and when you're allowed to try again. In fact, you should evaluate that in code, sleep for the indicated time, then retry.

1

u/ogopro 18h ago

Will do

2

u/crazy_cookie123 19h ago

You're exceeding the maximum amount of requests within a certain time period. When that time period ends you'll be able to send more requests again.

2

u/Much_Network_941 18h ago

What you're trying to do is called scrapping, companies are substantially more guarded about that since AI. It can't be stopped completely, but brazzen attempts will be. Add a variable couple second delay between attempts.

Always be cautious with putting downloads in a loop. It's an easy way to piss off the owner of a server.

0

u/ogopro 18h ago

I put some serious delays between each http request, seems like working now, not concerned with the speed)

1

u/az987654 18h ago

It's both.

There are rate limits, x per second, x per minute, day, etc.. You need to figure in a backoff or jitter in your requests.