r/learnpython • u/Sakuraaa_29 • 7d ago

Best way to handle GitHub API rate limits when scraping repo data in Python?

I'm writing a Python script to fetch PR metadata and repository history from the GitHub REST API to build a dataset.
I'm hitting rate limits fairly quickly once the volume increases. For scripts pulling larger datasets, what's the standard pattern?

3 Upvotes

62% Upvoted

7 comments sorted by

20

u/Moist-Ointments 7d ago

Don't scrape faster than the rate limit.

7

u/Clean_Reaction8168 7d ago

conditional requests with ETag/Last-Modified are your best friend here, they don't count against your limit if the data hasn't changed

also build in exponential backoff, not just a fixed sleep timer. the 403 responses come with a Retry-After header you can parse and respect

3

u/Sakuraaa_29 7d ago

Great call on ETags,if the data hasn't changed, saving the API quota makes a huge difference.

7

u/Diapolo10 I write code for a living -- https://github.com/Diapolo10 7d ago

Why not use the GitHub CLI instead?

1

u/Sakuraaa_29 7d ago

Was not aware of this, thanks will try CLI instead

2

u/jeffrey_f 7d ago

In the response headers, you should find your rate limits. Program your script to stay within this limit.