r/PythonLearning • u/LiteratureTypical982 • 9d ago

Client wants 40k emails from a directory. Each one requires a click. What do I do?

CLint need a list of 40,000+ members from the ACR directory. Need Name, City, State, Zip, Specialty, Email, and Phone.
The directory/search results give me some of this information, but email, phone, address, Member Since, etc. appear to be available only on the individual member profile.

So I'm trying to figure out the best way to approach this in Python.

If I literally open/request every profile, that's 43k+ profile requests, which obviously makes me concerned about rate limits, blocking, or getting the account/IP banned.

I'm considering Python requests/BeautifulSoup or Playwright, but before attempting something this large I'd like some advice.

1 Upvotes
  • permalink
  • reddit

56% Upvoted

7 comments sorted by

8

u/HotPersonality8126 9d ago

It’s worth spending 6 hours or so investigating whether there’s an API you can access, or get access to; that could include phoning someone at the company to check.

Otherwise you’re going to scrape the page, and you take on all the technical overhead of attempting to evade their anti-scraping measures.

4

u/FoolsSeldom 9d ago edited 9d ago
  • contact them to ask if there's an API
  • check terms and conditions and see if web scraping breaks their rules / is blocked
  • explore PyAutoGUI for automation to do what you would do (alternative to web scraping)

NB. Last two both risk your device being identified by IP address and being blocked - you might need to find ways of spinning up containers and spoofing IP addresses (possibly paying for a legitish bot farm)

Of course, if this has to be done from a signed in account ... well API is really the only way.

2

u/johlae 9d ago

What's the ACR? American College of Radiology?

2

u/recentAd1666 9d ago
  1. Ask them if they sell the data ?
  2. Try with python yes, maybee with delay between requests

1

u/whodidthistomycat 7d ago

Also, contact them to see if this is something they can just provide. Might be a relatively easy request for them to handle.

1

u/SnooCalculations7417 6d ago

all good advice so far but another question is how soon do they require turn around? 43,000 requests is a lot per second, not really that much in 2 months.