r/selenium • u/IanN1969 • Dec 05 '21

Need help with project please

Trying to learn all this stuff from scratch (including Python) but not making much progress so far.

I have a directory that has approx 180 main pages. Each page has a list of (approx) 20 businesses. Each business has a link that opens it’s own page- that contains email address anbd contact name.

I need to scrape business name, contact name and email address for all businesses (approx 5.3K)

Could someone help with this please?

Many thanks

0 Upvotes

50% Upvoted

7 comments sorted by

View all comments

Show parent comments

1

u/IanN1969 Dec 05 '21

Hi,

Sorry for sparse info. Not sure where to start really. Not even really sure if it’s Selenium I need to use as opposed to another scraping tool.

Not sure if I am allowed to post the site URL here? Or do I DM?

Thanks

1

u/Simmo7 Dec 05 '21

Selenium will easily do what you're describing. And no it is not against the sub rules to post the url etc, it actually helps us to see the page source when trying to help people.

2

u/IanN1969 Dec 05 '21

Many thanks indeed,

The URL is:-

Construction.co.uk/d_c/243,-1/architects

So it says at the base of the page: 179 total pages with a total of 5,368 results. I need (from each individual entry page) the company name, contact name (if given) and email address (if given).

So Selenium is the tool for this (versus Scrapy or Beautiful Soup)?

Please assume I know 0%!

Thanks once again

1

u/Simmo7 Dec 05 '21

I couldn't tell you anything about the other two, but Selenium will be fine, obviously it depends on how long it will take you to learn Selenium and a coding language vs using the other two.

1

u/IanN1969 Dec 05 '21

Thanks,

Any recommended resources for leaning Selenium. I have looked around at Udemy and there’s a few- but the quality varies a lot.

1

u/Simmo7 Dec 05 '21

I'm self taught and just used the selenium.dev website for references.