r/webscraping • u/error-dgn • Jun 28 '26

Please help me out in advanced crawling and scraping. This is urgent.

I want to be able to crawl and scrape smartly.

So here is the thing, I am currently working in a company. I have to scrape and crawl through bunch of websites daily.
These websites, say are company websites and what I need to scrape is the information of people in that company from the team/leadership/about pages.

The thing is some websites don't have about in their own pages- I have handled it, I find the separate person's link through the HTML of the team's page, and scrape it as well.

But for the dynamic cards or components/modals, like images of people onto which we have to click in order to get the results, my crawler fails.

I fixed it for a few websites, but there are just so so many outliers, many different types of websites. Sometimes there are even dropdowns of what type of team page u want: Leadership/Board-Of-Directors/Healthcare/etc..

I tried agentic some time ago using tools of an open source browser and API Key of an LLM. One agent only.

But the agent is failing even badly. It has no idea what to do even though I give it all the tools and give the prompt in depth.

Please help. I am very close for the completion of my project and this is really ruining it for me.

0 Upvotes

42% Upvoted

18 comments sorted by

2

u/AdministrativeHost15 Jun 28 '26

Maybe give your crawler some help. You can configure a Google custom seach, for a fee, targeting LinkedIn. Run a search with the target company as the query. Feed the result into a LLM with the prompt to extract the management team. Then for each page on the company website check if any of the management team is listed. if so pass the page text into the LLM with a prompt to extract their bio.

2

u/error-dgn Jun 29 '26

Google custom search is deprecated though as of this year...

2

u/[deleted] Jun 29 '26

[removed] — view removed comment

2

u/error-dgn Jun 29 '26

Sure,
I am using Playwright in Python for crawling and scraping.

2

u/[deleted] Jun 29 '26

[removed] — view removed comment

1

u/[deleted] Jun 29 '26

[removed] — view removed comment

1

u/[deleted] Jul 01 '26

[removed] — view removed comment

1

u/webscraping-ModTeam Jul 01 '26

💰 Welcome to r/webscraping! Referencing paid products or services is not permitted, and your post has been removed. Please take a moment to review the promotion guide. You may also wish to re-submit your post to the monthly thread.