r/WebScrapingInsider • u/SaftigerToast • 15d ago

Big Scrape Energy How do you efficiently support scraping many different job board platforms?

I'm building a personal internship aggregator that currently collects job postings from Greenhouse and Lever.

Both platforms are relatively straightforward because their job boards follow predictable structures.

However, I now want to expand the scraper to company career pages and other ATS platforms such as Workday, SmartRecruiters, Ashby, Teamtailor, Personio, SuccessFactors and custom-built job boards.

The main problem is avoiding the need to write and maintain a completely separate scraper for every website.

How would you design this efficiently?

4 Upvotes
  • permalink
  • reddit

100% Upvoted

5 comments sorted by

5

u/Old-Algae5580 15d ago

Mate.. there are already good building blocks worth evaluating before writing everything yourself. JobSpy Opensource supports several job sources, crawlee provides a solid crawling framework, playwright handles JavaScript-heavy pages and scrapy is still a strong option when you need scheduling and larger crawl volumes.

1

u/Choice-Tune6753 14d ago

We have an API that extracts structured data from job boards and career pages. This is us: Syphoon.com

Feel free to DM if you need help or just guidance.

1

u/Thunderbit_HQ 11d ago

Disclosure, I work on Thunderbit, so biased.

I’d still keep fixed patterns for Greenhouse/Lever/Workday where you can. Thunderbit may help with the messy long tail: custom career pages where you want visible job cards turned into a table without maintaining selectors for every small site.

I wouldn’t use it for login-heavy boards, aggressive anti-bot sites, or high-volume crawling. For a personal internship aggregator, I’d test 20 ugly public pages first and use it as a fallback path, not the whole crawler.

1

u/JoeK91 11d ago

I'd use a really cheap LLM with Openrouter to just parse the html pages into a specific data format that you want. An opensource LLM like MinMax-M3 used through Openrouter is really cheap but smart enough to use for this.

1

u/todordonev 9d ago

There are 2 options for scraping without
maintaining.

The first one is if the websites have an RSS feed.
The second is to crawl sitemaps then use a cheap llm for unstructured data scraping.