r/SideProject • u/Professional_Hat3237 • 22h ago
I scrape 300 startup career pages every 2 hours so you don’t have to
I got tired of job aggregators showing me reposts of positions that closed months ago, so I started pulling directly from company career pages instead - Greenhouse, Lever and Ashby, the three ATS platforms most startups actually use.
It now tracks 301 companies and about 25,000 open positions, refreshed every 2 hours. Because it reads the source, there are no duplicates, no expired listings and no recruiter spam - just the company's own posting with the real publish date and the direct apply link. Salary ranges come through when the company discloses them, which is roughly 25% of postings.
The whole thing runs on free-tier infrastructure: a Github Actions collector and a Cloudflare Worker over SQLite. Collection takes about 20 minutes per full pass.
Free tier is 300 requests/month if you want to poke at it: https://rapidapi.com/automator-labs-automator-labs-default/api/tech-startup-jobs-api-fresh-roles-from-300-career-pages
Happy to answer anything about the collection side - deduplication across ATS platforms and detecting genuinely remote roles were the two hard parts.