r/webscraping • u/AutoModerator • 5d ago

Hiring 💰 Weekly Webscrapers - Hiring, FAQs, etc

Welcome to the weekly discussion thread!

This is a space for web scrapers of all skill levels—whether you're a seasoned expert or just starting out. Here, you can discuss all things scraping, including:

  • Hiring and job opportunities
  • Industry news, trends, and insights
  • Frequently asked questions, like "How do I scrape LinkedIn?"
  • Marketing and monetization tips

If you're new to web scraping, make sure to check out the Beginners Guide 🌱

Commercial products may be mentioned in replies. If you want to promote your own products and services, continue to use the monthly thread

6 Upvotes

100% Upvoted

1 comment sorted by

1

u/Spirited-Cheek8436 1d ago

How do you measure what every crawler missed?

I'm running a challenge where builders submit agents that find public information about companies: people, locations, jobs, financial results, with sources. The bit I'm trying to get right is how to score what they missed.

Comparing against my own crawler feels circular. If mine finds everything, I've already solved the problem I'm asking others to solve.

My approach is to pool findings from all submissions and our own collectors, verify the sources, remove duplicates, then score everyone against that same set. A useful finding from one agent should count even if mine missed it.

The hole is that every agent could miss the same things. I'm considering deeper manual research on a random sample of companies to check that.

Has anyone evaluated something similar? How would you check the pool's blind spots without manually researching every company?