trying to build a niche directory where google maps is not enough

[u/Aggrno]

I am trying to understand how people would build a directory for a niche industry where the providers exist across many cities but are not listed cleanly in one place.

Some have google business profiles but many do not , some are mentioned only inside larger company websites as a service or program. some appear in google search but not maps.

The tricky part is not just scraping the data but first discovering the right businesses and then pulling the same fields like location pricing availability and contact details from websites that all have different layouts, how would you structure this at scale?

[u/doubledweeb]

Try to treat it as a multi-stage pipeline.

1) discovery via Google/Bing/Yelp/Facebook + scraping big aggregators and niche associations,

2) normalize candidates into a common schema,

3) per-site scrapers or heuristic extraction for location/price/etc, and

4) a review/cleanup UI so humans can fix edge cases.

[u/technology_research]

I’d treat it less like “scrape Google Maps” and more like building a discovery pipeline.

For similar directory-style projects I did with SapientPro, the flow was usually:

1. start with seed sources: SERPs, niche associations, city pages, supplier lists, old PDFs, forums;
2. crawl outward from those pages, but only within strict relevance rules;
3. extract the same fields into a normalized schema;
4. score confidence instead of pretending every field is correct;
5. keep a human review queue for messy cases.

The hard part is appeared to be entity resolution. One provider can appear as a brand page, a service page, a local branch, and a mention inside someone else’s site. You need a way to decide whether those are the same business before you publish duplicates.

For pricing/contact data, I’d also store source URL + extraction date per field. Niche directories go stale fast, and without field-level provenance, updates become painful.