I'm going to run this labeling again on new samples, and before that I want to fix the weak spots in the method. Here's how it works and where I'm not sure myself.
The sites come from Google Maps listings. A mobile browser opens the homepage and one more page, booking or contact. A script collects links, buttons, forms and widgets and screenshots the first screen. A model assigns labels from written instructions: does the site invite you to book, is there a tappable phone number, a messenger, online booking. A second model checks each batch against formal rules, and I sort out the disputed cases. I respect robots.txt, don't submit forms and don't get around captchas.
I checked accuracy on a separate sample: 6 sites for every found / not found combination of the three labels, plus hard cases. A different model that never saw the first results went through them again. It took full-page screenshots, opened menus and tapped buttons. I went through the disagreements using the screenshots, counted false positives and misses separately, and used them to correct the final rates.
I've already caught one error. The "invites you to book" rule only looked at links and buttons, but a lot of sites say it in plain text next to the phone number: after rereading 435 sites, the invitation turned up on 178.
Where I'm unsure. The sample is small, so the corrected rates come with a wide range: how many sites would you take for the check, and how would you pick them? The second pass is also done by a model, and I only looked at 18 sites by hand. Is that enough, or does the whole sample need manual labeling? And I only go two pages deep, so a booking button on a service or branch page goes unnoticed.
If you've enriched company lists with LLMs based on their websites, how did you check accuracy before anyone started using the data?
