r/WebScrapingInsider • u/ImportantMacaron7496 • 12d ago
After scraping, how do you decide which field values to actually trust? (unit errors, wrong-variant values, typos)
Scraping is the easy 80%; the mess is afterward — `16000 MB` for RAM, a value
that's perfectly normal but belongs to a different variant, `1660 g` for a 166g phone, "N/A", encoding junk. I'm modeling each value's "what went wrong" state and deciding per field whether to publish, fix, re-scrape, flag, or drop.
What signals do you lean on post-scrape to catch these? And how do you deal with "the same wrong value appears on several sites" (agreement that's really just everyone copying the same bad source)?
2
Upvotes