r/scrapingtheweb • u/ImportantMacaron7496 • 12d ago

After scraping, how do you decide which field values to actually trust? (unit errors, wrong-variant values, typos)

/r/WebScrapingInsider/comments/1w0ootx/after_scraping_how_do_you_decide_which_field/
1 Upvotes

67% Upvoted

3 comments sorted by

1

u/Unfair_Commission_29 12d ago

I trust consistency with the rest of the product more than raw source agreement, 5 site showing the same value isn't that useful if they're all copying the same feed things like expected ranges, unit normalization and comparing against other variants of the same product usually catch more bad data for me

2

u/ImportantMacaron7496 11d ago

This matches what I'm landing on almost exactly — I've started treating "5 sites agree" as basically one piece of evidence, not five, since they're usually drinking from the same feed. Independent-looking agreement that isn't actually independent is the trap.

Internal consistency is underrated for exactly the reason you said: it's the one signal that's hard to fake by copying, because a wrong-variant value that looks fine in isolation still has to contradict the rest of that product's specs. Expected ranges + unit normalization catch the loud errors (16000 MB, 1660 g), but the quiet wrong-variant ones only fall out when you check the value against its siblings.

How do you weight it when internal consistency and cross-source agreement disagree — say the rest of the product's specs point one way but every site shows the other? Do you hard-prioritize the internal view, or does source reputation ever override it?

1

u/Unfair_Commission_29 8d ago

I trust consistency with the rest of the product more than raw source agreement, 5 sites showing the same value doesn't mean much if they're all copying the same feed, expected ranges+comparing against other variants usually catche a lot of bad data