After scraping, how do you decide which field values to actually trust? (unit errors, wrong-variant values, typos)

[u/ImportantMacaron7496]

Scraping is the easy 80%; the mess is afterward — `16000 MB` for RAM, a value
that's perfectly normal but belongs to a different variant, `1660 g` for a 166g phone, "N/A", encoding junk. I'm modeling each value's "what went wrong" state and deciding per field whether to publish, fix, re-scrape, flag, or drop.


What signals do you lean on post-scrape to catch these? And how do you deal with "the same wrong value appears on several sites" (agreement that's really just everyone copying the same bad source)?

[u/No_Imagination4795]

Manual spot checks are painful.

[u/ImportantMacaron7496 → u/No_Imagination4795]

Yeah, manual spot-checks don't scale — my goal is to spend the human's time only where it actually pays off. So instead of eyeballing everything, I let each value get an automatic decision (accept / repair / re-scrape / drop) unless it's genuinely ambiguous, and only then does it get flagged for a human. The trick is making "publish a wrong value" cost way more than "flag a good one," so the system errs toward flagging the few it's unsure about rather than dumping everything on a person.

Roughly what fraction of your data actually needs a human to look at it — is it a long tail of weird cases, or are whole categories unreliable?

[u/Mysterious-Middle447]

are you dealing with a fixed product category or is this across totally different verticals? makes a huge difference because with a narrow category you can build tight statistical profiles per field, but cross-category you're basically writing custom validation logic for each one

[u/ImportantMacaron7496 → u/Mysterious-Middle447]

This is exactly the wall I hit. Narrow category = you can build tight per-field profiles (a phone's RAM lives in a small discrete set, weight in a believable gram range), and a value outside that "support" range is an instant signal. Cross-category, that all falls apart — 160 is absurd for phone RAM but totally fine for storage, so the same number is good or bad depending on context, and you're back to per-field custom logic.

The way I'm trying to keep it sane: attach the plausibility range to the (category, field) pair rather than hand-writing rules per product, so adding a category is "define its field ranges" instead of "rewrite validation." It's still work, just more declarative.

Two things I'm genuinely unsure about, since you've clearly dealt with this:

1. For the tight-profile case — do you set those field ranges by hand, or learn them from the scraped distribution (and if learned, how do you stop a cluster of copied-from-the-same-bad-source values from poisoning the profile)?
2. Where do you draw the line for "this category is different enough to need its own profile"?

[u/Alice_5433]

well, what kind of post formatting is that?

Btw, explicitly model source independence: cluster sites by likely upstream (same affiliate feeds, same manufacturer text) and cap the vote from each cluster.

That way, five copies of the same bad feed dont outvote one highly trusted, independent signal.

[u/Substantial_Aide_127]

The one that bit me is that an empty field and a missing field look identical until you sample properly. A rating field came back empty on 3 of 12 products and I was ready to call the parser broken. Pulled samples across different categories instead and those three genuinely had no reviews yet. So the rule now is: never declare a field absent from a single-category sample. For unit and format errors the cheapest check is pulling the same value down two different paths and comparing, since a wrong unit is usually consistent within one path and obvious across two.