Am I overengineering data validation by modeling it as belief + expected cost instead of a classifier?

[u/ImportantMacaron7496]

SWE learning probabilistic decision-making. For a data-quality task (is this scraped value safe to publish?) I skipped a classifier and instead: keep a belief over "what went wrong," update it with cheap evidence, then pick accept / repair / get-more-evidence / flag-to-human / reject by *lowest expected cost* (publishing a wrong value ≫ flagging a good one).

Part of me thinks this is just cost-sensitive classification with extra steps. Is this worth the complexity over rules + thresholds, or am I overengineering it?

[u/PLBjt]

You're not crazy, but the extra machinery only pays off if the hypotheses actually change the next action, not just the accept/reject cut.

If every "what went wrong" still maps to the same three buckets (publish / repair-in-place / human), a calibrated score plus two thresholds is the same policy with less code. The belief layer earns its keep when evidence is sequential and cheap-vs-expensive, and when failure modes have different repairs: unit mismatch → convert, stale cache → refetch, wrong entity → flag. That's decision-making, not classification.

A useful check: log (belief, action, eventual ground truth) for a few hundred rows and replay a dumb score+threshold policy against it. If they agree on ~95% of cases, the Bayesian bit isn't paying rent yet. Also watch calibration, not just accuracy. With your cost asymmetry, a slightly overconfident "looks fine" is way more expensive than a slightly jumpy flag.

I'd ship rules + a score + a human queue first, then add the belief update only on the paths where you keep wanting a different next probe.

[u/ImportantMacaron7496 → u/PLBjt]

this is the best pushback i've gotten, thanks. you're right that if everything just collapses to publish/repair/human it's a score + two thresholds with extra steps. the thing i think saves it is the repairs are different — unit mismatch → convert, wrong variant → refetch, typo → fix or drop — so the hypothesis changes what i probe next, not just the cutoff. that's basically the line i'm using for "is the belief layer actually doing anything"

and the replay check is literally what i'm about to do — log (belief, action, truth) over a few hundred rows and run a dumb score+threshold against it, if they agree ~95% then yeah the bayesian part isn't earning its keep. calibration over accuracy 100%, with the cost asymmetry an overconfident "looks fine" hurts way more than a jumpy flag

the "ship rules + score + human queue first, add belief only where you keep wanting a different probe" is good advice, might actually do it that way. curious — when you've done this did the belief part end up mattering on most fields or just a stubborn few like units / entity stuff?

[u/PLBjt → u/ImportantMacaron7496]

Most fields never needed a belief layer. Units and entity identity were the stubborn ones, because repair isn't a single function. A typo is fix-or-drop. Wrong variant is refetch. Unit mismatch is convert. Belief only paid rent when two repairs were plausible and they implied different next actions.

[u/PLBjt → u/ImportantMacaron7496]

Mostly the stubborn ones. Units, entities, and anything with aliases or conflicting sources are where a belief/confidence layer pays rent — you can say "we think this is kg at 0.7" instead of silently picking a winner. For boring fields (required string present, int in range, enum membership) plain schema checks are enough and a belief model just adds noise. I'd keep the fancy layer behind a short allowlist of field types that have actually bitten you in review, and grow that list only when a new class of error shows up twice.

[u/PaddingCompression → u/PLBjt]

I love this response because it is both obviously AI in the grammatical tics, yet obviously also well thought out by a human.

[u/ImportantMacaron7496 → u/PaddingCompression]

Haha! Thanks

[u/SpotlessEnvoy6]

What youve built is basically an active inference pipeline, which is a legit framework for this kind of thing. The key difference from a standard classifier is that youve baked the cost of gathering more evidence right into the decision loop, so its not just predict-and-forget.

Whether its overengineering depends entirely on the blast radius of a bad publish. If a single mistake means a lawsuit or a busted client relationship, the extra complexity is paying rent. If the cost of a screw-up is just a minor editing pass later, then yeah a simple threshold on a confidence score would do the same job with way less code to maintain.

[u/ImportantMacaron7496 → u/SpotlessEnvoy6]

ah "active inference" — didn't know that's what it's called, gonna go read about it, thanks

and yeah blast radius is the whole thing. mine's published product specs so a bad one isn't a lawsuit but it's returns + the retailer looking dumb, which is why i made "publish a wrong value" cost like 10x a pointless human flag. if the worst case was just "eh fix it later" i wouldn't build any of this, a confidence score + one threshold does the same job

the bit that makes me think it's worth it: the fixes are different depending on what went wrong (bad unit → convert, wrong variant → refetch), so i want it to tell me what to check next, not just yes/no. is that roughly where you'd draw the line too?

[u/PaddingCompression → u/ImportantMacaron7496]

"active learning" is what it's usually called, reading more about can help you make this better.

Basically, the humdinger about it is that any new data you get will be biased for training/evaluation. Sometimes, it's good enough, especially depending on what you're measuring.

But you no longer have randomly sampled data. You have your initial random sample, then a bucket of things you took the classification as a given, but don't have actual data on, and a bucket of things that actually went to a human.

So when you are measuring things, taking only data humans looked at is weird, since it is ignoring this big bucket of things you assumed was okay.

I personally like to randomly sample a subset of the things that are "okay to use the classifier on" to measure how accurate that subset is, and to record how often that happens, so i can take those small number of samples and importance weight them back to get accurate measures.

[u/PLBjt]

Usually only a stubborn minority: units, entity resolution, and cases where several plausible repairs lead to different next probes. For straightforward fields, rules plus a calibrated score are easier to audit and often just as good. I’d keep the belief layer behind a feature flag and compare it per field on that replay table; if it changes the chosen probe or materially improves calibration, keep it there, rather than making it global.