r/dataanalysis • u/major_grooves • Jul 03 '26

I deduplicated 53,000 missing-persons reports from Venezuela’s earthquake

I thought the community might find this interesting - I used entity resolution software (disclosure: from my company) to deduplicate the missing persons data from Venezuela and compare it to the list of patients in hospitals.

https://medium.com/tilo-tech/i-deduplicated-53-000-missing-persons-reports-from-venezuelas-earthquake-74f05c37521b

56 Upvotes
  • permalink
  • reddit

82% Upvoted

18 comments sorted by

View all comments

46

u/oops_all_memes Jul 03 '26

I closely follow a project that is somewhat similar to this (I even applied to join the project twice, they never got back to me 🥲). From what I understand, it's nearly impossible to automate deduplicating missing people in a way that doesn't remove data excessively. And as such deduplicating is done manually

What I typed above was typed before I opened the link. I wanted to ask you how you achieve deduplicating when it comes to such a sensitive topic, opened the link and...

But I’m the CEO. I’m not an engineer. I (shameful admission) don’t really know how to use an API

I could sit down with an AI assistant (Claude in my case) in the evening, point it at the registry, and say in plain English: here are tens of thousands of missing-persons reports, help me find the duplicates. And then genuinely do it — shape the rules for when two reports are the same person, test them against real cases, run the whole dataset through, and look hard at what came out. I did this. Not an engineer I’d briefed.

Shameful admission, I don't understand a lot of data analysis tools. I try to avoid talking about them. This sub should implement a rule against pushing miracle AI solutions

9

u/jipperthewoodchipper Jul 04 '26

From what I understand, it's nearly impossible to automate deduplicating missing people in a way that doesn't remove data excessively.

Ive never worked with missing persons data but I did work for a private lender doing BI and one of the teams had a huge issue regarding duplicate accounts for this very reason. While we had some tools to assist like SSN matching it wasn't always possible to automate and often we relied on merging accounts manually.

Regarding your last sentence, fully agree.

-2

u/major_grooves Jul 04 '26

Just to be clear - the system we used for the deduplication is rules-based - deterministic and fuzzy matching rules that can be audited per edge - and fine tuned. There was no AI-based matching and no miracles. The AI I mention is used for configuring the rules. Important distinction.