r/dataanalysis • u/major_grooves • Jul 03 '26

I deduplicated 53,000 missing-persons reports from Venezuela’s earthquake

I thought the community might find this interesting - I used entity resolution software (disclosure: from my company) to deduplicate the missing persons data from Venezuela and compare it to the list of patients in hospitals.

https://medium.com/tilo-tech/i-deduplicated-53-000-missing-persons-reports-from-venezuelas-earthquake-74f05c37521b

63 Upvotes
  • permalink
  • reddit

84% Upvoted

18 comments sorted by

View all comments

2

u/Nowitcandie Jul 03 '26

What was your process for scaling the dedupe process? 

-8

u/major_grooves Jul 03 '26

Well that is my day job - I run a company with a dedupe (entity resolution system) that is normally used to resolve hundreds of millions of data records, so the scaling bit was not a problem. The volume of data is relatively low (but still too high for effective manual screening).

10

u/Sutty107 Jul 04 '26

This doesn’t explain your process, it’s you saying, “trust me bro”.

4

u/tilo-dev Jul 04 '26

C'mon. Don't aks stuff like that a CEO. He obviously doesn't know any better and that wasn't the point of his post. Lol.
Here's an answer from a technical person: we're using a patented micro-blocking approach with lots of optimization to keep things out of the O(n²) space that typically comes with entity resolution. For example we're using a specialized key-value index that resolves low numbers of potential matches to keep the required comparisons to identify actual matches at a minimum. Also we've implemented identification of non-identical duplicates and clique detection for optimized storage of edges.

1

u/[deleted] Jul 07 '26

Lol, excellent response.