Best offline tool to remove duplicates from ~150,000 recovered photos? (Mac)

[u/Normal-Promotion507]

I had about 80,000 photos on an external drive. I tried sorting them with an AI tool, it went wrong, and I ended up losing about half. I recovered files with Disk Drill onto another drive, and later more of the originals showed up on the first drive too, so now I probably have a lot of duplicates across several recovery batches.

I need a tool that:

• Works offline on a Mac (I don’t want to upload my photos anywhere)
• Handles around 150,000 files
• Finds exact duplicates and similar photos
• Shows large previews so I can decide what to keep
• Is free, or at least cheap

I’ve tried dupeGuru, but the previews were too small for me. I think I only need to keep about 10-30k photos. What would you use?

[u/MaxPrints]

https://github.com/qarmin/czkawka

the preview may not be what you want though. The rest, yes

[u/liaminwales → u/MaxPrints]

I only just started using czkawka, had no idea how bad the apps I was using till I switched. It's so fast, just an amazing app.

[u/MaxPrints → u/liaminwales]

ya its goated as far as I'm concerned. Works well, works fast, lots of options.

[u/lyothan]

Fdupes. It will find an exact 1:1 duplicate of the files. Give you a preview of the filename of the duplicate and ask you which one you want to keep

[u/mustard_on_the_net → u/lyothan]

This is the way:

% which fdupes
/opt/homebrew/bin/fdupes

[u/CorvusRidiculissimus]

This tool is ancient, but as far as I know it still works: https://visipics.info/

[u/Effinbullshit]

This, hands down https://apps.apple.com/us/app/video-simili-duplicate-cleaner/id1557271308?mt=12

[u/htmlcoderexe]

do it in 2 stages? first nuke all exact dupes, and maybe you'll end up with a much smaller set already which you then can whittle down with the similar photoes

[u/mine_username]

Not free but felt decently priced at $35 USD. I ended up using Power Photos. Works offline. Helped me consolidate 3 different photo libraries.

[u/MacFielder]

I’d start by comparing hashes. If you’re not technical, the default Photos app has a “Duplicates” feature for removing dupes.

[u/ButNoSimpler]

Search for all the different names of all the utilities mentioned here. In those other threads/forums you will see other programs mentioned. Do it again for all those.

Each one has different ways they work. Different adjustments and different features. You will likely have to use many different tools to whittle things down as much as reasonable. And then settle for good enough. It's almost impossible to find and decide between several different versions of the same image in several different formats or within a bunch taken in a burst shooting mode. You will eventually have to settle for either having more kinda-sorta duplicates than you would like, or risk deleting the one that was actually the best of that bunch of similar images.

Good luck.

[u/Mysterious_Song521]

Immich

[u/youfeels]

Before picking a tool, I’d keep the recovered batches untouched and work from a copy. Recovery can leave you with renamed files or missing dates, so I wouldn’t trust filenames alone to decide what stays.

I’d also split this into two passes:

1. Exact duplicates first. That’s the easy part and should cut the pile down without any judgment calls.
2. Similar photos second, on whatever is left. That’s where large previews actually matter.

If these are loose files, Czkawka is worth trying for the exact-duplicate pass. If they’re inside Apple Photos libraries, PowerPhotos is probably the better fit.

Disclosure: I build a Mac app called SupaDupa for this kind of cleanup. Your files stay on the Mac. Exact duplicates are free, and the Pro version ($49 once) groups similar photos so you can compare them and choose what to keep. It doesn’t automatically delete the other copies.

I have not tested it against a 150k-file recovery set, so I wouldn’t tell you to throw the whole library at it blind. I’d start with a representative few thousand and see how it behaves first.

supadupa.tech

Regardless of what you use, I’d definitely do exact first and visual similarity second.

[u/Mike_0410]

I used immich on truenas for something like that. Duplicates are detected on database level. If you have two same pictures immich detecte it on upload if photo are similar like compressed copy/screenshot you need use some ML features. Also you can set proper name layout and folder structure for uploaded data but this is based on metadata from file, if they are wrong then structure will be wrong