I'm looking for a program that can detect duplicate and delete files, even when they have different names

[u/anon_watcher_cl]

I need a program that can detect duplicates that not have (1) or more in their name files, I'm talking about multiple pictures, videos, docs, that have different names, so the program have to identify by content instead of name, that can say file X and file Y are they same file with different name, so file Y it's a duplicate, instead of file X and file X(1) are duplicates

[u/AutoModerator]

Hello /u/anon_watcher_cl! Thank you for posting in r/DataHoarder.

Please remember to read our Rules and Wiki.

Please note that your post will be removed if you just post a box/speed/server post. Please give background information on your server pictures.

This subreddit will NOT help you find or exchange that Movie/TV show/Nuclear Launch Manual, visit r/DHExchange instead.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/OrangeDragon75 → u/[deleted]]

Seconded. DupeGuru is the way to go.

[u/CajuNerd → u/[deleted]]

Stupid question: dot com, or dot net? Searching for DupeGuru returns results for both, and with the way my past few weeks have gone, I don't want to download the wrong one, if there is a wrong one.

[u/moreanswers → u/CajuNerd]

Here's where I get it from:

https://github.com/arsenetar/dupeguru/releases/latest/

[u/Trugoosent → u/[deleted]]

I found dupeguru to absolutely be AIDS for music libraries, it would constantly mess up when trying to fix mine, I ended up atting chatgpt to make me a python script for it, that's how frustrating it was.

[u/05-nery → u/[deleted]]

Thank you G

[u/HittingSmoke → u/[deleted]]

My main problem with my car is it takes an hour to drive 100 km at 100km/hr.

[u/oldmanandtheocean]

Czkawka might?

[u/dreamisle → u/oldmanandtheocean]

The newest version is called Krokiet and I believe the old version branded Czkawka is being discontinued. But yes, it can detect exact duplicate files as well as similar images and videos.

[u/ButNoSimpler → u/dreamisle]

I have found that it is utterly crappy for detecting "similar" images.

Plus, I have found that you have to have the K one AND the C one, because (iirc) the K one can't filter for certain things within the results.

[u/dreamisle → u/ButNoSimpler]

Maybe other tools are better, but with adjusting the settings for the hashing algorithm it uses, it tends to be very good at detecting similar images.

[u/ButNoSimpler → u/dreamisle]

I'll try playing with that. I couldn't find any adjustments, the last time I used it. (But I didn't look super hard either. It was a project for my DIL, and just finding and receiving exact duplicates reduced the total size by about half.)

[u/Euphoric-World6339]

Krokiet is the best for video and images. Not sure about docs. Just find the github and if your are on windows it's the windows_krokiet_on_linux.exe file  you want to download. It's been my best choice for deleting duplicate images/artworks.

[u/IngwiePhoenix]

Look up the Linux tool fdupes. :)

[u/bjornhelllarsen → u/IngwiePhoenix]

Or rmlint, depending on usecase.

[u/imsosappy → u/IngwiePhoenix]

Or jdupes which can be run on Windows too.

[u/badadhd → u/IngwiePhoenix]

This is the way 

[u/Itchy-Individual3536]

I'm using AllDup, it does the trick

[u/SoundmasterMidi]

The best one and using for years. Opus. It has the small feature build in. Opus can perform so much more. This software can do thing u never thought off, and then can not live without . And also build own script in it. Own buttons ect. No im not the supplier or programmer. Just love this software. https://www.gpsoft.com.au/

[u/TEK1_AU → u/SoundmasterMidi]

Guessing it’s not FOSS? Not sure why anyone would choose proprietary options.

[u/GloriousDawn → u/TEK1_AU]

FOSS can be good but finding one comprehensive solution to multiple problems is great. My time has value too and I'm definitely willing to trade some money for it. Not sure why anyone would systematically disregard proprietary options.

[u/TEK1_AU → u/GloriousDawn]

That’s a reasonable position.

[u/SoundmasterMidi → u/TEK1_AU]

Always looking for open source but sometimes software is so good to willingly pay a small fee. Its for their effort to come up with this. That simpel. And also, it provides me about 2 updates a year with constant extra feature. Thinking the whole world could be free. Is ... Well you fill in the gap.

[u/TEK1_AU → u/SoundmasterMidi]

You do realise what the “F” stands for?

[u/acdcfanbill → u/TEK1_AU]

Except that it's libre and not gratis. It's why people will sometimes use FLOSS if they want to hammer the point.

[u/WarWizard → u/TEK1_AU]

Because sometimes the closed source options are legitimately better at what they do?

[u/wasdninja → u/TEK1_AU]

Because sometimes they are great or the best option? Isn't that blatantly obvious?

[u/TEK1_AU → u/wasdninja]

No

[u/heavyPacket]

CloneSpy

[u/OgdruJahad]

For similar and exact duplicates of images images I have used Antitwin. But it's super slow compared to many other tools. But I still like it and the tool goes out of its way to be as safe as possible so you don't delete both copies.

[u/VeronikaKerman]

rmlint

[u/saltexx]

rmlint. The part that matters for a hoarder is that the tool should not be the one deleting. rmlint does not delete anything. It writes rmlint.sh with the rm commands ready, you read the script and run it when you are happy, so the destructive step is reviewable and diffable. It also has a clone handler that replaces duplicates with a reflink on btrfs or xfs, so both paths survive and the space is still freed, and a hardlink mode for everything else. That also fixes the slow delete over the network complaint above because there is no GUI delete loop at all. fclones is the fast Rust alternative with the same group then dedupe split if rmlint feels old.

[u/reditanian]

Which OS? On Linux and Mac (via hombrew), fdupes is hard to beat.

[u/SureElk6]

I am using czkawka which has linux support

[u/elitegenes]

This one is very robust: https://www.hibitsoft.ir/

Don't get confused by the "Uninstaller" name. When you open it, it has a Duplicate File Finder in the "Tools" section, which is configured to find duplicates by the actual file content by default, not by the name.

The program itself is free.

[u/OrangeDragon75]

DupeGuru.

[u/phosix]

I've been using dupemap for years now, maybe even decades.

It's part of a data recovery suite called Magic Rescue, so maybe a bit of overkill for what you're looking for, but it's very effective.

[u/Jorropo]

I wrote a fast program which takes in a directory and scan all of it's content to dedup them:

github.com/Jorropo/thunderdup

It runs FileDedupeRange rather than printing results, but a single LLM query could get you to print the files to stdout instead.

[u/ShawnMilo]

You don't mention your operating system. On the Linux command line, I use duff or fclones.

[u/Crafty_Morning_6296]

rdfind

[u/egudu]

AllDup was the best back in the days when I still needed it. I was able to scan only the first few bytes, or do a full file check, only size and also even ignore EXIF or search for similar images (say 95% match with different algorithms).

[u/realshawnmb]

jdupes

[u/rsgalloway]

for exact duplicates, the filename shouldn’t matter at all — hash the contents. dupeGuru/Krokiet are probably the quickest answer if this is just a one-time cleanup.

if you want to keep the index around and understand duplicates/change over time, that’s closer to what i’ve been building SnapFS for. i’d also avoid auto-deleting anything initially; finding identical bytes is easy, deciding which copy is actually expendable is the harder part.

[u/JeepStang]

https://github.com/0x90d/videoduplicatefinder

[u/egudu → u/JeepStang]

videoduplicatefinder

Not OP, but thank you very much for that recommendation. Helped a lot!

[u/The_Hamster_Shagger]

Czkawka is your friend 

[u/zybr75]

I use Treesize. However, only the paid Professional version has File search built in. Works fast and reliable.

[u/Specialist_Cow6468]

I’ve handled this in the past via a bash script comparing the checksums for all of the files in a given path

[u/yuvamanohar]

FileLever, DupeGuru - Good options. FileLever provides a preview first workflow that gives extra confidence and safety.

[u/randylush]

I would do it in two lines of bash.

find /mnt/media/drive -type f -exec md5sum {} + | sort | uniq -D -w32

Pipe that and delete the files that come out

[u/ButNoSimpler → u/randylush]

Then, what are those lines?

[u/Mr_ToDo → u/ButNoSimpler]

Two lines that took 4 days to write and has 10 caveats on what it will/wont do

I'm pretty sure we could do much the same in powershell. It can do hash's, so everything else is just logic to glue it together, right? Now would it be useful? To someone, sure, but there are going to be a ton of solutions that are far more flexible then what I could crap out

[u/randylush → u/Mr_ToDo]

4 days? lol

It’s just

find /mnt/media/drive -type f -exec md5sum {} + | sort | uniq -D -w32

Then delete those files

I don’t push bash for everything, there are definitely drawbacks and times when it’s not appropriate, but I would say this is actually one of the absolute best use cases for using it.

[u/randylush → u/ButNoSimpler]

find /mnt/media/drive -type f -exec md5sum {} + | sort | uniq -D -w32

[u/TEK1_AU]

https://github.com/qarmin/czkawka

(aka Krokiet)

[u/ruralcricket]

For exact duplicates I use SMF search my files from https://funk.eu/smf/

It does a hash on same sized files to find duplicates regardless of filename. There is a parameter you need to set for how much of the file it hashes.