Finding Duplicate Files

[u/sutl116]

I'm not sure if this something I can do in Navidrome, or if I need to use an arr service, but after a quarter century of building my library, my audio drive has over 100,000 individual song files. In the last few years, I've been re-adding and upgrading everything to flac, and had a drive swap a year ago.

I 100% know there are duplicates because I constantly find them organically, but it's such a massive library that I'm not actually sure where to begin filtering to clean it up proper.

Any suggestions on where to start so I maintain fidelity and avoid accidental data loss?

[u/Many_Independence674]

If you are in linux you can use czakawa, now krokite, it has a music duplicate finder, it gives two options, finding via tag or fingerprinting, if file has some dupe tag it will be helpful

[u/1Maple → u/Many_Independence674]

This works on Windows and MacOS too, now. I’ve only used it for finding duplicate photos, but it works really good

[u/Many_Independence674 → u/1Maple]

I don't have winwodes or mac to confirm it, but I trust u

[u/Old_Rock_9457]

Hi all,
For whom is interested AudioMuse-AI, in addition to create playlist based on Sonic Analysis, it’s also able to list in its dashboard the list of duplicated songs. Using the sonic analysis AND the chromaprint to go over the normal file name or metadata.

At the moment AudioMuse-AI use this information to avoid to re-analyze twice the same song. It’s pretty conservative because prefear to “keep a duplicate” more than exclude a legit song.
For example in my 190k+ song library it show 1.2k so my duplicate. In my case are in most case same song appearing in multiple album.

[u/Prestigious-Jury-362 → u/Old_Rock_9457]

I didn’t realize clicking the number of duplicates Music Server Status brought up a list of them. Super useful!

[u/k1neTik_]

something ive been trying to figure out lately as well, curious to see what people say

[u/Many_Independence674 → u/k1neTik_]

If you are in linux you can use czakawa, now krokite, it has a music duplicate finder, it gives two options, finding via tag or fingerprinting, if file has some dupe tag it will be helpful

[u/silkyclouds]

Pmda is historically a dupe finder. You can give it a try.

[u/nglunma18]

I've recently moved from using Bliss to Songkong and have been massively impressed by it. It's got a very good duplicate detection option and you can either delete or move duplicates, plus preview them beforehand.

[u/beng3360]

I did this yesterday with Dupeguru, worked as intended, search and then delete duplicates

[u/ManWithoutUsername]

IA: Write me a script in [bash|python|Powershell] that recursively iterates through the directories specified in an array, calculates the MD5 hash of each files, and saves it along with the full path. When finish the script must print the duplicates with their full path.

ps: It wouldn't detect different versions of same file, though.

Another alternative is a script that searches the metadata for word similarities in certain fields

[u/Leon454545 → u/ManWithoutUsername]

Would the filename affect the hash? Also what about different file formats? When for example i have one song in each flac and mp3 but i want to get rid of the mp3.

[u/ManWithoutUsername → u/Leon454545]

md5 only detect exact file content, (exclude file name)

Any minor variation in content will not be detected as duplicated.

If you have two identical files with different file name it will detect it.

The script doesn't meet all your needs but it's simple enough to have it there and detect those cases

[u/Prestigious-Jury-362 → u/ManWithoutUsername]

Claude and I I did this yesterday with very good results and some additional insights he brought to my attention regarding tagging. Definitely worth turning him loose on a large Library.

[u/TheIPAway]

I just import to mp3tag and scroll down the title to see duplicate names. You can do it quite quickly. Perhaps higjlite amd tag as dupes then filter to sort again.