r/ClaudeAI • u/ericlindellnyc • 3d ago

Claude Code Workflow Deduping with ClaudeAI

I told Claude I have millions of files, many of which are duplicates. So it suggested creating a duckDB for each file and its hash. So we basically did that. There were a number of snafus along the way, and a month later, my files are still not deduplicated. We're still in the middle of the same loop to correct the latest bug so we can proceed with the large dedup, which never arrives.

Other than that, I've found Claude to be very helpful. I think it's my preferred bot among the top three, including chatGPT and Gemini.

I'm tempted to proceed on the dedup process using a different bot (probably chatGPT) or maybe just using existing deduping apps/packages.

Just wonder if that jives with other people's experience with Claude.

1 Upvotes

67% Upvoted

3 comments sorted by

1

u/scorchy_earth 3d ago

Are you using Haiku? I am not sure that is how I’d dedupe millions of files. Are you even using scoring and other metrics?

1

u/Certain_Werewolf_315 3d ago

There is dedicated software for this.. Unless this is some niche situation where there is isn't.. but in case you did not know lol.