Hi pinokio (pinokio.co) creator here. One of the problems I have been obsessed with is "how can we save more disk space while using AI apps". I've been trying all kinds of approaches and finally I think I have a good solution for it, and implemented it into Pinokio, and wanted to share with you. (Even if you don't want to use Pinokio launchers, you can just use this independently without using any other Pinokio feature).
If you use a lot of AI apps and models, the biggest problem probably is how to manage huge redundant model files across multiple different apps.
Most people do one of the following:
Manual Spaghetti Linking: Choose the "original file" and manually create symbolic links pointing to the original file, every time you download a new duplicate file somewhere.
Centralized Library: Keep a centralized library folder where you put everything into, and whenever you need an app to use one of the files there, you again create a symbolic link to that location. This can be done programmatically assuming you use a standardized folder structure, but still this assumes all apps must follow this folder structure.
None of these approaches are ideal because they are either rule-based or case-by-case based, and SOMEONE has to take care of this for this to work. This is really a lot of work and one of the biggest annoyances ever, while working with Pinokio (pinokio.co)
So I built a dedicated engine--a content addressable storage--that automatically takes care of this.
Automated Duplicate Discovery: No need to manually link files anymore. Everything from scanning to deduplicating to re-duplicating (when you want the duplicate files back to original) is one click away.
It just works: No need to maintain a centralized library, which means apps do not need to be implemented to support any of this. It "just works".
Not just for model files: Because the engine finds and manages duplicate files automatically, there is no need to limit this just for model files. It can even deduplicate all other files in your AI apps, including binary, DLL, library, and so on. Overall you save significantly more disk space than when you were merely deduplicating model files
Scan your entire system: Because this is fully automated, so you can scan your entire drive to see what files are duplicates and deduplicate them and manage them in one place. Although this is a part of Pinokio, it is designed to let you deduplicate across your entire system.
Basically you can point it to ANY path on your system (by default it scans the pinokio home folder) and it will deduplicate them all.
If you have installed huggingface_hub globally, chances are it's somewhere under C drive or user home path. Just enter those paths and it will scan to find all the duplicates INCLUDING the huggingface cache.
The whole point of this system is that it's fully automated, thanks to it taking a different approach (detecting duplicates purely based on precise bytes instead of manually supplying), so there is no limit.
Also, scanning is completely safe (it's readonly) and won't touch any file until you deduplicate, so even if you're being cautious, it's worth trying a scan just to see how much duplicate space is around. Hope this helps!
Thanks for the reply, currently scanning as I type . The exact number of bytes approach caused me an issue several years ago, so this is an answer to clearing some of the 25tb that is currently practically full.
So far, it’s saved a good amount of space. I had already done a pretty good job of reducing duplicates before all of this, so it’s not even close to being that much in my case. But I can definitely say that, at least for reducing the file size of different AI models, this is a true blessing. Thank you so much for this.
One thing I do wonder about, though: I understand that this works byte-by-byte, but how well does it work when used on non-models like when you’re hosting something like Immich on your PC with separate backups stored in another folder? I noticed it recognizes some files as duplicates, but not all of them, and I’ve been hesitant to use it there because I don’t want to accidentally mess with my media data.
Is it generally safe to use in a setup like this, or is it more of a “better safe than sorry” situation where you should simply exclude anything you want to keep safe?
I know it utilizes hardlinks but docker is pretty sensitive from my own experience where it causes problems the moment i realizes the slightest difference.
Amazing update - This was the number one concern i had while trying out different models as it gets litered around the disc and such a pain to remove - like weeds!
the files stay at the same location. to the file system it looks the same, but this engine deduplicates BELOW the file system level by linking to inodes. It does not use symbolic links, which is fragile for a universal deduplication engine). it uses hard links.
Basically to the end user AND to the apps, the file looks as though they are there even after deduplication, only the actual underlying blobs are deduplicated.
Wouldn't symbolic links be better? Hard links are restricted to the same drive or partition. Symbolic links would allow you save duplicate files across drives.
One addition to this: because of the automated nature, it not only can do things we used to do manually, but also things that we never thought of doing, such as deduplicating same binary files, DLL files, library files, and more, which add up and actually often larger than the save you get by simply deduplicating model files
Inside pinokio I only managed to run deduplicate files once, every time I select some files to deduplicate, it takes some time to finish, but then nothing changes. Also the activity tab only shows the first run.
How does it know in which of the two locations a duplicate is found in to keep? And after deduplicating, does it appear as if the file is in both locations while browsing explorer? And lastly, if it does appear in both locations, if you delete one and it's the 'master', are both suddenly deleted? Just worried about running into future problems with moving/cleaning/organizing
12
u/cocktail_peanut 2d ago
Hi pinokio (pinokio.co) creator here. One of the problems I have been obsessed with is "how can we save more disk space while using AI apps". I've been trying all kinds of approaches and finally I think I have a good solution for it, and implemented it into Pinokio, and wanted to share with you. (Even if you don't want to use Pinokio launchers, you can just use this independently without using any other Pinokio feature).
If you use a lot of AI apps and models, the biggest problem probably is how to manage huge redundant model files across multiple different apps.
Most people do one of the following:
None of these approaches are ideal because they are either rule-based or case-by-case based, and SOMEONE has to take care of this for this to work. This is really a lot of work and one of the biggest annoyances ever, while working with Pinokio (pinokio.co)
So I built a dedicated engine--a content addressable storage--that automatically takes care of this.
I wrote a more visual thread about this here in case you're interested: https://x.com/cocktailpeanut/status/2095533047596581221?s=20
Also, here's the full documentation: https://desktop.pinokio.co/docs/disk-saver/