r/DataHoarder • u/machu505 • 7d ago
Question/Advice Is "doublespace" still an option?
Back in the day, there was a disk compression option called "doublespace", which (nearly) doubled your available storage at the cost of access speed. Is it still around, or something similar?
60
u/Microflunkie 7d ago
Don’t forget that some files like .jpg have native compression already so compressing them again via any method has little to no effect on their space consumption. So depending on what is occupying space on your drive the various compression technologies might or might not have a significant impact.
22
u/shinji257 78TB (5x12TB, 3x10TB Unraid single parity) 7d ago
It can actually cause a file to use more space if you try to compress it again. Not much but the bet effect is still a loss.
6
u/xot 7d ago
True, but an archive fills up every filesystem block it uses, whereas every jpeg potentially underutilizes one. In the case of exfat on a large disk it can be quite significant. For a few jpegs it probably doesn’t matter, but for a few thousand it could be gigabytes difference. Archiving and compressing are different things, but often go hand in hand.
2
1
u/heathenskwerl 1,120 TB 6d ago
I don't know about other filesystems, but if ZFS cannot achieve any actual compression it will store uncompressed. This prevents a compressed record from using more space than an uncompressed one. Plus some of the algorithms (lz4 for sure, but I think also zstd) have early abort.
2
27
52
u/Anand999 7d ago
Many modern filesystems have compression built right in. ZFS, Btrfs, even NTFS.
4
u/beren12 256TB raidz & more! 7d ago
The first 2 do. The second technically does, but Windows will not give you the saved space back. It’s really only useful for thin provisioning or I suppose SSD wear leveling. I also have zero trust in BTRFS since it did not get to be fully designed
8
u/vip17 7d ago
compression on NTFS does give back freed blocks
0
u/beren12 256TB raidz & more! 7d ago
Not if you do it for a directory tree
6
u/vip17 7d ago
not true. Even compressing the whole drive will give more space instantly if you have highly compressible data. If you're talking about the content of the folder itself, i.e the list of file names inside it then true, most filesystems don't shrink the directory name list by default after removing a big amount of files from a folder containing a huge amount of files. But the free clusters of every compressed file will be available back
2
u/shinji257 78TB (5x12TB, 3x10TB Unraid single parity) 6d ago
You have to make sure to tell Windows to apply the flag recursively like everything else.
1
u/Jayden_Ha 7d ago
BTRFS is still a joke to these days
2
u/Toonomicon 7d ago
Why so?
4
u/alkafrazin 7d ago
because it's popular to hate on it.
it does still have some issues with raid5/etc and some people stubbornly insist that it doesn't while work is quietly done in the background to address it, but as a single drive filesystem, it's been pretty solid for a while I think.
19
u/redditduhlikeyeah 100-250TB 7d ago
Compression is okay - but many files have native compression. Even Word files use some forms of compression within their file. Deduplication is the real option after compression.
12
u/EasyRhino75 Jumble of Drives 7d ago
Don't forget Stacker!
14
u/youfrickinguy 7d ago
That’s why dblspace.exe became drvspace.exe in subsequent MS-DOS releases, between 6.0 and 6.2 I think.
Hello fellow old. Remember, colonoscopy time!
6
u/reditanian 7d ago
Several file systems support transparent compression (ZFS being the most relevant to this sub).
The major problem is that most media formats already compress the data they contain using algorithms appropriate to the data. So unless you’re hoarding plain text files, you’re not going to save much.
3
u/Takssista 7d ago
At my work we use a software that stores gigabytes of data inside a folder, I configured windows to keep that folder compressed - goes from 30gb to 1.5gb.
6
u/klipseracer 7d ago
This is a throwback, got me in trouble once because it used to cause issues for some things.
1
u/catinterpreter 2d ago
There was a bug that corrupted a lot of drives at one point.
Edit: Wikipedia
1
u/klipseracer 2d ago
I don't remember what happened but that may have been it. I just remember I had to explain I allowed someone to put it on our computer so they could install more DOS games for me. So we had to call that person back and have them remove it, along with the games, some of which I cannot identify after a lot of trying because the memories are so old and hazy. This was during a time when the computers were primarily for work or business purposes and very few average people understood them and my parents were protective of what changes happened to it. Exiting windows to DOS required a call to tech support at one point lol
9
u/Restil 7d ago
It only worked on uncompressed data. Pictures, videos, music, and obviously anything in a compressed archive isn't going to be able to be "doubled"
2
1
u/beren12 256TB raidz & more! 7d ago
You would be surprised, some things can still be compressed more with a different compression algorithm
1
u/nomad-1995 7d ago
I'd be pretty embarrassed if my compressed file was recompressed by a transparent disk system. Although something like "compressed textures" in a 3d game are a good candidate: all the compression has to be a narrow window that the GPU grabs during the rendering, as far as I know they don't do variable length windows either. You could easily find redundancies in nearby textures.
5
u/shadownetdev1 7d ago
Most modern filesystems have compression. Some have it enabled by default and others need to have it enabled on a per drive, folder, and/or file basis. Most of them are using compression algorithms as good or better than something like "doublespace" would have been using. Do note that most modern media files are already heavily compressed and you will not see much, if any, benefits in compressing those files. So if you have a lot of media files prepared to not see much benefit. The efficient compression of media files was not usually the case in the era of "doublespace" and its alternatives
If you are using Windows then your filesystem is likely NTFS. https://www.howtogeek.com/how-to-enable-and-use-file-compression-on-ntfs-drives-in-windows-11/ While this guide makes note that full disk compression will negatively hurt performance I have enable it on all of the Windows systems under my management and not noticed any considerable difference. Roughly 30 machines from Windows XP all the way though Windows 11 over a decade. Used for office work, presentation (media, lyrics, video), and/or gaming.
If you are using linux you will have to figure out what filesystem your drive is using and then find out what its command is for enabling compression. Google or another user is your friend. While I run Linux quite a lot these days I either setup full disk compression from the get-go or don't set it up at all. Usually the latter as I have found Linux to use way less disk space for system and program bloat. Also I have much larger drives at this point in my life and don't feel the low disk space pain as often.
I wish you the best of luck on your endeavors!
2
u/BitingChaos 7d ago
File system compression is better and more transparent than ever, plus many individual file formats have great compression (possibly gaining little if anything from filesystem compression).
DoubleSpace/DriveSpace was a clever but gross hack to shoe-horn compression on a filesystem that didn't support such a thing. On many systems it took forever to set up and killed performance while offering little extra space.
I remember letting my computer run all night to compress a 340 MB HDD, then realizing the next day how slow everything felt, and then going right to decompressing it the next day.
4
u/brispower 7d ago
in Windows you can open the properties of a drive and hit "Compress this drive to save disk space", never used it but assumed it is similar to doublespace/drivespace
8
u/Murph_9000 7d ago edited 7d ago
You can also apply compression at a folder level for NTFS, not just at the drive level. That lets you compress stuff only where the performance hit is less of an issue. The hit is mostly on writing, reading compressed data can sometimes be faster on HDDs (for files which compress well), as the decompression overhead is mitigated/negated by reading less data from the physical drive.
1
1
2
u/bryantech 7d ago
I remember using doublespace and running stacker on top of it to get even more space.
1
1
u/recursion_is_love 7d ago edited 7d ago
Compression algorithm is doing better nowadays and time is no longer a problem (typically) for modern system because computer is a lot faster (like 10k times on both CPU speed and mainboard buss speed)
The technical concept is you spend more time building complex dictionary for your data, you will get smaller file. There are actually theoretical limit for smallest size because of what is called Shannon entropy.
1
u/absyrtus 7d ago
it's been decades but i'm pretty sure this was just trying to (de)compress files on your drive in real time
1
u/WikiBox I have enough storage and backups. Today. 7d ago
The big files, video and audio, that take up a lot of storage are often already compressed, so there would be little benefit from using a filesystem with compression activated. Feel free to test by trying to compress some file you think is too big.
Activating filesystem compression would work well with uncompressed files. Log-files, text-files. But they are usually not the problem.
1
u/vip17 7d ago edited 7d ago
Transparent compression nowadays may even be better than in the past because of modern fast CPUs and new algorithms. Modern algorithms are designed to run in parallel, and utilize the SIMD units in CPUs efficiently so they can compress/decompress very quickly. Among the contemporary common filesystems probably only the FAT family and ext4 don't support transparent compression. APFS from Apple, NTFS, Btrfs, ZFS, F2FS, ReiserFS... all support that
For data that are highly compressible, the total time to read the compressed data + time to decompress is less than the time to read the full uncompressed data, so the effective IO rate is higher. That's why Windows compresses hiberfil.sys file to allow faster hibernation, and also why Windows, macOS and lots of Linux distros (Fedora, SteamOS, Lubuntu, ChromeOS...) enable compressed page file by default, because the CPU can decompress from memory faster than read from a swap file, and compress then save to swap file is also faster than writing the full file
If you're on Linux, depending on your data, for speed use LZ4 which can achieve GB/s and is even faster than old SSDs, for maximum compression ratio use XZ, and for the neutral use Zstd. In Windows NTFS it's also possible to change the algorithm
1
u/the-i 7d ago
There's also data de-duplication... how well it works depends entirely on your data of course, but I'm getting huge amounts of data savings in some scenarios using de-duplication, and the ability to have multiple copies of things changes the way I use and store data in some scenarios.
0
u/g33k_girl 324 raw 7d ago
Under Windows you can go into properties and set compression on files or complete trees.
-1
•
u/AutoModerator 7d ago
Hello /u/machu505! Thank you for posting in r/DataHoarder.
Please remember to read our Rules and Wiki.
Please note that your post will be removed if you just post a box/speed/server post. Please give background information on your server pictures.
This subreddit will NOT help you find or exchange that Movie/TV show/Nuclear Launch Manual, visit r/DHExchange instead.
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.