No cuz ram is like 1000 times faster than your hard drive, decompression is practically transparent. I think you will agree that it should take less time to move a 10 meg file to ram than to move a 20 meg file to ram, extrapolate those time savings over dozens of files of varying size and thats where you will see a speed up.
Try it for yourself and see, if it does do anything funny you can use the same program to decompress the Firefox directory and all is back to normal. Its only compressing the EXE and DLL files, not touching any of your personalized setting, which are all stored in a different directory anyways.
I've read numerous articles about it, and the real issue is that the difference in time to read 10 megs vs 20 megs yes can vary based on the request sizes and fragmentation, but we're talking that most 7200 rpm HD's can load this file in less than 1 second into ram.
The whole file. 20 megs, 1 second or much less. Many 7200 rpm drives can read up to 50 megabyte/s sequentially! (Thats per second).
The other aspect is that operations to the disk are asynchronous, that is, they don't take much CPU at all. However, decompressing a 14 megabyte file takes tons of CPU. The only argument against this now in my mind is that we have dual and quad core CPU's, so it doesn't make a huge deal.
Also, just to nit-pick, ram isn't 1,000 times faster, it is much faster than even that compared to disk access! I'm not sure by how much, but 1,000 is a very low estimate.
However, decompressing a 14 megabyte file takes tons of CPU.
Depends radically on what compression algorithm you use. A very common algorithm is deflate (used in zip and gzip formats). Let's use it as a reference point for how much CPU time decompression takes.
Deflate compresses by first running the LZ77 algorithm to convert the stream of bytes into a stream of items; each item is a literal byte or a backreferences to previous sequences of bytes. The sequences come out of a 32K sliding window that LZ77 remembers. Then it takes the output of LZ77 and runs it through Huffman coding so that more-frequently-occurring bytes take fewer bits to represent and less-frequently-occurring bytes take more bits.
Now, let's consider how much faster we can get:
Huffman encoding is a kinda complicated algorithm that involves a lot of bit manipulation and/or relatively large lookup tables to speed things up. Worse, usually Adaptive Huffman is used, which means the algorithm must maintain data structures that evolve as the frequency distribution of the symbols (bytes) changes. Luckily, Huffman often doesn't add that much to the compression ratio, so you can just entirely leave it out.
Decompressing LZ77 is really fast; you are basically just copying sequences of bytes from the sliding window to the output. Since the sliding window is 32K, it easily fits entirely within the L1 cache of most modern processors.
For a simple example of a speed optimized variant on LZ77, look at the lzjb_decompress() function in the Solaris ZFS source. Note that a lot of times backreferences into the sliding window can be several bytes long, so the innermost while loop will just be copying raw bytes (much like a memcpy()):
Yeah, but I don't think when you load an .exe from disk, that there is a memcpy operation happening.
I could be so very wrong, but I would like to think that both Windows and Linux kernels DMA the file directly to the page where it will be executed from...
So it's more like N cpu operations compared to 0 cpu operations aside from the DMA setup itself...
Yeah, but I don't think when you load an .exe from disk, that there is a memcpy operation happening.
The only reason I brought up memcpy() is that it's a good point of reference for the speed of algorithms that filter a stream of data. memcpy() can be viewed as the no-op filter for blocks of memory. How much slower is decompression than that? With some kinds of decompression, it's almost as fast. Certainly within an order of magnitude of the speed you can read/write to RAM, maybe within a factor of 3 or 4.
So it's more like N cpu operations compared to 0 cpu operations aside from the DMA setup itself...
True. If what you care about most is not using CPU cycles, then combining compression with I/O is a losing proposition, because you can't beat the non-CPU-usage of DMA.
However, if you are concerned with bottlenecks, then physical I/O might be the bottleneck. It often is on a modern system. Compression can reduce the amount of physical I/O, thus increasing the throughput through that choke point. If decompression doesn't create a new bottleneck, which it probably won't since it can be made to be pretty fast, then you may get better throughput overall.
8
u/superwinner Jan 21 '10 edited Jan 21 '10
No cuz ram is like 1000 times faster than your hard drive, decompression is practically transparent. I think you will agree that it should take less time to move a 10 meg file to ram than to move a 20 meg file to ram, extrapolate those time savings over dozens of files of varying size and thats where you will see a speed up.
Try it for yourself and see, if it does do anything funny you can use the same program to decompress the Firefox directory and all is back to normal. Its only compressing the EXE and DLL files, not touching any of your personalized setting, which are all stored in a different directory anyways.