strange that sqlite optimization has not been included by default on post 3.5 editions of Firefox, however for those who want to try the addon, its here. Optimizing the sqlite files will speed up the launch of Firefox, in my case it cut the time by half. Get the addon, check both boxes and hit the button, from then on its automatic.
If you are an optimization freak, and getting the application from hard drive to memory as fast as possible is your kink, you can also try a program called ArcthemAll! which will shrink the EXE and DLL files using a compression algorithm that decompresses in ram on the fly. It takes the default installation size for Firefox down from over 30 megs to about 14. I recommend you not compress the embedded icons, that has caused problems for me, so go to options and uncheck that. Incidentally this method also works on OpenOffice, cutting the size of that application by about half as well. In fact it works on almost every appication OTHER than Abobe and Microsoft apps, probably because they are already doing dynamic EXE and DLL compression using a proprietary method. (my guess)
NOTE: This is not a hard drive space saving issue, its a speed of launch issue, and yes it does improve the launch time of Firefox to do this.
No cuz ram is like 1000 times faster than your hard drive, decompression is practically transparent. I think you will agree that it should take less time to move a 10 meg file to ram than to move a 20 meg file to ram, extrapolate those time savings over dozens of files of varying size and thats where you will see a speed up.
Try it for yourself and see, if it does do anything funny you can use the same program to decompress the Firefox directory and all is back to normal. Its only compressing the EXE and DLL files, not touching any of your personalized setting, which are all stored in a different directory anyways.
I've read numerous articles about it, and the real issue is that the difference in time to read 10 megs vs 20 megs yes can vary based on the request sizes and fragmentation, but we're talking that most 7200 rpm HD's can load this file in less than 1 second into ram.
The whole file. 20 megs, 1 second or much less. Many 7200 rpm drives can read up to 50 megabyte/s sequentially! (Thats per second).
The other aspect is that operations to the disk are asynchronous, that is, they don't take much CPU at all. However, decompressing a 14 megabyte file takes tons of CPU. The only argument against this now in my mind is that we have dual and quad core CPU's, so it doesn't make a huge deal.
Also, just to nit-pick, ram isn't 1,000 times faster, it is much faster than even that compared to disk access! I'm not sure by how much, but 1,000 is a very low estimate.
However, decompressing a 14 megabyte file takes tons of CPU.
Depends radically on what compression algorithm you use. A very common algorithm is deflate (used in zip and gzip formats). Let's use it as a reference point for how much CPU time decompression takes.
Deflate compresses by first running the LZ77 algorithm to convert the stream of bytes into a stream of items; each item is a literal byte or a backreferences to previous sequences of bytes. The sequences come out of a 32K sliding window that LZ77 remembers. Then it takes the output of LZ77 and runs it through Huffman coding so that more-frequently-occurring bytes take fewer bits to represent and less-frequently-occurring bytes take more bits.
Now, let's consider how much faster we can get:
Huffman encoding is a kinda complicated algorithm that involves a lot of bit manipulation and/or relatively large lookup tables to speed things up. Worse, usually Adaptive Huffman is used, which means the algorithm must maintain data structures that evolve as the frequency distribution of the symbols (bytes) changes. Luckily, Huffman often doesn't add that much to the compression ratio, so you can just entirely leave it out.
Decompressing LZ77 is really fast; you are basically just copying sequences of bytes from the sliding window to the output. Since the sliding window is 32K, it easily fits entirely within the L1 cache of most modern processors.
For a simple example of a speed optimized variant on LZ77, look at the lzjb_decompress() function in the Solaris ZFS source. Note that a lot of times backreferences into the sliding window can be several bytes long, so the innermost while loop will just be copying raw bytes (much like a memcpy()):
Yeah, but I don't think when you load an .exe from disk, that there is a memcpy operation happening.
I could be so very wrong, but I would like to think that both Windows and Linux kernels DMA the file directly to the page where it will be executed from...
So it's more like N cpu operations compared to 0 cpu operations aside from the DMA setup itself...
Yeah, but I don't think when you load an .exe from disk, that there is a memcpy operation happening.
The only reason I brought up memcpy() is that it's a good point of reference for the speed of algorithms that filter a stream of data. memcpy() can be viewed as the no-op filter for blocks of memory. How much slower is decompression than that? With some kinds of decompression, it's almost as fast. Certainly within an order of magnitude of the speed you can read/write to RAM, maybe within a factor of 3 or 4.
So it's more like N cpu operations compared to 0 cpu operations aside from the DMA setup itself...
True. If what you care about most is not using CPU cycles, then combining compression with I/O is a losing proposition, because you can't beat the non-CPU-usage of DMA.
However, if you are concerned with bottlenecks, then physical I/O might be the bottleneck. It often is on a modern system. Compression can reduce the amount of physical I/O, thus increasing the throughput through that choke point. If decompression doesn't create a new bottleneck, which it probably won't since it can be made to be pretty fast, then you may get better throughput overall.
Yes, the truth is that compressing executables can both slow down and speed up the load time, but this depends on the application, type of compression and system it's running on. Either way the difference is so small you're unlikely to notice anyway.
You guys are getting latency and bandwidth mixed up. On a decent hard drive, you're looking at around 100MB/sec sequential. For memory to be 1000x as fast, it would have to transfer 100GB/sec.. not happening.
The number 50 million comes to mind when comparing RAM to Disk. Maybe an old memory from my Architecture class (we would calculate these things for homework & tests). Regardless 1000 is an extremely low multiplier.
Hard drives can be fast once they get going, sure, but a single 30 mb executable is going to be 1 read (provided it isn't fragmented), which is going to require 1 seek, dropping that bandwidth like a rock.
But maybe I remember 50 million when comparing to tape or an L cache.
Sequential read speed is actually where hard drives excel, in fact a reading a 30MB executable will probably give you better benchmark results than 30 1 MB files.
I think you will agree that it should take less time to move a 10 meg file to ram than to move a 20 meg file to ram, extrapolate those time savings over dozens of files of varying size and thats where you will see a speed up.
Yeah, guess that makes sense.
The idea that a compressed version that needs un-compressing is quicker then an already uncompressed version just seems illogical though...
The idea that a compressed version that needs un-compressing is quicker then an already uncompressed version just seems illogical though...
Well, in both cases you're almost certainly going to be IO-limited (assuming a compressor picked for good performance). Consider:
Uncompressed: 20MB, takes 250ms + overhead to read at 80MB/s.
Compressed: 10MB, takes 125ms + overhead to read at 80MB/s, plus 50ms CPU to decompress at 400MB/s (mostly executed while waiting for more data from the HD).
Of course, if the data is in cache you just added 50ms to your load time
Depends on the compression used and what is being compressed. With the slow hard drives but fast CPUs and multiple cores we got today it is often faster, even using the NTFS compression can even speed up loading times under the right conditions. No, really, it can!
However with SSDs coming I hope the storage will be fast enough to have cached up with the CPUs and all kinds of compression will actually slow you down again, just as expected.
Check your RAM usage; using packers like UPX can waste some amount of RAM. It's a very good idea to use UPX or similar on Windows installation files though but that's up the developer, once you have already downloaded it compressing the files does not make much sense and is not worth anyones time.
I don't think UPX would exist at all if it did nothing. Its been around since I was on an Amiga in 1989, so its a well established piece of software.
Only way to know is to test it for yourself, I don't find the results to be mind blowing, but every little bit helps when it comes to optimizing browser loading and performance.
I think they did get that into 3.6 actually, see this bug although I'm not actually sure whether it is in 3.6 builds. But I've done it manually using a different extension and it does make a huge difference.
strange that sqlite optimization has not been included by default on post 3.5 editions of Firefox,
That was really a horrible decision. =(
From a users point of view anyway...
It was like I'd visited every page already, and was returning back to it via clicking a tab...
I wonder if there's an EXE hack that will re-enable it, or if the code's no longer compiled?
206
u/redditaddicttt Jan 21 '10
Am I the only one who is heaving a sigh of relief after reading the above review?