Closed Bug 55930 Opened 20 years ago Closed 9 years ago
Use UPX to dramatically reduce startup time
I found this in netscape.public.mozilla.performance.size-matters written by Fernando Cassia <email@example.com>. If what's said is correct, it sounds like an easy way to get a great win: -------------------------------------------- Anyone tried UPX? It is an EXE and DLL compressor, like good old "PKLITE" for DOS or "LXLITE" for OS/2, but it works both on Win32 and Linux binaries. I was able to shrink Netscape 6 (win32) PR3 to about 45%, and improve load times dramatically (on my old trusty Pentium 233MMX with 128mb ram). Find this hidden gem at http://www.upx.org BTW: this only makes the distribution smaller and faster to load. It does NOT replace much needed code optimization. :-) Comments? Regards Fernando ---------------------------------------------
Nominating for rtm to at least get an investigation of the feasability of using upx.
Summary: Use UPX to dramatically reduce startup time. → Use UPX to dramatically reduce startup time.
I think the url should be http://upx.tsx.org
Assignee: cls → leaf
rtm- no way we're going to have time to make this major a change and fully test it this close to rtm.
set to future, clear keywords.
Severity: normal → enhancement
Target Milestone: --- → Future
OK, I just did a quick, 10 minute test of this, and I think it warrants further testing. I compressed mozilla.exe and all dlls in \bin and \bin\components from a recent nightly. Before, these files took up 11480400 bytes (roughly 11.2 MB). Now they take up 4730368 bytes (about 4.6 MB). The app still seems to run perfectly fine. I browsed a few pages, including a few requiring the Flash plug-in, created a new news account, subscribed to and read a couple of mozilla.org groups, created and saved a page in composer, fired up the address book, and I've got choffman's browser buster running right now. I can't really speak to startup speeds because I'm also playing around with the preloader right now. But anything that can squish gkcontent.dll down to 32% of its original size definitely deserves a look.
you don't have to convince us, you have to convince n.p.m.performance. I suspect this will slow down startup times, and possibly others, noticeably and last I heard they weren't trading runtime performance for disk space footprint, hence "future".
Updating URL.. Changing OS-> All since this will do Win32 and Linux. We could also consider compressing for distro and decompressing upon decompression of the XPIs (during setup).. I've seen stuff compressed to about 5% when compressed using UPX and then some ZIP equivalent.. but I say leave it compressed with UPX, the benefits usually outweigh the drawbacks, however more testing is required. The only real adverse effect of using UPX is that Mozilla may take more memory.. load times for the most part will decrease. I'll compress the latest build and report here.
Any increased memory usage is very bad for Mozilla because it uses so much memory already and for people without much RAM the performance is limited by swapping.
Here are the stats. Mozilla build 2001073103-trunk. System: PIII-600/128MB RAM/Win2K SP2. Homepage set to http://www.swva.net Uncompressed: 24.7/26.5MB minimum load/peak memory usage on first page load and 5 mins of browsing Bugzilla 8-10s loading time to first page load memory usage did not drop below initial state 16.6MB used on disk Compressed with UPX 1.20 (flags: --best --compress-icons:0): 27.5/28.8MB minimum load/peak memory usage on first page load and 5 mins of browsing Bugzilla 9-11s loading time to first page load memory usage dropped to 9.6MB (not reproducibly) 9.5MB used on disk Drawbacks: -1s loss in _local load time on my machine_ -~2MB extra memory usage on load Benefits: -Network load time would be substantially decreased and load time on slow systems may be decreased -Memory usage has potential to drop off to 35% initial state after a few minutes, however I could not determine what caused it to drop that much. -Size on disk is only 57% what it used to be.. This means distro size could be about 1/2 what it is now. Netscape 6 will see a greater benefit here over Mozilla. Opinion: Use UPX, the benefits outweigh the drawbacks.. perhaps Mozilla develops could take time to work on UPX on sourceforge to make it work even better. Now, does anybody have any Linux stats? The compression works differently on Linux.
I think you might have minimized Mozilla. That causes the OS to put Mozilla's memory on the standby list so that memory usage is decreased. As for the compression, the only benefit I see is that the packaged product becomes smaller than normal zip, but if that size is a problem, isn't there lots of better compression systems? RAR and ACE and bzip2 for instance.
performance is key, so I doubt you'll find anyone at mozilla.org or netscape willing to tolerate higher memory use and longer load times to save disk space. Alternatively, I bet you'd find lots of people to support using more disk space to lower memory usage and reduce page load times if you can find a way to do that...
Then I suggest at minimum the possibility (mentioned in my first comment) of compressing with UPX for packaging and decompressing upon unpackaging of the XPIs. But I also quote the initial report "I was able to shrink Netscape 6 (win32) PR3 to about 45%, and improve load times dramatically (on my old trusty Pentium 233MMX with 128mb ram)." Load time really depends on the machine, but it should never be slowed by more than a second or two, it will be increased in other cases. So the real issue here is the 2MB overhead.. but it is really negligible on actual use because basic use easily makes Moz use over 30MB and minimizing once reduces the overhead.. to the point that it may actually use _less than the uncompressed version_! The initial overhead may be higher, but a simple minimize negates that. (Which makes me wonder if we can reduce footprint by going into and out of standby immediately after load). These things must be considered.. I'll post a compressed build if anybody wants to experiment (Is there any way to modify installer-sea?). I can give some more statistics, but I'd like to see how it performs on a variety of machines.
ironically, windows installers have the least documentation: http://mozilla.org/projects/xpinstall/wizard/ *but* you can use mozilla/xpinstall/wizard/windows/builder/build.pl to make installers from your local build; alternatively you can look over the script and see where upx can be used.
bug cleanup - all leaf's bugzilla bugs should be assigned to firstname.lastname@example.org (not email@example.com), now and any future bugs created. this should be a one time change, apologies for the spam.
Assignee: leaf → leaf
I'm going to take this bug.. though I may not get to it for several weeks.
Assignee: leaf → ask
*** Bug 138640 has been marked as a duplicate of this bug. ***
The .so (shared libs) files for linux can't compressed so this doesn't help. Or better ... it isn't nice to compress the libs. See comment 5 on http://upx.sourceforge.net/phpBB/viewtopic.php?t=251 I think this will also the problem on other systems. If we later have one GRE and different apps that use it, then we also have memory footprint on win if we use UPX. So I think this bug should be losed as wontfix
The comment also says: "Of course this only concerns those shared libraries, which are known to actually be "shared". If you are simply writing a module for you own application and don't expect many instances of it to run at the same time, then the overhead involved is just the usual one as with the exes (the memory consumption is increased by the size of compressed file) and you can safely compress it." In general this isn't a good solution for Linux anyway (due to the fact that it takes uncompressed+compressed in memory), but it's still good for Windows.
OS: All → Windows 95
Just to update the last comment, I think v1.90 beta resolves the Linux problem: "The main news in the unstable versions are support for bootable Linux kernels ("vmlinuz/386"), direct Linux ELF-to-memory decompression ("linux/elf386") and Playstation exes ("ps1/exe")."
Even if we can't do this on all platforms, we can at least do it on Windows. Firefox Win32 installer is based on 7zip, which isn't cross platform.
...and seven and a half years later, the committee for the difficult decisions which reports to the Controversy Committee has done nothing with this. (sorry couldn't resist... ;). Ain't democracy wonderful? At least we can all reach a concensus on doing nothing. :) FC
Fernando - Obviously no one has had time to work on this. Would you be willing to take it over and contribute a patch? The Mozilla organization has advanced quite a bit since 2001 and the results should be easily measurable now.
Just a little spot test on xul.dll from the current firefox release (3.0.1) shows a compression ratio of 38.5% using UPX 3.03. Command used: upx --best --ultra-brute -f -v xul.dll Result: 9704960 -> 3736064 bytes
Assignee: ask → nobody
Product: SeaMonkey → Core
QA Contact: granrosebugs → build-config
With UPX the OS can't map the DLL memory directly to disk, which is important in terms of memory usage. WONTFIX.
Status: NEW → RESOLVED
Closed: 11 years ago
Resolution: --- → WONTFIX
Re-opening. We shouldn't write this off without real numbers showing what the tradeoff actually is. If the tradeoff is even vaguely reasonable, we should start broader community discussion about it.
Status: RESOLVED → REOPENED
OS: Windows 95 → All
Priority: P3 → P1
Hardware: x86 → All
Resolution: WONTFIX → ---
Target Milestone: Future → ---
In my testing upx can reduce firefox startup from 2.6s to 1.7 when windows prefetch isn't working right. When prefetch is working, then there isn't a measurable difference. UPX adds 100ms to warm startup. I think deploying with UPX may be nice in some niche situations, but in general this just goes to show that compression is best when it's well-implemented at the filesystem level(bug 514083).
What about its' use in reducing download sizes?
(In reply to comment #29) > In my testing upx can reduce firefox startup from 2.6s to 1.7 when windows > prefetch isn't working right. When prefetch is working, then there isn't a > measurable difference. Talked to Taras on IRC, and he said that prefetch was failing for him randomly. I've seen similar behavior, where Firefox gets pushed out of the prefetch cache somehow. This is more argument that we should not rely on Windows to solve this for us.
(In reply to comment #30) > What about its' use in reducing download sizes? We already UPX compress the Firefox installer.
Is this moot now that we do omnijar?
(In reply to comment #33) > Is this moot now that we do omnijar? yeah sorta. UPX turned out to slow down our warm startup too much.
(In reply to comment #34) > (In reply to comment #33) > > Is this moot now that we do omnijar? > > yeah sorta. UPX turned out to slow down our warm startup too much. What do you mean by "sorta"?
I think this can be closed now that we use omnijar, as UPX slows down warm startup.
(In reply to Marco Castelluccio from comment #36) > I think this can be closed now that we use omnijar, as UPX slows down warm > startup. Yes, thank you.
Status: REOPENED → RESOLVED
Closed: 11 years ago → 9 years ago
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.