Closed Bug 55930 Opened 24 years ago Closed 13 years ago

Use UPX to dramatically reduce startup time.

Categories

(Firefox Build System :: General, enhancement, P1)

enhancement

Tracking

(Not tracked)

RESOLVED WONTFIX

People

(Reporter: bratell, Unassigned)

References

(Blocks 1 open bug, )

Details

(Keywords: perf, Whiteboard: [ts])

Attachments

(1 file)

I found this in netscape.public.mozilla.performance.size-matters written by 
Fernando Cassia <fcassia@compuserve.com>. If what's said is correct, it sounds 
like an easy way to get a great win:

--------------------------------------------
Anyone tried UPX? It is an EXE and DLL compressor, like good old
"PKLITE" for DOS or "LXLITE" for OS/2, but it works both on Win32 and
Linux binaries.

I was able to shrink Netscape 6 (win32) PR3 to about 45%, and improve
load times dramatically (on my old trusty Pentium 233MMX with 128mb
ram).

Find this hidden gem at http://www.upx.org

BTW: this only makes the distribution smaller and faster to load. It
does NOT replace much needed code optimization. :-)

Comments?

Regards
Fernando
---------------------------------------------
Nominating for rtm to at least get an investigation of the feasability of using 
upx.
Keywords: rtm
Summary: Use UPX to dramatically reduce startup time. → Use UPX to dramatically reduce startup time.
I think the url should be http://upx.tsx.org
Updating URL.
windows.
Assignee: cls → leaf
rtm-

no way we're going to have time to make this major a change and fully test it
this close to rtm.
Whiteboard: [rtm-]
set to future, clear keywords.
Severity: normal → enhancement
Keywords: rtm
Whiteboard: [rtm-]
Target Milestone: --- → Future
OK, I just did a quick, 10 minute test of this, and I think it warrants further 
testing.  I compressed mozilla.exe and all dlls in \bin and \bin\components from 
a recent nightly.  Before, these files took up 11480400 bytes (roughly 11.2 MB). 
 Now they take up 4730368 bytes (about 4.6 MB).

The app still seems to run perfectly fine.  I browsed a few pages, including a 
few requiring the Flash plug-in, created a new news account, subscribed to and 
read a couple of mozilla.org groups, created and saved a page in composer, fired 
up the address book, and I've got choffman's browser buster running right now.

I can't really speak to startup speeds because I'm also playing around with the 
preloader right now.  But anything that can squish gkcontent.dll down to 32% of 
its original size definitely deserves a look.
you don't have to convince us, you have to convince n.p.m.performance.  I
suspect this will slow down startup times, and possibly others, noticeably and
last I heard they weren't trading runtime performance for disk space footprint,
hence "future".
Updating URL..  Changing OS-> All since this will do Win32 and Linux.  We could
also consider compressing for distro and decompressing upon decompression of the
XPIs (during setup).. I've seen stuff compressed to about 5% when compressed
using UPX and then some ZIP equivalent.. but I say leave it compressed with UPX,
the benefits usually outweigh the drawbacks, however more testing is required. 
The only real adverse effect of using UPX is that Mozilla may take more memory..
load times for the most part will decrease.  I'll compress the latest build and
report here.
Keywords: perf
OS: Windows NT → All
Any increased memory usage is very bad for Mozilla because it uses so much
memory already and for people without  much RAM the performance is limited by
swapping. 
Here are the stats.  Mozilla build 2001073103-trunk.  System: PIII-600/128MB
RAM/Win2K SP2.  Homepage set to http://www.swva.net

Uncompressed:
24.7/26.5MB minimum load/peak memory usage on first page load and 5 mins of
browsing Bugzilla
8-10s loading time to first page load
memory usage did not drop below initial state
16.6MB used on disk

Compressed with UPX 1.20 (flags: --best --compress-icons:0):
27.5/28.8MB minimum load/peak memory usage on first page load and 5 mins of
browsing Bugzilla
9-11s loading time to first page load
memory usage dropped to 9.6MB (not reproducibly)
9.5MB used on disk

Drawbacks:
-1s loss in _local load time on my machine_
-~2MB extra memory usage on load

Benefits:
-Network load time would be substantially decreased and load time on slow
systems may be decreased
-Memory usage has potential to drop off to 35% initial state after a few
minutes, however I could not determine what caused it to drop that much.
-Size on disk is only 57% what it used to be.. This means distro size could be
about 1/2 what it is now.  Netscape 6 will see a greater benefit here over Mozilla.

Opinion:
Use UPX, the benefits outweigh the drawbacks.. perhaps Mozilla develops could
take time to work on UPX on sourceforge to make it work even better.

Now, does anybody have any Linux stats?  The compression works differently on Linux.
I think you might have minimized Mozilla. That causes the OS to put Mozilla's
memory on the standby list so that memory usage is decreased.

As for the compression, the only benefit I see is that the packaged product
becomes smaller than normal zip, but if that size is a problem, isn't there lots
of better compression systems? RAR and ACE and bzip2 for instance.
performance is key, so I doubt you'll find anyone at mozilla.org or netscape
willing to tolerate higher memory use and longer load times to save disk space.
 Alternatively, I bet you'd find lots of people to support using more disk space
to lower memory usage and reduce page load times if you can find a way to do that...
Then I suggest at minimum the possibility (mentioned in my first comment) of 
compressing with UPX for packaging and decompressing upon unpackaging of the 
XPIs.

But I also quote the initial report "I was able to shrink Netscape 6 (win32) 
PR3 to about 45%, and improve load times dramatically (on my old trusty Pentium 
233MMX with 128mb ram)."

Load time really depends on the machine, but it should never be slowed by more 
than a second or two, it will be increased in other cases.  So the real issue 
here is the 2MB overhead.. but it is really negligible on actual use because 
basic use easily makes Moz use over 30MB and minimizing once reduces the 
overhead.. to the point that it may actually use _less than the uncompressed 
version_!  The initial overhead may be higher, but a simple minimize negates 
that.  (Which makes me wonder if we can reduce footprint by going into and out 
of standby immediately after load).  These things must be considered.. I'll 
post a compressed build if anybody wants to experiment (Is there any way to 
modify installer-sea?).  I can give some more statistics, but I'd like to see 
how it performs on a variety of machines.
ironically, windows installers have the least documentation:
http://mozilla.org/projects/xpinstall/wizard/

*but* you can use mozilla/xpinstall/wizard/windows/builder/build.pl to make 
installers from your local build; alternatively you can look over the script and 
see where upx can be used.
bug cleanup - all leaf's bugzilla bugs should be assigned to leaf@mozilla.org
(not leaf@netscape.com), now and any future bugs created.

this should be a one time change, apologies for the spam.
Assignee: leaf → leaf
I'm going to take this bug.. though I may not get to it for several weeks.
Assignee: leaf → ask
*** Bug 138640 has been marked as a duplicate of this bug. ***
The .so (shared libs) files for linux can't compressed so this doesn't help. Or
better ... it isn't nice to compress the libs.

See comment 5 on http://upx.sourceforge.net/phpBB/viewtopic.php?t=251

I think this will also the problem on other systems.
If we later have one GRE and different apps that use it, then we also have
memory footprint on win if we use UPX.

So I think this bug should be losed as wontfix
The comment also says: "Of course this only concerns those shared libraries,
which are known to actually be "shared". If you are simply writing a module for
you own application and don't expect many instances of it to run at the same
time, then the overhead involved is just the usual one as with the exes (the
memory consumption is increased by the size of compressed file) and you can
safely compress it."

In general this isn't a good solution for Linux anyway (due to the fact that it
takes uncompressed+compressed in memory), but it's still good for Windows.
OS: All → Windows 95
Blocks: 171082
Just to update the last comment, I think v1.90 beta resolves the Linux problem:
"The main news in the unstable versions are support for bootable Linux kernels
("vmlinuz/386"), direct Linux ELF-to-memory decompression ("linux/elf386") and
Playstation exes ("ps1/exe")."
Even if we can't do this on all platforms, we can at least do it on Windows. 
Firefox Win32 installer is based on 7zip, which isn't cross platform.
Product: Browser → Seamonkey
...and seven and a half years later, the committee for the difficult decisions which reports to the Controversy Committee has done nothing with this. (sorry couldn't resist... ;).

Ain't democracy wonderful? At least we can all reach a concensus on doing nothing. :)

FC
Fernando -

Obviously no one has had time to work on this.  Would you be willing to take it over and contribute a patch?  The Mozilla organization has advanced quite a bit since 2001 and the results should be easily measurable now.
Just a little spot test on xul.dll from the current firefox release (3.0.1) shows a compression ratio of 38.5% using UPX 3.03.

Command used: upx --best --ultra-brute -f -v xul.dll

Result: 9704960  ->   3736064 bytes
Whiteboard: [ts]
Assignee: ask → nobody
Product: SeaMonkey → Core
QA Contact: granrosebugs → build-config
With UPX the OS can't map the DLL memory directly to disk, which is important in terms of memory usage. WONTFIX.
Status: NEW → RESOLVED
Closed: 15 years ago
Resolution: --- → WONTFIX
Re-opening. We shouldn't write this off without real numbers showing what the tradeoff actually is.

If the tradeoff is even vaguely reasonable, we should start broader community discussion about it.
Status: RESOLVED → REOPENED
OS: Windows 95 → All
Priority: P3 → P1
Hardware: x86 → All
Resolution: WONTFIX → ---
Target Milestone: Future → ---
In my testing upx can reduce firefox startup from 2.6s to 1.7 when windows prefetch isn't working right. When prefetch is working, then there isn't a measurable difference.

UPX adds 100ms to warm startup. I think deploying with UPX may be nice in some niche situations, but in general this just goes to show that compression is best when it's well-implemented at the filesystem level(bug 514083).
What about its' use in reducing download sizes?
(In reply to comment #29)
> In my testing upx can reduce firefox startup from 2.6s to 1.7 when windows
> prefetch isn't working right. When prefetch is working, then there isn't a
> measurable difference.

Talked to Taras on IRC, and he said that prefetch was failing for him randomly. I've seen similar behavior, where Firefox gets pushed out of the prefetch cache somehow. This is more argument that we should not rely on Windows to solve this for us.
(In reply to comment #30)
> What about its' use in reducing download sizes?

We already UPX compress the Firefox installer.
Blocks: 447581
Is this moot now that we do omnijar?
(In reply to comment #33)
> Is this moot now that we do omnijar?

yeah sorta. UPX turned out to slow down our warm startup too much.
(In reply to comment #34)
> (In reply to comment #33)
> > Is this moot now that we do omnijar?
> 
> yeah sorta. UPX turned out to slow down our warm startup too much.

What do you mean by "sorta"?
I think this can be closed now that we use omnijar, as UPX slows down warm startup.
(In reply to Marco Castelluccio from comment #36)
> I think this can be closed now that we use omnijar, as UPX slows down warm
> startup.

Yes, thank you.
Status: REOPENED → RESOLVED
Closed: 15 years ago13 years ago
Resolution: --- → WONTFIX
Product: Core → Firefox Build System
You need to log in before you can comment on or make changes to this bug.