171082 - Do everything possible to minimize the build size

Status: RESOLVED INCOMPLETE

(Firefox :: General, defect)

Description

[Blake Ross]

We should do everything we can to reduce size.  For starters, there's
eliminating the modern jar and the composer UI, as well as samples stuff and
embedding crap.  Once we fork completely, we can get rid of comm.jar. This is
kind of a tracking bug; it'd be great if we could get this work going for by 0.2.

Comment 1

[Asa Dotzler [:asa]]

rginda asked kindly if we would not remove jsd3250.dll. that is platform
specific stuff for venkman (or any other project that might want to use it) that
would require him to make multiple different XPIs for venkman if we removed it.
It's about 75k and I think we should try to keep it. 

my preliminary investigations show that we can shave about 1MB from static size
by removing modern and embedding stuff. 

Comment 2

[Blake Ross]

We should try to remove composer UI, modern and embedding crap for 0.2.

Comment 3

[Asa Dotzler [:asa]]

how about we get what's already got for 0.2 and really attack this in 0.3?

Comment 4

[Hampton Maxwell]

You could zip the linux builds with bzip2 rather than gzip.  That will shave
about 5% off the download size according to my testing.  bzip2 is pretty much
guaranteed to be on any linux distro from the last 2-3 years I'd imagine.

Comment 5

[Pierre Chanial]

yeah, we should target comm.jar eradication for 0.3

Comment 6

[Sébastien Delahaye]

I've made a Win installer/uninstaller for Phoenix (using NSIS 2.0a7) and it
reduces the size to 7.01 MB, thanks to the bzip2 compression. The .zip from the
same build weights about 7.44 MB.
The installer (based on the build from october 02) is here:
http://seb.delahaye.net/phoenix/Phoenix-build20021002-setup.exe and the script
used to make it is here: http://seb.delahaye.net/phoenix/phoenix.nsi
I plan to make a build everyday, 'cause I think it's a lot more user-friendly
than unzipping a file (it can add shortcuts, etc.), and it's smaller :)

Comment 7

[Bernard Alleysson]

"Do everything possible" ?

bug 127985: Build phoenix with Visual C++ 7.0 !
10% win on binaries size, Windows only
(as measured in bug 136999, just by comparing VC6 O2 vs VC7 O2)

bug 118455: Use -0 (no compression) for jar files !
read comment by [e-mail]

Comment 8

[Sébastien Delahaye]

Not sure if this belongs to bug 161749 or this one.
I saw on the Phoenix forum that the full Mozilla preferences panel is still
available in Phoenix by pasting this URI into the addressbar:
chrome://communicator/content/pref/pref.xul
The full message is here:
http://www.mozillazine.org/talkback/read.php?f=11&i=836&t=836

In fact, in the windows nightlies (I haven't looked at the linux ones),
chrome/comm.jar (540KB) as well as chrome/modern.jar (17.2KB) are still present.
I'm not sure they are really needed if Phoenix doesn't use them.

Comment 9

[Sébastien Delahaye]

Attachment: pngcrush'd .png files

I used pngcrush (http://pmt.sourceforge.net/pngcrush/) to reduce the size of
the .png files that are in the skin/classic/browser/ folder of the browser.jar
file.
It should make browser.jar slighty smaller without altering the quality of the
.png files.
The modified .png files are attached.

Comment 10

[Blake Ross]

Some things to do:

- Disable wallet (waiting on Satchel)
- Disable embed-sample.jar
- Disable en-mac.jar
- Disable en-win.jar on Unix, en-unix.jar on Windows
- Disable modern.jar (if disabling wallet doesn't do this)
- Disable comm.jar (waiting on total fork)
- Disable building mozilla/themes/classic (waiting on total fork)
- Disable forms.jar (if that's what I think it is -- form controls)
- Disable various components we don't need (a cursory glance reveals
nsDictionary.js, nsSetDefaultMail.js, nsUnsetDefaultMail.js,
nsDownloadProgressListener.js, urlwidgt.xpt
- Disable a lot of the stuff in the /res directory

Comment 11

[André Dahlqvist]

Just some info on the kind of savings that bzip2 can provide. I created a tar of
a build from today, and then used both gzip and bzip2 on it. The savings are
quite big:

Size with gzip-compresson: 9,525,442
Size with bzip2-compression: 8,889,198

The makefile xpinstall/packager/Makefile already has the target required to
build a bzip2-build, so the change should be easy to make. The command to create
a tar.bz2-build is:

make MOZ_PKG_APPNAME=phoenix MOZILLA_BIN="\$(DIST)/bin/phoenix-bin"
MOZ_PKG_FORMAT="BZ2"

Comment 12

[greg]

What about the help system(help.jar)? Is it needed?

I think it can be removed safely, it is Mozilla-specific after all. It hasn't
got much use for phoenix.

Comment 13

[greg]

just saw help.jar is in some way stripped down - it has only abou 20k size!

sorry for that last post from me

(but strangely enough, I can still use the help! For example, I can access it
through the preferences panel or Page Info window. And it all works.)

Comment 14

[Blake Ross]

*** Bug 172426 has been marked as a duplicate of this bug. ***

Comment 15

[Stephen Walker]

/profile/defaults/panels.rdf is still copied to the Phoenix profile dir even
though it isn't used by Phoenix.

Comment 16

[Simon Paquet [:sipaq]]

While going through all the pref stuff, I made a little list concerning which
prefs are obsolete. I placed two question marks in front of pref entries, of
which I'm not entirely sure if they are totally obsolete.

In all.js
=========
?? pref("advanced.mailftp",                    false);

// sspitzer:  change this back to "news" when we get to beta.
// for now, set this to news.mozilla.org because you can only
// post to the server specified by this pref.
pref("network.hosts.nntp_server",           "news.mozilla.org");

pref("network.proxy.news",                  "");
pref("network.proxy.news_port",             0);


In WinPref.js
=============
// Unread mail count timer. Value to be specified in seconds
// default is 5 minutes, i.e., 5 * 60 seconds = 300
pref("mail.windows_xp_integration.unread_count_interval", 300);


editor.js
=========
?? All of it


mailnews.js
===========
All of it

Comment 17

[Stephen Walker]

Attachment: patch

Nuke res/gfx, res/rdf, res/samples, res/throbber, res/viewer.properties,
xpfe/components/prefwindow, xpfe/components/updates, xpfe/components/urlwidget
and panels.rdf.

Comment 18

[Brian Ryner (not reading)]

Comment on attachment 105912 [details] [diff] [review]
patch

Looks good, except that I already filed bug 179414 for the rdf/resources stuff
and I'd prefer to fix that without resorting to ifdefs.  I'd also like to get
rid of the images in gfx/src entirely if they aren't used anywhere, rather than
ifdefing.  For the stuff in xpfe/browser, it would make more sense to ifdef it
with ENABLE_TESTS.

Comment 19

[Stephen Walker]

I filed bug 113971 close to a year ago about removing the unused gfx/src images.

Comment 20

[Stephen Walker]

Attachment: patch v1.1

Removed the parts that bug 179414 & bug 113971 already cover. 
s/ifndef MOZ_PHOENIX/ifdef ENABLE_TESTS

Comment 21

[Asa Dotzler [:asa]]

Comment on attachment 106275 [details] [diff] [review]
patch v1.1

bryner, can you review this patch please? Thanks.

Comment 22

[Stephen Walker]

Attachment: patch v1.2

Disable embed-sample.jar also

Comment 23

[Blake Ross]

Another thing for the list:
- Disable help.jar

Comment 24

[Stephen Walker]

bryner, can you change the .mozconfig's to not build help? Adding an ifdef for
an extension that can be disabled is unnecessary.

Comment 25

[André Dahlqvist]

*** Bug 180702 has been marked as a duplicate of this bug. ***

Comment 26

[André Dahlqvist]

About building help as en extension; doesn't bug 89603 affect Phonix? 

Comment 27

[Simon Paquet [:sipaq]]

*** Bug 180704 has been marked as a duplicate of this bug. ***

Comment 28

[Stephen Walker]

bryner, blake: can one of you checkin my patch?

Comment 29

[Michael Beatty]

Use UPX (http://upx.sf.net/) to compress x86 builds.

Win32 4.0 default install:
13.9 MB
Win32 4.0 default install, `upx --best *' ran in root, components directories:
7.74 MB

This is the installed size, not the installer package size.

Comment 30

[Blake Ross]

Attachment: patch to disable more stuff

Comment 31

[Blake Ross]

That patch isn't complete, I need to deal with the jar.mn.

Comment 32

[Stephen Walker]

Using UPX 1.24 (upx --best, in dir's with dll's/exe's) with the latest nightly
build decreased the zip size from 6.09MB to 5.47MB while the unzipped size went
from 12.3MB to 7.39MB. However, some problems with using UPX have already been
stated in bug 55930 that need to be looked at.

Comment 33

[laszlo]

Bug 182695 might be a side effect of some clean-ups that happened on or after 11/26.

Comment 34

[Jeremy M. Dolan]

Just thought I'd point out defaults/pref/mailnews.js still exists, it's 26k.

Also, there are 23 GIFs in the tree. Might save negligable space converting them
to PNG, but if PNGs take longer to parse/uncompress, may not by worth the savings.

Comment 35

[Simon Paquet [:sipaq]]

Jeremy, I already mentioned that in comment 16. There is a lot of bloat still
there. For example in classic.jar we still package icons and markup for composer
and messenger.

Comment 36

[Stephen Walker]

Not all of the prefs in editor.js go unused by Phoenix, so it cannot be removed 
entirely.  The prefs that are used could be copied to the Phoenix version of 
all.js, which would allow Phoenix builds to skip modules/libpref/src/init 
during the build process.

The unneeded images are there because classic is still waiting on a total fork, 
see comment 10.

Comment 37

[Christian 'CeeJay' Jensen]

If you prefer it you can find a simple windows frontend for PNGcrush at :
http://pngcrushgui.saurabhkumar.com/

PNGcrush is good , however I find that with some images, Irfanview with it's PNG
compression set to 9 (the max), can compress even better.
I tried it on Toolbar.png in classic.jar.
Before : 54.796 bytes
After  : 46.120 bytes

The remaining gif's should probably be converted to PNG's but if needed you
could optimize them with GifOptimizer:
http://www.flashfxp.com/freeware.php?Program=GifOptimizer

All ZIP archives (.zip and .xpi i believe) can be compressed better with 7-Zip :
http://www.7-zip.org/
The programmer have found a way to make compatible zip-archives that are smaller
than any other program can produce.

All Jar packages can be compressed better.
I use WinAce with max compression for this, but I haven't tested other programs.

Comment 38

[scratch]

*.jar is the same as *.zip.

Comment 39

[Minh Tran]

RAR it
I use Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.3b) Gecko/20030211
Phoenix/0.5
phoenix-win32.zip original nighlybuilt 6,468,727 bytes
phoenux.zip compressed with WinRAR zip mode 6,452,061 bytes
phoenix.rar compressed with WinRAR, Best-solid-Dict Size 4096KB 5,395,644 bytes


17% smaller with RAR

to [e-mail]

*.jar is not the same with *.rar. I can RAR classic.rar to 82% ratio (from
470,008 to 387,125 )

Comment 40

[Rainer Koschnick]

Nightly from 20th of Feb:
6,471,499 bytes

The same archive compressed with 7-zip (Open Source) ZIP modus:
6,329,806 bytes

Comment 41

[Rainer Koschnick]

If you compile using Visual Studio 6 or 7 the "Aggressive Optimize" header file
might prove useful.  I use it in all of my projects and so far it always helped
reducing the executable file sizes.  Can be found at http://www.nopcode.com

Comment 42

[Pierre Chanial]

morphing it into a tracking bug where the dependencies will go.

Comment 43

[Simon Paquet [:sipaq]]

Adding dependencies

Comment 44

[Roland Mainz]

bryner:
What do you think about the idea of using uncompressed JARs that the resulting
*.tar.gz/*.zip shrinks (see bug 118455 ("libjar performance / RFE: Replace
{compressed JAR, read(), uncompress()} with {uncompressed JARs, mmap()}")) ? 

That may save up to 500k in the resulting binary release tarball...

Comment 45

[alanjstr]

*** Bug 207315 has been marked as a duplicate of this bug. ***

Comment 46

[alanjstr]

Does anyone know of anything in editor.js or mailnews.js that should be kept?  I
started bug 207315 and bug 207317 for them since this seems to be a tracker.  

Comment 47

[Neil Parks]

I deleted both files from June 14 nightly and there is no problem whatsoever.  I
don't think FB (at least on Win ME) needs anything from either one.

Mozilla/5.0 (Windows; U; Win 9x 4.90; en-US; rv:1.4b) Gecko/20030614 Mozilla
Firebird/0.6

Comment 48

[alanjstr]

Stephen-

In comment 36 you said some of mailnews and editor.js were still needed.  Can
you please specify which lines on the respective bugs?

Comment 49

[Karsten Düsterloh]

[e-mail]: who do you think you are?

Comment 50

[Stefan Borggraefe]

I think Bug 68686 "Shrink .jar files by stripping out whitespace, comments"
looks very promising. Fixing it would benefit all Mozilla applications.

Comment 51

[Simon Paquet [:sipaq]]

taking over qa abandoned bugs.

Comment 52

[Glenn Randers-Pehrson]

Add dependency on bug #208607 (reduction of libpng size)

Comment 53

[Brian Ryner (not reading)]

Moving out, this is an ongoing bug.

Comment 54

[Mike Connor [:mconnor]]

*** Bug 240774 has been marked as a duplicate of this bug. ***

Comment 55

[Pierre Ducroquet]

Attachment: Compacter for the installer

I just wrote two scripts  for linux which compacts the size of firefox. It
works great and allow to raise the size of the installed firefox from 19Mb to
10mb!
Here is the script which compact the installer (change the third line of the
file !!!)
It uses upx and zip...
It produces a different firefox, compacted with upx too...

Comment 56

[Pierre Ducroquet]

Attachment: Compacter for the firefox folder itself

Here is the script which compact the firefox folder (change the third line of
the file !!!)

Comment 57

[Pierre Ducroquet]

Attachment: Compacter for the firefox folder itself

Here is the script which compact the firefox folder (change the third line of
the file !!!)

Comment 58

[alanjstr]

That means that files have to be decompressed at runtime which has already been
deemed unacceptable.  This bug is about changing code, not the compressor du jour

Comment 59

[Aaron Kaluszka]

Pierre, you might want to continue your work in bug 55930.  What we could do
here is compact the build with UPX BEFORE packaging and decompress it upon
installation, gaining the decrease in size for network transport, but not
harming execution time.  I don't know how well LZMA compresses UPX'd
executables, but ZIP archive sizes are significantly improved.

Comment 60

[Troodon]

Just a thing I've noted with my Win32 installs:
Mozilla Seamonkey 1.6 [Browser+Composer+Chatzilla+the two default themes  
=11.184.354 bytes
Mozilla Firefox 0.9.1 [Browser+the default theme]=13.917.451 bytes

!!!

Comment 61

[Justin Kerk]

There's another PNG compressor called OptiPNG at
http://www.cs.toronto.edu/~cosmin/pngtech/optipng/ that outperforms pngcrush.
It's also more convenient to use because it compresses all the files in-place.

Comment 62

[Jo Hermans]

*** Bug 254033 has been marked as a duplicate of this bug. ***

Comment 63

[Jo Hermans]

*** Bug 254034 has been marked as a duplicate of this bug. ***

Comment 64

[Thomas]

Can we introduce some of the techniques as used in "Portable Firefox" and "pPp"?
Like recompressing EXE, DLL and JAR files?
Maybe also change "Target Milestone" to a more current version :p

Comment 65

[Nigel McFarlane]

*** Bug 256415 has been marked as a duplicate of this bug. ***

Comment 66

[Cameron]

Does this bug have a target? as in.. how could it ever be resolved? 

Seems to me this bug could be closed and just throw a simple one line page on devmo - "Keep builds small."

Comment 67

[alanjstr]

Well, this is a meta bug

Comment 68

[Dimosthenis Stamatiou]

Whatever you guys decide, DON'T try compressing seamonkey.exe with UPX... After a while, it crashes user32.dll, which takes explorer.exe with it, and keeps doing that unless you open task manager, kill explorer.exe and then uncompress seamonkey.exe, either by command line or UPX-iy...

Comment 69

[Cpuroast]

Hi,

I noticed that Firefox 2.0 final for windows is packed with UPX 1.24 instead of 1.25.

Also, A few things have changed in the UPX world since 1.x.

The best compression is always achieved through: upx --brute filename 
instead of upx --best filename.
This option was added in upx 2.0. It will try every possible compression method and keep the smallest one. 

Also I just tested the new UPX 2.03 by de-upxing the official firefox 2.0 windows installer, and re applying upx with UPX 2.03 with the --brute parameter and the installer shrunk by 2KB compared to with UPX 1.24.

Also I did the same with UPX 2.90 beta which whill be called 3.0 final once it's finished, and it shrunk by 3KB compared to 1.24 and 1KB compared to 2.03.

So shrinking it by an extra 2KB now with UPX 2.03 and then by another 1KB when UPX 3.0 is final, seems like a good idea to me.

UPX 2.03 with the new --brute parameter instead of --best should put in place. I don't think there is a big risk involved.

Ciao,

Comment 70

[Glenn Randers-Pehrson]

There is some discussion in bug #386585 about the tradeoff between code size
and speed of decoding PNG images.  The speed can be increased somewhat on
PCs by using assembler code to access Intel MMX functions.  The difference
in speed can be measured in very large (10k by 10k) images, but for reasonably
sized images the difference is hardly perceptible.  The size of the compiled
code depends on platform, and amounts to about 8Kbytes on a FreeBSD system.
It was reported that on a Windows XPSP2 platform there was *no* increase in
filesize which I find difficult to believe.

Comment 71

[Martin Walch]

I think this meta bug can be broken up into five parts, and IMO a distinction like this should be made when discussing about build size:

1. Changes that do not affect any function or readability. Most of this are some sort of cleanups.

some examples:
- remove whitespaces before EOL in css, js and other files (e.g. bash/sed: for i in `find . -name "*.css"`; do sed -i s/"\( \)*$"//g $i; done)
- the same with newlines before EOF if there is more than one
- strip chunks from png files that do not really give information (e.g. tEXt chunk "Created with The GIMP")
- code cleanups in js or c++ files like removing unused functions that are obsolete (and it is sure that they will never be used again)

These are permanent changes and submitting them to the tree should be safe. Most of them are easy and can be made semi-automatically. They have a lot of small positive effects and it is very unlikely that they have any negative side effects.

2. Automatic changes when making the distribution. These are changes that may have significant impact on the readability of code and cause loss of information that is not needed for normal usage of the distribution.

this could be sth. like
- remove all non-mandatory whitespace and newline characters in html, css and js files
- remove all comments from all files that are not compiled (e.g. except licenses)
- give variables from these files autogenerated shortnames
- optimize layout of interpreted code for scanning, parsing and interpreting
- remove optional chunks from png files that do not have any effects on the distribution
- strip functions and variables that will not be used in the current distribution but might be used in the future

Most of these changes are rather difficult as they must be fully automated and parse files the same way as it is done when using the distribution. They are only applied for the distribution and will never be submitted to the tree. Their main purpose is to reduce file sizes and to increase performance. However, this could also come in handy for some profiling.

(of course the code that is doing such changes should not be shipped with the distribution)

3. The effects of choices for basic character encoding, formatting and file formats may be considerable. They may slightly increase readability or automated processing.

this could be things like
- improve structure for Huffman and LZ* compression
-- make sure license texts, that are equivalent are also identical (identation, newlines and whitespaces are exactly the same
-- use the maximum line length for license texts
-- sort authors in alphabetical order
-- use always LF, never CR LF
-- use only 7 bit us-ascii
- use png instead of gif
- do not use an alpha channel, this is only bloat and will cost cpu time
- do use an alpha channel, as the background will be defined in css anyway
- use only one pixel graphics format
- set up exact and obligatory coding standards for css, js, html... this could even include things like alphabetic sorting of attributes in css files (most of these things can be automated with auto-formatting tools or one of those big IDEs)
- re-arrange variables and functions in js as well as in c++ code
- think about code alignment

These are extensive permanent changes with high potential. E. g. it is possible that a duplicate of a complete license text will fit it only few bytes. Or a smart code layout could reduce the used memory of L1 or L2 caches or how much must be kept in RAM or may be swapped. However, most of these changes are hard to automate and some of them have complex side effects, so they can become dangerous and should be carefully analyzed before applying them to the tree.

4. Find the best compression strategy, compression formats and tools.

- do compress png files as good as possible
- use optipng (..) for png compression
- do not compress png files at all
- compress jar files as good as possible
- do not compress jar files
- use upx for overall compression
- group files by type and choose compression format by type
- group files by type, so similar files get into one compression window
- use a special compiler or compiler version
- pass special parameters to the compiler

These changes are very complex. Some can be committed, but most not, because they can only be applied when finally making the distribution. It is vital to balance the costs for compression, decompression, file size and so on. Depending on the system, the system load and the available hardware, loading an uncompressed file from harddisk might be much faster as well as much slower than loading a compressed file from harddisk and uncompressing it in RAM. A time-consuming compression over several days on a single computer might be acceptable if it gives a 5% gain in file size without loosing speed in decompression or execution. Compiling with gcc's -fprofile-arcs, using the program and recompiling it with -fbranch-probabilities might increase the binary size by a few bytes, but increase performance by several percent.

5. Making changes to the content itself.

this could be e. g.
- reuse more code
- make svgs instead of pngs
- create pngs with colors that fit into a 256 color palette (but still look good ;))
- create graphics that are easy to compress
- use different, more compact algorithms
- remove functionality
- replace a feature with something similar that is slimmer
- <whatever comes to your mind>

These are the really big changes that are probably far beyond a bug report like this one. I only list it for the sake of completeness.

Just my thoughts on this. Thanks for reading.

Comment 72

[Worcester12345]

Just wanted to bump this bug. It is my oldest open Firefox bug.

Comment 73

[:Gavin Sharp [email: [e-mail]]]

This bug isn't tracking anything useful at this point.

Comment 74

[Worcester12345]

(In reply to Blake Ross from comment #0)
> We should do everything we can to reduce size.  For starters, there's
> eliminating the modern jar and the composer UI, as well as samples stuff and
> embedding ****.  Once we fork completely, we can get rid of comm.jar. This is
> kind of a tracking bug; it'd be great if we could get this work going for by
> 0.2.

How much of this original description has actually been done, and what is left?

Also, what is or was the original intent of this bug. In other words, the "why" part of it.

Thanks.

Comment 75

[Glenn Randers-Pehrson]

(In reply to Worcester12345 from comment #74)
> (In reply to Blake Ross from comment #0)
> > We should do everything we can to reduce size.
> Also, what is or was the original intent of this bug. In other words, the
> "why" part of it.

Looking at the comments above (e.g., comment #40), it appears that the goal
was to reduce download size of the builds, which was then about 6Mb.  Now
it's about 130Mb.
	firefox-0.8-i686-pc-linux-gnu.tar.gz	09-Feb-2004 05:53 	8.8M
	firefox-28.0b6.tar.bz2	25-Feb-2014 08:14 	29M

Comment 76

[Glenn Randers-Pehrson]

Now it's about 30Mb not 130Mb (I fixed that typo but somehow it survived).

Comment 77

[Dimosthenis Stamatiou]

(In reply to Glenn Randers-Pehrson from comment #75)
> (In reply to Worcester12345 from comment #74)
> > (In reply to Blake Ross from comment #0)
> > > We should do everything we can to reduce size.
> > Also, what is or was the original intent of this bug. In other words, the
> > "why" part of it.
> 
> Looking at the comments above (e.g., comment #40), it appears that the goal
> was to reduce download size of the builds, which was then about 6Mb.  Now
> it's about 130Mb.
> 	firefox-0.8-i686-pc-linux-gnu.tar.gz	09-Feb-2004 05:53 	8.8M
> 	firefox-28.0b6.tar.bz2	25-Feb-2014 08:14 	29M

Back then though, Firefox/Firebird/Phoenix was much more bare than now. Since then, stuff were added like animations, graphics, third-party libraries, the 3D inspector etc., so it has piled up. The update files for 64-bit versions of nightly, for example, sit at around 40MB.