179533 - combine mail dlls

Status: RESOLVED FIXED

(MailNews Core :: Backend, defect)

Description

[(not reading, please use [e-mail] instead)]

combine mail dlls

paraphrasing what alecf and I discussed over aim:

"I noticed you recently added more DLLs to the product.  (recently, bayesian
filter plugin and mailviews dll.)

so I've had some experience dealing with DLLs and such, and I talked a bit with 
ulrich drepper yesterday (the guy who owns libc for linux).  if you guys are
concerned about performance and footprint, you should REALLY consider
consolidating your dlls


for instance, did you know there is a 32k overhead for every DLL on windows, and
a 32-100k overhead on linux?

there is the "static" build which produces mailapp.dll, but we've yet to ship
anything with it

we should work towards that as the default.


the other thing is, that cross-DLL calls are more expensive than inter-dll calls
especially when you're calling across symbol boundaries like msgbsutil.dll or 

a call into msgbaseutil.dll is as expensive as a vtable call


whats nice is that you don't really have to rearrange any code

my suggestion would be to add two new directories:

mailnews/build and mailnews/extensions/build
and just declare some giant factories there

leave all the other stuff (like nsMsgBaseCID.h) in the original directories, and
just #include them

and stop build all the individual build dirs

as ulrich drepper said in his talk "DLLs should not be used as a means to
organize  your code, instead they should be used to decide on granularity of
what should and shouldn't be loaded

one thing you might want to optionally load is the import stuff.


but if I remember mail's dependencies correctly, every DLL gets loaded 

in addition, I was seeing great savings from consolidating dlls, just because
you  only have one factory, and you lose the on-disk overhead of each dll


just consolidating you should expect to get a ~25k savings on disk
I count 25 mail DLLs - if you consolidated this into 3 dlls (mail, import, mail
extensions) then you'd save 550k of disk space, at a minimum

Comment 1

[Dan Mosedale (:dmosedale, :dmose)]

> there is the "static" build which produces mailapp.dll, but we've yet to ship
> anything with it
>
> we should work towards that as the default.

Agreed.

> the other thing is, that cross-DLL calls are more expensive than inter-dll 
> calls 

I'm guessing you meant "intra-dll calls" here, in which case that would be correct.

> especially when you're calling across symbol boundaries like 
> msgbsutil.dll or 

Symbol boundaries?

> a call into msgbaseutil.dll is as expensive as a vtable call

This is true for C code and non-virtual C++ calls.  I think a virtual C++ call
doesn't actually get more expensive, because it goes through the vtable instead
of (not in addition to) the PLT/GOT.  I'm not positive about this last, however,
perhaps bbaetz or drepper can confirm.

> mailnews/build and mailnews/extensions/build
> and just declare some giant factories there

It might even be reasonable to just do it all in mailnews/build.  We want it to
be possible to build without the extensions, but that doesn't necessarily mean
that it's important to give users the ability to throw away all the extensions
in one chunk at runtime in the default build (since most users are likely to
want the extensions).

> just consolidating you should expect to get a ~25k savings on disk
> I count 25 mail DLLs - if you consolidated this into 3 dlls (mail, import,
> mail extensions) then you'd save 550k of disk space, at a minimum

Why not into one?  The code will all be demand-paged in anyway, so we don't have
to worry about import code getting loaded when it's not needed.  Since we export
so few symbols anyway, the number of relocations that need to happen at startup
   isn't likely to grow much, I don't think.

I just added a pointer to Ulrich's home page (where the paper and other related
info can be found).  Thanks for the talk, Ulrich!

Seth, feel free to reassign this to me, if you wish.

Comment 2

[Ulrich Drepper]

> This is true for C code and non-virtual C++ calls.  I think a virtual C++ call
> doesn't actually get more expensive, because it goes through the vtable instead
> of (not in addition to) the PLT/GOT.

Once resolved, functions in virtual function tables indeed have the same cost in
intra- and inter-DSO calls.  But the initial relocation still has to be done and
since the virtual function table is data the relocation cannot be delayed. 
Therefore virtual functions have an immediate influence on the startup time. 
Even if the number of relocations is not high the fact that they cannot be
delayed using the lazy relocation thought PLT entries is significant. 
Especially in large projects (with many and/or large DSOs) every lookup is costly.

Comment 3

[Alec Flett]

wow! so that means vtables in general need to be relocated for each dll loaded,
at DLL load time? Yikes... but I guess not surprising.

Ulrich - you had mentioned in your talk about pre-processing dso's so that they
were loaded at the right address the first time, to avoid the relocation cost
(i.e. assuming they don't conflict with any other DSO's in the set, etc) - where
is that tool? (And for those windows people listening, this is like rebase.exe,
if I'm understanding the tool correctly)

Comment 4

[Ulrich Drepper]

> wow! so that means vtables in general need to be relocated for each dll loaded,
> at DLL load time? Yikes... but I guess not surprising.

Just to be pedantic: this isn't true for DLLs, but it's true for DSOs.  DLLs are
these horrible Windows things which have, as far as we are concerned here, not
much in common with DSOs.  I really think we should use the right terminology
since it does make a difference.

Ignoring this, vtables of course have to have relocations.  The vtable entries
are either address/function descriptors or little code stub.  In any case is
there an absolute address somewhere.  Absolute addresses (or even cross-DSO
relative addresses) cannot be computed at link-time.

vtables are relatively cheap if the linker knows the target functions are in the
same DSO.  In that case relative relocations are generated which don't require
the expensive name lookup.  If the target function could be in another target
(note: the function could be defined locally) then it's a full relocation which
can be very costly.  If a relocation cannot be avoided (which is the case for
vtables) the next best thing you can do is to convert them into relative
relocations.  If the target symbol isn't used externally at all, use a linker
map/version script.  If it is used somewhere else as well use a local alias (see
 section 2.2 in my DSO paper).  If the symbol is only defined in another DSO
you're out of luck.  Only rearranging and merging DSOs can help.

> Ulrich - you had mentioned in your talk about pre-processing dso's so that
> they were loaded at the right address the first time, to avoid the relocation
> cost (i.e. assuming they don't conflict with any other DSO's in the set, etc)
> - where is that tool?

ftp://people.redhat.com/jakub/prelink/

Note that you need a recent glibc.  AFAIK, the only distribution featuring the
necessary prerequisites is Red Hat 8.

Comment 5

[Alec Flett]

*** Bug 200607 has been marked as a duplicate of this bug. ***

Comment 6

[Jo Hermans]

Any progress ? This should be a huge improvement, without much trouble.

Comment 7

[Scott MacGregor]

re-assigning as I need this work for Thunderbird. 

I've added a new directory under mozilla/mailnews called build which contains
the factory for mail.dll. See build/nsMailModule and build/Makefile.in for more
details. 

Will start posting the associated build and packaging changes that go with this
new directory shortly. 

Comment 8

[Scott MacGregor]

to give a rough idea of savings, I made an optimized version of mail.dll and
compared it to the current size of the various mail dlls. Keep in mind optimized
DLLS on my maching are always larger than what the release team generates (never
could figure out why).

mail.dll --> 2.56 MB

vs. 
msgbaseutl.dll --> 286,720
msgcompo.dll --> 285,024
msgdb  ---> 110,592
msgimap  --> 528,384
msglocal --> 249,856
msgmdn --> 40,960
msgnews--> 262,144
msgmime -> 40,960
msgbase ---> 581,632
addrbook --> 475,136
emitters --> 45,056
bayesian --> 36,864
mailview --> 28,672
---------------------
          total: 2,972,000

savings of 412 kb. 

Comment 9

[Scott MacGregor]

All of the various import DLLs will become a single import.dll file. Please see:
mailnews/import/build/nsImportModule.cpp for more details.

What's left:
1) Packaging / installer changes
2) smime

Comment 10

[Scott MacGregor]

Attachment: mail changes

This patch contains the mail specific changes for optionally building a static
mail build. It currently generates:
mail.dll
smime.dll
import.dll

Comment 11

[Scott MacGregor]

Attachment: --enable-static-mail config changes

This patch adds a build config option to generate a static build verion of
mailnews. 

It is off by default. I also added a small line of code to enable it by default
for Thunderbird.

Comment 12

[hacker formerly known as [e-mail]]

So what makes this option any different than the --enable-meta-components=mail
option? (And why am I always the last one to know when someone decides to land a
new build option?)

Comment 13

[Alec Flett]

yeah, I was going to say - you need to clear the configure.in changes through
seawood.. I don't think this is deserving of a configure option - you should
just turn this on now, before 1.4beta closes.

Comment 14

[Matteo Settenvini]

If tested, why shouldn't it be enabled by default? (I refer to
http://www.mozilla.org/status/2003-04-25.html). If it is an improvement, it
should be on, not just for Firebird, but for Mozilla 1.4b also.

Comment 15

[(not reading, please use [e-mail] instead)]

one of the reasons mscott was waiting on the trunk was (I think) to avoid any
possible ns tree issues / installer issues.  (upgrade, and you can't have all
those old mail modules lying around, right?)

for the trunk, would 1.5 be a better time to turn this on?

Comment 16

[Sean Su]

Don't worry about cleaning up obsolete files on upgrade.  All one has to do
(after this fix has been applied - or at the same time) is to add obsolete mail
files to the list in mail.jst for win32, see link:
  http://lxr.mozilla.org/seamonkey/source/xpinstall/packager/windows/mail.jst#351

Mac doesn't have an installer, so nothing to worry about there.

Linux's installer deletes the entire directory before installing anything, so
it's safe there.

Win32 is the only platform to worry about.

Comment 17

[timeless]

and os2/mail.jst ?

Comment 18

[Peter Van der Beken [:peterv]]

Please cvs remove mozilla/mailnews/addrbook/build/nsAbBaseCID.h since that was
moved to src.

Comment 19

[Benjamin Smedberg]

I'm working on a configurable packaging system in bug 20640... please don't land
any packaging changes for this bug without talking to me. Also, was there ever
an answer how this build options is different from --enable-meta-components=mail ?

Comment 20

[Wayne Mery (:wsmwk)]

Per Scott "Thunderbird (and even the suite right now) use a static mail build." So this should be closed, no?

Comment 21

[David :Bienvenu]

yes, marking fixed.