46775 - [tracking] need ability to generate a statically linked build

Status: RESOLVED WORKSFORME

(SeaMonkey :: Build Config, defect, P3)

Description

[Judson Valeski]

we should be able to build a single binary, statically linking everything
together. This will allow systems that can't afford shared lib overhead to load
a single bin into memory.

Global GetFactory/library functions will collide so we'll need a way to resolve
these conflicts.

Comment 1

[Judson Valeski]

blizzard, any idea on how much overhead each .so takes up on linux? Would this
be a big win?

Comment 2

[Christopher Blizzard (:blizzard)]

You want to do what to my whom?

I have no idea what the pain of loading shared libraries is.  It's not painless,
to be sure.  There's all sorts of information stored in those shared libraries
that we could save on.  No dynamic string information, no shared PIC info, etc. 
Of course, I don't have to tell you what the downside is to this but I can see
why you might want it for embedded type applications.

Comment 3

[Brendan Eich [:brendan]]

Adding xpcom buddies.

/be

Comment 4

[Judson Valeski]

ideas I've seen fly around.
- #ifdef the .cpp's and makefiles to do what we want. quick and dirty. not very 
flexible. XP. (warren)
- post build processing. build a tool that would take a list of shared libs and 
strip out the dll entry points and regenerate consolodated entry points in a new 
root lib. very flexible. not necessarily XP (we'd need a tool per platform). 
(dougt)
- build processing. use auto-conf/build flags to redefine the NSGetModule token 
w/ $(module-name)_GetModule() when building a lib, and force it to be static. 
That would get rid of all the name collisions, then you could link all the 
statics however you wanted. very flexible. not necessarily XP. (shaver)

Comment 5

[cls]

Attachment: build tweaks to override NSGetModule tokens

Comment 6

[cls]

Attachment: template used to statically load modules

Comment 7

[cls]

I've attached the results of my attempt to make Shaver's idea work.  I was able
to get necko to compile statically and load before the other modules were
autoregistered..  I hit a couple of assertions complaining that viewer couldn't
1) create a hidden window or 2) create a scratch URI which makes me think that
necko wasn't registered even though it said it was.

 The first attachment contains the changes to make
--enable-static-modules=necko,layout work as advertised.  There's an ifdef
DEBUG_cls in the first patch that someone will want to check out where I tried
to get around xpcom complaining about registering a library with null nsIFile. 
The second attachment contains the template file used when generating the calls
to module_NSGetModule.

nsNativeComponentLoader: autoregistering begins.
nsNativeComponentLoader: autoregistering succeeded
nNCL: registering deferred (0)
*** Registering necko core and primary protocols components (all right -- a
generic module!)
###!!! ASSERTION: HiddenWindow not created: 'NS_SUCCEEDED(rv)', file
../../../../mozilla/xpfe/appshell/src/nsAppShellService.cpp, line 223
###!!! Break: at file
../../../../mozilla/xpfe/appshell/src/nsAppShellService.cpp, line 223
Initialized app shell component {18c2f989-b09f-11d2-bcde-00805f0e1353},
rv=0x00000000
Initialized app shell component {33e569b0-40f8-11d4-9a41-000064657374},
rv=0x00000000
###!!! ASSERTION: Couldn't create scratch URI: 'mScratchUri', file
../../../../mozilla/intl/strres/src/nsStringBundle.cpp, line 698
###!!! Break: at file ../../../../mozilla/intl/strres/src/nsStringBundle.cpp,
line 698
###!!! ASSERTION: failed to Ensure1Window: 'NS_SUCCEEDED(rv)', file
../../../mozilla/xpfe/bootstrap/nsAppRunner.cpp, line 932
###!!! Break: at file ../../../mozilla/xpfe/bootstrap/nsAppRunner.cpp, line 932
PREF_Cleanup()

Comment 8

[Mike Shaver (:shaver emeritus)]

I think this is a case of needing --whole-archive to pull in all the
unreferenced-by-executable symbols that we need to really make it all go.

cls is on it.

Comment 9

[cls]

Unfortunately, --whole-archive didn't help.  I switched to testing with nsjpg
instead of necko and regardless of the --whole-archive setting, I still cannot
load jpgs.

Comment 10

[Yannick Koehler]

The BeOS is actually going to require this out.  I'm going to see if I can help 
out.

Comment 11

[Colin Blake]

The overhead of loading this many shareable images (DLL's) on OpenVMS is very 
high. If everything could be linked as one huge image, activation would I'm sure 
be much faster. Adding myself to the cc list, as I'm very interested in 
following this effort.

Comment 12

[Doug Turner (:dougt)]

help cls out on this.

Comment 13

[Doug Turner (:dougt)]

helping cls out on this.

Comment 14

[Scott Collins]

might this be easier if every library built as a static lib and had a separate 
sharedlib target based on that (at least on Mac)?

Comment 15

[cls]

I talked to dougt and valeski on irc for a bit and I think I've made some
progress.  First let me do a coredump of my understanding of the situation atm. 
There are basically 3 different linking / loading scenarios that we have to
worry about:

1) Dynamically loading components at runtime. This is what we do by default.
2) Link all components into a single shared lib to be loaded at runtime.
3) Link all components into a library to be linked against the app.

(1) is pretty much covered.  (3) is what we are going to need in the long run
for most platforms that cannot work in (1).  The only benefit of (2) that I can
see is that it will save us the dynamic loading overhead.  (2) won't necessarily
increase our platform coverage. Nevertheless, dougt and I decided to get (2)
working as (3) may require a new loader.

I've actually managed to get (2) working on a single component.  I created a new
component, libstaticmods.so, which contains a static component, libnsjpg.a. 
After a few hours of digging thru xpcom, I was able to get mozilla to load
libstaticmods.so at runtime and then go load a jpg from a website. 

Unfortunately, things fell apart after that.  I tried running with a static
nsjpg, nsgif & nspng and libstaticmods.so wouldn't link.  The problem is that
each of these modules define some global logging functions, il_log_module, which
cause conflicts when everything is linked statically.  I have a feeling that we
are going to run into that general problem with most modules.

Comment 16

[Andrew Thompson]

The adition win for startegy 2) on the Mac is it would reduce the total open file 
count. Mac OS < 9 can only have about 350 files open. (Mac OS 9 ups this to 
8000). So people with older MacOS really need this too, and any performance gain 
wouldn't hurt!

Please see bug 26659 for more info.

Comment 17

[Andrew Thompson]

Hmmm, the other solution thats been suggested to solve the open files problem is 
to put all the shared libraries into a JAR file, like the chrome. 

Would no doubt need linker/loader changes though...

Comment 18

[cls]

Attachment: working staticmod changes

Comment 19

[cls]

Attachment: tar.gz of modules/staticmod/

Comment 20

[cls]

Attachment: changes to layout & htmlparser that should work but don't

Comment 21

[cls]

Ok, I've attached the changes I made to get method (2) to work.  In retrospect,
I think I cheated a bit.  Instead of trying to load each component using the
standard NSGetModule() call & dealing with the RegisterSelf calls, I took each
component's nsModuleComponentInfo list and appended to the one used by
libstaticmod.so.   The plus side is that this method did not require any changes
to the loader or the generic factory routines.  The downside is that it only
seems to work with generic modules.  Also, I changed the configure option to
--enable-static-components as it made more sense to deal with these changes on a
component level rather than a module level. 

In order for the current scheme to work, each component library needs to export
these 3 symbols which will have $(LIBRARY_NAME)_ prepended to them for static
builds:
NSGetModule
NSGetModule_components
NSGetModule_components_count

The first is the standard xpcom entry point for components.  The second is the
nsModuleComponentInfo struct that is usually passed to the new generic module
macro.  The third is the number of components in the previous struct.

Comment 22

[Colin Blake]

With respect to the symbol names being:

  NSGetModule
  NSGetModule_components
  NSGetModule_components_count

with $(LIBRARY_NAME)_ prepended to them, I would like to remind people that some
operating systems require symbol names to be unique within a certain length. On
OpenVMS that length is 31 characters. I would like to suggest changing the 
naming scheme to:

  NSGetModule
  nsModuleComponentInfo
  count_nsModuleComponentInfo

or something similar.

Comment 23

[cls]

dveditz has an explanation for the apparent generic/non-generic factory mismatch
that I'm seeing:

<dveditz> cls: generic factories means it was built using the macros and a
+standard structure. Otherwise it's a custom factory 
> dveditz: hmmm.  it may be coincidental, but I haven't been able to get a
+component that doesn't use the macros to work with the proposed fix for that
+bug
<dveditz> cls: how are non-generic vs generic factories involved in that bug.
+Both should have the same entry points, right? 
> dveditz: yes, they should be.  which is why I'm confused 
<dveditz> cls: there are also new-style and old-style modules 
<dveditz> cls: the new style has everything gotten from NSGetModule 
<dveditz> cls; the old style has 4 entry points, something like
+NSRegisterSelf, NSUnregisterSelf, etc. 
<dveditz> cls: all the generic modules use new-style, a *few* non-generic ones
+do as well, but mostly the non-generic ones are the old-style ones that have
+not been converted 
<dveditz> cls: therefore your macro-magic aimed at NSGetModule isn't going to
+work 

Comment 24

[Judson Valeski]

yes, there are pleanty of non-macro'ized factories out there.

Comment 25

[lchiang]

per email with Jud, changing nsbeta3+ to nsbeta3- on all "embed" keyword bugs
since embedding changes will not be made in the mn6 branch. If you feel this bug
fix needs to go into mn6 branch, please list the reasons/user impact/risk and
nominate for rtm. Thanks.

Comment 26

[cls]

Adding warren to cc: 

Comment 27

[Doug Turner (:dougt)]

Attachment: Build system changes to make static modules work (without configure diff)

Comment 28

[Doug Turner (:dougt)]

Attachment: Static Module code (tar.gz)

Comment 29

[Doug Turner (:dougt)]

The last two attachments will allow you to do something like:

./configure --enable-static-components=necko,necko2,cookie,...

and it will build a library called libstaticmod.so which includes necko, necko2,
cookie.

Comment 30

[cls]

dougt, wrt to your review request:

Suggestion for NSGetModule_components* renaming: 
NSGetModule_comps & NSGetModule_comp_count

The calloc() in modules/staticmod/nsStaticModule.cpp.in needs to be switched to
PR_Calloc().  Also, we never free those structures once they've been
allocated.   

And what's our game plan for non-generic factory modules?  Do we need to add the
loading of non-generics to nsStaticModule or do we need to convert non-generics
to generic modules?  

Comment 31

[Brendan Eich [:brendan]]

I know next to nothing, but why should the prefix hog the number of letters in
the names (NSGetModule_comps & NSGetModule_comp_count)?  NSGM_components is more
like it, no?

/be

Comment 32

[Doug Turner (:dougt)]

well, note that the build will redefine these defines to [module name]_*

I would be happy with:

comps
comp_count

which would translate to (in the static module case)

[module]_comps
[module]_comp_c

Also, we should only export these symbols when generating a static module.
Thoughts?  

Comment 33

[Colin Blake]

I've said it before, but I'll say it again. OpenVMS only has 31 significant 
characters in a name. I would MUCH rather you turned:

  some_very_long_symbol_foo
  some_very_long_symbol_foo2

into

  svls_foo
  svls_foo2

or similar.

Comment 34

[cls]

Well, we'd still need some sort of prefix (NSGM will do) so that the build
system doesn't go off prepending the module name to every count & components
token in the code.  Short of adding another ifdef, nothing immediately comes to
mind about fixing the excessive exporting problem.

Comment 35

[Doug Turner (:dougt)]

okay:  |NSGM_comps| and |NSGM_comp_count| it is.  I am not going to touch
NSGetModule.  Colin, you are going to be okay unless some evil person uses a 20+
character module name.

Comment 36

[Warren Harris]

Subject: 
             Combining Modules
        Date: 
             19 Oct 2000 07:11:53 GMT
       From: 
             [e-mail] (Warren Harris)
 Organization: 
             Another Netscape Collabra Server User
         To: 
             [e-mail]
         CC: 
             Judson Valeski <[e-mail]>, mozilla-embedding 
<[e-mail]>
 Newsgroups: 
             netscape.public.mozilla.embedding




Doug, 

I spent a little time tonight working out what I was trying to describe on the 
phone today for combining modules together. I think the system I came up with 
has a few advantages over what
you showed me, primarily that the build system changes are minimized, and also 
that it doesn't have the same code explosion problem due to replicating the 
module bodies. Here's what I
came up with... 

The way it works is that you compile a standard module file with 
-DNS_COMBINED_MODULES. The module file looks pretty much the same except that 
the first argument to
NS_IMPL_NSGETMODULE is now unquoted: 

     NS_IMPL_NSGETMODULE(nsRegistryViewerModule, components)

This ends up generating a specially named NSGetModule function that can later be 
called by a "meta module." 

Next, in some other directory, we implement the meta module that combines the 
other modules together. This is done by first putting the NSGetModule functions 
into an array, and then
using the NS_IMPL_META_NSGETMODULE macro: 

     #include "nsIGenericFactory.h" 

     NS_DECL_SUBMODULE(nsRegistryViewerModule) 
     NS_DECL_SUBMODULE(nsSearchModule) 

     static nsGetModuleFun gCombModuleFuns[] = { 
         NSGetModule_nsRegistryViewerModule, 
         NSGetModule_nsSearchModule 
     }; 

     NS_IMPL_META_NSGETMODULE(nsAppComponents, gCombModuleFuns)

I think that's all we have to do, but I have to admit that I haven't fully 
tested the code yet. I'll enclose it below. 

Warren 
  

Comment 37

[Warren Harris]

Attachment: Warren's "meta module" patch

Comment 38

[Judson Valeski]

hmm, although warren's concept minimizes build changes, the build changes are
what give us build time flexiblity. if you wanted to selectively
-DNS_COMBINED_MODULES, you'd have to modify the build system anyway. IMO, we
should run w/ what doug and cls have come up w/. However, we're gonna want to do
this on windows too, and it isn't obvious to me which method is going to port
over to windows better; thoughts?

also, the stripping of quotations is semantically strange and will lead to
confusion I suspect.?

Comment 39

[vidur (gone)]

cls' and dougt's system allows the determination of which modules should be
combined to happen in a single place - from the make (actually config) command
line. The command line list determines which modules should be built as static
libraries as well as what goes into the meta module. Because of the config
dependencies, it is going to be a bit of work translating the scheme to Windows
and Mac (let alone other platforms), but I think we should run with it now until
we can come up with a tried and true way for all platforms.

Comment 40

[Doug Turner (:dougt)]

warren,

We have a working solution for our embedding needs.  I have checked this in on
our embedding branch.  Since the bug is solved for me, I am going to reassign
this bug to you.

If things don't pan out wrt the meta module implmentation, I would like you to
reconsider the solution that we have created.

thanks.



/cvsroot/mozilla/Makefile.in,v  <--  Makefile.in
new revision: 1.110.8.1; previous revision: 1.110
done
Checking in allmakefiles.sh;
/cvsroot/mozilla/allmakefiles.sh,v  <--  allmakefiles.sh
new revision: 1.251.4.1; previous revision: 1.251
done
Checking in configure;
/cvsroot/mozilla/configure,v  <--  configure
new revision: 1.616.2.1; previous revision: 1.616
done
Checking in configure.in;
/cvsroot/mozilla/configure.in,v  <--  configure.in
new revision: 1.723.2.1; previous revision: 1.723
done
Checking in config/autoconf.mk.in;
/cvsroot/mozilla/config/autoconf.mk.in,v  <--  autoconf.mk.in
new revision: 3.162.10.1; previous revision: 3.162
done
Checking in config/config.mk;
/cvsroot/mozilla/config/config.mk,v  <--  config.mk
new revision: 3.170.8.1; previous revision: 3.170
done
Checking in extensions/xmlextras/build/src/nsXMLExtrasModule.cpp;
/cvsroot/mozilla/extensions/xmlextras/build/src/nsXMLExtrasModule.cpp,v  <--
nsXMLExtrasModule.cpp
new revision: 1.4.10.1; previous revision: 1.4
done
Checking in js/src/xpconnect/src/xpcmodule.cpp;
/cvsroot/mozilla/js/src/xpconnect/src/xpcmodule.cpp,v  <--  xpcmodule.cpp
new revision: 1.21.10.1; previous revision: 1.21
done
Checking in mailnews/import/eudora/src/nsEudoraFactory.cpp;
/cvsroot/mozilla/mailnews/import/eudora/src/nsEudoraFactory.cpp,v  <--
nsEudoraFactory.cpp
new revision: 1.8.10.1; previous revision: 1.8
done
Checking in mailnews/import/oexpress/nsOEFactory.cpp;
/cvsroot/mozilla/mailnews/import/oexpress/nsOEFactory.cpp,v  <--  nsOEFactory.cpp
new revision: 1.11.10.1; previous revision: 1.11
done
Checking in mailnews/import/outlook/src/nsOutlookFactory.cpp;
/cvsroot/mozilla/mailnews/import/outlook/src/nsOutlookFactory.cpp,v  <--
nsOutlookFactory.cpp
new revision: 1.8.10.1; previous revision: 1.8
done
Checking in mailnews/import/src/nsImportFactory.cpp;
/cvsroot/mozilla/mailnews/import/src/nsImportFactory.cpp,v  <--  nsImportFactory.cpp
new revision: 1.8.10.1; previous revision: 1.8
done
Checking in mailnews/import/text/src/nsTextFactory.cpp;
/cvsroot/mozilla/mailnews/import/text/src/nsTextFactory.cpp,v  <--
nsTextFactory.cpp
new revision: 1.7.10.1; previous revision: 1.7
done
Checking in modules/libimg/gifcom/gif.cpp;
/cvsroot/mozilla/modules/libimg/gifcom/gif.cpp,v  <--  gif.cpp
new revision: 1.32.4.1; previous revision: 1.32
done
Checking in modules/libimg/pngcom/ipng.cpp;
/cvsroot/mozilla/modules/libimg/pngcom/ipng.cpp,v  <--  ipng.cpp
new revision: 1.29.4.1; previous revision: 1.29
done
RCS file: /cvsroot/mozilla/modules/staticmod/Makefile,v
done
Checking in modules/staticmod/Makefile;
/cvsroot/mozilla/modules/staticmod/Makefile,v  <--  Makefile
initial revision: 1.1
done
RCS file: /cvsroot/mozilla/modules/staticmod/nsStaticModule.cpp.in,v
done
Checking in modules/staticmod/nsStaticModule.cpp.in;
/cvsroot/mozilla/modules/staticmod/nsStaticModule.cpp.in,v  <--
nsStaticModule.cpp.in
initial revision: 1.1
done
Checking in netwerk/build/nsNetModule.cpp;
/cvsroot/mozilla/netwerk/build/nsNetModule.cpp,v  <--  nsNetModule.cpp
new revision: 1.26.8.1; previous revision: 1.26
done
Checking in xpcom/components/nsGenericFactory.cpp;
/cvsroot/mozilla/xpcom/components/nsGenericFactory.cpp,v  <--  nsGenericFactory.cpp
new revision: 1.21.10.1; previous revision: 1.21
done
Checking in xpcom/components/nsIGenericFactory.h;
/cvsroot/mozilla/xpcom/components/nsIGenericFactory.h,v  <--  nsIGenericFactory.h
new revision: 1.20.10.1; previous revision: 1.20
done


(note that you may have to manually pull modules/staticmod.  I am not sure if it
is part of the cvs module)

Comment 41

[Warren Harris]

Fair enough.

Comment 42

[Brendan Eich [:brendan]]

This sounds like a fork in the making.  Shouldn't we agree fast, or else get 
dougt's changes on the trunk too?

warren: how bad is the "code explosion problem" in dougt's scheme?

I don't think we should expect an XP solution here, since our build system is a 
set of platform-specific subsystems.  Maybe we can move toward the holy grail of 
autoconf'd Windows builds (works on OS2!), but I'm not holding my breath.

/be

Comment 43

[Doug Turner (:dougt)]

The code "explosion" termed above is nothing more than copying the module info
stucture which is then passed to the component manager.  It really sounds
scarier than it is.

The patch to the NS_IMPL_NSGETMODULE define is more complicated with meta module
diffs.  Now we have two implmentations depending on which mode we are building
for.  The patch I wrote simply includes two additional lines to the original define.

Since I already have the module owner permission for the build changes, as well
as reviewers of this code, I could check in the build specific changes.
However, I have made changes to the xpcom module.  For that I need warren's
approval.  The changes to xpcom are twofold.

(1) exportings two symbols from the generic module defines.  This allows me to
access the module info struct and determine its size.

(2) pass the module info struct and size to the module destructor.  This is so
that I can deallocate the structure that I pass to the NS_IMPL_NSGETMODULE
macro.  We never needed this before since most of the structs pass to this are
statically allocated.

Warren are you happy with these two changes?  What if I ifdef (1)?

--doug

Comment 44

[Doug Turner (:dougt)]

*** Bug 51114 has been marked as a duplicate of this bug. ***

Comment 45

[cls]

*** Bug 39382 has been marked as a duplicate of this bug. ***

Comment 46

[Doug Turner (:dougt)]

Warren, I am reassigning some bugs which you own over to me (dougt).  If you 
think that you will have time to address them please reassign them back to 
yourself.  This reassignment was suggested by choffman.

Comment 47

[Doug Turner (:dougt)]

I have checked my patches into the truck.  I am marking this bug fixed.  If 
there are alternatives to building a static modules in mozilla, please track 
them in a different bug.

Comment 48

[cls]

Hold up.  Did your changes resolve *all* of the static linking issues?  Namely,
statically linking non-generic modules & registering modules that are statically
linked into the app itself instead of the staticmod component.  Also, was there
any resolution reached on the ability to delete the components list that was now
dynamically allocated?

Comment 49

[Doug Turner (:dougt)]

AH.  this bug is for the entire static library problem, not just the generic 
module linkage.  Sorry about that.  I am reopening based on your comments.  

Comment 50

[Doug Turner (:dougt)]

over to cls.  I am not going to have too much time to work on this.  If you 
don't have time either, please assign to jud valeski.

Comment 51

[Colin Blake]

When are we going to see some of this work appear in the trunk? I can't wait
to do some meta-building on OpenVMS. Should improve startup time a LOT.

Comment 52

[cls]

The --enable-static-components work has already landed on the trunk.  The
monolithic binary work has yet to be done.

Comment 53

[Colin Blake]

Looks to me like --enable-static-components just defines MOZ_STATIC_COMPONENTS,
and the only thing that does is to cause modules/staticmod to get added to
DIRS in the root Makefile. But since modules/staticmod doesn't exist, the
whole things a no-op, right?

Comment 54

[cls]

Attachment: Land the rest of the staticmod changes on the tip

Comment 55

[Colin Blake]

Chris, which modules currently get built into the mega-module, and when do you 
hope to turn this on as the default build method (for those platforms that want 
it)?

Comment 56

[cls]

Colin, you have to specify which components you wish to merge into the
meta-component.  The current embed.mk uses this:
--enable-static-components=necko,necko2,cookie,psmglue,caps,docshell,editor,txmgr,txtsvc,jar50,jsurl,nspng,nsgif,pref,shistory,strres,uriloader

I have no plans to turn it on by default for anyone yet.  We still have a ways
to go before *all* of the components can be merged together.  We still need to
convert the non-generic/old-style components into generic components (ie, ones
that use NSGetModule).  Along the way, we will need to figure out how to resolve
any global symbol conflicts.

Comment 57

[Brian Ryner (not reading)]

r=bryner on the last patch

Comment 58

[cls]

Patch has been checked in.

Comment 59

[cls]

Attachment: Build changes from Ron Guilmette's static link mega-patch updated against the current trunk.

Comment 60

[Chris Waterson]

Branch cut yesterday; STATIC_BUILD_20010418_BRANCH. I'll start filing and adding 
dependent bugs.

Comment 61

[Chris Waterson]

For those interested in helping make this work, here are checkout instructions:

cvs co -r STATIC_BUILD_20010418_BRANCH mozilla/client.mk
cd mozilla
cat <<-DONE > .mozconfig
ac_add_options --disable-shared
ac_add_options --enable-static
ac_add_options --disable-gtktest
ac_add_options --disable-glibtest
ac_add_options --disable-tests
ac_add_options --without-jpeg
ac_add_options --without-zlib
ac_add_options --without-png
DONE
make -fclient.mk

Comment 62

[Roger Chickering]

I had an idea that we could use to get rid of the need for NSGetModule
altogether and give us a lot more flexibility in terms of how we
aggregate code for XPCOM classes into dlls.

The idea is to have the linker and C++ runtime do the work for us.
For each dynamic factory that we'd like to register, we instantiate a
static "DynamicFactoryRegistration" object.  A
DynamicFactoryRegistration object contains the information we'll need
later to call RegisterFactory, and the DynamicFactoryRegistration
constructor adds "this" to a list of pending registrations.

After initializing XPCOM, we can then process each of these
DynamicFactoryRegistration objects to register all the factories from
DLLs that we implicitly loaded.  In addition, we process the list
after explicitly loading a dll to register all of the explicit dll's
factories.

I'm going to attach a tar file to this bug which contains a README
with a much more long-winded explanation of the idea as well as
example code which demonstrates how this could work.

If this seems like a good idea, I'd be happy to charge off and
implement it and see how it works out.

Comment 63

[Roger Chickering]

Attachment: Demonstration of new approach to dynamically registering XPCOM factories

Comment 64

[Roger Chickering]

That attachment I just added is a tar file.  Save it as
dynreg.tar, unpack it like this:

    tar xvf dynreg.tar

It contains windows-specific code, but the idea should be
applicable on all platforms.

Comment 65

[Chris Waterson]

Should that stuff be filed under a different bug?

Comment 66

[Roger Chickering]

OK, between the time that I came up with that great DynamicFactoryRegistration
idea and now I've been reading all about how evil static intializers
are with respect to C++ portability.  So for now, never mind.  I
still think it's a good idea but I'm not ready to charge off and
implement it anymore.  :-)

Sorry for the spam!

-Roger

Comment 67

[Randell Jesup [:jesup] (needinfo me)]

Roger's idea is/was a good one, even if there are portability problems with it
(care to expand or provide a URL or two on that issue?  Perhaps Scott
([e-mail]) would have some comments on this; he seems to be very up on
c++ portability.)

Perhaps we can "roll our own" to avoid the compatibility issues for static
initializations, but otherwise keep the same structure.  I don't know how
different this would be from the current FindSymbol/GetModule stuff, though.

(Don't mind me, I'm just guessing off-the-cuff.)

Comment 68

[Roger Chickering]

rjesup: Here's a URL about the portability of static initializers:

http://www.mozilla.org/hacking/portable-cpp.html#dont_use_static_constructors

There were also a number of threads on netscape.public.mozilla.xpcom
where this was discussed.

I'd like to hope that someday all of our platforms will have support
for C++ language features like this, but I'm new around here and I'm
trying to pick my battles...  :-)

-Roger

Comment 69

[motrei]

I tried the instructions listed here; the build completed but on invoke, nothing
happened. THinking that I needed to list the portions being statically linked
together for now, I tried using --enable-static-components, but I couldnt find
any documentation of what valid values for its argument is, and, looking at the
configure file, I dont believe --enable-static-components=all would work. I am
using configure on the command line. The command line I am using is basically:

configure --prefix=/home/moz/cvs/staticobjs/install --with-gtk
--with-libIDL-prefix=/home/moz/libidl/install --disable-logging
--disable-mailnews --disable-editor-api-log --disable-tests --enable-lea
--disable-debug --enable-strip-libs --enable-elf-dynstr-gc --enable-crypto
--enable-optimize='-O9 -funroll-loops -ffast-math -malign-double
-march=pentiumpro -mcpu=pentiumpro -fexpensive-optimizations -fno-exceptions
-fno-rtti' --disable-shared --disable-cpp-rtti --disable-cpp-exceptions
--enable-idlc --enable-svg --disable-dtd-debug --enable-static
--disable-pedantic --without-extensions --disable-ldap --disable-bidi

I've tried disabling jpeg etc as listed below without success. Any suggestions
on how to try linking statically? I've tried on the tip as well as on the
branch mentioned here.

Comment 70

[Cathleen]

Here is a break down list of libs required per install option (linux mozilla)

minimal xpcom lib (required for installer)
-------------------------
bin/libmozjs.so
bin/libnspr4.so
bin/libplc4.so
bin/libplds4.so
bin/libxpcom.so
bin/libxpistub.so
bin/libmozz.so
bin/components/libxpinstall.so
bin/components/libjar50.so
bin/components/libunicharutil.so

browser libs
------------------------
bin/libcmt.so
bin/libgtksuperwin.so
bin/libgtkxtbin.so
bin/libmozjpeg.so
bin/libjsj.so
bin/libmsgbaseutil.so
bin/libprotocol.so
bin/libgkgfx.so
bin/libgtkembedmoz.so
bin/components/libappcomps.so
bin/components/libjsdom.so
bin/components/libnsappshell.so
bin/components/libcaps.so
bin/components/libchardet.so
bin/components/libchrome.so
bin/components/libcookie.so
bin/components/libdocshell.so
bin/components/libeditor.so
bin/components/libembedcomponents.so
bin/components/libgfx2.so
bin/components/libgfx_gtk.so
bin/components/libgfxps.so
bin/components/libimglib2.so
bin/components/libimggif.so
bin/components/libimgjpeg.so
bin/components/libimgpng.so
bin/components/libimgppm.so
bin/components/libjsurl.so
bin/components/liblwbrk.so
bin/components/libmork.so
bin/components/libmozbrwsr.so
bin/components/libmozfind.so
bin/components/libmozucth.so
bin/components/libmozxfer.so
bin/components/libnsgif.so
bin/components/libnsjpg.so
bin/components/libnslocale.so
bin/components/libnsmng.so
bin/components/libnspng.so
bin/components/liboji.so
bin/components/libpref.so
bin/components/libprofile.so
bin/components/libgkcontent.so
bin/components/libgklayout.so
bin/components/libgkplugin.so
bin/components/libhtmlpars.so
bin/components/libgkview.so
bin/components/librdf.so
bin/components/libshistory.so
bin/components/libstrres.so
bin/components/libtimer_gtk.so
bin/components/libtxmgr.so
bin/components/libtxtsvc.so
bin/components/libuconv.so
bin/components/libucvcn.so
bin/components/libucvibm.so
bin/components/libucvja.so
bin/components/libucvko.so
bin/components/libucvlatin.so
bin/components/libucvtw.so
bin/components/libucvtw2.so
bin/components/liburiloader.so
bin/components/libwebbrwsr.so
bin/components/libwallet.so
bin/components/libwalletviewers.so
bin/components/libwidget_gtk.so
bin/components/libxpconnect.so
bin/components/libnecko.so
bin/components/libnecko2.so
bin/components/libnkcache.so
bin/components/libjsloader.so
bin/components/libnsprefm.so
bin/components/libregviewer.so
bin/components/libxremote_client.so
bin/components/libxmlextras.so
bin/components/libtransformiix.so
bin/plugins/libnullplugin.so


mail libs
-------------------
bin/libldap40.so
bin/liblber40.so
bin/components/libaddrbook.so
bin/components/liblocalmail.so
bin/components/libmailnews.so
bin/components/libmime.so
bin/components/libmimeemitter.so
bin/components/libmozldap.so
bin/components/libmsgcompose.so
bin/components/libmsgdb.so
bin/components/libmsgimap.so
bin/components/libmsgnews.so
bin/components/libvcard.so
bin/components/libsigned.so
bin/components/libsmime.so
bin/components/libimport.so
bin/components/libimpText.so
bin/components/libabsyncsvc.so

psm libs
-------------------
bin/libnssckbi.so
bin/components/libpipnss.so
bin/components/libpippki.so

Comment 71

[Tomi Leppikangas]

With help from cls i get working static build:
../configure  --enable-mathml --enable-svg --with-extensions --with-xprint \
   --disable-debug --enable-optimize --disable-shared --enable-static
make

One problem i get is that build machine doesnt have zip installed and
i run to bug #76526 which seems to happend just when branch was cut. This
was fixed by just copying personalToolbar.js in place:
cp ../xpfe/browser/resources/content/personalToolbar.js \
   dist/bin/chrome/comm/content/navigator/personalToolbar.js

Comment 72

[Cathleen]

update:
----------------

We're going to cut a new branch and merge all changes from the old changes into 
the new branch.

Here is a to do list:
* make a new branch, both mozilla and commercial (mcafee)
  branch tag: STATIC_BUILD_20010523_BRANCH
              STATIC_BUILD_20010523_BASE
* land the follow changes:
  - xpcom changes (waterson)
  - XPFE changes from danm (waterson)
  - win32 changes, client.mak, big patch (dprice)
  - mac changes, NGLayoutBuildList etc. (thesteve)
  - absync changes from sspitzer/chuang (cathleen)
  - figure how to build mail.so (cls)
  - figure how to build xpinstall (cls)

Comment 73

[Chris McAfee]

base & branch tag has been cut as Cathleen described:

  STATIC_BUILD_20010523_BRANCH

both mozilla & ns trees.

Comment 74

[jhp (no longer active)]

Trying to get this working on OS/2.  I have it building completely, but when I
go to start up, the appshell initialization fails.  For some reason, the os/2
widget library does not register (from widget/src/os2).  The module registration
stuff at the bottom of widget/src/os2/nsWidgetFactory.cpp is very similar to the
gtk version, so I don't understand why this is failing.  Any ideas?

Comment 75

[Chris McAfee]

Static build tbox is now up:
http://tinderbox.mozilla.org/showbuilds.cgi?tree=SeaMonkey-StaticBuild

openwound is normal non-static build, worms is my first attempt
at building static (--enable-static --disable-shared).

Comment 76

[cls]

Right now, the new branch builds but exits immediately on startup.  The xpcom
diffs need to be landed on the new branch before anything useful can happen.

Comment 77

[Chris Waterson]

checked in xpcom changes. there are still some XPFE changes that need to go in.
I'll get them in later tonight.

Comment 78

[Wan-Teh Chang]

The three NSPR libraries (libnspr4, libplc4, and libplds4) must
not be statically linked.  Doing so will lose the automatic native
thread attachment feature on Windows and lose the ability to use
true atomic routines (to modify XPCOM object reference counts,
for example) on Solaris SPARC.  The NSPR static libraries are not
supported and will be deleted in a future release.

Comment 79

[cls]

I landed the --enable-meta-components changes last night.  Right now, the only
option that works is --enable-meta-components=mail .  There are several new
makefile variables that are required to make this scheme work.

In config.mk, the following variables are required for each meta component:
MOZ_META_COMPONENTS_<metamodule>
MOZ_META_COMPONENTS_<metamodule>_comps
MOZ_META_COMPONENTS_<metamodule>_libs

The first variable is the list of module names to be used by the build system. 
The second is the corresponding list of libraries where those modules can be
found.  The third is the list of non-component libs that may be required to
resolve link dependencies.

In each Makefile.in of a module to be built into a static module,
META_COMPONENT=<metamodule> needs to be set.  Each Makefile.in that creates a
component library needs to have COMPONENT_NAME (should be MODULE_NAME?) set. 
This name must be the same as the name passed to the NS_IMPL_NSGETMODULE macro.

As with the --enable-static-components setup, the NSGetModule_components &
NSGetModule_components_count symbols need to be global and mangled in the same
fashion as NSGetModule.  The mangling is triggered by the
XPCOM_TRANSLATE_NSGM_ENTRY_POINT define.  Also, this scheme only works with
generic modules (preferrably using the NS_IMPL_ macros).

Originally, we toyed with the idea of making the meta modules completely
user-specified (ala --enable-static-components), but this seems fairly
impractical for 2 reasons.  First, each meta component has it's own CID,  if we
made them completely dynamic then we couldn't guarantee (or even know) what
service is supposed to be provided by a specific CID.   Second, we would somehow
have to translate the list of libs into a list of component names (or
vice-versa) from the user-input which is basically a pita.  For those reasons,
I'm proposing getting rid of --enable-static-components (currently being used by
embed makefiles) and sticking with hardcoded meta components.

Other misc stuff:
Per wtc's comment, NSPR libs are only built shared now until the platform
doesn't support shared libs. 

NO_STATIC_LIB & NO_SHARED_LIB have been replaced by BUILD_SHARED_LIBS,
BUILD_STATIC_LIBS & FORCE_STATIC_LIB.  (Good-bye double-negative ifdefs and good
riddance).

The list of components for the final link is dynamically created during the
build.  Plus side is that we don't need a ton of Makefile & src ifdefs for every
configure option combination.  However, if something doesn't get added, then you
don't notice until it's too late (see late checkin for ldap).  

Comment 80

[Roger Chickering]

I built STATIC_BUILD_20010523_BRANCH on Linux, and had to resolve
a couple of problems.  There was a windowsy include directive in
nsAppRunner.cpp (I'll post a patch) that was causing problems, and
I had to install libmozz.a, libjpeg.a, and libmozjpeg.a into dist/lib
by running "make install" in the appropriate directories; this
didn't happen automatically.

-Roger

Comment 81

[Roger Chickering]

Attachment: Patch to fix '\' path separator in #include directive

Comment 82

[timeless]

http://www.mozilla.org/hacking/portable-cpp.html#include_simple_filename

3. #include statements should include only simple filenames.
Non-portable example: 
  #include "directory/filename.h"

Comment 83

[Chris Waterson]

Win32 should now (almost) build correctly on the branch if you

  set MOZ_STATIC_COMPONENT_LIBS=1

The build will fail linking mozilla.exe; remove the ``transformiix'' lines from
mozilla/dist/win32_[d|o].obj/[final-link-comps|final-link-comp-names]. (This
problem will be resolved when when merge to the trunk.)

Caveats: PSM still doesn't build; you can sort of build the mail ``meta-module''
by hand (run nmake in mozilla/modules/staticmod after removing the import
modules from mail-link-comps and mail-link-comp-names -- symbol conflicts);
there is some true butchery in some of the makefiles, most notably JS, libreg,
xpinstall, and webshell.

  

Comment 84

[Chris Waterson]

rogc: I reverted the nsAppRunner.cpp changes; we don't need the kooky include
path anymore.

Comment 85

[cls]

Attachment: unix makefile changes for static branch 0523 against trunk

Comment 86

[cls]

Attachment: Removing redundant symbols from intl. From 20010523 -> trunk

Comment 87

[cls]

Attachment: Make components use RegisterComponentWithType & NSGETMODULE_ENTRY_POINT. 20010523 -> trunk

Comment 88

[cls]

Attachment: static component loader changes. needs xpcom/components/nsStaticComponent*. 20010523 -> trunk

Comment 89

[Roy Yokoyama]

/r=yokoyama for the attachment (06/11/01 23:38)

Comment 90

[cls]

A new branch has been cut from this morning's trunk and the previous branch's
changes have been rolled forward.  The branch tag is:
STATIC_BUILD_20010612_BRANCH

Comment 91

[Asa Dotzler [:asa]]

a= [e-mail] for checkin to the trunk.
(on behalf of drivers)

Comment 92

[Christopher Blizzard (:blizzard)]

sr=blizzard on 38031.

Comment 93

[Chris Waterson]

I broke the entirety of the RegisterComponentWithType() junk out into bug 86030,
and picked up all the changes in the patch there. shaver, you wrote it, I r='d.
I'll make scc sr= it. Or maybe blizzard.

Comment 94

[Daniel (Leaf) Nunes]

a=leaf for the build system patch.

Comment 95

[Roger Chickering]

Attachment: Build changes for function reordering on Linux

Comment 96

[Roger Chickering]

Attachment: order.zip: zip file containing files to add to build for function reordering

Comment 97

[Dan Mosedale (:dmosedale, :dmose)]

Note that if you use "cvs add" (without doing a "cvs commit") to add the files,
you can then use "cvs diff -uN" to make your patch include new files, rather
than having to zip or tar them up separately.

Comment 98

[Chris Waterson]

rogc: looks good. What happens if mozilla-bin.order is out-of-date with respect
to the symbols in the build? (E.g., a new symbol not in the order file is
created, or a symbol in the order file goes away.)

Also, we should generate a version of the mozilla-bin.order file with the
autoregistration turned _off_, and see how it affects startup...I noticed a
whole bunch of RegSelf calls! :-)

r=waterson

Comment 99

[Roger Chickering]

The two attachments I just added contain changes relative to
STATIC_BUILD_20010612_BRANCH for reordering the functions in
mozilla-bin on Linux.  I added a new configure option --enable-reorder
that causes us to pass -ffunction-sections to gcc and which
causes the xpfe/bootstrap Makefile to do the actual reordering
based on mozilla-bin.order.

-ffunction-sections causes the compiler to put each function in
its own section in its object file, which is necessary for
linker order specification because the linker deals with things
in sections not functions.  My experiments have not shown any
runtime penalty for specifying -ffunction-sections with no reordering.

I have found that on Linux function reordering saves us between
10% and 13% on startup time and between 1.5 and 2 MB of memory
at startup.

cls, waterson, and cathleen: do these changes look OK?  If so,
will you please check them in Cathleen (I don't have CVS write
access yet)?  I will gladly incorporate feedback and do another 
patch tomorrow.

Thanks!
-Roger

Comment 100

[Roger Chickering]

waterson: functions that are not in the .order file go at the
end, and functions in the .order file that are not in any of the
objects don't go anywhere but the link still succeeds (without
warnings like we see on Windows).

As the functions get added and removed the .order file gets less
and less relevant and thus does less and less good.  Ideally,
we can check in a new order file closer to when we actually ship
to get the full benefit.

Also note that specifying the order in the GNU ld script causes
the link to take a *really* long time.

dmose: Thanks for the tip.  Please bear with me; I will figure it
out eventually :-)

Comment 101

[Cathleen]

rogc, everthing is landed in static branch for you.  :-)

Comment 102

[cls]

The static build branch landed on the trunk last Wednesday so it's also on the
0.9.2 branch.  Moving target milestone out to 0.9.3 for the various cleanup issues.

Comment 103

[Juan José Mata]

Seawood: can you please summarize in a new comment what all the remaining
cleanup issues are?

Comment 104

[hacker formerly known as [e-mail]]

We're still waiting on a resolution for bug 88038.  Distributors of multi-user
installations will need to run regxpcom after installing Mozilla so that the
user doesn't attempt to write to a directory when Mozilla is first run.   The
security-conscious do not want to run an app the size of Mozilla as root just to
register some components (which also requires a valid display because of the
gtk_init calls).  I believe bryner hacked something up for chimino which
registers the internal components from within the app (not sure how it handles
external components).

I believe work on the mac cfm static port was dropped a while ago and now the
mac cfm port itself has been dropped so static builds should work on all of our
supported platforms.

The static build options should be changed to reflect the actual configuration,
which is just static-components.  Some people are still expecting a true static
build with no shared library dependencies.

Oh, and we need to reimplement the gecko meta-component build changes for gmake.

Comment 105

[Ben Bucksch (:BenB)]

Do I understand it correctly that if we don't give users the choice of installed
applications (just install everything incl. Mailnews and Chatzilla), we can use
static builds? Because in that case, you can run the registriation on the build
machine as the building user (not root) and just ship components.reg et al with
the binaries. I do that for Beonex Communicator (we have a simple tarball on
Unix, only to be extracted somewhere) and it works fine (meanwhile).
If you are using the installer, where people select the installed components,
you have to have write access to the install dir and run Mozilla binaries this
way anyways, so there should not be a problem either.
This leaves only distro packages (rpm, deb) as problem, right?

Comment 106

[hacker formerly known as [e-mail]]

When using the --enable-meta-components options, certain bits of the browser are
still optional (mailnews/crypto).  Heck, even w/o that option, jsd & inspector
are still shared/optional for some reason.  Then there's the problem of third
party components.  At some point, the administrator will need to register
components so shipping a pre-built registry isn't practical.

> If you are using the installer, where people select the installed components,
> you have to have write access to the install dir and run Mozilla binaries this

That works fine for a single-user setup.  For a multi-user setup, people may not
feel comfortable running Mozilla as root / admin user.  I don't think we have
per-user component registries yet.

Comment 107

[Dennis Haney]

Isnt debian's update-mozilla-chrome the solution to this problem?

Comment 108

[hacker formerly known as [e-mail]]

Can you elaborate on that?

Comment 109

[Dennis Haney]

They ship the individual parts of installed-chrome.txt on the component level.
Based on what is actually installed they create the actual installed-chrome.txt
and . Wouldn't it be possible to do the same with compreg.dat (even though
debian runs regxpcom to generate it)?

Comment 110

[Daniel Veditz [:dveditz]]

installed-chrome.txt is the trigger for creating the chrome registry (chrome.rdf
and overlayinfo/*), and you have to have write access to create those. If their
solution means we're processing installed-chrome.txt each startup (because the
processed chrome.rdf can't be written) then that's a serious startup performance
hit. If you have write access you don't need a pre-generated compreg.dat

Comment 111

[cls]

Mass reassign to default build config owner

Comment 112

[(default assignee for unassigned bugs. does not receive bugmail)]

Mass reassign of Build/Config bugs to Leaf.

Comment 113

[Benjamin Smedberg]

Whatever we have now with --enable-static is good... statically linking the rest (NSPR/JS) is separate.