217772 - build GRE components as a meta-component

Status: RESOLVED DUPLICATE of bug 274605

(Core Graveyard :: Embedding: GRE Core, defect)

Description

[Benjamin Smedberg]

*** From email ***
>Doug, I have a patch in one of my trees which builds most of the GRE
>components as a single meta-component.
>
>1) do you think this might help startup time/performance?

Yes, it should.  We can easily measure this.

>2) do you think this might help codesize? I'm having trouble measuring
>this between multiple trees, because the codsighs tool won't compare
>easily.

probably not.  Although, on linux, there is a bit of overhead per DLL.

>3) are you interested in having this patch in the tree (as an optional
>configure flag, of course).

Sure.  this is probably the way the GRE should be shipped.  Put it in a 
bug, cc me, darin, and brendan.
*** end email ***

So here it is... if this is destined to become a default build option, it should
probably be release-only. Linking a debug build with the gigantic meta-component
takes forever (on my 1Ghz/1GB RAM linux box, almost 20 minutes). A release build
is much quicker.

I don't really like the whole system of putting meta-component linking
information in config.mk. It's error-prone, and not a "normal" place too look
for such information. It actually looks pretty easy to recurse through the
makefiles and extract the information automatically (it's all in the makefiles
already). I haven't done anything about it yet, but I'll give it a shot and see
what it costs in terms of build time.

cls was very negative about this proposal, but I asked him why and didn't
understand his answer.

Patch this evening, hopefully. This is definitely alpha material. Doug, I also
had a few questions about some modules that were being packaged as part of the
GRE (on windows) but didn't have GRE_MODULE = 1 in the Makefiles. I'm not
exactly clear on what GRE_MODULE does... it doesn't look like dist/gre is used
for anything obvious.

Comment 1

[Darin Fisher]

i think we need to see some numbers to help us decide how worthwhile this
approach is.  off hand, it sounds like something good, but numbers would help
affirm that.    this could also just be a configure option that is disabled by
default.  kind of like the way static builds are currently handled.  you can
then just leave it up to folks to decide if they want this or not, independent
of whether or not they are compiling with debug symbols.

Comment 2

[Benjamin Smedberg]

oops, forgot to cc cls.  Chris, since you presented some objections on IRC,
could you comment here about what you think is wrong with my plan?

Comment 3

[Benjamin Smedberg]

Attachment: allow configurable --enable-meta-components=gre

OK, here's what I have. It builds and apparently run smoothly on linux/gtk and
win32. I would like to go ahead and land this or something similar when the
tree opens for 1.6. This would *not* build by default... you need to specify
--enable-meta-components=gre to configure. I am pretty sure that this will not
work on mac without additional linker flags (I would need to test, and I don't
have a mac). But once I land, I can get somebody with a mac to try and tweak
the last few link flags.

Comment 4

[Benjamin Smedberg]

Comment on attachment 130637 [details] [diff] [review]
allow configurable --enable-meta-components=gre

Also, once it is landed, people can also play with performance numbers... none
of my machines are "quiet" enough to make TS/TXUL/TP numbers useful.

Comment 5

[cls]

> cls was very negative about this proposal, but I asked him why and didn't
understand his answer. 

bsmedberg: "what do you think about linking all of the GRE modules into a
meta-component?"
cls: "I think it's a bad idea."

That's about as far into the proposal as we got.  Having seen the full proposal
now, I still think it's a bad idea.  But remember that I detest both useless
additional build options and meta-components, so you're two strikes down with
the get-go.  

1) Management issue.  Can we get a standard m.o GRE distribution on all
platforms *before* we start making weird build customizations for it?  The list
of modules that should go into the GRE (bug 201149) is, afaik, based upon the
list of dlls that were used in one of the AOL-related client releases.  Afaik,
that list has not been reviewed or otherwise authorized in any way.  Should PSM,
cookie, xmlextras & wallet really be part of the GRE?

2) Custom build options. Why make this another general build option that we're
only going to test in a specific situation?  Thanks to minimo, we already have
more than enough options which only work in select undocumented cases.  Anyone
who attmepts to use those options for anything else (like say a SeaMonkey or
FireBird build) are told unceremoniously to stop using those options (bug
210791).  Given our track record, I don't see this case being any better. 
Looking at the patch, it'll break if I build using --disable-extensions . 
Jumping back up to the first point, why is an extension part of our base GRE anyway?

3) Meta-components.  You have the pain of combining code to link a single huge
library (ala gklayout), without the benefits.  Sure, you shave off some dll
overhead for the linking penalty.  Because those modules have to be treated as
separate for other build configurations, you must use xpcom to cross the
imaginary module boundaries within the dll.  IIRC, the overhead of xpcom calls
is what's driving our decomification efforts in layout.  If the GRE is going to
be used by multiple applications, in theory, we should decomify the combined dll
to increase performance as well.

Will combining these libs give us some performance gain? Sure, we've already
seen that to a certain degree with the other meta components & static build
variants.  Is that gain worth the additional pain caused by adding the build
option, the link penalty and needing to maintain another matrix of
meta_component configs? IMO, no.  

As an aside, since darin mentioned static builds, has anyone given any real
thought as to what's going to happen to the static builds when the GRE is
shipped separately?

Comment 6

[Benjamin Smedberg]

> list of dlls that were used in one of the AOL-related client releases.  Afaik,
> that list has not been reviewed or otherwise authorized in any way.  Should PSM,
> cookie, xmlextras & wallet really be part of the GRE?

I was taking my list of modules from the manifest files in embedding/config.  I
think that dougt or drivers need to make an "executive list" of modules that
are/aren't in the GRE, and post it (updated as necessary, of course). Not just a
list of win32 dlls, but a "feature list" if you will of things that ought to be
in a GRE, across all platforms.

Personally, I think that various things that live in the extensions directory
still ought be part of the GRE, viz xmlextras. Moving the cookie backend to
necko will make it unnecessary to include extensions/cookie in the GRE.

If we generate the meta-module configuration information dynamically (see bug
217847) then this meta-module could obey special configuration options such as
disabling xmlextras.

Comment 7

[Benjamin Smedberg]

Comment on attachment 130637 [details] [diff] [review]
allow configurable --enable-meta-components=gre

There's a better way.

Comment 8

[Benjamin Smedberg]

*** This bug has been marked as a duplicate of 274605 ***