201149 - build libxul (formerly called the GRE) separately from apps

Status: RESOLVED DUPLICATE of bug 272783

(SeaMonkey :: Build Config, defect)

Description

[hacker formerly known as [e-mail]]

With the new roadmap and the various embedding clients that keep invading the
mozilla/ cvs space, we really need a way to build just the GRE, preferably via a
separate mechanism than we're currently using to avoid confusion.  I'm thinking
of using a gre.mk to pull and build just the parts of the tree that compose the
GRE.  Ideally, this would mean that we also actually define what goes into the
GRE rather than letting the embeddor of the month decide what shape and flavor
they want their magic gecko pill to be.  In reality, we're shooting for letting
the developer build just what they need so GREs will probably still be as
inconsistent from app to app as they are right now.  We're not going to be able
to get around that if we seriously want to accommodate embeddors' needs,
however, we should define what the authoritative GRE is (ie, the one distributed
by mozilla.org).   Starting with packaging manifests and working backwards is
*not* the way to do this.

With the move to gre.mk, we can take the opportunity to completely move away
from cvs modules (e.g. SeaMonkeyAll).  Besides dropping the application
directories, this would allow us to further trim our GRE tree by allowing us to
drop the cruft that have accumulated under the directories defined by our
current cvs modules.  This also means that if we drop a component from cvs, we
can stop pulling from that directory immediately rather than having to search it
forever.  This has a big advantage over cvs modules as we cannot remove a
directory from the cvs module definition without breaking pulls of previous
builds.  Right now, I'm tasked to provide a new toplevel makefile for this
barebones minimo configuration (bug 199345).  I'd rather not duplicate those
lists of directories in minimo.mk if they can be added to a gre.mk with the
appropriate ifdefs.

One downside that I haven't worked out yet is how to handle building these
multiple apps from the single toplevel mozilla/ dir.  The current approach used
by phoenix (and soon camino if objdir support is to be added) is to add options
for those application targets to the toplevel configure script and have it
generate the makefiles, etc for the application as well as gecko. I think that's
a losing proposition and contributes heavily to the monolithic feel of our build
process.  The GRE should be a separate standalone build process that's used by
the applications much like NSPR & LDAP.  This confusion would be better avoided
by not lumping everything into the mozilla/ directory and/or creating
application specific subdirs to contain the toplevel build files needed for that
application.

Comment 1

[timeless]

this sounds good, but it will mean that the cvs modules that are exposed in
bonsai will become less useful with time. perhaps no one cares (after all we
still have mozilla/lib/* [ignoring mozilla/lib/mac which we actually need]).

Comment 2

[hacker formerly known as [e-mail]]

Attachment: win32 GRE file list

This is the list of files that dougt sent me which compose the authoritative
GRE.

Comment 3

[Benjamin Smedberg]

dougt: I was under the impression that the "toolkit" was going to be part of the
GRE, as it is shared by all XUL applications.

At the very least it should contain html.css, probably ua.css

IMO since we are doing this we should ship all of toolkit.jar as part of the GRE
immediately. Otherwise, each XUL app that uses the GRE is going to have to
include their own copy, which seems rather pointless.

Comment 4

[hacker formerly known as [e-mail]]

Hrm.  Let's not start confusing the GRE & the XRE.  If the GRE were to contain
"toolkit", then how would it be possible to share a single GRE between
SeaMonkey, the Birds and whatever other embeddors were using the non-XUL
portions of the GRE?

Comment 5

[Benjamin Smedberg]

Toolkit is *supposed* to be a common toolkit for multiple applications. It
should not change for different applications.

Really, what is the distinction between GRE and XRE? All XRE needs in addition
to the base GRE is toolkit...

Comment 6

[Christian :Biesinger (don't email me, ping me on IRC)]

bsmedberg: no, xre also needs a way to run the xul applications. gre is just a
collection of .dlls/.xpts basically (as I understand it)

Comment 7

[Brendan Eich [:brendan]]

See bug 206358 for a generic XUL apprunner in the new roadmap world.

If the difference between "GRE" and "XRE" is a small enough epsilon, there's no
point in having two things.

/be

Comment 8

[hacker formerly known as [e-mail]]

Right now, the distinction between the GRE and XRE is XUL.  Last I heard, there
were going to be no chrome files in the GRE as the GRE is targetting native
application embeddors.  Also, since the toolkit not part of the tree identified
as gecko (ie, the engine), it doesn't belong in the GRE.  

Brendan, if you're proposing skipping the GRE in favor of an XRE that contains
toolkit, wouldn't that cause problems for the people who still use the SeaMonkey
app suite.  Or do they just lose out on the ability to share the rendering
engine with the rest of the applications?

Comment 9

[Doug Turner (:dougt)]

the GRE can not (today) contain resource files (.css, etc) because necko doesn't
support res/chrome/ urls that can look in both the GRE directory and the
application directory.  When this is fix, you would expect that the required
pieces for basic web browsing to be in the GRE.  Of course, we would allow
embedders to replaces these files by having a modified copy in their local
application res directory.

Comment 10

[Benjamin Smedberg]

cls: even if we can't/won't include all of toolkit/ at this time, certain
"chrome-like" files such as html.css/quirks.css are definitely part of the gecko
rendering engine and should be included in the GRE when the
"multiple-chrome-locations" bug has been resolved.

Comment 11

[Brendan Eich [:brendan]]

I'm not proposing anything yet; I'm here to say it doesn't pay to split *RE
hairs if the hairs are few and small.

Also, in case there's confusion: "XUL" is supported by the GRE.  The content and
layout code for XUL and XBL is compiled in, AFAIK ("Minimo" and other embeddings
may compile it out, or not, but mozilla.org is not going to ship a GRE that does
not support XUL and XBL).

BTW, the toolkits are compatible, last I heard.  That doesn't mean the
integrated app could use mozilla/toolkit without changes -- I don't know for
sure that it couldn't.  It would probably need other stuff not in
mozilla/toolkit.  It would probably need different back ends (satchel, e.g.).

/be

Comment 12

[Brendan Eich [:brendan]]

Chris said I could mess with the summary a bit to match current jargon (see
http://weblogs.mozillazine.org/roadmap/archives/005370.html#more for the
libxul/xulrunner pair).  We should start now on this, it'll take several steps
and some sweat equity.  It's a priority for 1.8 and beyond.

Pragmatically, per Pike, we would like whatever libxul gets built for the new
apps to be built once, then linked against when building Firefox and
Thunderbird.  But we want static builds for Firefox on most platforms, for perf.
 Until we get libxul consolidated into one library, with minimal exports and
fast intra-lib call linkage, we shouldn't lose the release builds' performance
gain from being static.  But developers need to test Fx and Tb, and can't stand
the cycles and lost disk space to pull and build two trees.

Another step would unify for SeaMonkey, but that'll take more work on the ifdef
front.  We should size up the problem before deciding how and when to attack it,
I think.  Can someone remind me of a bug on that, if there is one?

/be

Comment 13

[cls]

Attachment: split up DIST v1

So, we're trying to separate libxul from the big fat baby we call Mozilla.  We
start to do that by first decoupling the bits which are already separate apps,
NSPR, NSS & LDAP.  This first patch does that by putting each of the subapps
into separate dist/<subapp> dirs & dist/seamonkey.  There's some hackery added
to xpinstall/packager/Makefile to bundle these separate dirs into a single
directory (dist/seamonkey/mozilla) for testing & distribution.	

This is definitely a work in progress.	It has only been tested with a Linux
SeaMonkey tree. YMMV.  

As you can see from the incomplete config/autoconf.mk & config/config.mk
changes, there is still some work to be done to split libxul from the front-end
apps.  Right now, the idea is to set MOZ_APP as appropriate in each of the
1000+ makefiles.  We will also need to convert REQUIRES into gecko_REQUIRES and
split it up into <app>_REQUIRES as necessary.  I haven't figured out how to
automatically detect *_REQUIRES variables so the makefiles will have to
explicitly set REQUIRES_APPS as well which will cause the <app>_REQUIRES vars
to be used.  It's a bit complex for my liking at the moment but it will make
the split explicit and make it more likely that inter-app dependencies aren't
accidentally introduced.

I haven't followed up on the massive makefile changes yet (though the perl
script has been written) but I think we're going to run into problems with
directories that aren't in gecko but span apps, like toolkit or xpinstall. 
bsmedberg has suggested that extensions be given their own MOZ_APP setting so
there shouldn't be a huge problem there.  

Another area that I haven't even begun to look into is how to merge the chrome
jar files into the final package.  With this first patch, it's not an issue as
the chrome is all under dist/seamonkey/.

Comment 14

[alanjstr]

Is this going to be a phased approach, or are you going to try to do it all at
once?  

Comment 15

[cls]

*** This bug has been marked as a duplicate of 272783 ***