107302 - mozilla/Makefile.* needs to support componentized builds

Status: RESOLVED WONTFIX

(Firefox Build System :: General, defect)

Description

[Judson Valeski]

Currently the only way to build XPCOM standalone is via modules.mk. We need to
migrate the specific dir pulling/building mechanism to the top level
mozilla/Makefile.* build system.

Eventually, this will allow for "layered" builds that will let the developer
determine how much they want to build (just the core, core+embedding,
core+embedding+mozilla-proper, etc).

Comment 1

[Judson Valeski]

To clarify, I didn't mean that we pull the existing dir pulling/building
mechanism over, rather, that we just wind up w/ some model defining what to
pull/build. Ideally that's dynamically built up at build time (perhaps based of
REQUIRES).

Comment 2

[Chris McAfee]

We can do the pull part of this via some static form
of alecf's module-graph.pl script + module2dir.pl lookup.

Comment 3

[hacker formerly known as [e-mail]]

Attachment: First basic cut at layering

Comment 4

[hacker formerly known as [e-mail]]

Attachment: last minute tweaks are bad

Comment 5

[hacker formerly known as [e-mail]]

As long as the perl script generated makefile operates on a module level, then
we're fine (I think).  I do not want to use the current modules.mk mechanism of
cherry-picking directories from various modules in order to get a particular
module to build.  That mechanism was meant as a workaround for the module
dependency mess not as the final solution.

Right now, the only things that can be "layered" are nspr, xpidl & js.  xpcom
isn't going to make it into the base list until the follow deps are cut:
xpcom -> string -> unicharutil -> necko
xpcom -> necko
xpcom -> libjar -> caps
xpcom -> uconv

Comment 6

[Alec Flett]

as discussed with cls on irc (but recorded here for the record)
xpcom->unicharutil is bug 100214
xpcom->necko is bug 100212
xpcom->uconv is bug 100676
jar<->uconv is bug 102943
xpcom->jar is bug 107289 (but this is lower priority than bug 102943)
string->unicharutil is bug 10575 (just filed)

you might also want to see bug 66759, which is the tracking bug for XPCOM_STANDALONE

Comment 7

[Alec Flett]

Comment on attachment 55702 [details] [diff] [review]
last minute tweaks are bad

looks like a good start. sr=alecf

Comment 8

[hacker formerly known as [e-mail]]

*** Bug 107301 has been marked as a duplicate of this bug. ***

Comment 9

[hacker formerly known as [e-mail]]

Ugh, so after some extended testing, I discovered that this could potentially
screwup our REQUIRES dependency system since the purge script doesn't know about
the split.  Running the purge script for each portion of the split cases headers
to be missing and only running the purge script for the final portion of the
split (currently the largest segment) could cause dependency problems in the
first portions to be hidden until the build is clobbered.

Comment 10

[hacker formerly known as [e-mail]]

Ok, crisis averted.  Turns out that I was doing a boneheaded thing by putting
xpcom/typelib (xpidl) into the base group w/o the rest of xpcom.  REQUIRES will
force us to group on module boundaries, which is what we want...I was just
jumping the gun a bit with xpidl.

Comment 11

[hacker formerly known as [e-mail]]

Attachment: use generic rule to handle multiple tiers of dirs

I added a generic rule to handle multiple lists of "tier" dirs.  No more
copy-n-paste ugliness.	I had to invoke a separate make process for each
toplevel tier compile so that parallel builds still worked.  No clue how to
make this work for win32.  I'm tempted to just make win32 use a gmakefile for
the toplevel makefile.

Comment 12

[Brian Ryner (not reading)]

Comment on attachment 58073 [details] [diff] [review]
use generic rule to handle multiple tiers of dirs

r=bryner

Comment 13

[timeless]

in comment #6 alecf meant string->unicharutil is bug 107575

Comment 14

[hacker formerly known as [e-mail]]

Attachment: Add tiers for psm, mailnews, calendar & final binary.

Comment 15

[Brian Ryner (not reading)]

Comment on attachment 60731 [details] [diff] [review]
Add tiers for psm, mailnews, calendar & final binary. 

r=bryner

Comment 16

[hacker formerly known as [e-mail]]

Attachment: Add separate tier for extensions

Comment 17

[Alec Flett]

Comment on attachment 74626 [details] [diff] [review]
Add separate tier for extensions

cool. sr=alecf

Comment 18

[Asa Dotzler [:asa]]

Comment on attachment 74626 [details] [diff] [review]
Add separate tier for extensions

a=asa (on behalf of drivers) for checkin to the 1.0 trunk

Comment 19

[Vidar Haarr (not reading bugmail)]

Has this been checked in ?

Comment 20

[hacker formerly known as [e-mail]]

Attachment: move over Beyonce, xpcom is going solo!

This patch cleans up the xpcom entry for modules.mk since all of xpcom's major
external dependencies have been broken. (Go Team!)  It also rearranges the
tiers so that tier_1 is the 3rd party libs that we've sucked into the the tree
and tier_2 is the base application level libs (that have no other
dependencies).

Comment 21

[Alec Flett]

Comment on attachment 81970 [details] [diff] [review]
move over Beyonce, xpcom is going solo!

sr=alecf
nice.

Comment 22

[Garrett Arch Blythe]

Comment on attachment 81970 [details] [diff] [review]
move over Beyonce, xpcom is going solo!

r=blythe

Comment 23

[hacker formerly known as [e-mail]]

Attachment 81970 [details] [diff] has been checked in on the trunk.

Comment 24

[hacker formerly known as [e-mail]]

Mass reassign to new default build assignee

Comment 25

[hacker formerly known as [e-mail]]

Attachment: split xpfe & toolkit into separate tiers

Comment 26

[Alec Flett]

Comment on attachment 127478 [details] [diff] [review]
split xpfe & toolkit into separate tiers

this is great stuff! can I sr=alecf this or do we have to wait for leaf? :) I
especially like the renaming of xpfe/components/history/public's module to
"history" - I've had similar changes in my tree for a long time.

Comment 27

[hacker formerly known as [e-mail]]

Well, I was hoping to get the module owner's input on this but I won't squabble
over a free sr. :) Bryner's a module peer anyways.

Comment 28

[Daniel (Leaf) Nunes]

Comment on attachment 127478 [details] [diff] [review]
split xpfe & toolkit into separate tiers

sr=leaf

Comment 29

[Roland Mainz]

Can anyone check whether bug 212909 ('Build failure due missing
"nsToolkitCompsCID.h"') is a regression due the last checkin, please ?

Comment 30

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

This caused bustage in Firebird and Thunderbird in three different ways.  See
bug 212509 for details.  (more problems with the forked nsIGlobalHistory,
xpfe/components/search/src/ depends on browser/components/bookmarks/public/, and
toolkit/xre/ depends on xpinstall/.)

Comment 31

[Alec Flett]

ack! who forked nsIGlobalHistory? that interface is FROZEN!

Comment 32

[Alec Flett]

ack! who forked nsIGlobalHistory? that interface is FROZEN!

Comment 33

[(default assignee for unassigned bugs. does not receive bugmail)]

Mass reassign of Build/Config bugs to Leaf.

Comment 34

[Chase Phillips]

Mass reassign of open bugs for [e-mail] to [e-mail].

Comment 35

[J. Paul Reed [:preed]]

Mass re-assign of bugs that aren't on the build team radar, so bugs assigned to [e-mail] reflects reality.

If there is a bug you really think we need to be looking at, please *email* [e-mail] with a bug number and explanation.