Closed Bug 463385 Opened 11 years ago Closed 10 years ago

sprinkle magic PARALLEL_DIRS fairy dust about the build system

Categories

(MailNews Core :: Build Config, defect)

defect
Not set

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: neil, Assigned: neil)

References

Details

Attachments

(2 files, 4 obsolete files)

mailnews/ version of bug 462381
Attached patch Proposed patch (obsolete) — Splinter Review
I also bypassed some makefiles which only one one child in DIRS.
Assignee: nobody → neil
Status: NEW → ASSIGNED
Attachment #346650 - Flags: superreview?(bienvenu)
Attachment #346650 - Flags: review?(gozer)
Comment on attachment 346650 [details] [diff] [review]
Proposed patch

I'll file a follow up bug for the TB-specific dirs
Attachment #346650 - Flags: superreview?(bienvenu) → superreview+
Comment on attachment 346650 [details] [diff] [review]
Proposed patch

a full clobber build with -j8 and this patch breaks for me on Linux 4 times before finally working. Summary of the errors are :

gmake[6]: Leaving directory `/home/gozer/opt/src/mozilla.org/objdir-x86_64-unknown-linux-gnu-bloat/mailnews/base/test'
gmake[6]: Leaving directory `/home/gozer/opt/src/mozilla.org/objdir-x86_64-unknown-linux-gnu-bloat/mailnews/base/search/src'
nsICopyMessageListener.idl
../../../mozilla/dist/bin/xpidl -m header -w -I/home/gozer/opt/src/mozilla.org/comm-central/mailnews/base/public -I../../../mozilla/dist/idl -o _xpidlgen/nsICopyMessageListener /home/gozer/opt/src/mozilla.org/comm-central/mailnews/base/public/nsICopyMessageListener.idl
Creating .deps
/home/gozer/opt/src/mozilla.org/objdir-x86_64-unknown-linux-gnu-bloat/mozilla/config/nsinstall -D ../../../../mozilla/dist/include/msgbase
/home/gozer/opt/src/mozilla.org/objdir-x86_64-unknown-linux-gnu-bloat/mozilla/config/nsinstall -D ../../../../mozilla/dist/idl
gmake[6]: Leaving directory `/home/gozer/opt/src/mozilla.org/objdir-x86_64-unknown-linux-gnu-bloat/mailnews/base/src'
Creating _xpidlgen/.done
/home/gozer/opt/src/mozilla.org/objdir-x86_64-unknown-linux-gnu-bloat/mozilla/config/nsinstall -D ../../../../mozilla/dist/idl
nsIMsgFilter.idl
../../../../mozilla/dist/bin/xpidl -m header -w -I/home/gozer/opt/src/mozilla.org/comm-central/mailnews/base/search/public -I../../../../mozilla/dist/idl -o _xpidlgen/nsIMsgFilter /home/gozer/opt/src/mozilla.org/comm-central/mailnews/base/search/public/nsIMsgFilter.idl
/home/gozer/opt/src/mozilla.org/comm-central/mailnews/base/search/public/nsMsgFilterCore.idl:39: can't open included file MailNewsTypes2.idl for reading

input callback returned failure
gmake[6]: *** [_xpidlgen/nsIMsgFilter.h] Error 2
gmake[6]: Leaving directory `/home/gozer/opt/src/mozilla.org/objdir-x86_64-unknown-linux-gnu-bloat/mailnews/base/search/public'
nsICopyMsgStreamListener.idl
../../../mozilla/dist/bin/xpidl -m header -w -I/home/gozer/opt/src/mozilla.org/comm-central/mailnews/base/public -I../../../mozilla/dist/idl -o _xpidlgen/nsICopyMsgStreamListener /home/gozer/opt/src/mozilla.org/comm-central/mailnews/base/public/nsICopyMsgStreamListener.idl
gmake[5]: *** [search/public_export] Error 2
gmake[5]: *** Waiting for unfinished jobs....


[...]

c++ -o nsMsgBodyHandler.o -c -I../../../../mozilla/dist/include/system_wrappers -include /home/gozer/opt/src/mozilla.org/comm-central/mozilla/config/gcc_hidden.h -DMOZILLA_INTERNAL_API -DMOZ_THUNDERBIRD=1 -DOSTYPE=\"Linux2.6.26.6-49\" -DOSARCH=Linux  -I/home/gozer/opt/src/mozilla.org/comm-central/mailnews/base/search/src -I. -I../../../../mozilla/dist/include/xpcom -I../../../../mozilla/dist/include/string -I../../../../mozilla/dist/include/msgdb -I../../../../mozilla/dist/include/msgbaseutil -I../../../../mozilla/dist/include/necko -I../../../../mozilla/dist/include/mime -I../../../../mozilla/dist/include/msgimap -I../../../../mozilla/dist/include/rdf -I../../../../mozilla/dist/include/msglocal -I../../../../mozilla/dist/include/intl -I../../../../mozilla/dist/include/unicharutil -I../../../../mozilla/dist/include/locale -I../../../../mozilla/dist/include/msgcompose -I../../../../mozilla/dist/include/pref -I../../../../mozilla/dist/include/msgnews -I../../../../mozilla/dist/include/txmgr -I../../../../mozilla/dist/include/addrbook -I../../../../mozilla/dist/include/docshell -I../../../../mozilla/dist/include   -I../../../../mozilla/dist/include/msgbase `../../../../mozilla/nsprpub/config/nspr-config --prefix=../../../../mozilla/dist --includedir=../../../../mozilla/dist/include/nspr --cflags`        -fPIC   -fno-rtti -fno-exceptions -Wall -Wpointer-arith -Woverloaded-virtual -Wsynth -Wno-ctor-dtor-privacy -Wno-non-virtual-dtor -Wcast-align -Wno-invalid-offsetof -Wno-long-long -pedantic -gstabs+ -fno-strict-aliasing -fshort-wchar -pthread -pipe  -DDEBUG -D_DEBUG -DDEBUG_gozer -DTRACING -g -fno-inline   -DMOZILLA_CLIENT -include ../../../../mozilla-config.h -Wp,-MD,.deps/nsMsgBodyHandler.pp /home/gozer/opt/src/mozilla.org/comm-central/mailnews/base/search/src/nsMsgBodyHandler.cpp
c++ -o nsMimeObjectClassAccess.o -c -I../../../mozilla/dist/include/system_wrappers -include /home/gozer/opt/src/mozilla.org/comm-central/mozilla/config/gcc_hidden.h -DENABLE_SMIME -DMOZILLA_INTERNAL_API -DMOZ_THUNDERBIRD=1 -DOSTYPE=\"Linux2.6.26.6-49\" -DOSARCH=Linux  -I/home/gozer/opt/src/mozilla.org/comm-central/mailnews/mime/src -I. -I../../../mozilla/dist/include/xpcom -I../../../mozilla/dist/include/string -I../../../mozilla/dist/include/msgbase -I../../../mozilla/dist/include/msgbaseutil -I../../../mozilla/dist/include/pref -I../../../mozilla/dist/include/js -I../../../mozilla/dist/include/necko -I../../../mozilla/dist/include/uconv -I../../../mozilla/dist/include/unicharutil -I../../../mozilla/dist/include/msgcompose -I../../../mozilla/dist/include/dom -I../../../mozilla/dist/include/intl -I../../../mozilla/dist/include/msgimap -I../../../mozilla/dist/include/exthandler -I../../../mozilla/dist/include/chardet -I../../../mozilla/dist/include/caps -I../../../mozilla/dist/include/mimetype -I../../../mozilla/dist/include/xpconnect -I../../../mozilla/dist/include/nkcache -I../../../mozilla/dist/include/pipnss -I../../../mozilla/dist/include/imglib2 -I../../../mozilla/dist/include/content -I../../../mozilla/dist/include/htmlparser -I../../../mozilla/dist/include/layout -I../../../mozilla/dist/include/msgsmime -I../../../mozilla/dist/include   -I../../../mozilla/dist/include/mime `../../../mozilla/nsprpub/config/nspr-config --prefix=../../../mozilla/dist --includedir=../../../mozilla/dist/include/nspr --cflags`        -fPIC   -fno-rtti -fno-exceptions -Wall -Wpointer-arith -Woverloaded-virtual -Wsynth -Wno-ctor-dtor-privacy -Wno-non-virtual-dtor -Wcast-align -Wno-invalid-offsetof -Wno-long-long -pedantic -gstabs+ -fno-strict-aliasing -fshort-wchar -pthread -pipe  -DDEBUG -D_DEBUG -DDEBUG_gozer -DTRACING -g -fno-inline   -DMOZILLA_CLIENT -include ../../../mozilla-config.h -Wp,-MD,.deps/nsMimeObjectClassAccess.pp /home/gozer/opt/src/mozilla.org/comm-central/mailnews/mime/src/nsMimeObjectClassAccess.cpp
gmake[6]: Entering directory `/home/gozer/opt/src/mozilla.org/objdir-x86_64-unknown-linux-gnu-bloat/mailnews/db/msgdb/build'
gmake[6]: *** No rule to make target `-lmsgbaseutil', needed by `libmsgdb.so'.  Stop.
gmake[6]: *** Waiting for unfinished jobs....


[...]

c++ -o nsMsgSend.o -c -I../../../mozilla/dist/include/system_wrappers -include /home/gozer/opt/src/mozilla.org/comm-central/mozilla/config/gcc_hidden.h -DMOZ_APP_VERSION="3.0b1pre" -DMOZILLA_INTERNAL_API -DMOZ_THUNDERBIRD=1 -DOSTYPE=\"Linux2.6.26.6-49\" -DOSARCH=Linux  -I/home/gozer/opt/src/mozilla.org/comm-central/mailnews/compose/src -I. -I../../../mozilla/dist/include/xpcom -I../../../mozilla/dist/include/string -I../../../mozilla/dist/include/dom -I../../../mozilla/dist/include/layout -I../../../mozilla/dist/include/editor -I../../../mozilla/dist/include/uriloader -I../../../mozilla/dist/include/msgbase -I../../../mozilla/dist/include/necko -I../../../mozilla/dist/include/mime -I../../../mozilla/dist/include/pref -I../../../mozilla/dist/include/intl -I../../../mozilla/dist/include/lwbrk -I../../../mozilla/dist/include/uconv -I../../../mozilla/dist/include/locale -I../../../mozilla/dist/include/unicharutil -I../../../mozilla/dist/include/content -I../../../mozilla/dist/include/msgbaseutil -I../../../mozilla/dist/include/msgdb -I../../../mozilla/dist/include/rdf -I../../../mozilla/dist/include/appshell -I../../../mozilla/dist/include/toolkitcomps -I../../../mozilla/dist/include/appcomps -I../../../mozilla/dist/include/msgimap -I../../../mozilla/dist/include/msgnews -I../../../mozilla/dist/include/docshell -I../../../mozilla/dist/include/addrbook -I../../../mozilla/dist/include/widget -I../../../mozilla/dist/include/htmlparser -I../../../mozilla/dist/include/txmgr -I../../../mozilla/dist/include/caps -I../../../mozilla/dist/include/exthandler -I../../../mozilla/dist/include/nkcache -I../../../mozilla/dist/include/mimetype -I../../../mozilla/dist/include/windowwatcher -I../../../mozilla/dist/include/gfx -I../../../mozilla/dist/include/zipwriter -I../../../mozilla/dist/include/xulapp -I../../../mozilla/dist/include   -I../../../mozilla/dist/include/msgcompose `../../../mozilla/nsprpub/config/nspr-config --prefix=../../../mozilla/dist --includedir=../../../mozilla/dist/include/nspr --cflags`        -fPIC   -fno-rtti -fno-exceptions -Wall -Wpointer-arith -Woverloaded-virtual -Wsynth -Wno-ctor-dtor-privacy -Wno-non-virtual-dtor -Wcast-align -Wno-invalid-offsetof -Wno-long-long -pedantic -gstabs+ -fno-strict-aliasing -fshort-wchar -pthread -pipe  -DDEBUG -D_DEBUG -DDEBUG_gozer -DTRACING -g -fno-inline   -DMOZILLA_CLIENT -include ../../../mozilla-config.h -Wp,-MD,.deps/nsMsgSend.pp /home/gozer/opt/src/mozilla.org/comm-central/mailnews/compose/src/nsMsgSend.cpp
chmod +x libmsgmdn.so
/home/gozer/opt/src/mozilla.org/objdir-x86_64-unknown-linux-gnu-bloat/mozilla/config/nsinstall -R -m 755 libmsgmdn.so ../../../../mozilla/dist/bin/components
: ../../../../mozilla/dist/bin/components/libmsgmdn.so
gmake[7]: Leaving directory `/home/gozer/opt/src/mozilla.org/objdir-x86_64-unknown-linux-gnu-bloat/mailnews/extensions/mdn/build'
c++: ../src/libmsgimap_s.a: No such file or directory
gmake[6]: *** [TestImapHdrXferInfo] Error 1
gmake[6]: Leaving directory `/home/gozer/opt/src/mozilla.org/objdir-x86_64-unknown-linux-gnu-bloat/mailnews/imap/test'
gmake[5]: *** [test_libs] Error 2
gmake[5]: *** Waiting for unfinished jobs....

[...]
Attachment #346650 - Flags: review?(gozer) → review-
Attached patch Revised patch (obsolete) — Splinter Review
* mailnews/base can't be parallelised, so make it explicitly in build.mk
* imap/test can't be parallelised
Attachment #346650 - Attachment is obsolete: true
Attachment #346897 - Flags: superreview?(bienvenu)
Attachment #346897 - Flags: review?(gozer)
Comment on attachment 346897 [details] [diff] [review]
Revised patch

Still something broken, with a full clobber and make -j8, I get:

gmake[4]: Leaving directory `[objdir]/mail/branding/nightly'
gmake[4]: Entering directory `[objdir]/mailnews/base'
gmake[4]: *** No rule to make target `[comm-central]/mailnews/base/search/Makefile.in', needed by `search/Makefile'.  Stop.
gmake[4]: Leaving directory `[objdir]/mailnews/base'
gmake[3]: *** [export_tier_app] Error 2
gmake[3]: Leaving directory `[objdir]'
gmake[2]: *** [tier_app] Error 2
gmake[2]: Leaving directory `[objdir]'
make[1]: *** [default] Error 2
make[1]: Leaving directory `[objdir]'
make: *** [build] Error 2
Ah, my bad, I didn't apply that patch to the tip, disregard my comment #5, sorry.
Actually, my double-bad. I've tried this patch against the tip, and I get the same failure. Seems something is depending on the now removed mailnews/base/search/Makefile.in
Attached patch Restored patch (obsolete) — Splinter Review
The diff to mailnews/base/Makefile.in is the same as in the original patch.
Attachment #346897 - Attachment is obsolete: true
Attachment #346916 - Flags: superreview?(bienvenu)
Attachment #346916 - Flags: review?(gozer)
Attachment #346897 - Flags: superreview?(bienvenu)
Attachment #346897 - Flags: review?(gozer)
Comment on attachment 346916 [details] [diff] [review]
Restored patch

Still failing on clobber & make -j8

../../../../mozilla/dist/bin/xpidl -m header -w -I[comm-central]/mailnews/base/search/public -I../../../../mozilla/dist/idl -o _xpidlgen/nsIMsgFilter [comm-central]/mailnews/base/search/public/nsIMsgFilter.idl
gmake[5]: Leaving directory `[objdir]/mailnews/base/src'
nsIMsgFilterHitNotify.idl
../../../../mozilla/dist/bin/xpidl -m header -w -I[comm-central]/mailnews/base/search/public -I../../../../mozilla/dist/idl -o _xpidlgen/nsIMsgFilterHitNotify [comm-central]/mailnews/base/search/public/nsIMsgFilterHitNotify.idl
nsIFolderListener.idl
../../../mozilla/dist/bin/xpidl -m header -w -I[comm-central]/mailnews/base/public -I../../../mozilla/dist/idl -o _xpidlgen/nsIFolderListener [comm-central]/mailnews/base/public/nsIFolderListener.idl
[comm-central]/mailnews/base/search/public/nsMsgFilterCore.idl:39: can't open included file MailNewsTypes2.idl for reading

input callback returned failure
gmake[5]: *** [_xpidlgen/nsIMsgFilter.h] Error 2
gmake[5]: *** Waiting for unfinished jobs....
/usr/bin/gmake -C search/src export
gmake[5]: Entering directory `[objdir]/mailnews/base/search/src'
Creating .deps
nsIFolderLookupService.idl
../../../mozilla/dist/bin/xpidl -m header -w -I[comm-central]/mailnews/base/public -I../../../mozilla/dist/idl -o _xpidlgen/nsIFolderLookupService [comm-central]/mailnews/base/public/nsIFolderLookupService.idl
gmake[5]: Leaving directory `[objdir]/mailnews/base/search/src'
/usr/bin/perl -I[comm-central]/mozilla/config [comm-central]/mozilla/config/build-list.pl ../../../mozilla/config/final-link-libs msgbaseutil
[comm-central]/mailnews/base/search/public/nsMsgFilterCore.idl:39: can't open included file MailNewsTypes2.idl for reading

input callback returned failure
gmake[5]: *** [_xpidlgen/nsIMsgFilterHitNotify.h] Error 2
gmake[5]: Leaving directory `[objdir]/mailnews/base/search/public'
gmake[4]: *** [search/public_export] Error 2
gmake[4]: *** Waiting for unfinished jobs....
Attachment #346916 - Flags: review?(gozer) → review-
OK, that should be as easy as moving search/* back into DIRS

In theory I could move base/search/* into mailnews/Makefile's PARALLEL_DIRS and base/build into mailnews/Makefile's DIRS for optimum parallelisation ;-)
Attachment #346916 - Attachment is obsolete: true
Attachment #346936 - Flags: superreview?(bienvenu)
Attachment #346936 - Flags: review?(gozer)
Attachment #346916 - Flags: superreview?(bienvenu)
(In reply to comment #10)
> Created an attachment (id=346936) [details]
>[...]
> 
> In theory I could move base/search/* into mailnews/Makefile's PARALLEL_DIRS and
> base/build into mailnews/Makefile's DIRS for optimum parallelisation ;-)

Then why not just go ahead and do that ? Parallelism is good!
Attached patch Your wish is my command (obsolete) — Splinter Review
I won't cancel the previous reviews just in case ;-)

I moved base/search/src into mailnews/Makefile's PARALLEL_DIRS so that it gets built after base/util, and base/build into mailnews/Makefile's DIRS so that it gets built after base/search/src, but I kept search/public in base/Makefile.
Attachment #346943 - Flags: superreview?(bienvenu)
Attachment #346943 - Flags: review?(gozer)
Comment on attachment 346943 [details] [diff] [review]
Your wish is my command

sr=me, as long as it works for gozer :-)
Attachment #346943 - Flags: superreview?(bienvenu) → superreview+
Comment on attachment 346943 [details] [diff] [review]
Your wish is my command

/opt/src/mozilla.org/comm-central/mailnews/base/search/public/nsIMsgSearchTerm.idl
/usr/bin/perl -I/home/gozer/opt/src/mozilla.org/comm-central/mozilla/config /home/gozer/opt/src/mozilla.org/comm-central/mozilla/config/build-list.pl ../../../mozilla/config/final-link-libs msgbaseutil
/home/gozer/opt/src/mozilla.org/comm-central/mailnews/base/search/public/nsMsgFilterCore.idl:39: can't open included file MailNewsTypes2.idl for reading

input callback returned failure
/home/gozer/opt/src/mozilla.org/comm-central/mailnews/base/search/public/nsMsgFilterCore.idl:39: can't open included file MailNewsTypes2.idl for reading

input callback returned failure
gmake[5]: *** [_xpidlgen/nsIMsgFilter.h] Error 2
gmake[5]: *** Waiting for unfinished jobs....
Attachment #346943 - Flags: review?(gozer) → review-
It looks like search/public doesn't parallelise either. Fingers crossed ;-)
Attachment #346943 - Attachment is obsolete: true
Attachment #347091 - Flags: review?(gozer)
Attachment #347091 - Flags: review?(gozer) → review+
Comment on attachment 347091 [details] [diff] [review]
Slightly less parallel

clobber + make -j8 worked for me.
Comment on attachment 347091 [details] [diff] [review]
Slightly less parallel

I'm going to take the liberty of moving the sr request to this patch
Attachment #347091 - Flags: superreview+
If anyone runs into chrome problems due to incomplete .jar packaging in parallel builds, see bug 464080.
Comment on attachment 346936 [details] [diff] [review]
Fourth time lucky?

clearing sr request - I think this is obsolete if not, let me know...
Attachment #346936 - Flags: superreview?(bienvenu) → superreview?
Depends on: 488838
Depends on: 509367
Comment on attachment 346936 [details] [diff] [review]
Fourth time lucky?

I believe these requests are obsolete, please re-request if necessary.
Attachment #346936 - Flags: superreview?
Attachment #346936 - Flags: review?(gozer)
Neil this bug hasn't been touched in a while; more to do or should we resolve this and do any additional cleanup in separate bugs?
Yes, I guess we can mark this fixed, and file followup bugs as necessary.
Status: ASSIGNED → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.