914560 - Package XPI extensions during tools phase so that all required files are in place when running the packaging step

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[Frank Wein [:mcsmurf]]

Since the changes from Bug 913967/Bug 912856 and related bugs SeaMonkey fails to compile because the Venkman build fails. Example log from Linux SeaMonkey build box:
make -C extensions libs
make[4]: Entering directory `/builds/slave/c-cen-t-lnx/build/objdir/mozilla/extensions'
make[5]: Entering directory `/builds/slave/c-cen-t-lnx/build/objdir/mozilla/extensions'
make[6]: Entering directory `/builds/slave/c-cen-t-lnx/build/objdir/mozilla/extensions/inspector'
mkdir -p "../../dist/xpi-stage/inspector/chrome/"
make[6]: Entering directory `/builds/slave/c-cen-t-lnx/build/objdir/mozilla/extensions/venkman'
Packaging venkman-0.9.89.xpi...
cd ../../dist/xpi-stage/venkman && /usr/bin/zip -qr ../venkman-0.9.89.xpi *
rm -f "../../dist/xpi-stage/inspector/defaults/preferences/inspector.js"

zip error: Nothing to do! (try: zip -qr ../venkman-0.9.89.xpi . -i *)
make[6]: *** [libs] Error 12
make[6]: Leaving directory `/builds/slave/c-cen-t-lnx/build/objdir/mozilla/extensions/venkman'
make[5]: *** [venkman_libs] Error 2

Comment 1

[Sladky Vladimir]

I (provisionally) compiled with "ac_add_options --enable-extensions=default,-venkman"
- i have now (vaious) errors and canot compile from 4. september ...

Comment 2

[Hartmut Figge]

We have to wait at least until this current bustage is fixed:

/home/hafi/moz-work/src/obj-x86_64-unknown-linux-gnu/mozilla/dom/bindings/UnionTypes.h:1506:3: error: candidate is: bool mozilla::dom::TelephonyCallOrTelephonyCallGroupReturnValue::TrySetToTelephonyCallGroup(int*, int)
/home/hafi/moz-work/src/obj-x86_64-unknown-linux-gnu/mozilla/dom/bindings/UnionTypes.cpp:1253:1: error: prototype for ‘bool mozilla::dom::TelephonyCallOrTelephonyCallGroupReturnValue::ToJSVal(JSContext*, JS::Handle<JSObject*>, JS::MutableHandle<JS::Value>) const’ does not match any in class ‘mozilla::dom::TelephonyCallOrTelephonyCallGroupReturnValue’
/home/hafi/moz-work/src/obj-x86_64-unknown-linux-gnu/mozilla/dom/bindings/UnionTypes.h:1515:3: error: candidate is: bool mozilla::dom::TelephonyCallOrTelephonyCallGroupReturnValue::ToJSVal(int*, int) const
make[6]: *** [UnionTypes.o] Error 1

Comment 3

[Frank Wein [:mcsmurf]]

We dont have to wait, this error is independent of that other build problem.

Comment 4

[:Gijs (he/him)]

Has anyone done any kind of investigation about what's needed to fix this? (and, is there a similar CZ bug? https://bugzilla.mozilla.org/show_bug.cgi?id=913967#c6 would make it seem so, but I haven't seen any new bugs appear)

Comment 5

[Frank Wein [:mcsmurf]]

I think people are bit (too) lazy filing bugs... :)
I'm not sure sure what's up here. I think this is from Bug 912292.
I'm a bit confused why there's both a XPI_NAME inside venkman/Makefile.in and venkman/resources/Makefile.in

Comment 6

[Frank Wein [:mcsmurf]]

Mike: Can you suggest a fix/workaround for the problem here? I'm not very experienced with the build system and need a bit of help. For me it looks like this bug happens due to the reordering in Bug 912292. It tries to export/create the .xpi file because of this rule http://hg.mozilla.org/comm-central/annotate/52ce01123f03/config/rules.mk#l1480 when running "make libs" inside extensions/venkman/. This fails as it first would need to traverse through all the DIRS to copy the files into the xpi staging area (so for example in extensions/venkman/resources). Does this need to be fixed in the venkman Makefiles or is a rules.mk change required for that?
Code for this can be found at http://mxr.mozilla.org/comm-central/source/mozilla/extensions/venkman/Makefile.in

Comment 7

[[e-mail]]

Well in attachment 799480 [details] [diff] [review] he worked around the problem by moving the affected rules from libs:: to tools:: if that helps.

Comment 8

[[e-mail]]

(the affected rule in this case seems to be on line 1490 of mozilla/config/rules.mk)

Comment 9

[Mike Hommey [:glandium]]

What Neil said.

Comment 10

[Philip Chee]

Does https://hg.mozilla.org/comm-central/rev/4cd118a58040 help or hurt?

Comment 11

[Frank Wein [:mcsmurf]]

I've pushed a patch to TB try to test what was proposed in Comment 7: https://tbpl.mozilla.org/?tree=Thunderbird-Try&rev=cebc32134361

Comment 12

[Frank Wein [:mcsmurf]]

(and yes, I'm "abusing" TB try to build SeaMonkey in this case)

Comment 13

[Hartmut Figge]

With https://bugzilla.mozilla.org/show_bug.cgi?id=915100#c16 there is no need for abusing. ;)

Comment 14

[Frank Wein [:mcsmurf]]

(In reply to Hartmut Figge from comment #13)
> With https://bugzilla.mozilla.org/show_bug.cgi?id=915100#c16 there is no
> need for abusing. ;)

I don't have Linux handy, Windows is currently broken due to another issue.

Comment 15

[Frank Wein [:mcsmurf]]

Attachment: mozilla-central patch

This fixes the execution of the XPI_PKGNAME rule by moving it to tools::. The patch seems to fix the SeaMonkey build (https://tbpl.mozilla.org/?tree=Thunderbird-Try&rev=cebc32134361). I've also submitted the patch to Try to create a Firefox build (https://tbpl.mozilla.org/?tree=Try&rev=506a83664116), but I suspect this one will take quite a while to finish.

Comment 16

[Ian Neal]

Comment on attachment 803258 [details] [diff] [review]
mozilla-central patch

Are you sure this is the correct patch? Almost looks like a diff of a diff!

Comment 17

[Frank Wein [:mcsmurf]]

Attachment: Better mozilla-central patch

Yeah, I attached the wrong patch. That was the one I used for try.

Comment 18

[[e-mail]]

I looked into this and that isn't quite enough; it's ok for developers but it doesn't stage the XPIs correctly because that's done in a separate rule.

Comment 19

[[e-mail]]

Attachment: Includes additional fix

Comment 20

[Frank Wein [:mcsmurf]]

You mean the move/creation of the xpi file to dist/xpi-stage? The try-build seems to contain all extensions in distribution/extensions/. But then I only had the try server to test this as I have no Linux build env handy atm.

Comment 21

[[e-mail]]

I'm developing on Windows where the build doesn't fail but it does produce incorrect results. Checking a file listing of objdir/mozilla/dist/ against another build showed the two sets of differences and your patch only fixed one set, so I just assumed that the other fix was necessary.

Comment 22

[Hartmut Figge]

With the fix of Neil i am getting for SM on Linux x86_64:

/home/hafi/moz-work/src/obj-x86_64-unknown-linux-gnu/mozilla/_virtualenv/bin/python /home/hafi/moz-work/src/mozilla/toolkit/mozapps/installer/packager.py -DMOZ_GLUE_IN_PROGRAM -DOSTYPE=\"Linux3.10\" -DOSARCH=Linux -DNO_NSPR_10_SUPPORT -DAB_CD=en-US -DMOZ_APP_NAME=seamonkey -DPREF_DIR=defaults/pref -DJAREXT= -DMOZ_ENABLE_GNOME_COMPONENT=1 -DMOZ_GTK2=1 -DMOZ_PACKAGE_DEBUGQA=1 -DMOZ_URL_CLASSIFIER=1 -DMOZ_UPDATER=1 -DMOZ_MOVEMAIL=1 -DNECKO_WIFI=1 -DMOZ_CHILD_PROCESS_NAME=plugin-container -DDLL_PREFIX=lib -DDLL_SUFFIX=.so -DBIN_SUFFIX= -DBINPATH=bin \
        --format omni \
        --removals /home/hafi/moz-work/src/suite/installer/removed-files.in \
         \
         \
         \
        --optimizejars \
         \
        package-manifest ../../mozilla/dist ../../mozilla/dist/seamonkey \
        --non-resource defaults/messenger/mailViews.dat defaults/profile/localstore.rdf defaults/profile/panels.rdf 
Error: /home/hafi/moz-work/src/obj-x86_64-unknown-linux-gnu/suite/installer/package-manifest:582: Missing file(s): bin/distribution/extensions/[e-mail]

Comment 23

[[e-mail]]

(In reply to Hartmut Figge from comment #22)
> Error:
> /home/hafi/moz-work/src/obj-x86_64-unknown-linux-gnu/suite/installer/package-
> manifest:582: Missing file(s):
> bin/distribution/extensions/[e-mail]

Is this as part of the build? (Maybe I wasn't seeing that locally because of my mozconfig.)

Comment 24

[Hartmut Figge]

It is near the end of my build log. The complete log: http://www.triffids.de/pub/sm/venk/
My .mozconfig, also contained in the build log:
  --enable-application=suite
  --enable-optimize
  --disable-tests
  --enable-crypto
  --disable-calendar
  --enable-extensions=default
  --disable-crashreporter
  --disable-debug-symbols

Comment 25

[[e-mail]]

Thanks, I see what the problem is now, suite/app/Makefile.in packages extensions in the libs:: phase (line 196), when of course the extensions haven't been fully built yet.

So maybe changing this line to tools:: would work.

Or maybe it would be better to build chrome in the compile phase, but that would then need changes to the various extensions as they expect to build chrome in the libs phase.

Comment 26

[Hartmut Figge]

(In reply to [e-mail] from comment #25)
> Thanks, I see what the problem is now, suite/app/Makefile.in packages
> extensions in the libs:: phase (line 196), when of course the extensions
> haven't been fully built yet.
> 
> So maybe changing this line to tools:: would work.

It does. :) Hartmut, now using a new SM.

Comment 27

[Mike Hommey [:glandium]]

(In reply to [e-mail] from comment #25)
> Or maybe it would be better to build chrome in the compile phase

No, compile is for compilation. Leave new targets alone.

Comment 28

[Frank Wein [:mcsmurf]]

Pushed the patch to get things going again: https://hg.mozilla.org/mozilla-central/rev/92bea49b46b4

Comment 29

[Hartmut Figge]

Attachment: Move SeaMonkey extension packaging step to tools phase

Well, to get things going the correction, Neil mentioned, is required. Ideally it should have been contained in a new fix.

Comment 30

[Frank Wein [:mcsmurf]]

Comment on attachment 803720 [details] [diff] [review]
Move SeaMonkey extension packaging step to tools phase

Pushed this patch as bustage fix: https://hg.mozilla.org/comm-central/rev/f3e59e50874b

Comment 32

[Frank Wein [:mcsmurf]]

Looks like it's compiling again. But at least on Mac there's some error left, the Windows and Linux builds seem to be fine (at least the official builds contain all the extensions). Maybe a clobber is needed, not sure.

Comment 33

[[e-mail]]

(In reply to Frank Wein from comment #32)
> Looks like it's compiling again. But at least on Mac there's some error
> left, the Windows and Linux builds seem to be fine (at least the official
> builds contain all the extensions). Maybe a clobber is needed, not sure.

Strange, the log suggests that some of the files are missing, but I can see them being installed to dist in both halves of the build, so I don't know what's going on there...

Comment 34

[Philip Chee]

http://tbpl-dev.callek.net/php/getParsedLog.php?id=20806047&tree=SeaMonkey#error0

Error: /builds/slave/c-cen-t-osx64/build/objdir/i386/suite/installer/package-manifest:47: Missing file(s): /SeaMonkey.app/Contents/MacOS/plugin-container.app/
Error: /builds/slave/c-cen-t-osx64/build/objdir/i386/suite/installer/package-manifest:361: Missing file(s): /SeaMonkey.app/Contents/MacOS/extensions/{972ce4c6-7e08-4474-a285-3208198ce6fd}.xpi
Error: /builds/slave/c-cen-t-osx64/build/objdir/i386/suite/installer/package-manifest:364: Missing file(s): /SeaMonkey.app/Contents/MacOS/extensions/[e-mail]
Error: /builds/slave/c-cen-t-osx64/build/objdir/i386/suite/installer/package-manifest:529: Missing file(s): /SeaMonkey.app/Contents/MacOS/distribution/extensions/{59c81df5-4b7a-477b-912d-4e0fdf64e5f2}.xpi
Error: /builds/slave/c-cen-t-osx64/build/objdir/i386/suite/installer/package-manifest:532: Missing file(s): /SeaMonkey.app/Contents/MacOS/distribution/extensions/{f13b157f-b174-47e7-a34d-4815ddfdfeb8}.xpi
Error: /builds/slave/c-cen-t-osx64/build/objdir/i386/suite/installer/package-manifest:535: Missing file(s): /SeaMonkey.app/Contents/MacOS/distribution/extensions/[e-mail]
Error: /builds/slave/c-cen-t-osx64/build/objdir/i386/suite/installer/package-manifest:538: Missing file(s): /SeaMonkey.app/Contents/MacOS/distribution/extensions/[e-mail]

Comment 35

[Frank Wein [:mcsmurf]]

(In reply to [e-mail] from comment #33)
> (In reply to Frank Wein from comment #32)
> > Looks like it's compiling again. But at least on Mac there's some error
> > left, the Windows and Linux builds seem to be fine (at least the official
> > builds contain all the extensions). Maybe a clobber is needed, not sure.
> 
> Strange, the log suggests that some of the files are missing, but I can see
> them being installed to dist in both halves of the build, so I don't know
> what's going on there...

I think the problem is the rsync build step running too early (in libs phase) on OS X:
http://hg.mozilla.org/comm-central/annotate/303c7574e706/suite/app/Makefile.in#l224 see the attached log where I included the (what I think) relevant parts of a OS X build log.

Comment 36

[Frank Wein [:mcsmurf]]

Attachment: SeaMonkey Mac OS X build log, rsync running too early

Comment 37

[Frank Wein [:mcsmurf]]

Attachment: Move OS X packaging to tools phase

This patch basically copies this change http://hg.mozilla.org/mozilla-central/diff/fc9fb405796e/browser/app/Makefile.in to our Makefile.

Comment 38

[Frank Wein [:mcsmurf]]

Comment on attachment 804265 [details] [diff] [review]
Move OS X packaging to tools phase

Pushed as bustage fix: https://hg.mozilla.org/comm-central/rev/a3d827851227

Comment 39

[Tony Mechelynck [:tonymec]]

SeaMonkey hourly builds have appeared for W32, L32 and L64. Not on the Mac (yet) but there the current problem seems to have something to do with packaging the omnijar.

For instance on L64:
Mozilla/5.0 (X11; Linux x86_64; rv:26.0) Gecko/20100101 Firefox/26.0 SeaMonkey/2.23a1 ID:20130912192824 c-c:303c7574e706 m-c:b9029b1de410
No nightlies yet.

Comment 40

[Frank Wein [:mcsmurf]]

Closing as fixed. All three platforms have working nightly builds again. Please check that all builds are working and extensions are included in all builds (especially on Mac).