Closed Bug 914560 Opened 6 years ago Closed 6 years ago

Package XPI extensions during tools phase so that all required files are in place when running the packaging step

Categories

(Firefox Build System :: General, defect)

defect
Not set

Tracking

(Not tracked)

RESOLVED FIXED
mozilla26

People

(Reporter: mcsmurf, Assigned: neil)

References

Details

Attachments

(4 files, 2 obsolete files)

Since the changes from Bug 913967/Bug 912856 and related bugs SeaMonkey fails to compile because the Venkman build fails. Example log from Linux SeaMonkey build box:
make -C extensions libs
make[4]: Entering directory `/builds/slave/c-cen-t-lnx/build/objdir/mozilla/extensions'
make[5]: Entering directory `/builds/slave/c-cen-t-lnx/build/objdir/mozilla/extensions'
make[6]: Entering directory `/builds/slave/c-cen-t-lnx/build/objdir/mozilla/extensions/inspector'
mkdir -p "../../dist/xpi-stage/inspector/chrome/"
make[6]: Entering directory `/builds/slave/c-cen-t-lnx/build/objdir/mozilla/extensions/venkman'
Packaging venkman-0.9.89.xpi...
cd ../../dist/xpi-stage/venkman && /usr/bin/zip -qr ../venkman-0.9.89.xpi *
rm -f "../../dist/xpi-stage/inspector/defaults/preferences/inspector.js"

zip error: Nothing to do! (try: zip -qr ../venkman-0.9.89.xpi . -i *)
make[6]: *** [libs] Error 12
make[6]: Leaving directory `/builds/slave/c-cen-t-lnx/build/objdir/mozilla/extensions/venkman'
make[5]: *** [venkman_libs] Error 2
I (provisionally) compiled with "ac_add_options --enable-extensions=default,-venkman"
- i have now (vaious) errors and canot compile from 4. september ...
We have to wait at least until this current bustage is fixed:

/home/hafi/moz-work/src/obj-x86_64-unknown-linux-gnu/mozilla/dom/bindings/UnionTypes.h:1506:3: error: candidate is: bool mozilla::dom::TelephonyCallOrTelephonyCallGroupReturnValue::TrySetToTelephonyCallGroup(int*, int)
/home/hafi/moz-work/src/obj-x86_64-unknown-linux-gnu/mozilla/dom/bindings/UnionTypes.cpp:1253:1: error: prototype for ‘bool mozilla::dom::TelephonyCallOrTelephonyCallGroupReturnValue::ToJSVal(JSContext*, JS::Handle<JSObject*>, JS::MutableHandle<JS::Value>) const’ does not match any in class ‘mozilla::dom::TelephonyCallOrTelephonyCallGroupReturnValue’
/home/hafi/moz-work/src/obj-x86_64-unknown-linux-gnu/mozilla/dom/bindings/UnionTypes.h:1515:3: error: candidate is: bool mozilla::dom::TelephonyCallOrTelephonyCallGroupReturnValue::ToJSVal(int*, int) const
make[6]: *** [UnionTypes.o] Error 1
We dont have to wait, this error is independent of that other build problem.
Has anyone done any kind of investigation about what's needed to fix this? (and, is there a similar CZ bug? https://bugzilla.mozilla.org/show_bug.cgi?id=913967#c6 would make it seem so, but I haven't seen any new bugs appear)
I think people are bit (too) lazy filing bugs... :)
I'm not sure sure what's up here. I think this is from Bug 912292.
I'm a bit confused why there's both a XPI_NAME inside venkman/Makefile.in and venkman/resources/Makefile.in
Mike: Can you suggest a fix/workaround for the problem here? I'm not very experienced with the build system and need a bit of help. For me it looks like this bug happens due to the reordering in Bug 912292. It tries to export/create the .xpi file because of this rule http://hg.mozilla.org/comm-central/annotate/52ce01123f03/config/rules.mk#l1480 when running "make libs" inside extensions/venkman/. This fails as it first would need to traverse through all the DIRS to copy the files into the xpi staging area (so for example in extensions/venkman/resources). Does this need to be fixed in the venkman Makefiles or is a rules.mk change required for that?
Code for this can be found at http://mxr.mozilla.org/comm-central/source/mozilla/extensions/venkman/Makefile.in
Flags: needinfo?(mh+mozilla)
Well in attachment 799480 [details] [diff] [review] he worked around the problem by moving the affected rules from libs:: to tools:: if that helps.
(the affected rule in this case seems to be on line 1490 of mozilla/config/rules.mk)
What Neil said.
Flags: needinfo?(mh+mozilla)
I've pushed a patch to TB try to test what was proposed in Comment 7: https://tbpl.mozilla.org/?tree=Thunderbird-Try&rev=cebc32134361
(and yes, I'm "abusing" TB try to build SeaMonkey in this case)
With https://bugzilla.mozilla.org/show_bug.cgi?id=915100#c16 there is no need for abusing. ;)
(In reply to Hartmut Figge from comment #13)
> With https://bugzilla.mozilla.org/show_bug.cgi?id=915100#c16 there is no
> need for abusing. ;)

I don't have Linux handy, Windows is currently broken due to another issue.
Attached patch mozilla-central patch (obsolete) — Splinter Review
This fixes the execution of the XPI_PKGNAME rule by moving it to tools::. The patch seems to fix the SeaMonkey build (https://tbpl.mozilla.org/?tree=Thunderbird-Try&rev=cebc32134361). I've also submitted the patch to Try to create a Firefox build (https://tbpl.mozilla.org/?tree=Try&rev=506a83664116), but I suspect this one will take quite a while to finish.
Attachment #803258 - Flags: review?(mh+mozilla)
Comment on attachment 803258 [details] [diff] [review]
mozilla-central patch

Are you sure this is the correct patch? Almost looks like a diff of a diff!
Attached patch Better mozilla-central patch (obsolete) — Splinter Review
Yeah, I attached the wrong patch. That was the one I used for try.
Attachment #803258 - Attachment is obsolete: true
Attachment #803258 - Flags: review?(mh+mozilla)
Attachment #803274 - Flags: review?(mh+mozilla)
Attachment #803274 - Flags: review?(mh+mozilla) → review+
I looked into this and that isn't quite enough; it's ok for developers but it doesn't stage the XPIs correctly because that's done in a separate rule.
Assignee: nobody → neil
Attachment #803274 - Attachment is obsolete: true
Status: NEW → ASSIGNED
Attachment #803552 - Flags: review?(mh+mozilla)
You mean the move/creation of the xpi file to dist/xpi-stage? The try-build seems to contain all extensions in distribution/extensions/. But then I only had the try server to test this as I have no Linux build env handy atm.
I'm developing on Windows where the build doesn't fail but it does produce incorrect results. Checking a file listing of objdir/mozilla/dist/ against another build showed the two sets of differences and your patch only fixed one set, so I just assumed that the other fix was necessary.
Attachment #803552 - Flags: review?(mh+mozilla) → review+
With the fix of Neil i am getting for SM on Linux x86_64:

/home/hafi/moz-work/src/obj-x86_64-unknown-linux-gnu/mozilla/_virtualenv/bin/python /home/hafi/moz-work/src/mozilla/toolkit/mozapps/installer/packager.py -DMOZ_GLUE_IN_PROGRAM -DOSTYPE=\"Linux3.10\" -DOSARCH=Linux -DNO_NSPR_10_SUPPORT -DAB_CD=en-US -DMOZ_APP_NAME=seamonkey -DPREF_DIR=defaults/pref -DJAREXT= -DMOZ_ENABLE_GNOME_COMPONENT=1 -DMOZ_GTK2=1 -DMOZ_PACKAGE_DEBUGQA=1 -DMOZ_URL_CLASSIFIER=1 -DMOZ_UPDATER=1 -DMOZ_MOVEMAIL=1 -DNECKO_WIFI=1 -DMOZ_CHILD_PROCESS_NAME=plugin-container -DDLL_PREFIX=lib -DDLL_SUFFIX=.so -DBIN_SUFFIX= -DBINPATH=bin \
        --format omni \
        --removals /home/hafi/moz-work/src/suite/installer/removed-files.in \
         \
         \
         \
        --optimizejars \
         \
        package-manifest ../../mozilla/dist ../../mozilla/dist/seamonkey \
        --non-resource defaults/messenger/mailViews.dat defaults/profile/localstore.rdf defaults/profile/panels.rdf 
Error: /home/hafi/moz-work/src/obj-x86_64-unknown-linux-gnu/suite/installer/package-manifest:582: Missing file(s): bin/distribution/extensions/inspector@mozilla.org.xpi
(In reply to Hartmut Figge from comment #22)
> Error:
> /home/hafi/moz-work/src/obj-x86_64-unknown-linux-gnu/suite/installer/package-
> manifest:582: Missing file(s):
> bin/distribution/extensions/inspector@mozilla.org.xpi

Is this as part of the build? (Maybe I wasn't seeing that locally because of my mozconfig.)
It is near the end of my build log. The complete log: http://www.triffids.de/pub/sm/venk/
My .mozconfig, also contained in the build log:
  --enable-application=suite
  --enable-optimize
  --disable-tests
  --enable-crypto
  --disable-calendar
  --enable-extensions=default
  --disable-crashreporter
  --disable-debug-symbols
Thanks, I see what the problem is now, suite/app/Makefile.in packages extensions in the libs:: phase (line 196), when of course the extensions haven't been fully built yet.

So maybe changing this line to tools:: would work.

Or maybe it would be better to build chrome in the compile phase, but that would then need changes to the various extensions as they expect to build chrome in the libs phase.
(In reply to neil@parkwaycc.co.uk from comment #25)
> Thanks, I see what the problem is now, suite/app/Makefile.in packages
> extensions in the libs:: phase (line 196), when of course the extensions
> haven't been fully built yet.
> 
> So maybe changing this line to tools:: would work.

It does. :) Hartmut, now using a new SM.
(In reply to neil@parkwaycc.co.uk from comment #25)
> Or maybe it would be better to build chrome in the compile phase

No, compile is for compilation. Leave new targets alone.
Pushed the patch to get things going again: https://hg.mozilla.org/mozilla-central/rev/92bea49b46b4
Well, to get things going the correction, Neil mentioned, is required. Ideally it should have been contained in a new fix.
Comment on attachment 803720 [details] [diff] [review]
Move SeaMonkey extension packaging step to tools phase

Pushed this patch as bustage fix: https://hg.mozilla.org/comm-central/rev/f3e59e50874b
Duplicate of this bug: 914491
Looks like it's compiling again. But at least on Mac there's some error left, the Windows and Linux builds seem to be fine (at least the official builds contain all the extensions). Maybe a clobber is needed, not sure.
Component: Venkman JS Debugger → Build Config
OS: Windows 8 → All
Product: Other Applications → Core
Hardware: x86_64 → All
Target Milestone: --- → mozilla26
Summary: Venkman fails to compile on trunk ("zip error: Nothing to do!") → Package XPI extensions during tools phase so that all required files are in place when running the packaging step
Attachment #803720 - Attachment description: add_to_fix → Move SeaMonkey extension packaging step to tools phase
Attachment #803720 - Flags: checkin+
Attachment #803552 - Flags: checkin+
(In reply to Frank Wein from comment #32)
> Looks like it's compiling again. But at least on Mac there's some error
> left, the Windows and Linux builds seem to be fine (at least the official
> builds contain all the extensions). Maybe a clobber is needed, not sure.

Strange, the log suggests that some of the files are missing, but I can see them being installed to dist in both halves of the build, so I don't know what's going on there...
http://tbpl-dev.callek.net/php/getParsedLog.php?id=20806047&tree=SeaMonkey#error0

Error: /builds/slave/c-cen-t-osx64/build/objdir/i386/suite/installer/package-manifest:47: Missing file(s): /SeaMonkey.app/Contents/MacOS/plugin-container.app/
Error: /builds/slave/c-cen-t-osx64/build/objdir/i386/suite/installer/package-manifest:361: Missing file(s): /SeaMonkey.app/Contents/MacOS/extensions/{972ce4c6-7e08-4474-a285-3208198ce6fd}.xpi
Error: /builds/slave/c-cen-t-osx64/build/objdir/i386/suite/installer/package-manifest:364: Missing file(s): /SeaMonkey.app/Contents/MacOS/extensions/modern@themes.mozilla.org.xpi
Error: /builds/slave/c-cen-t-osx64/build/objdir/i386/suite/installer/package-manifest:529: Missing file(s): /SeaMonkey.app/Contents/MacOS/distribution/extensions/{59c81df5-4b7a-477b-912d-4e0fdf64e5f2}.xpi
Error: /builds/slave/c-cen-t-osx64/build/objdir/i386/suite/installer/package-manifest:532: Missing file(s): /SeaMonkey.app/Contents/MacOS/distribution/extensions/{f13b157f-b174-47e7-a34d-4815ddfdfeb8}.xpi
Error: /builds/slave/c-cen-t-osx64/build/objdir/i386/suite/installer/package-manifest:535: Missing file(s): /SeaMonkey.app/Contents/MacOS/distribution/extensions/inspector@mozilla.org.xpi
Error: /builds/slave/c-cen-t-osx64/build/objdir/i386/suite/installer/package-manifest:538: Missing file(s): /SeaMonkey.app/Contents/MacOS/distribution/extensions/debugQA@mozilla.org.xpi
(In reply to neil@parkwaycc.co.uk from comment #33)
> (In reply to Frank Wein from comment #32)
> > Looks like it's compiling again. But at least on Mac there's some error
> > left, the Windows and Linux builds seem to be fine (at least the official
> > builds contain all the extensions). Maybe a clobber is needed, not sure.
> 
> Strange, the log suggests that some of the files are missing, but I can see
> them being installed to dist in both halves of the build, so I don't know
> what's going on there...

I think the problem is the rsync build step running too early (in libs phase) on OS X:
http://hg.mozilla.org/comm-central/annotate/303c7574e706/suite/app/Makefile.in#l224 see the attached log where I included the (what I think) relevant parts of a OS X build log.
Comment on attachment 804265 [details] [diff] [review]
Move OS X packaging to tools phase

Pushed as bustage fix: https://hg.mozilla.org/comm-central/rev/a3d827851227
Attachment #804265 - Flags: checkin+
SeaMonkey hourly builds have appeared for W32, L32 and L64. Not on the Mac (yet) but there the current problem seems to have something to do with packaging the omnijar.

For instance on L64:
Mozilla/5.0 (X11; Linux x86_64; rv:26.0) Gecko/20100101 Firefox/26.0 SeaMonkey/2.23a1 ID:20130912192824 c-c:303c7574e706 m-c:b9029b1de410
No nightlies yet.
Closing as fixed. All three platforms have working nightly builds again. Please check that all builds are working and extensions are included in all builds (especially on Mac).
Status: ASSIGNED → RESOLVED
Closed: 6 years ago
Resolution: --- → FIXED
Blocks: 888478
Product: Core → Firefox Build System
You need to log in before you can comment on or make changes to this bug.