653971 - Mac XULRunner packaging broken by bug 386760

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[Phil Ringnalda (:philor)]

http://tinderbox.mozilla.org/showlog.cgi?log=XULRunner/1304157982.1304168790.8784.gz

Compressing...
cd ../../dist && (cd universal/xulrunner/xulrunner-6.0a1.en-US.mac-pkg.pkg/Contents/MacOS && /tools/buildbot/bin/python2.6 /builds/slave/cen-osx64-xr/build/config/createprecomplete.py) && /builds/slave/cen-osx64-xr/build/build/package/mac_osx/pkg-dmg --source "universal/xulrunner-pkg" --target "xulrunner-6.0a1.en-US.mac-pkg.dmg" --volname "XULRunner" 
/bin/sh: line 0: cd: universal/xulrunner/xulrunner-6.0a1.en-US.mac-pkg.pkg/Contents/MacOS: No such file or directory

http://tinderbox.mozilla.org/showlog.cgi?log=XULRunner/1302516325.1302527024.4933.gz&fulltext=1
(the last one before bug 386760 landed)

Compressing...
cd ../../dist && /builds/slave/cen-osx64-xr/build/build/package/mac_osx/pkg-dmg --source "universal/xulrunner-pkg" --target "xulrunner-2.2a1pre.en-US.mac-pkg.dmg" --volname "XULRunner" 
mktemp -d pkg-dmg.71204.XXXXXXXX
etc.

Comment 1

[Shankar Chittoor]

Can we prioritize this bug... It is preventing us as addon developers to be ready for the Firefox 5 launch on the Mac environment.

Comment 2

[Robert Strong (they/them - no direct email)]

Nick, I'm going to take a look at what's going on here but I wondered if you might already know since I'm unfamiliar with Mac XULRunner?

Comment 3

[Robert Strong (they/them - no direct email)]

Now that omnijar is on by default this is failing earlier on when trying to optimize the jar files. :(

Comment 4

[:Gavin Sharp [email: [e-mail]]]

That would be bug 659718, right?

Comment 5

[Robert Strong (they/them - no direct email)]

I don't think so... I'm just compiling xulrunner and I suspect that fixes a similar issue when compiling xullrunner + app.

Comment 6

[Robert Strong (they/them - no direct email)]

Errors with Mac xulrunner universal latest trunk

Creating package directory...
/opt/local/bin/python2.6 /moz/_1_mozilla-central/mozilla/config/optimizejars.py --optimize /moz/_1_mozilla-central/xr-universal/i386/xulrunner/installer/../../jarlog/ ../../dist/bin/chrome ../../dist/universal/xulrunner/xulrunner-7.0a1.en-US.mac-pkg.pkg/Contents/MacOS/chrome
signing nss libraries
moduleSpec configdir='' certPrefix='' keyPrefix='' secmod='' flags=noCertDB, noModDB
Generate a DSA key pair ... 
../../dist/universal/xulrunner/xulrunner-7.0a1.en-US.mac-pkg.pkg/Contents/MacOS/libsoftokn3.dylib: -5950: File not found
moduleSpec configdir='' certPrefix='' keyPrefix='' secmod='' flags=noCertDB, noModDB
Generate a DSA key pair ... 
../../dist/universal/xulrunner/xulrunner-7.0a1.en-US.mac-pkg.pkg/Contents/MacOS/libnssdbm3.dylib: -5950: File not found
Removing unpackaged files...
cd ../../dist/universal/xulrunner/xulrunner-7.0a1.en-US.mac-pkg.pkg/Contents/MacOS; rm -rf xulrunner-config regchrome* regxpcom*  core bsdecho gtscc js js-config jscpucfg nsinstall viewer TestGtkEmbed codesighs* elf-dynstr-gc mangle* maptsv* mfc* mkdepend* msdump* msmap* nm2tsv* nsinstall* res/samples res/throbber shlibsign* ssltunnel* certutil* pk12util* winEmbed.exe chrome/chrome.rdf chrome/app-chrome.manifest chrome/overlayinfo components/compreg.dat components/xpti.dat content_unit_tests necko_unit_tests *.dSYM 
/moz/_1_mozilla-central/xr-universal/i386/config/nsinstall -D ../../dist/
Compressing...
cd ../../dist && (cd universal/xulrunner/xulrunner-7.0a1.en-US.mac-pkg.pkg/Contents/MacOS && rm -f omni.jar components/binary.manifest && grep -h '^binary-component' components/*.manifest > binary.manifest ; sed -e 's/^binary-component/#binary-component/' components/components.manifest > components.manifest && mv components.manifest components && zip -r9m omni.jar chrome chrome.manifest components/*.js components/*.xpt components/*.manifest modules res defaults greprefs.js jsloader  -x chrome/icons/\* defaults/pref/channel-prefs.js res/cursors/\* res/MainMenu.nib/\*  && true && /opt/local/bin/python2.6 /moz/_1_mozilla-central/mozilla/config/optimizejars.py --optimize /moz/_1_mozilla-central/xr-universal/i386/xulrunner/installer/../../jarlog/ ./ ./ && mv binary.manifest components && printf "manifest components/binary.manifest\n" > chrome.manifest) && (cd universal/xulrunner/xulrunner-7.0a1.en-US.mac-pkg.pkg/Contents/MacOS && /opt/local/bin/python2.6 /moz/_1_mozilla-central/mozilla/config/createprecomplete.py) && /moz/_1_mozilla-central/mozilla/build/package/mac_osx/pkg-dmg --source "universal/xulrunner-pkg" --target "xulrunner-7.0a1.en-US.mac-pkg.dmg" --volname "XULRunner" 
grep: components/*.manifest: No such file or directory
sed: components/components.manifest: No such file or directory
make[2]: *** [make-package] Error 1
make[1]: *** [default] Error 2
make: *** [package] Error 2

Comment 7

[Asa Dotzler [:asa]]

tracking 5 because this breaks major add-ons.

Comment 8

[Robert Strong (they/them - no direct email)]

Attachment: patch rev1

This fixes building with omnijar which I believe is the default now and the generation of the precomplete file. Still need to check building with omnijar disabled but I'd like to get your take on this since I'm not very familiar with universal builds.

Comment 9

[Robert Strong (they/them - no direct email)]

Comment on attachment 536993 [details] [diff] [review]
patch rev1

I think that the changes to config/createprecomplete.py aren't needed though I am not positive.

Comment 10

[Robert Strong (they/them - no direct email)]

It also appears that omnijar being turned on by default broke the generation of the binary.manifest from the components.manifest

sed: components/components.manifest: No such file or directory

cc'ing glandium

Mike, the Mac xulrunner nightly builds are broken in part due to bug 620931. I believe the attached patch minus the changes to createprecomplete.py are all that is needed to fix this bug for both MOZ_CHROME_FILE_FORMAT=omni and MOZ_CHROME_FILE_FORMAT=jar builds but I haven't looked into why it is having problem with the components.manifest with just MOZ_CHROME_FILE_FORMAT=omni.

Comment 11

[Robert Strong (they/them - no direct email)]

Attachment: patch rev2

Comment 12

[Robert Strong (they/them - no direct email)]

Comment on attachment 537053 [details] [diff] [review]
patch rev2

Nick, is there any way you can check to see if this fixes the xulrunner build and mar generation.

Comment 13

[Mike Hommey [:glandium]]

(In reply to comment #10)
> Mike, the Mac xulrunner nightly builds are broken in part due to bug 620931.
> I believe the attached patch minus the changes to createprecomplete.py are
> all that is needed to fix this bug for both MOZ_CHROME_FILE_FORMAT=omni and
> MOZ_CHROME_FILE_FORMAT=jar builds but I haven't looked into why it is having
> problem with the components.manifest with just MOZ_CHROME_FILE_FORMAT=omni.

Probably due to how PACK_OMNIJAR is defined in toolkit/mozapps/installer/packager.mk (it should be using $(ZIP) and $(UNZIP), btw)

Comment 14

[Nick Thomas [:nthomas] (UTC+12)]

I can certainly try a xulrunner build to get a dmg + SDKs with this patch, but I'm not sure what you mean about mar generation. We don't create those for XR.

Comment 15

[Robert Strong (they/them - no direct email)]

(In reply to comment #14)
> I can certainly try a xulrunner build to get a dmg + SDKs with this patch,
> but I'm not sure what you mean about mar generation. We don't create those
> for XR.
Thanks Nick... since mars aren't generated I'm tempted not to create the precomplete file. What do you think?

Comment 16

[Robert Strong (they/them - no direct email)]

btw: no need to try out the patch since mars aren't generated.

Comment 17

[Robert Strong (they/them - no direct email)]

I've verified that both MOZ_CHROME_FILE_FORMAT=jar and MOZ_CHROME_FILE_FORMAT=omni build and package correctly with the patch though the second time packaging with MOZ_CHROME_FILE_FORMAT=omni fails. This is due to it modifying the files in dist/universal/xulrunner/XUL.framwork. I'll take a look at what we do for app omnijar packaging to see if there is an obvious fix.

Comment 18

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 537053 [details] [diff] [review]
patch rev2

I took this patch for a spin in a staging instance of the buildbot, on top of m-c rev a52b04dc29ea. It completed make package, and the following step 'make -f client.mk sdk', without errors. It does expose a problem with the buildbot setup which is filed as bug 661792,

FWIW, I think we're also hitting the manifest error
 sed: can't read components/components.manifest: No such file or directory
on windows m-c xulrunner, but not on linux.

Comment 19

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to comment #17)
> I've verified that both MOZ_CHROME_FILE_FORMAT=jar and
> MOZ_CHROME_FILE_FORMAT=omni build and package correctly with the patch
> though the second time packaging with MOZ_CHROME_FILE_FORMAT=omni fails.
> This is due to it modifying the files in
> dist/universal/xulrunner/XUL.framwork. I'll take a look at what we do for
> app omnijar packaging to see if there is an obvious fix.

Bug 600832 has a work around for this - making the postflight_all target again before calling package.

Comment 20

[Robert Strong (they/them - no direct email)]

Thanks a lot for doing all of this Nick!

Comment 21

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

Comment on attachment 537053 [details] [diff] [review]
patch rev2

Hardcoding it might be a little ugly but we need to fix this for 5 and this is good enough IMO.

Comment 22

[Robert Strong (they/them - no direct email)]

Pushed to mozilla-central
http://hg.mozilla.org/mozilla-central/rev/6ee7ab2989ba

note: it is hardcoded here as well
http://mxr.mozilla.org/mozilla-central/source/xulrunner/installer/mac/Makefile.in#67

I also pushed to try to check if this would have any adverse affect on mobile packaging (didn't think it would... just wanted to be safe)
http://tbpl.mozilla.org/?tree=Try&rev=2b0f7c6aaf11

Comment 23

[Robert Strong (they/them - no direct email)]

Not related to this bug but bug 661910 also has a problem with reading the components.manifest except it is on windows where xulrunner doesn't have any binary components.

Comment 24

[Robert Strong (they/them - no direct email)]

Comment on attachment 537053 [details] [diff] [review]
patch rev2

Drivers, this is a xulrunner only one line change and is needed so the sdk is built.

Comment 25

[Asa Dotzler [:asa]]

Comment on attachment 537053 [details] [diff] [review]
patch rev2

Yes, we want this for [5] Beta and [6] Aurora.

Comment 26

[Robert Strong (they/them - no direct email)]

Pushed to mozilla-beta
http://hg.mozilla.org/releases/mozilla-beta/rev/9171cee0df6e

Pushed to mozilla-aurora
http://hg.mozilla.org/releases/mozilla-aurora/rev/39a95a10faf0

Comment 27

[Robert Strong (they/them - no direct email)]

Before the patch landed

cd ../../dist && (cd universal/xulrunner/xulrunner-6.0a2.en-US.mac-pkg.pkg/Contents/MacOS && rm -f omni.jar components/binary.manifest && grep -h '^binary-component' components/*.manifest > binary.manifest ; sed -e 's/^binary-component/#binary-component/' components/components.manifest > components.manifest && mv components.manifest components && zip -r9m omni.jar chrome chrome.manifest components/*.js components/*.xpt components/*.manifest modules res defaults greprefs.js jsloader  -x chrome/icons/\* defaults/pref/channel-prefs.js res/cursors/\* res/MainMenu.nib/\*  && true && /tools/buildbot/bin/python2.6 /builds/slave/aurora-osx64-xr/build/config/optimizejars.py --optimize /builds/slave/aurora-osx64-xr/build/obj-firefox/i386/xulrunner/installer/../../jarlog/ ./ ./ && mv binary.manifest components && printf "manifest components/binary.manifest\n" > chrome.manifest) && (cd universal/xulrunner/xulrunner-6.0a2.en-US.mac-pkg.pkg/Contents/MacOS && /tools/buildbot/bin/python2.6 /builds/slave/aurora-osx64-xr/build/config/createprecomplete.py) && /builds/slave/aurora-osx64-xr/build/build/package/mac_osx/pkg-dmg --source "universal/xulrunner-pkg" --target "xulrunner-6.0a2.en-US.mac-pkg.dmg" --volname "XULRunner" 
/bin/sh: line 0: cd: universal/xulrunner/xulrunner-6.0a2.en-US.mac-pkg.pkg/Contents/MacOS: No such file or directory
sed: components/components.manifest: No such file or directory
<a name='err1'></a><font color='000080'>make[2]: *** [make-package] Error 1
</font><font color='000080'>make[1]: *** [default] Error 2
</font><font color='000080'>make: *** [package] Error 2
</font>program finished with exit code 2


After the patch landed

cd ../../dist/universal/xulrunner/XUL.framework/Versions/Current; rm -rf xulrunner-config regchrome* regxpcom*  core bsdecho gtscc js js-config jscpucfg nsinstall viewer TestGtkEmbed codesighs* elf-dynstr-gc mangle* maptsv* mfc* mkdepend* msdump* msmap* nm2tsv* nsinstall* res/samples res/throbber shlibsign* ssltunnel* certutil* pk12util* winEmbed.exe chrome/chrome.rdf chrome/app-chrome.manifest chrome/overlayinfo components/compreg.dat components/xpti.dat content_unit_tests necko_unit_tests *.dSYM 
/builds/slave/aurora-osx64-xr/build/obj-firefox/i386/config/nsinstall -D ../../dist/
Compressing...
cd ../../dist &amp;&amp; (cd universal/xulrunner/XUL.framework/Versions/Current &amp;&amp; rm -f omni.jar components/binary.manifest &amp;&amp; grep -h '^binary-component' components/*.manifest > binary.manifest ; sed -e 's/^binary-component/#binary-component/' components/components.manifest > components.manifest &amp;&amp; mv components.manifest components &amp;&amp; zip -r9m omni.jar chrome chrome.manifest components/*.js components/*.xpt components/*.manifest modules res defaults greprefs.js jsloader  -x chrome/icons/\* defaults/pref/channel-prefs.js res/cursors/\* res/MainMenu.nib/\*  &amp;&amp; true &amp;&amp; /tools/buildbot/bin/python2.6 /builds/slave/aurora-osx64-xr/build/config/optimizejars.py --optimize /builds/slave/aurora-osx64-xr/build/obj-firefox/i386/xulrunner/installer/../../jarlog/ ./ ./ &amp;&amp; mv binary.manifest components &amp;&amp; printf "manifest components/binary.manifest\n" > chrome.manifest) &amp;&amp; (cd universal/xulrunner/XUL.framework/Versions/Current &amp;&amp; /tools/buildbot/bin/python2.6 /builds/slave/aurora-osx64-xr/build/config/createprecomplete.py) &amp;&amp; /builds/slave/aurora-osx64-xr/build/build/package/mac_osx/pkg-dmg --source "universal/xulrunner-pkg" --target "xulrunner-6.0a2.en-US.mac-pkg.dmg" --volname "XULRunner" 
etc...

There is also the following that appears to be breaking uploading the build which is not related to this bug or bug 386760 that also affects 1.9.1, 1.9.2, and 2.0 where bug 386760 did not land.

======== BuildStep started ========
find filepath failed
=== Output ===
bash -c 'find build/obj-firefox/i386/dist -maxdepth 1 -type f -name *.dmg'
 in dir /builds/slave/aurora-osx64-xr/. (timeout 1200 secs)
 watching logfiles {}
 argv: ['bash', '-c', 'find build/obj-firefox/i386/dist -maxdepth 1 -type f -name *.dmg']
 environment:
  Apple_PubSub_Socket_Render=/tmp/launch-u6x8Ty/Render
  CVS_RSH=ssh
  DISPLAY=/tmp/launch-C7ZmUA/:0
  HOME=/Users/cltbld
  LOGNAME=cltbld
  PATH=/tools/buildbot/bin:/tools/python/bin:/opt/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/X11/bin
  PWD=/builds/slave/aurora-osx64-xr
  PYTHONPATH=/tools/buildbot/lib/python2.6/site-packages:/tools/twisted/lib/python2.6/site-packages/:/tools/twisted-core/lib/python2.6/site-packages:/tools/zope-interface/lib/python2.6/site-packages
  SHELL=/bin/bash
  SSH_AUTH_SOCK=/tmp/launch-3kaUB2/Listeners
  TMPDIR=/var/folders/7I/7I253dv+HLesSBUBGCX08E+++TM/-Tmp-/
  USER=cltbld
  VERSIONER_PYTHON_PREFER_32_BIT=no
  VERSIONER_PYTHON_VERSION=2.6
  __CF_USER_TEXT_ENCODING=0x1F6:0:0
 using PTY: False
program finished with exit code 0
elapsedTime=0.284242
=== Output ended ===
======== BuildStep ended ========
======== BuildStep started ========
maybe rebooting slave lost
=== Output ===
=== Output ended ===
======== BuildStep ended ========

Comment 28

[Nick Thomas [:nthomas] (UTC+12)]

The filepath failure is bug 661792, which should be fixed in tomorrows builds.