Last Comment Bug 653971 - Mac XULRunner packaging broken by bug 386760
: Mac XULRunner packaging broken by bug 386760
Status: RESOLVED FIXED
: regression
Product: Core
Classification: Components
Component: Build Config (show other bugs)
: Trunk
: x86_64 Mac OS X
: -- normal with 3 votes (vote)
: mozilla5
Assigned To: Robert Strong [:rstrong] (use needinfo to contact me)
:
: Gregory Szorc [:gps]
Mentors:
Depends on:
Blocks: 386760 653432 661792
  Show dependency treegraph
 
Reported: 2011-04-30 21:49 PDT by Phil Ringnalda (:philor)
Modified: 2013-12-27 14:33 PST (History)
12 users (show)
See Also:
Crash Signature:
(edit)
QA Whiteboard:
Iteration: ---
Points: ---
Has Regression Range: ---
Has STR: ---
+
fixed
fixed


Attachments
patch rev1 (2.07 KB, patch)
2011-06-02 15:02 PDT, Robert Strong [:rstrong] (use needinfo to contact me)
no flags Details | Diff | Splinter Review
patch rev2 (601 bytes, patch)
2011-06-02 18:08 PDT, Robert Strong [:rstrong] (use needinfo to contact me)
nthomas: review+
khuey: review+
asa: approval‑mozilla‑aurora+
asa: approval‑mozilla‑beta+
Details | Diff | Splinter Review

Description Phil Ringnalda (:philor) 2011-04-30 21:49:06 PDT
http://tinderbox.mozilla.org/showlog.cgi?log=XULRunner/1304157982.1304168790.8784.gz

Compressing...
cd ../../dist && (cd universal/xulrunner/xulrunner-6.0a1.en-US.mac-pkg.pkg/Contents/MacOS && /tools/buildbot/bin/python2.6 /builds/slave/cen-osx64-xr/build/config/createprecomplete.py) && /builds/slave/cen-osx64-xr/build/build/package/mac_osx/pkg-dmg --source "universal/xulrunner-pkg" --target "xulrunner-6.0a1.en-US.mac-pkg.dmg" --volname "XULRunner" 
/bin/sh: line 0: cd: universal/xulrunner/xulrunner-6.0a1.en-US.mac-pkg.pkg/Contents/MacOS: No such file or directory

http://tinderbox.mozilla.org/showlog.cgi?log=XULRunner/1302516325.1302527024.4933.gz&fulltext=1
(the last one before bug 386760 landed)

Compressing...
cd ../../dist && /builds/slave/cen-osx64-xr/build/build/package/mac_osx/pkg-dmg --source "universal/xulrunner-pkg" --target "xulrunner-2.2a1pre.en-US.mac-pkg.dmg" --volname "XULRunner" 
mktemp -d pkg-dmg.71204.XXXXXXXX
etc.
Comment 1 Shankar Chittoor 2011-06-01 10:25:58 PDT
Can we prioritize this bug... It is preventing us as addon developers to be ready for the Firefox 5 launch on the Mac environment.
Comment 2 Robert Strong [:rstrong] (use needinfo to contact me) 2011-06-01 10:29:30 PDT
Nick, I'm going to take a look at what's going on here but I wondered if you might already know since I'm unfamiliar with Mac XULRunner?
Comment 3 Robert Strong [:rstrong] (use needinfo to contact me) 2011-06-01 16:22:22 PDT
Now that omnijar is on by default this is failing earlier on when trying to optimize the jar files. :(
Comment 4 :Gavin Sharp [email: gavin@gavinsharp.com] 2011-06-01 16:26:55 PDT
That would be bug 659718, right?
Comment 5 Robert Strong [:rstrong] (use needinfo to contact me) 2011-06-01 16:29:26 PDT
I don't think so... I'm just compiling xulrunner and I suspect that fixes a similar issue when compiling xullrunner + app.
Comment 6 Robert Strong [:rstrong] (use needinfo to contact me) 2011-06-01 16:32:38 PDT
Errors with Mac xulrunner universal latest trunk

Creating package directory...
/opt/local/bin/python2.6 /moz/_1_mozilla-central/mozilla/config/optimizejars.py --optimize /moz/_1_mozilla-central/xr-universal/i386/xulrunner/installer/../../jarlog/ ../../dist/bin/chrome ../../dist/universal/xulrunner/xulrunner-7.0a1.en-US.mac-pkg.pkg/Contents/MacOS/chrome
signing nss libraries
moduleSpec configdir='' certPrefix='' keyPrefix='' secmod='' flags=noCertDB, noModDB
Generate a DSA key pair ... 
../../dist/universal/xulrunner/xulrunner-7.0a1.en-US.mac-pkg.pkg/Contents/MacOS/libsoftokn3.dylib: -5950: File not found
moduleSpec configdir='' certPrefix='' keyPrefix='' secmod='' flags=noCertDB, noModDB
Generate a DSA key pair ... 
../../dist/universal/xulrunner/xulrunner-7.0a1.en-US.mac-pkg.pkg/Contents/MacOS/libnssdbm3.dylib: -5950: File not found
Removing unpackaged files...
cd ../../dist/universal/xulrunner/xulrunner-7.0a1.en-US.mac-pkg.pkg/Contents/MacOS; rm -rf xulrunner-config regchrome* regxpcom*  core bsdecho gtscc js js-config jscpucfg nsinstall viewer TestGtkEmbed codesighs* elf-dynstr-gc mangle* maptsv* mfc* mkdepend* msdump* msmap* nm2tsv* nsinstall* res/samples res/throbber shlibsign* ssltunnel* certutil* pk12util* winEmbed.exe chrome/chrome.rdf chrome/app-chrome.manifest chrome/overlayinfo components/compreg.dat components/xpti.dat content_unit_tests necko_unit_tests *.dSYM 
/moz/_1_mozilla-central/xr-universal/i386/config/nsinstall -D ../../dist/
Compressing...
cd ../../dist && (cd universal/xulrunner/xulrunner-7.0a1.en-US.mac-pkg.pkg/Contents/MacOS && rm -f omni.jar components/binary.manifest && grep -h '^binary-component' components/*.manifest > binary.manifest ; sed -e 's/^binary-component/#binary-component/' components/components.manifest > components.manifest && mv components.manifest components && zip -r9m omni.jar chrome chrome.manifest components/*.js components/*.xpt components/*.manifest modules res defaults greprefs.js jsloader  -x chrome/icons/\* defaults/pref/channel-prefs.js res/cursors/\* res/MainMenu.nib/\*  && true && /opt/local/bin/python2.6 /moz/_1_mozilla-central/mozilla/config/optimizejars.py --optimize /moz/_1_mozilla-central/xr-universal/i386/xulrunner/installer/../../jarlog/ ./ ./ && mv binary.manifest components && printf "manifest components/binary.manifest\n" > chrome.manifest) && (cd universal/xulrunner/xulrunner-7.0a1.en-US.mac-pkg.pkg/Contents/MacOS && /opt/local/bin/python2.6 /moz/_1_mozilla-central/mozilla/config/createprecomplete.py) && /moz/_1_mozilla-central/mozilla/build/package/mac_osx/pkg-dmg --source "universal/xulrunner-pkg" --target "xulrunner-7.0a1.en-US.mac-pkg.dmg" --volname "XULRunner" 
grep: components/*.manifest: No such file or directory
sed: components/components.manifest: No such file or directory
make[2]: *** [make-package] Error 1
make[1]: *** [default] Error 2
make: *** [package] Error 2
Comment 7 Asa Dotzler [:asa] 2011-06-01 16:45:45 PDT
tracking 5 because this breaks major add-ons.
Comment 8 Robert Strong [:rstrong] (use needinfo to contact me) 2011-06-02 15:02:07 PDT
Created attachment 536993 [details] [diff] [review]
patch rev1

This fixes building with omnijar which I believe is the default now and the generation of the precomplete file. Still need to check building with omnijar disabled but I'd like to get your take on this since I'm not very familiar with universal builds.
Comment 9 Robert Strong [:rstrong] (use needinfo to contact me) 2011-06-02 17:55:04 PDT
Comment on attachment 536993 [details] [diff] [review]
patch rev1

I think that the changes to config/createprecomplete.py aren't needed though I am not positive.
Comment 10 Robert Strong [:rstrong] (use needinfo to contact me) 2011-06-02 18:05:59 PDT
It also appears that omnijar being turned on by default broke the generation of the binary.manifest from the components.manifest

sed: components/components.manifest: No such file or directory

cc'ing glandium

Mike, the Mac xulrunner nightly builds are broken in part due to bug 620931. I believe the attached patch minus the changes to createprecomplete.py are all that is needed to fix this bug for both MOZ_CHROME_FILE_FORMAT=omni and MOZ_CHROME_FILE_FORMAT=jar builds but I haven't looked into why it is having problem with the components.manifest with just MOZ_CHROME_FILE_FORMAT=omni.
Comment 11 Robert Strong [:rstrong] (use needinfo to contact me) 2011-06-02 18:08:12 PDT
Created attachment 537053 [details] [diff] [review]
patch rev2
Comment 12 Robert Strong [:rstrong] (use needinfo to contact me) 2011-06-02 18:09:09 PDT
Comment on attachment 537053 [details] [diff] [review]
patch rev2

Nick, is there any way you can check to see if this fixes the xulrunner build and mar generation.
Comment 13 Mike Hommey [:glandium] 2011-06-02 18:55:49 PDT
(In reply to comment #10)
> Mike, the Mac xulrunner nightly builds are broken in part due to bug 620931.
> I believe the attached patch minus the changes to createprecomplete.py are
> all that is needed to fix this bug for both MOZ_CHROME_FILE_FORMAT=omni and
> MOZ_CHROME_FILE_FORMAT=jar builds but I haven't looked into why it is having
> problem with the components.manifest with just MOZ_CHROME_FILE_FORMAT=omni.

Probably due to how PACK_OMNIJAR is defined in toolkit/mozapps/installer/packager.mk (it should be using $(ZIP) and $(UNZIP), btw)
Comment 14 Nick Thomas [:nthomas] 2011-06-02 18:57:37 PDT
I can certainly try a xulrunner build to get a dmg + SDKs with this patch, but I'm not sure what you mean about mar generation. We don't create those for XR.
Comment 15 Robert Strong [:rstrong] (use needinfo to contact me) 2011-06-02 19:54:29 PDT
(In reply to comment #14)
> I can certainly try a xulrunner build to get a dmg + SDKs with this patch,
> but I'm not sure what you mean about mar generation. We don't create those
> for XR.
Thanks Nick... since mars aren't generated I'm tempted not to create the precomplete file. What do you think?
Comment 16 Robert Strong [:rstrong] (use needinfo to contact me) 2011-06-02 19:55:24 PDT
btw: no need to try out the patch since mars aren't generated.
Comment 17 Robert Strong [:rstrong] (use needinfo to contact me) 2011-06-02 20:33:08 PDT
I've verified that both MOZ_CHROME_FILE_FORMAT=jar and MOZ_CHROME_FILE_FORMAT=omni build and package correctly with the patch though the second time packaging with MOZ_CHROME_FILE_FORMAT=omni fails. This is due to it modifying the files in dist/universal/xulrunner/XUL.framwork. I'll take a look at what we do for app omnijar packaging to see if there is an obvious fix.
Comment 18 Nick Thomas [:nthomas] 2011-06-03 04:19:29 PDT
Comment on attachment 537053 [details] [diff] [review]
patch rev2

I took this patch for a spin in a staging instance of the buildbot, on top of m-c rev a52b04dc29ea. It completed make package, and the following step 'make -f client.mk sdk', without errors. It does expose a problem with the buildbot setup which is filed as bug 661792,

FWIW, I think we're also hitting the manifest error
 sed: can't read components/components.manifest: No such file or directory
on windows m-c xulrunner, but not on linux.
Comment 19 Nick Thomas [:nthomas] 2011-06-03 04:24:35 PDT
(In reply to comment #17)
> I've verified that both MOZ_CHROME_FILE_FORMAT=jar and
> MOZ_CHROME_FILE_FORMAT=omni build and package correctly with the patch
> though the second time packaging with MOZ_CHROME_FILE_FORMAT=omni fails.
> This is due to it modifying the files in
> dist/universal/xulrunner/XUL.framwork. I'll take a look at what we do for
> app omnijar packaging to see if there is an obvious fix.

Bug 600832 has a work around for this - making the postflight_all target again before calling package.
Comment 20 Robert Strong [:rstrong] (use needinfo to contact me) 2011-06-03 04:37:32 PDT
Thanks a lot for doing all of this Nick!
Comment 21 Kyle Huey [:khuey] (Exited; not receiving bugmail, email if necessary) 2011-06-03 13:15:17 PDT
Comment on attachment 537053 [details] [diff] [review]
patch rev2

Hardcoding it might be a little ugly but we need to fix this for 5 and this is good enough IMO.
Comment 22 Robert Strong [:rstrong] (use needinfo to contact me) 2011-06-03 20:46:01 PDT
Pushed to mozilla-central
http://hg.mozilla.org/mozilla-central/rev/6ee7ab2989ba

note: it is hardcoded here as well
http://mxr.mozilla.org/mozilla-central/source/xulrunner/installer/mac/Makefile.in#67

I also pushed to try to check if this would have any adverse affect on mobile packaging (didn't think it would... just wanted to be safe)
http://tbpl.mozilla.org/?tree=Try&rev=2b0f7c6aaf11
Comment 23 Robert Strong [:rstrong] (use needinfo to contact me) 2011-06-03 22:50:00 PDT
Not related to this bug but bug 661910 also has a problem with reading the components.manifest except it is on windows where xulrunner doesn't have any binary components.
Comment 24 Robert Strong [:rstrong] (use needinfo to contact me) 2011-06-06 10:07:38 PDT
Comment on attachment 537053 [details] [diff] [review]
patch rev2

Drivers, this is a xulrunner only one line change and is needed so the sdk is built.
Comment 25 Asa Dotzler [:asa] 2011-06-06 15:09:24 PDT
Comment on attachment 537053 [details] [diff] [review]
patch rev2

Yes, we want this for [5] Beta and [6] Aurora.
Comment 26 Robert Strong [:rstrong] (use needinfo to contact me) 2011-06-06 16:52:49 PDT
Pushed to mozilla-beta
http://hg.mozilla.org/releases/mozilla-beta/rev/9171cee0df6e

Pushed to mozilla-aurora
http://hg.mozilla.org/releases/mozilla-aurora/rev/39a95a10faf0
Comment 27 Robert Strong [:rstrong] (use needinfo to contact me) 2011-06-07 12:08:24 PDT
Before the patch landed

cd ../../dist && (cd universal/xulrunner/xulrunner-6.0a2.en-US.mac-pkg.pkg/Contents/MacOS && rm -f omni.jar components/binary.manifest && grep -h '^binary-component' components/*.manifest > binary.manifest ; sed -e 's/^binary-component/#binary-component/' components/components.manifest > components.manifest && mv components.manifest components && zip -r9m omni.jar chrome chrome.manifest components/*.js components/*.xpt components/*.manifest modules res defaults greprefs.js jsloader  -x chrome/icons/\* defaults/pref/channel-prefs.js res/cursors/\* res/MainMenu.nib/\*  && true && /tools/buildbot/bin/python2.6 /builds/slave/aurora-osx64-xr/build/config/optimizejars.py --optimize /builds/slave/aurora-osx64-xr/build/obj-firefox/i386/xulrunner/installer/../../jarlog/ ./ ./ && mv binary.manifest components && printf "manifest components/binary.manifest\n" > chrome.manifest) && (cd universal/xulrunner/xulrunner-6.0a2.en-US.mac-pkg.pkg/Contents/MacOS && /tools/buildbot/bin/python2.6 /builds/slave/aurora-osx64-xr/build/config/createprecomplete.py) && /builds/slave/aurora-osx64-xr/build/build/package/mac_osx/pkg-dmg --source "universal/xulrunner-pkg" --target "xulrunner-6.0a2.en-US.mac-pkg.dmg" --volname "XULRunner" 
/bin/sh: line 0: cd: universal/xulrunner/xulrunner-6.0a2.en-US.mac-pkg.pkg/Contents/MacOS: No such file or directory
sed: components/components.manifest: No such file or directory
<a name='err1'></a><font color='000080'>make[2]: *** [make-package] Error 1
</font><font color='000080'>make[1]: *** [default] Error 2
</font><font color='000080'>make: *** [package] Error 2
</font>program finished with exit code 2


After the patch landed

cd ../../dist/universal/xulrunner/XUL.framework/Versions/Current; rm -rf xulrunner-config regchrome* regxpcom*  core bsdecho gtscc js js-config jscpucfg nsinstall viewer TestGtkEmbed codesighs* elf-dynstr-gc mangle* maptsv* mfc* mkdepend* msdump* msmap* nm2tsv* nsinstall* res/samples res/throbber shlibsign* ssltunnel* certutil* pk12util* winEmbed.exe chrome/chrome.rdf chrome/app-chrome.manifest chrome/overlayinfo components/compreg.dat components/xpti.dat content_unit_tests necko_unit_tests *.dSYM 
/builds/slave/aurora-osx64-xr/build/obj-firefox/i386/config/nsinstall -D ../../dist/
Compressing...
cd ../../dist &amp;&amp; (cd universal/xulrunner/XUL.framework/Versions/Current &amp;&amp; rm -f omni.jar components/binary.manifest &amp;&amp; grep -h '^binary-component' components/*.manifest > binary.manifest ; sed -e 's/^binary-component/#binary-component/' components/components.manifest > components.manifest &amp;&amp; mv components.manifest components &amp;&amp; zip -r9m omni.jar chrome chrome.manifest components/*.js components/*.xpt components/*.manifest modules res defaults greprefs.js jsloader  -x chrome/icons/\* defaults/pref/channel-prefs.js res/cursors/\* res/MainMenu.nib/\*  &amp;&amp; true &amp;&amp; /tools/buildbot/bin/python2.6 /builds/slave/aurora-osx64-xr/build/config/optimizejars.py --optimize /builds/slave/aurora-osx64-xr/build/obj-firefox/i386/xulrunner/installer/../../jarlog/ ./ ./ &amp;&amp; mv binary.manifest components &amp;&amp; printf "manifest components/binary.manifest\n" > chrome.manifest) &amp;&amp; (cd universal/xulrunner/XUL.framework/Versions/Current &amp;&amp; /tools/buildbot/bin/python2.6 /builds/slave/aurora-osx64-xr/build/config/createprecomplete.py) &amp;&amp; /builds/slave/aurora-osx64-xr/build/build/package/mac_osx/pkg-dmg --source "universal/xulrunner-pkg" --target "xulrunner-6.0a2.en-US.mac-pkg.dmg" --volname "XULRunner" 
etc...

There is also the following that appears to be breaking uploading the build which is not related to this bug or bug 386760 that also affects 1.9.1, 1.9.2, and 2.0 where bug 386760 did not land.

======== BuildStep started ========
find filepath failed
=== Output ===
bash -c 'find build/obj-firefox/i386/dist -maxdepth 1 -type f -name *.dmg'
 in dir /builds/slave/aurora-osx64-xr/. (timeout 1200 secs)
 watching logfiles {}
 argv: ['bash', '-c', 'find build/obj-firefox/i386/dist -maxdepth 1 -type f -name *.dmg']
 environment:
  Apple_PubSub_Socket_Render=/tmp/launch-u6x8Ty/Render
  CVS_RSH=ssh
  DISPLAY=/tmp/launch-C7ZmUA/:0
  HOME=/Users/cltbld
  LOGNAME=cltbld
  PATH=/tools/buildbot/bin:/tools/python/bin:/opt/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/X11/bin
  PWD=/builds/slave/aurora-osx64-xr
  PYTHONPATH=/tools/buildbot/lib/python2.6/site-packages:/tools/twisted/lib/python2.6/site-packages/:/tools/twisted-core/lib/python2.6/site-packages:/tools/zope-interface/lib/python2.6/site-packages
  SHELL=/bin/bash
  SSH_AUTH_SOCK=/tmp/launch-3kaUB2/Listeners
  TMPDIR=/var/folders/7I/7I253dv+HLesSBUBGCX08E+++TM/-Tmp-/
  USER=cltbld
  VERSIONER_PYTHON_PREFER_32_BIT=no
  VERSIONER_PYTHON_VERSION=2.6
  __CF_USER_TEXT_ENCODING=0x1F6:0:0
 using PTY: False
program finished with exit code 0
elapsedTime=0.284242
=== Output ended ===
======== BuildStep ended ========
======== BuildStep started ========
maybe rebooting slave lost
=== Output ===
=== Output ended ===
======== BuildStep ended ========
Comment 28 Nick Thomas [:nthomas] 2011-06-07 15:31:27 PDT
The filepath failure is bug 661792, which should be fixed in tomorrows builds.

Note You need to log in before you can comment on or make changes to this bug.