923979 - XULRunner on Mac won't launch: Library not loaded (libmozglue.dylib)

Status: RESOLVED INCOMPLETE

(Toolkit Graveyard :: XULRunner, defect)

Description

[Alex Vincent [:WeirdAl]]

As requested, I've boiled down a simple set of steps to reproduce a XULRunner-on-Macintosh bustage.

1. hg clone http://hg.code.sf.net/p/verbosio/jasmine/code verbosio-jasmine-code
2. cd verbosio-jasmine-code/
# This gets you a branch where XULRunner on Mac is preferred over "firefox --app"
3. hg update xr-mac


# This downloads the XULRunner SDK and places it in ./libraries/xulrunner-sdk.  It also 
# stores a path to the SDK in ./local.config .
4. python project.py --update-sdk=release

5. # Open xr-app/application.ini, make sure the MaxVersion field is greater than the SDK version.

6. python project.py test-xul

# Expected result: XULRunner application launches and runs 3 quick Jasmine tests in a XUL window.
# Actual result:
dyld: Library not loaded: @executable_path/libmozglue.dylib
  Referenced from: .../dist/app/JasmineXR.app/Contents/MacOS/xulrunner
  Reason: image not found

libmozglue.dylib lives under dist/app/JasmineXR.app/Contents/Frameworks/XUL.framework/ .

Other notes:
* https://developer.mozilla.org/en-US/docs/Getting_started_with_XULRunner recommends copying XUL.framework from the SDK to /Library/Frameworks, to use --install-app.  However, --install-app is broken in general.  The verbosio-jasmine-code repository uses a Python script to execute the install-app code.  This Python script is also offered in bug 747597, so you shouldn't need to install the framework or run --install-app.

Comment 1

[Mike de Boer [:mikedeboer]]

Attachment: Patch v1: re-add xulrunner-stub to the OSX builds of XULRunner

Dave, we talked about this on IRC in #xulrunner. I now fixed the stub to be more future-proof when it comes to path handling.

With XUL.framework in a proper structure, complying to OSX conventions, I think that looking for the binaries in `<bundle>/Contents/Frameworks/XUL.framework/Versions/Current/` is the best way forward.

Second part of this patch is to force building the stub binary by providing a different name than `xulrunner` (xulrunner-stub), because the app binary itself is also called `xulrunner` (and not `xulrunner-bin` anymore).

Comment 2

[Mike de Boer [:mikedeboer]]

Dave, if you're not the right peer to review this, could you proxy the request if you know who is?

Comment 3

[Dave Townsend [:mossop]]

Comment on attachment 822268 [details] [diff] [review]
Patch v1: re-add xulrunner-stub to the OSX builds of XULRunner

This isn't really more future proof, you just change one hardcoded path for another. I'm not sure I agree with the new path either since it has the potential to break existing apps. Either way I think bsmedberg is a better reviewer here

Comment 4

[Alex Vincent [:WeirdAl]]

Is there any reasonable chance of getting this in for FF27 Aurora?  There's a couple very useful changes already slated for that (Log.jsm, XULRunner as devtools debuggee), so having this would make a nice trifecta.

Comment 5

[Mike de Boer [:mikedeboer]]

Thanks for the review Benjamin!

Alex, I think its ok to uplift this, since the risk is minimal. I'll request approval once this has lived on m-c for a day or two.

Comment 7

[C Farrell]

I was also having a failure in all verisons of xulrunner 18+ on OSX.  I fixed this in xulrunner 25.0 by patching the stub to look for XUL.framework/xulrunner instead of XUL.framework/libxpcom.dylib.  Additionally I had to create a symlink for XUL.framework/xulrunner to XUL.framework/Versions/Current/xulrunner (this link existed in archives for older versions).

Comment 8

[Mike de Boer [:mikedeboer]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/fa39e1828489

Comment 9

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/fa39e1828489

Comment 10

[Mike de Boer [:mikedeboer]]

Comment on attachment 822268 [details] [diff] [review]
Patch v1: re-add xulrunner-stub to the OSX builds of XULRunner

[Approval Request Comment]
Bug caused by (feature/regressing bug #): n/a
User impact if declined: XULRunner consumers (still) won't be able to use it on OSX without considerable workarounds
Testing completed (on m-c, etc.): 2 days in the tree
Risk to taking this patch (and alternatives if risky): none.
String or IDL/UUID changes made by this patch: n/a

Comment 11

[bhavana bajaj [:bajaj]]

(In reply to Mike de Boer [:mikedeboer] from comment #10)
> Comment on attachment 822268 [details] [diff] [review]
> Patch v1: re-add xulrunner-stub to the OSX builds of XULRunner
> 
> [Approval Request Comment]
> Bug caused by (feature/regressing bug #): n/a
> User impact if declined: XULRunner consumers (still) won't be able to use it
> on OSX without considerable workarounds
Has this been broken for a while now(how many versions of firefox does this date back to ) ? If so can we know what regressed it ?

Also since this is a mozconfig change , not sure what impact will this have on our mergedays, NI releng for input here.

> Testing completed (on m-c, etc.): 2 days in the tree
> Risk to taking this patch (and alternatives if risky): none.
> String or IDL/UUID changes made by this patch: n/a

Comment 12

[Aki Sasaki (not active)]

(In reply to bhavana bajaj [:bajaj] from comment #11)
> Also since this is a mozconfig change , not sure what impact will this have
> on our mergedays, NI releng for input here.

I think we don't check xulrunner mozconfigs for releases, and I think this can ride the trains.

Comment 13

[Mike de Boer [:mikedeboer]]

bajaj: this has been broken since XULRunner/ Fx 15b3: http://ftp.mozilla.org/pub/mozilla.org/xulrunner/releases/15.0b3/sdk/xulrunner-15.0b3.en-US.mac-x86_64.sdk.tar.bz2

This is the first time the SDK was distributed without the stub executable on OSX. In other words, XULRunner 15b2 still contained the stub executable.

Comment 14

[bhavana bajaj [:bajaj]]

(In reply to Mike de Boer [:mikedeboer] from comment #13)
> bajaj: this has been broken since XULRunner/ Fx 15b3:
> http://ftp.mozilla.org/pub/mozilla.org/xulrunner/releases/15.0b3/sdk/
> xulrunner-15.0b3.en-US.mac-x86_64.sdk.tar.bz2
> 
> This is the first time the SDK was distributed without the stub executable
> on OSX. In other words, XULRunner 15b2 still contained the stub executable.

Given this has been an issue since 15.x and keeping in mind release engineering's opinion on the moz config changes involved here, lets let this ride the trains.

Comment 15

[Mike Kaply [:mkaply]]

I'm confused. I just grabbed the latest nightly of Firefox 28 XULRunner, but I do not see xulrunner-stub in the zip.

Am I missing something?

Comment 16

[Alex Vincent [:WeirdAl]]

Mike de Boer, how did you test the fix for this bug?  I've tried XR 29 from mozilla-central, including the install_app.py from bug 747597, and I *still* can't get it to work.

dyld: Library not loaded: @executable_path/libmozglue.dylib
  Referenced from: /Users/ajvincent/verbosio/xr-test/./dist/app/JasmineXR.app/Contents/MacOS/xulrunner
  Reason: image not found

Comment 17

[Mike de Boer [:mikedeboer]]

Mike, Alex, I only fixed compiling the stub executable on Mac and changed the check for libmozglue.dylib.

I've been poking at WHY the stub won't be included in the package, but without any luck. It's hellishly frustrating. IOW, I'm looking for pointers!

I can give you a correct xulrunner_stub that I compiled myself, but that'd be cheating, I guess. Still, if you want it, please let me know.

Comment 18

[Alex Vincent [:WeirdAl]]

No, I want this fixed properly.  But I have some ideas on a plan of attack.  I started investigating last night.

First off, I saw signs of hope with these commands:
cp dist/app/JasmineXR.app/Contents/Frameworks/XUL.framework/* dist/app/JasmineXR.app/Contents/MacOS/
cp -r dist/app/JasmineXR.app/Contents/Frameworks/XUL.framework/res dist/app/JasmineXR.app/Contents/MacOS/res

./dist/app/JasmineXR.app/Contents/MacOS/xulrunner /Users/ajvincent/verbosio/xr-test/dist/app/JasmineXR.app/Contents/Resources/application.ini --no-remote -profile /Users/ajvincent/verbosio/xr-test/dist/profile -python /Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python

That almost worked.  Then I hit bug 958814 and realized that my install_app.py script was missing something which caused a crash at startup.  Once I added in the following lines to Info.plist manually:

<key>CFBundleIdentifier</key>
<string>org.mozdev.verbosio</string>

the test finally passed.

On another path, when I didn't do any of the above, XCode clearly showed breakage happening before int main() began running.  So whatever's going on, we're in Mac-specific implementation details, and deep.

---

So in digging around a little more while writing this, I found another link which may provide clues:
https://developer.apple.com/library/mac/documentation/macosx/Conceptual/BPFrameworks/Concepts/FrameworkAnatomy.html

Apparently, the way we build XUL.framework is wrong.  At the very least, the framework appears to be missing an Info.plist file.  This may be the key that we've been missing - after all, there has to be something that tells the Mac to load files in another directory, and Info.plist appears to be it.

There appears to be lots of useful information around this website.  Maybe we've been attacking this problem the wrong way the whole time.  It's certainly worth doing a structural review of XUL.framework's organization and contents.

---

One possibility is that we may want to dump the XUL frameworks model entirely and just put all the binaries in one place.  I'm not sure I want to entertain that yet, given what I just discovered above, but it's an option to put on the table.

Comment 19

[Alex Vincent [:WeirdAl]]

xulrunner/app/Makefile.in::tools generates Info.plist and puts it in dist/XUL.framework/Versions/$(FRAMEWORK_VERSION), so that works correctly.

dist/xulrunner/XUL.framework/Versions/Current/ doesn't have it, so it doesn't end up in the XULRunner SDK tarball.

packager.mk invokes packager.py to transform dist/XUL.framework/Versions/Current into dist/xulrunner/XUL.framework/Versions/Current/, so the missed Info.plist file is dropped there.

Comment 20

[Alex Vincent [:WeirdAl]]

(In reply to Mike de Boer [:mikedeboer] from comment #17)
> Mike, Alex, I only fixed compiling the stub executable on Mac and changed
> the check for libmozglue.dylib.
> 
> I've been poking at WHY the stub won't be included in the package, but
> without any luck. It's hellishly frustrating. IOW, I'm looking for pointers!

I think I found a partial answer to this question.
http://hg.mozilla.org/mozilla-central/annotate/f356b409d710/configure.in#l4111

The stub is compiled from nsXULStub.cpp.

On Linux & Windows, the stub is called xulrunner-stub.  On Mac, it's called xulrunner.  Then it gets replaced by the nsXULRunnerApp.cpp version of xulrunner.

Comment 21

[Mike de Boer [:mikedeboer]]

Please see http://mxr.mozilla.org/mozilla-central/source/xulrunner/config/mozconfigs/macosx-universal/xulrunner#7, which I added in the patch for this bug and fixes what you 'found'.

Like I said, the reason it's not present in the xulrunner package is in the packaging scripts. The packaging script needs to start explicitly copying in the xulrunner_stub executable, but I don't know (yet) where, thus how, to fix it.

Comment 22

[Alex Vincent [:WeirdAl]]

Yeah, last night I was working on those packaging scripts and had it basically working.  Then I found out it's just not going to matter:

https://groups.google.com/forum/#!topic/mozilla.dev.platform/o99wQZBjIJw

Comment 23

[Mike Kaply [:mkaply]]

(In reply to Alex Vincent [:WeirdAl] from comment #22)
> Yeah, last night I was working on those packaging scripts and had it
> basically working.  Then I found out it's just not going to matter:
> 
> https://groups.google.com/forum/#!topic/mozilla.dev.platform/o99wQZBjIJw

You've got to be kidding me. I just built something new for a client on XUL Runner. ARGH.

Comment 24

[Mike de Boer [:mikedeboer]]

Gents, please don't worry just yet... these decisions aren't taken overnight and we're just discussing it atm.

Alex, please share your updates to the packaging scripts, as I will take it if it works. Giving up on Xulrunner at this point really isn't necessary (I'm not!).

Comment 25

[Alex Vincent [:WeirdAl]]

Attachment: work in progress patch (with some cruft)

This is an attempt to resolve the xulrunner-stub bustage and the packaging bugs with --enable-tests.  But it also has some stuff that shouldn't go into a final patch.  

* build/unix/run-mozilla.sh shouldn't be there, because now we have a stub.
* python/mozbuild/mozpack/files.py changes probably shouldn't be there.  (Diagnostic only)
* toolkit/mozapps/installer/packager.mk:  My version of tar stopped supporting --owner.  However, I also have disabled stripping of debug symbols for now.  (I think.)
* toolkit/mozapps/installer/packager.py:  I was having problems with --enable-tests; this change should no longer be necessary.
* xulrunner/app/nsXULRunnerApp.cpp: Diagnostic print lines.
* xulrunner/stub/nsXULStub.cpp: Diagnostic print lines.

Comment 26

[Alex Vincent [:WeirdAl]]

Attachment: Fix XULRunner application bundles for Mac

The install_app.py change is much smaller than the patch indicates; unfortunately the version currently in mozilla-central has Windows new-line endings.

This patch *fixes* XULRunner on Mac, including app installation and execution, per manual tests.

Comment 27

[Alex Vincent [:WeirdAl]]

Attachment: hg diff -w xulrunner/app/install_app.py

For reference, I'm posting the changes to install_app, excluding whitespace.

Comment 28

[Alex Vincent [:WeirdAl]]

Comment on attachment 8360177 [details] [diff] [review]
Fix XULRunner application bundles for Mac

Benjamin, I'd really appreciate your reviewing this for check-in, even if XULRunner is on the way out.  Think of it as transitional code to get us by while the Firefox SDK effort ramps up.  In particular, install_app.py will probably move over to the Firefox SDK (it's still good, at least), and I already posted the whitespace-ignorant changes in another attachment.

Comment 29

[Benjamin Smedberg]

Why are we adding a package-manifest to xulrunner now? That seems fragile at best, which is why we didn't have one before. I'd prefer not to have one.

Comment 30

[Alex Vincent [:WeirdAl]]

The reason for that was primarily to allow building XULRunner SDK with tests.  Without the manifest, XR can only be bundled as an SDK when --disable-tests is part of the configuration.

Given a choice between this bug and bug 961936, I'd really prefer reviews on bug 961936.

Comment 31

[Alex Vincent [:WeirdAl]]

Comment on attachment 8360177 [details] [diff] [review]
Fix XULRunner application bundles for Mac

Canceling review request in deference to bug 961936.  Also, bug 933111 was the bug about --enable-tests breaking the XULRunner SDK.

Comment 32

[Mike Kaply [:mkaply]]

Probably should have put this here. I'm using this patch to try to build a mac XULRunner. Trying to use install_app.py, I get:

I'm getting an error running install_app.py:

Traceback (most recent call last):
  File "install_app.py", line 221, in <module>
    handleCommandLine()
  File "install_app.py", line 215, in handleCommandLine
    except exn:
NameError: global name 'exn' is not defined

Should I be running it from ./bin/install_app.py or ./XUL.framework/Versions/29.0.1/install_app.py?

Comment 33

[Mike Kaply [:mkaply]]

I believe it should be:

    try:
        installApp(cmds.appLocation, cmds.installDir, cmds.appName, cmds.greDir)
    except Exception as exn:
        shutil.rmtree(cmds.installDir)
        raise exn

Comment 34

[Mike Kaply [:mkaply]]

So I have things built, but I'm getting "Couldn't load XPCOM"

I added some debug code, and the return from XPCOMGlueStartup is 80040111.

When I printf the GRE directory, I'm getting /Users/mkaply/Projects/mozilla/mozilla-release/obj-x86_64-apple-darwin13.2.0/mozglue/build/libmozglue.dylib instead of the path inside my .app file.

Could that be an issue?

Any ideas how to debug app startup on Mac using XCode?

Comment 35

[Mike Kaply [:mkaply]]

This was running the stub directly. Trying to launch the .app gives:

LSOpenURLsWithRole() failed with error -10810 for the file /Users/mkaply/Projects/foo/WorkSmart.app.

Comment 36

[Mike Kaply [:mkaply]]

Interesting. If I take the generated .app file and move it to another Mac, it works.

So it seems to have to do with the symlinks into the build version of XULRunner.

Comment 37

[Abhishek K]

Is there any update o this issue

Comment 39

[Tom Peeters]

Is this problem even going to be fixed? It has been forever since we had a working osx stub in the xulrunner releases provided on ftp.mozilla.org. I'm able to build the stub myself using the source code, but for me it always results in font-issues (different font-size/font).

Comment 40

[Benjamin Smedberg]

No, it is unlikely that this will ever be fixed unless somebody here steps up and diagnoses/fixes it.

Comment 41

[Benjamin Smedberg]

XULRunner has been removed from the Mozilla tree: see https://groups.google.com/forum/#!topic/mozilla.dev.platform/_rFMunG2Bgw for context.

I am closing all the bugs currently in the XULRunner bugzilla component, in preparation for moving this component to the graveyard. If this bug is still valid in a XULRunner-less world, it will need to be moved to a different bugzilla component to be reopened.