Closed Bug 1125394 Opened 5 years ago Closed 5 years ago

unable to run chrome test suite on Mac OSX

Categories

(Firefox Graveyard :: Webapp Runtime, defect, P1)

31 Branch
x86
macOS
defect

Tracking

(firefox40 fixed)

RESOLVED FIXED
Firefox 40
Tracking Status
firefox40 --- fixed

People

(Reporter: nick, Assigned: myk)

References

Details

Attachments

(3 files, 1 obsolete file)

When I run `./mach webapprt-test-chrome` on OSX 10.8.5, I cannot complete the test harness.  It hangs at `must wait for focus` yet no windows are open from the test harness.
Myk, Marco, any advice here?
Priority: -- → P1
This is fallout from moving the webapprt-stub executable from the Contents/MacOS/ to the Contents/Resources/ subdirectory of the Firefox app bundle in bug 1091109.

When the test runner launches the runtime in order to run the tests, it invokes the webapprt-stub binary from inside the Firefox app bundle via a command like the following:

/Users/myk/Projects/gecko-dev/obj-x86_64-apple-darwin13.4.0/dist/Nightly.app/Contents/Resources/webapprt-stub -test-mode http://mochi.test:8888/tests/ -foreground -profile /var/folders/lp/8t_7y24119720_hjp5wc_4cr0000gn/T/tmpkokIcJ.mozrunner

The binary starts and opens two windows (mochitest.xul and webapp.xul), but they never appear.  If I copy the stub from Contents/Resources/ to Contents/MacOS/ invoke it from there, however, then they do appear.

Perhaps Mac (at least version 10.9+) requires an executable to be invoked from Contents/MacOS/ in order for it to create UI.

As for how to fix it, we may be able to create an app bundle for running tests and use that to run them.  Or copy the stub to Contents/MacOS/ for the purpose of running tests.

spohl: do you have a suggestion?
Blocks: 1091109
Flags: needinfo?(spohl.mozilla.bugs)
My best guess is that webapprt-stub can't find one of its dependencies. Does the console print anything useful? If not, moving files one-by-one from Contents/MacOS to Contents/Resources may show what file it can't find. We could then fix webapprt-stub to load this file out of Contents/MacOS, which may avoid a lot of extra work.
Flags: needinfo?(spohl.mozilla.bugs)
(In reply to Stephen Pohl [:spohl] from comment #3)
> My best guess is that webapprt-stub can't find one of its dependencies. Does
> the console print anything useful?

It doesn't print any error messages, and instrumenting its code shows that the expected code is executed, and both windows are opened successfully (and their code is successfully executed).  The windows just don't actually appear.
(In reply to Stephen Pohl [:spohl] from comment #3)
> If not, moving files one-by-one from
> Contents/MacOS to Contents/Resources may show what file it can't find. We
> could then fix webapprt-stub to load this file out of Contents/MacOS, which
> may avoid a lot of extra work.

I tried copying all files from Contents/MacOS to Contents/Resources, which should have resolved the problem if this was the case; but it didn't have any effect.

Note that webapprt-stub doesn't exhibit any symptoms of not finding dependencies.  It loads and executes webapprt/CommandLineHandler.js, which means it must have found and loaded libxul (and other libraries).  CommandLineHandler.js then opens chrome://webapprt/content/mochitest.xul successfully, which in turn loads mochitest.js, which itself appears to execute correctly.

The only thing that doesn't happen is display of a native application window.

Stephen: is it possible that an executable in the Resources/ directory of an application bundle is treated differently from one in the MacOS/ directory, such that the OS doesn't enable it to open native windows (or other native graphic elements)?
Flags: needinfo?(spohl.mozilla.bugs)
Summary: unable to run chrome test suite on OSX → unable to run chrome test suite on Mac OSX
(In reply to Myk Melez [:myk] [@mykmelez] from comment #5)
> Stephen: is it possible that an executable in the Resources/ directory of an
> application bundle is treated differently from one in the MacOS/ directory,
> such that the OS doesn't enable it to open native windows (or other native
> graphic elements)?

Interesting question, and not one I have an immediate answer to. It sounds like this is happening with OSX 10.8 and above? Does it change anything if you select "Anywhere" under System Preferences > Security & Privacy > Allow apps downloaded from?

Does the System Console print anything when this is happening? I think you mentioned that Terminal doesn't, is this also true for System Console?
Flags: needinfo?(spohl.mozilla.bugs)
(In reply to Stephen Pohl [:spohl] from comment #6)
> (In reply to Myk Melez [:myk] [@mykmelez] from comment #5)
> > Stephen: is it possible that an executable in the Resources/ directory of an
> > application bundle is treated differently from one in the MacOS/ directory,
> > such that the OS doesn't enable it to open native windows (or other native
> > graphic elements)?
> 
> Interesting question, and not one I have an immediate answer to. It sounds
> like this is happening with OSX 10.8 and above?

I'm on 10.9.5; unsure about others.


> Does it change anything if
> you select "Anywhere" under System Preferences > Security & Privacy > Allow
> apps downloaded from?

That's the current value of that pref on my system, and I'm still experiencing the problem.


> Does the System Console print anything when this is happening? I think you
> mentioned that Terminal doesn't, is this also true for System Console?

The only message in System Console that wasn't in Terminal is this one, which seems unrelated:

2015/02/16 4:04:32.000 PM kernel[0]: webapprt-stub (map: 0xffffff80308a6d20) triggered DYLD shared region unnest for map: 0xffffff80308a6d20, region 0x7fff90800000->0x7fff90a00000. While not abnormal for debuggers, this increases system memory footprint until the target exits.
Here is the output I get when I'm trying to launch this with a local debug build. Does this match what you're seeing?

spohl:mozilla-central Stephen$ ./mach webapprt-test-chrome
From _tests: Kept 34797 existing; Added/updated 0; Removed 0 files and 0 directories.
SUITE-START | Running 17 tests
pk12util: PKCS12 IMPORT SUCCESSFUL
MochitestServer : launching [u'/Users/Stephen/Documents/objdir-ff-debug/dist/bin/xpcshell', '-g', u'/Users/Stephen/Documents/objdir-ff-debug/dist/NightlyDebug.app/Contents/Resources', '-v', '170', '-f', u'/Users/Stephen/Documents/objdir-ff-debug/dist/bin/components/httpd.js', '-e', "const _PROFILE_PATH = '/var/folders/pv/q6h74svd5cz7l8t2r3nx492c0000gn/T/tmpkAkS77.mozrunner'; const _SERVER_PORT = '8888'; const _SERVER_ADDR = '127.0.0.1'; const _TEST_PREFIX = undefined; const _DISPLAY_RESULTS = false;", '-f', '/Users/Stephen/Documents/objdir-ff-debug/_tests/testing/mochitest/server.js']
runtests.py | Server pid: 79363
runtests.py | Websocket server pid: 79364
runtests.py | SSL tunnel pid: 79365
[79363] WARNING: Re-registering a CID?: file /Users/Stephen/Documents/mozilla-central/xpcom/components/nsComponentManager.cpp, line 531
runtests.py | Running tests: start.

runtests.py | Application pid: 79366
2015-02-17 13:13:41.241 webapprt-stub[79366:14533547] MY WEBAPPRT BUILDID: 20150217131224
2015-02-17 13:13:41.243 webapprt-stub[79366:14533547] found override firefox binary: (null)
2015-02-17 13:13:41.243 webapprt-stub[79366:14533547] SEARCHING for webapprt, trying: org.mozilla.nightly
2015-02-17 13:13:41.246 webapprt-stub[79366:14533547] USING FIREFOX : /Users/Stephen/Documents/objdir-ff-release/dist/Nightly.app
2015-02-17 13:13:41.247 webapprt-stub[79366:14533547] Looking for firefox ini file here: /Users/Stephen/Documents/objdir-ff-release/dist/Nightly.app/Contents/Resources/application.ini
2015-02-17 13:13:41.249 webapprt-stub[79366:14533547] FIREFOX WEBAPPRT BUILDID: 20150213214513
2015-02-17 13:13:41.249 webapprt-stub[79366:14533547] ### This Application has an old webrt. Updating it.
2015-02-17 13:13:41.249 webapprt-stub[79366:14533547] ### My webapprt path: /Users/Stephen/Documents/objdir-ff-debug/dist/NightlyDebug.app/Contents/Resources/webapprt
2015-02-17 13:13:41.249 webapprt-stub[79366:14533547] ### Trying Firefox webapprt path: /Users/Stephen/Documents/objdir-ff-release/dist/Nightly.app/Contents/Resources/webapprt-stub
2015-02-17 13:13:41.254 webapprt-stub[79366:14533547] ### Successfully updated webapprt, relaunching
2015-02-17 13:13:41.254 webapprt-stub[79366:14533547]  launching webrt at path: /Users/Stephen/Documents/objdir-ff-debug/dist/NightlyDebug.app/Contents/Resources/webapprt
2015-02-17 13:13:41.255 webapprt-stub[79366:14533547] COMMAND LINE: '/Users/Stephen/Documents/objdir-ff-debug/dist/NightlyDebug.app/Contents/Resources/webapprt /Users/Stephen/Documents/objdir-ff-debug/dist/NightlyDebug.app/Contents/Resources/webapprt'
2015-02-17 13:13:41.269 webapprt[79366:14533547] MY WEBAPPRT BUILDID: 20150213214513
2015-02-17 13:13:41.270 webapprt[79366:14533547] found override firefox binary: (null)
2015-02-17 13:13:41.270 webapprt[79366:14533547] SEARCHING for webapprt, trying: org.mozilla.nightly
2015-02-17 13:13:41.272 webapprt[79366:14533547] USING FIREFOX : /Users/Stephen/Documents/objdir-ff-release/dist/Nightly.app
2015-02-17 13:13:41.272 webapprt[79366:14533547] Looking for firefox ini file here: /Users/Stephen/Documents/objdir-ff-release/dist/Nightly.app/Contents/Resources/application.ini
2015-02-17 13:13:41.272 webapprt[79366:14533547] FIREFOX WEBAPPRT BUILDID: 20150213214513
2015-02-17 13:13:41.272 webapprt[79366:14533547] This Application has the newest webrt.  Launching!
2015-02-17 13:13:41.272 webapprt[79366:14533547] Set XUL_APP_FILE to: /Users/Stephen/Documents/objdir-ff-debug/dist/NightlyDebug.app/Contents/Resources/Contents/MacOS/webapp.ini
2015-02-17 13:13:41.367 webapprt[79366:14533547] WebappRT application.ini path: /Users/Stephen/Documents/objdir-ff-release/dist/Nightly.app/Contents/Resources/webapprt/webapprt.ini
2015-02-17 13:13:41.372 webapprt[79366:14533547] /Users/Stephen/Documents/objdir-ff-debug/dist/NightlyDebug.app/Contents/Resources/Contents/MacOS/webapp.ini was not found
2015-02-17 13:13:41.373 webapprt[79366:14533547] got exception: Unable to parse environment files for application startup
TEST-INFO | Main app process: exit 0

runtests.py | Application ran for: 0:00:00.171457
zombiecheck | Reading PID log: /var/folders/pv/q6h74svd5cz7l8t2r3nx492c0000gn/T/tmpCq4yM4pidlog
Stopping web server
Stopping web socket server
Stopping ssltunnel
WARNING | leakcheck | refcount logging is off, so leaks can't be detected!
runtests.py | Running tests: end.
SUITE-END | took 2s
At first glance I see two issues. One is known, one might be responsible for the issue in this bug.

(In reply to Stephen Pohl [:spohl] from comment #8)
> 2015-02-17 13:13:41.243 webapprt-stub[79366:14533547] SEARCHING for
> webapprt, trying: org.mozilla.nightly
> 2015-02-17 13:13:41.246 webapprt-stub[79366:14533547] USING FIREFOX :
> /Users/Stephen/Documents/objdir-ff-release/dist/Nightly.app
> 2015-02-17 13:13:41.247 webapprt-stub[79366:14533547] Looking for firefox
> ini file here:
> /Users/Stephen/Documents/objdir-ff-release/dist/Nightly.app/Contents/
> Resources/application.ini
> 2015-02-17 13:13:41.249 webapprt-stub[79366:14533547] FIREFOX WEBAPPRT
> BUILDID: 20150213214513
> 2015-02-17 13:13:41.249 webapprt-stub[79366:14533547] ### This Application
> has an old webrt. Updating it.
> 2015-02-17 13:13:41.249 webapprt-stub[79366:14533547] ### My webapprt path:
> /Users/Stephen/Documents/objdir-ff-debug/dist/NightlyDebug.app/Contents/
> Resources/webapprt
> 2015-02-17 13:13:41.249 webapprt-stub[79366:14533547] ### Trying Firefox
> webapprt path:
> /Users/Stephen/Documents/objdir-ff-release/dist/Nightly.app/Contents/
> Resources/webapprt-stub
> 2015-02-17 13:13:41.254 webapprt-stub[79366:14533547] ### Successfully
> updated webapprt, relaunching

The way we scan for an updated webapprt stub here causes us to use an incorrect webapprt-stub. We should have been using the one in objdir-ff-debug. Instead, we think that the one in objdir-ff-release is the correct one. This is the known issue that I mentioned above.


> 2015-02-17 13:13:41.272 webapprt[79366:14533547] This Application has the
> newest webrt.  Launching!
> 2015-02-17 13:13:41.272 webapprt[79366:14533547] Set XUL_APP_FILE to:
> /Users/Stephen/Documents/objdir-ff-debug/dist/NightlyDebug.app/Contents/
> Resources/Contents/MacOS/webapp.ini

This path is definitely incorrect. A few lines later, this results in:

> 2015-02-17 13:13:41.372 webapprt[79366:14533547]
> /Users/Stephen/Documents/objdir-ff-debug/dist/NightlyDebug.app/Contents/
> Resources/Contents/MacOS/webapp.ini was not found
> 2015-02-17 13:13:41.373 webapprt[79366:14533547] got exception: Unable to
> parse environment files for application startup
> TEST-INFO | Main app process: exit 0
(In reply to Stephen Pohl [:spohl] from comment #9)
> The way we scan for an updated webapprt stub here causes us to use an
> incorrect webapprt-stub. We should have been using the one in
> objdir-ff-debug. Instead, we think that the one in objdir-ff-release is the
> correct one. This is the known issue that I mentioned above.

Is there a bug filed for this known issue?  It should be marked as a blocker for this bug.  It will not be practical to have to uninstall all other versions of Firefox to run the test harness.
Flags: needinfo?(spohl.mozilla.bugs)
(In reply to Nick Desaulniers [:\n] from comment #10)
> (In reply to Stephen Pohl [:spohl] from comment #9)
> > The way we scan for an updated webapprt stub here causes us to use an
> > incorrect webapprt-stub. We should have been using the one in
> > objdir-ff-debug. Instead, we think that the one in objdir-ff-release is the
> > correct one. This is the known issue that I mentioned above.
> 
> Is there a bug filed for this known issue?  It should be marked as a blocker
> for this bug.  It will not be practical to have to uninstall all other
> versions of Firefox to run the test harness.

Well, I called it an issue but it's been termed a feature at other times. Myk, I think you'd be much better at explaining why we do this.
Flags: needinfo?(spohl.mozilla.bugs) → needinfo?(myk)
(In reply to Stephen Pohl [:spohl] from comment #11)
> Well, I called it an issue but it's been termed a feature at other times.
> Myk, I think you'd be much better at explaining why we do this.

We do it because the mechanism for finding the path to an application on Mac OS X is to search for it by its CFBundleIdentifier via NSWorkspace:absolutePathForAppBundleWithIdentifier <https://developer.apple.com/library/mac/documentation/Cocoa/Reference/ApplicationKit/Classes/NSWorkspace_Class/index.html#//apple_ref/occ/instm/NSWorkspace/absolutePathForAppBundleWithIdentifier:>.

This is because the path to an application can change, since you can drag an application bundle to any location on your filesystem.  So we can't simply assume that the bundle exists in a canonical location.

Of course we might be able to assume that for tests, but I chatted about this issue with smichaud a while ago, and we ultimately determined that bug 914754 would be the best way to address it.  See that bug for more details about the problem and proposed solution.
Flags: needinfo?(myk)
This should work, but I'm going to do some more testing before requesting review.
Assignee: nobody → myk
Status: NEW → ASSIGNED
This is essentially the same as patch v1, but with some minor style improvements, plus a fix for the XRE path on Linux.

The problem with running the runtime tests on Mac is that the webapprt-stub executable has been moved to the Resources/ subdirectory of the Nightly app bundle, and Mac OS apparently only displays windows for executables in the MacOS/ directory of an app bundle.

This fix copies the executable to a new TestApp bundle, which it uses to run the tests.  Ideally, the build system would copy the executable, but it isn't clear how to do that, so I've made the mach command do the copying (which also means that we delay copying the executable to the TestApp bundle until you run a test suite that needs it).

Requesting review from Marco for the webapprt/ changes and from Joel for the testing/ changes.

Joel: you're one of two test harness peers <https://wiki.mozilla.org/Modules/All#Test_Harness> with "mochitest" in your list of specialties, and robcee doesn't seem to be around anymore, but let me know if someone else should be looking at these changes.
Attachment #8601019 - Attachment is obsolete: true
Attachment #8601333 - Flags: review?(mar.castelluccio)
Attachment #8601333 - Flags: review?(jmaher)
Comment on attachment 8601333 [details] [diff] [review]
patch v2: fix style nits and XRE path on Linux

Review of attachment 8601333 [details] [diff] [review]:
-----------------------------------------------------------------

I don't understand the need for webapp.ini, and info.plist is foreign to me.  The rest looks good.

Keep in mind that bug 1155338 will be landing soon (today) on mozilla-inbound and cause a conflict/bitrot with this due to changes in mach_commands.py.
Attachment #8601333 - Flags: review?(jmaher) → review+
Comment on attachment 8601333 [details] [diff] [review]
patch v2: fix style nits and XRE path on Linux

Review of attachment 8601333 [details] [diff] [review]:
-----------------------------------------------------------------

The webapprt/ changes look good to me. Have you tested the patch on Windows as well?

The "Info.plist" file is the same we'd create on install. We could modify it at
some point and we would need to keep the testing one up to date. It shouldn't
really be a problem though, because, even if we forgot, it'd hardly affect the
results of the tests.
Attachment #8601333 - Flags: review?(mar.castelluccio) → review+
(In reply to Marco Castelluccio [:marco] from comment #16)
> Have you tested the patch on Windows as well?

Not yet, I'll do so next.


In the meantime, I've pushed to try:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=976b9c917d39


> The "Info.plist" file is the same we'd create on install. We could modify it
> at
> some point and we would need to keep the testing one up to date. It shouldn't
> really be a problem though, because, even if we forgot, it'd hardly affect
> the
> results of the tests.

Indeed.  There might be ways to make this better (generating the file at test time via the app installer, or basing it on a template that the app installer also uses), but I'm leery of their complexity.
(In reply to Myk Melez [:myk] [@mykmelez] from comment #17)
> (In reply to Marco Castelluccio [:marco] from comment #16)
> > Have you tested the patch on Windows as well?
> 
> Not yet, I'll do so next.

The tests are failing on Windows for an unrelated reason, bug 1162196.
(In reply to Joel Maher (:jmaher) from comment #15)
> I don't understand the need for webapp.ini, and info.plist is foreign to me.
> The rest looks good.

Apps that run in the runtime have a webapp.ini file with app-specific configuration.  Typical properties are Name (which the runtime uses to display the name of the app in various native interfaces) and Profile, which the runtime uses to find the app's data profile.  For the Test App, we only need to specify the Name, as the Profile is determined by the mochitest runner.

Info.plist is a Mac-specific file that is required to be in an app bundle.  Firefox creates one dynamically for apps it installs, but we can specify one statically for the Test App.

More info about the file format here: https://developer.apple.com/library/ios/documentation/General/Reference/InfoPlistKeyReference/Articles/AboutInformationPropertyListFiles.html


> Keep in mind that bug 1155338 will be landing soon (today) on
> mozilla-inbound and cause a conflict/bitrot with this due to changes in
> mach_commands.py.

This patch contains the same changes as patch v2, it just resolves the conflicts (no bitrot) with those changes.  This is the version I'll land once Inbound reopens.
https://hg.mozilla.org/mozilla-central/rev/9110fe02f095
Status: ASSIGNED → RESOLVED
Closed: 5 years ago
Resolution: --- → FIXED
Target Milestone: --- → Firefox 40
Cool, the tests now complete for me, though an error window is left open and I see the tests not picking the correct host version of FF to use:

2015-05-07 16:00:31.559 webapprt[77448:1929764] found override firefox binary: org.mozilla.nightly
2015-05-07 16:00:31.561 webapprt[77448:1929764] USING FIREFOX : /Applications/FirefoxNightly.app
(In reply to Nick Desaulniers [:\n] from comment #22)
> Cool, the tests now complete for me, though an error window is left open and
> I see the tests not picking the correct host version of FF to use:
> 
> 2015-05-07 16:00:31.559 webapprt[77448:1929764] found override firefox
> binary: org.mozilla.nightly
> 2015-05-07 16:00:31.561 webapprt[77448:1929764] USING FIREFOX :
> /Applications/FirefoxNightly.app

This is a different issue, for which I've filed bug 1168737.
Product: Firefox → Firefox Graveyard
You need to log in before you can comment on or make changes to this bug.