894162 - webapprt-test-chrome|content tests hang after opening WebappRT Test Shim

Status: RESOLVED FIXED

(Firefox Graveyard :: Webapp Runtime, defect)

Description

[Myk Melez [:myk] [@mykmelez]]

After building, I ran the webapprt-test-chrome tests, which hung after opening the WebappRT Test Shim window (webapprt-test-content tests similarly hang):

  ./mach build
  …
  make -C obj-x86_64-apple-darwin12.4.0/ webapprt-test-chrome
  07-15 17:20 > make -C obj-x86_64-apple-darwin12.4.0/ webapprt-test-chrome 
  Build configuration changed. Regenerating backend.
  …
  rm -f ./webapprt-test-chrome.log && /Users/myk/Mozilla/central/obj-x86_64-apple-darwin12.4.0/_virtualenv/bin/python _tests/testing/mochitest/runtests.py --autorun --close-when-done --console-level=INFO --log-file=./webapprt-test-chrome.log --file-level=INFO --failure-file=/Users/myk/Mozilla/central/obj-x86_64-apple-darwin12.4.0/_tests/testing/mochitest/makefailures.json --testing-modules-dir=/Users/myk/Mozilla/central/obj-x86_64-apple-darwin12.4.0/_tests/modules --extra-profile-file=./dist/plugins --symbols-path=./dist/crashreporter-symbols   --webapprt-chrome --appname ./dist/Nightly.app/Contents/MacOS/webapprt-stub --browser-arg -test-mode
  …
  INFO | runtests.py | Running tests: start.
  …
  2013-07-15 18:04:36.442 webapprt-stub[9965:707] MY WEBAPPRT BUILDID: 20130715171742
  2013-07-15 18:04:36.443 webapprt-stub[9965:707] found override firefox binary: (null)
  2013-07-15 18:04:36.444 webapprt-stub[9965:707] USING FIREFOX : /Users/myk/Mozilla/central/obj-x86_64-apple-darwin12.4.0/dist/Nightly.app
  2013-07-15 18:04:36.444 webapprt-stub[9965:707] Looking for firefox ini file here: /Users/myk/Mozilla/central/obj-x86_64-apple-darwin12.4.0/dist/Nightly.app/Contents/MacOS/application.ini
  2013-07-15 18:04:36.445 webapprt-stub[9965:707] FIREFOX WEBAPPRT BUILDID: 20130715171742
  2013-07-15 18:04:36.445 webapprt-stub[9965:707] This Application has the newest webrt.  Launching!
  2013-07-15 18:04:36.446 webapprt-stub[9965:707] Set XUL_APP_FILE to: /Users/myk/Mozilla/central/obj-x86_64-apple-darwin12.4.0/dist/Nightly.app/Contents/MacOS/webapp.ini
  2013-07-15 18:04:36.457 webapprt-stub[9965:707] WebappRT application.ini path: /Users/myk/Mozilla/central/obj-x86_64-apple-darwin12.4.0/dist/Nightly.app/Contents/MacOS/webapprt/webapprt.ini
  2013-07-15 18:04:36.458 webapprt-stub[9965:707] Profile specified with -profile: /var/folders/lp/8t_7y24119720_hjp5wc_4cr0000gn/T/tmp0rwOax/
  Server listening on port 4443 with cert pgo server certificate
  2013-07-15 18:04:36.662 webapprt-stub[9965:707] *** WARNING: Method userSpaceScaleFactor in class NSWindow is deprecated on 10.7 and later. It should not be used in new applications. Use convertRectToBacking: instead.

Comment 1

[Myk Melez [:myk] [@mykmelez]]

The rest of that output (after the test run times out):

  TEST-UNEXPECTED-FAIL | automation.py | application timed out after 330 seconds with no output
  args: ['/usr/sbin/screencapture', '-C', '-x', '-t', 'png', '/var/folders/lp/8t_7y24119720_hjp5wc_4cr0000gn/T/mozilla-test-fail_OtRLHT']
  libpng warning: zero length keyword
  libpng warning: Empty language field in iTXt chunk
  SCREENSHOT: data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAC…
  Can't trigger Breakpad, just killing process
  INFO | automation.py | Application ran for: 0:05:30.789174
  INFO | zombiecheck | Reading PID log: /var/folders/lp/8t_7y24119720_hjp5wc_4cr0000gn/T/tmpXr4zBRpidlog
  WARNING | leakcheck | refcount logging is off, so leaks can't be detected!
  
  INFO | runtests.py | Running tests: end.
  make: *** [webapprt-test-chrome] Error 247

Comment 2

[Marco Castelluccio [:marco]]

Looks like the problem is an incorrect content-type for the manifest.

Comment 3

[Marco Castelluccio [:marco]]

Attachment: Un-hang

This is going to remove the hangs.

In the chrome tests, there are REINSTALL_FORBIDDEN errors.
In the content test, getSelf is returning a null value, and *I think* that's because we aren't associating the <browser> element to an app principal.

I haven't looked at the errors in detail, so further investigation is necessary.
The chrome tests aren't failing anymore with my local patch queue, so I hope my patch queue contains something that actually fixes them and not that cheats them.

Why don't we run these tests automatically?

Comment 4

[Marco Castelluccio [:marco]]

(In reply to Marco Castelluccio [:marco] from comment #3)
> In the content test, getSelf is returning a null value, and *I think* that's
> because we aren't associating the <browser> element to an app principal.

My hypothesis was correct. This will be fixed by bug 892837.

Comment 5

[Marco Castelluccio [:marco]]

(In reply to Marco Castelluccio [:marco] from comment #3)
> In the chrome tests, there are REINSTALL_FORBIDDEN errors.

These fail because we don't allow reinstalls (we used to allow reinstalls when we introduced the tests).
With my local patch queue they're working because we're allowing multiple apps per origin.

Comment 6

[Marco Castelluccio [:marco]]

Myk, so while the content tests will be fixed when we'll correctly set the principal appId, the chrome tests could be fixed in different ways:
1) Allow multiple apps per origin
2) Skip the reinstallation check while we're testing
3) Serve apps from different origins (it's ok for now, but what if we add a lot of tests?)
4) With two origins: serve the "container" app from an origin and all the actual test apps from the other origin, uninstall each test app after each test.

Comment 7

[Myk Melez [:myk] [@mykmelez]]

Comment on attachment 776195 [details] [diff] [review]
Un-hang

Looks good, r=myk!


> Why don't we run these tests automatically?

I'm not sure, but probably we simply haven't gone through the process yet.  The web runtime tests use a new harness, and adding a harness to Mozilla's continuous integration framework requires some work on the part of the Release Engineering team to test and integrate the harness.


> Myk, so while the content tests will be fixed when we'll correctly set the 
> principal appId, the chrome tests could be fixed in different ways:
> 1) Allow multiple apps per origin

We'll do this soon enough, but I hope we can fix the tests in the meantime.


> 2) Skip the reinstallation check while we're testing

Hmm, I'd prefer to avoid skipping checks as much as possible.  The closer the test environment is to the real environment, the more reliable the tests are.


> 3) Serve apps from different origins (it's ok for now, but what if we add a lot of tests?)

There are a number of domains available for serving apps:

https://developer.mozilla.org/en-US/docs/Mochitest#How_do_I_test_issues_which_only_show_up_when_tests_are_run_across_domains.3F
http://mxr.mozilla.org/mozilla-central/source/build/pgo/server-locations.txt

And we can add more.  But not all those domains are appropriate to reuse, and adding domains is cumbersome.


> 4) With two origins: serve the "container" app from an origin and all the actual test apps from the other origin, uninstall each test app after each test.

This seems like the best option.  Besides scaling with the number of tests, it also encourages tests to clean up after themselves, which is valuable in and of itself (lack of such cleanup leads to issues that are subtle and difficult to diagnose).

Comment 8

[Myk Melez [:myk] [@mykmelez]]

Setting checkin-needed while waiting for Inbound to reopen.

Comment 9

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/0791d8310519

Comment 10

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/0791d8310519