Closed Bug 797164 Opened 12 years ago Closed 9 years ago

Add support for mochitest-chrome to B2G

Categories

(Testing :: Mochitest, defect, P1)

Product:
Testing
Component:
Mochitest
Platform:
All
Gonk (Firefox OS)
Type:
defect

Tracking

(blocking-basecamp:+)

Status:
RESOLVED FIXED
Project Flags:
blocking-basecamp +

People

(Reporter: jgriffin, Assigned: jgriffin)

References

Details

(Whiteboard: [LOE:M])

Attachments

(3 files, 6 obsolete files)

add "mochitest-chrome-remote" mach command
4.17 KB, patch
ahal
: review+
Details | Diff | Splinter Review
push chrome test files to device
6.26 KB, patch
ahal
: review+
Details | Diff | Splinter Review
instead of loading via browser-test-overlay.xul, simply load harness on b2g
4.62 KB, patch
ahal
: review+
Details | Diff | Splinter Review 
Looks like we'll need to add support for mochitest-chrome for B2G.
Noming since this blocks getting the existing updater mochitests running for B2G
blocking-basecamp: --- → ?
Blocking all these suites is important enough to block on IMHO.
blocking-basecamp: ? → +
jgriffin: I'm potentially interested in helping here. Can you provide more details? ping on IRC if needed.
Flags: needinfo?(jgriffin)
(In reply to Gregory Szorc [:gps] from comment #3)
> jgriffin: I'm potentially interested in helping here. Can you provide more
> details? ping on IRC if needed.

mochitest-plain on B2G works by using Marionette to navigate to the mochitest test page here:  http://mxr.mozilla.org/mozilla-central/source/build/mobile/b2gautomation.py#271

I imagine for mochitest chrome, we'd do something similar, except that instead of loading the test page into the existing content window, we'd load it in place of shell.xul.  This probably would look like this:

self.marionette.set_context(self.marionette.CONTEXT_CHROME)
self.marionette.navigate(self.testURL)

Performing the navigation in the 'chrome' context causes us to replace the primary XUL page that's currently loaded, which in the case of B2G is shell.xul.

Btw, mochitest-plain is currently broken on B2G due to bug 798580.  This probably won't impact efforts to get mochitest-chrome working, AFAICT.
Flags: needinfo?(jgriffin)
It turns out, of course, that things aren't quite so simple.  I've been looking into this and am running into this error:

E/GeckoConsole(  429): [JavaScript Error: "TypeError: linkArray is undefined" {file: "chrome://mochikit/content/server.js" line: 534}]

Investigation continues.
The problem is that when the chrome harness tries to build a test list, it takes the url chrome://mochitests/content/mochitests.xul and resolves it into a local file url.  However, the way it does this (http://mxr.mozilla.org/mozilla-central/source/testing/mochitest/chrome-harness.js#32) resolves to a file url on the machine used to compile the build, not the device (e.g., in my case file:///home/jgriffin/armB2G/objdir-gecko/_tests/testing/mochitest/mochitests.xul).

I'd been working under the assumption that mochitest-chrome was run at some point in the past on mobile, but maybe I'm wrong.  Do you know Joel?  If it did, I'm not sure how it worked around this.  If it didn't, then we'll have to adapt this somehow for the remote case.

Either way it's more work than I thought.
mochitest-chrome has never run on android.  We did run it on n900, but that was a local machine.  Likewise mochitest-browser-chrome has never run remotely.

Is there a need for mochitest-chrome?  iirc, many of the tests are 'Firefox' specific as they have been rolled into that suite for higher privileges.
(In reply to Joel Maher (:jmaher) from comment #7)
> mochitest-chrome has never run on android.  We did run it on n900, but that
> was a local machine.  Likewise mochitest-browser-chrome has never run
> remotely.
> 
> Is there a need for mochitest-chrome?  iirc, many of the tests are 'Firefox'
> specific as they have been rolled into that suite for higher privileges.

Unfortunately, there is a need.  QA assures me that some app permissions tests cannot be run in mochitest-plain, and marshall_law wants mochitest-chrome support for update tests.
I would suggest copying the test files to the device and we could hack the harness to make it work with a different root directory.
I thought all the work you did to run mochitest-chrome from a jar file (bug 543800) was in support of running it on mobile? In any event, that's how they're run nowadays, so it should just be a matter of getting the jar file installed to the profile and maybe some fiddling with the harness. The tests all run from chrome: URLs which map back to jar URLs.
All P1s need an assignee. Please re-assign or change the LOE as appropriate, jgriffin.
Assignee: nobody → jgriffin
Priority: -- → P1
Whiteboard: [LOE:M]
Ah, I see.  The chrome JAR file only gets made during 'make package-tests', so it isn't possible to run remote chrome tests via 'make mochitest-chrome', or directly via the runner in the objdir/_tests/testing/mochitest dir.
you could generate the .jar file from 'make mochitest-chrome' and copy it to the device/emulator without much trouble.
Depends on: 805430
No longer blocks: 787578
It appears there are no longer any use cases requiring this.  If one appears, we'll re-open this.
Status: NEW → RESOLVED
Closed: 12 years ago
Resolution: --- → WONTFIX
We have a use case now. See bug 1021060 comment 16 and onward.
Status: RESOLVED → REOPENED
Flags: needinfo?(jgriffin)
Resolution: WONTFIX → ---
Bug 1021060 makes it sound as if mochitest-chrome on Android is a higher priority than on B2G.  I'll plan for this in early Q3, but we won't be able to implement in the next couple of weeks.
Flags: needinfo?(jgriffin)
That's because requestAutocomplete is being implemented for desktop and android at the moment. Bug 964295 tracks a Firefox OS UI but I don't know of the timeline on that. Does a separate bug need to be filed for Android?

Perhaps there should be a newsgroup discussion on whether it's worth the infra resources to run chrome tests on every platform. Since we seem to run mochitest-plain everywhere it would seems like we think it is worth it since they aren't that different in terms of what they're testing.
I would question the value of mochitest-chrome on android.  I suspect about half of the test cases will not work due to lack of features (note: different UI).  Is this worth the effort?  Likewise will we have many chrome cases for b2g?
Half is a lot of tests though. I suspect a lot of the chrome tests in dom/, content/, and some in toolkit/ aren't related to the UI and could run on mobile. I'm not sure how common it would be for these tests to fail on mobile but not desktop.
The issue, I think, is that some tests need to execute some setup code in chrome before they do something in content.  SpecialPowers works ok in mochitest-plain for some cases, but the more complex the chrome setup code required, the harder it is to use SpecialPowers.

Yes, we should file a separate bug for mochitest-chrome on Android, since that's a completely different code path, harness-wise.  And yes, I think a newsgroup thread is appropriate; I volunteer to start one tomorrow.
please be clear that we will be adding harness support for mochitest-chrome to android/b2g instead of turning on mochitest-chrome which indicates the majority of the chrome tests.
(In reply to Joel Maher (:jmaher) from comment #21)
> please be clear that we will be adding harness support for mochitest-chrome
> to android/b2g instead of turning on mochitest-chrome which indicates the
> majority of the chrome tests.

Agreed.
See Also: → 1026290
(In reply to Joel Maher (:jmaher) from comment #18)
> I would question the value of mochitest-chrome on android.  I suspect about
> half of the test cases will not work due to lack of features (note:
> different UI).

Aside from tests written before the introduction of mochitest-browser, what kind of mochitest-chrome tests would be depending on UI feature? The point of mochitest-chrome is to run privileged stuff _without_ the dependency on the Firefox browser-chrome (which is why, for example, we can run them on Thunderbird CI).
Most of the mochitest-chrome tests won't work in B2G because they use .xul files, which aren't well-supported in B2G.  I don't know what level of support we have for .xul files in Android.
I would be happy to require/encourage that new mochitest-chrome tests use .html, and write a PSA to dev-platform. Most mochitest-chrome files just use .xul by convention.

Does the standard mochitest-plain template (created via gen_template.pl) work out-of-the-box in mochitest-chrome?
Probably not, chrome tests need chrome:// url's for SimpleTest.js and friends, whereas the mochitest-plain template uses a relative url.
(In reply to Jonathan Griffin (:jgriffin) from comment #26)
> Probably not, chrome tests need chrome:// url's for SimpleTest.js and
> friends, whereas the mochitest-plain template uses a relative url.

Ok. Can we switch the template generated by gen_template.pl -type chrome to spit out an html file?
Yes, just filed bug 1026785 for this.
Unassigning since I'm not actively working on this.  We're in the process of deciding whether we should pick this up for Q3.
Assignee: jgriffin → nobody
See https://groups.google.com/d/msg/mozilla.dev.platform/rdaX9q33nGc/R-qQbiG2My4J

jgriffin, can we get an update on this?
Flags: needinfo?(jgriffin)
We haven't started yet, and might not be able to until early Q4.
Flags: needinfo?(jgriffin)
(In reply to Jonathan Griffin (:jgriffin) from comment #31)
> We haven't started yet, and might not be able to until early Q4.

Ok - good to know! As long as it happens sometime in the not-too-distant future, I can feel a lot better about writing mochitest-chrome tests. :-)
Assignee: nobody → gbrown
Depends on: 1026290
See Also: 1026290 → 
I'm treating 1026290 as a dependency -- getting chrome tests working on Android first. I'm making good progress there and hope to apply what I'm learning to b2g very soon now.
We can run mochitest-chrome on Android now (via runtestsremote.py with --chrome specified); not all tests pass, but all tests can be run! There were 3 issues on Android:
 - there was an initialization race sometimes preventing the "contentEvent" from being received
 - logFile (and maybe other) parameters were not passed in testConfig.js
 - chrome test files were missing from device and tests.manifest was pointing to a non-remote location.

If I address these issues for b2g, mochitest-chrome still fails. On b2g, redirect.html is loaded and sends the contentEvent, but contentEvent is never received:

http://hg.mozilla.org/mozilla-central/annotate/29d086b32a26/testing/mochitest/browser-test.js#l72

In fact, it appears that browser-test.js is never loaded.

browser-test.js is in mochikit.jar and referenced from browser-test-overlay.xul - http://hg.mozilla.org/mozilla-central/annotate/29d086b32a26/testing/mochitest/browser-test-overlay.xul#l10, which is also in mochikit.jar.

mochikit.jar is pushed to device at /data/local/tests/profile/extensions/staged/mochikit@mozilla.org. Unexpectedly, I find that it appears to never be moved from "staged"...is mochikit.jar pushed but somehow not used on b2g?

(Also, :jmaher noted that mochitest-chrome on desktop Firefox behaves very similarly with --e10s: redirect.html is loaded but nothing else happens; we wonder if there is a common cause.)
Another possible point of failure: browser-test-overlay.xul is referenced from the mochikit chrome.manifest, which would normally contain:

  overlay chrome://browser/content/browser.xul chrome://mochikit/content/browser-test-overlay.xul
  overlay chrome://browser/content/shell.xhtml chrome://mochikit/content/browser-test-overlay.xul
  overlay chrome://navigator/content/navigator.xul chrome://mochikit/content/browser-test-overlay.xul
  overlay chrome://webapprt/content/webapp.xul chrome://mochikit/content/browser-test-overlay.xul

I've tried variations such as 

  overlay chrome://b2g/content/shell.html chrome://mochikit/content/browser-test-overlay.xul

but I'm not certain what this should be.
mochikit seems to be copied to /system/b2g/distribution/bundles/mochikit@mozilla.org, complete with chrome.manifest and the jar.
For B2G and mochitest-plain, we run the tests in a child process, since we run them inside an out-of-process Gaia app, 'test-container'.  In the context of this app, shell.html doesn't exist (it only exists for the top-level context).

> I've tried variations such as 
> 
>   overlay chrome://b2g/content/shell.html chrome://mochikit/content/browser-test-overlay.xul

That looks right, but I'm also not positive that chrome overlays work in B2G.  Let's figure out whether we need these tests to run in the parent process or in a child process, and then we can figure out how to get there.

Bobby, should mochitest-chrome for B2G run tests in the top-level process, or in an OOP app?
Flags: needinfo?(bobbyholley)
They should run in the parent process (so that they have access to the usual non-sandboxed stuff), and they should not be using XUL or overlays at all. They should simply be html-based - just like mochitest-plain, but loaded from a chrome:// URI rather than an http:// URI.
Flags: needinfo?(bobbyholley)
I think what we should do then, gbrown, is make a version of shell.html that we package with the mochitest-browser files that loads the same .js files as browser-test-overlay.xul.  We can then use Marionette to navigate to that file.
Thanks jgriffin.

First I checked to see what would happen if I tried loading those .js files from <script> tags in shell.html itself. That reports:

I/GeckoConsole(   44): No chrome package registered for chrome://mochikit/content/chrome-harness.js
I/GeckoConsole(   44): No chrome package registered for chrome://mochikit/content/browser-test.js
I/GeckoConsole(   44): No chrome package registered for chrome://mochikit/content/cc-analyzer.js
I/GeckoConsole(   44): No chrome package registered for chrome://mochikit/content/mochitest-e10s-utils.js

but appears to successfully load those files and call the onLoad handler in browser-test.js.
 
I also tried loading from b2g_start_script.js via loadSubScript(). That reports no errors, but the onLoad handler in browser-test.js is not called.
I found a minor change to browser-test.js was needed (but this does not solve the problem from comment 41):

diff --git a/testing/mochitest/browser-test.js b/testing/mochitest/browser-test.js
index a989d15..e6da999 100644
--- a/testing/mochitest/browser-test.js
+++ b/testing/mochitest/browser-test.js
@@ -5,7 +5,11 @@ var gConfig;
 
 if (Cc === undefined) {
   var Cc = Components.classes;
+}
+if (Ci === undefined) {
   var Ci = Components.interfaces;
+}
+if (Cu === undefined) {
   var Cu = Components.utils;
 }
(In reply to Geoff Brown [:gbrown] from comment #41)
> Thanks jgriffin.
> 
> First I checked to see what would happen if I tried loading those .js files
> from <script> tags in shell.html itself. That reports:
> 
> I/GeckoConsole(   44): No chrome package registered for
> chrome://mochikit/content/chrome-harness.js
> I/GeckoConsole(   44): No chrome package registered for
> chrome://mochikit/content/browser-test.js
> I/GeckoConsole(   44): No chrome package registered for
> chrome://mochikit/content/cc-analyzer.js
> I/GeckoConsole(   44): No chrome package registered for
> chrome://mochikit/content/mochitest-e10s-utils.js
> 
> but appears to successfully load those files and call the onLoad handler in
> browser-test.js.
>  
> I also tried loading from b2g_start_script.js via loadSubScript(). That
> reports no errors, but the onLoad handler in browser-test.js is not called.

If you load it via loadSubScript(), I think you'll need to then call testInit(); I wouldn't expect the onLoad handler to be triggered.
After talking with :ahal, I'm calling loadSubScript() from b2g_start_script to load the js normally loaded via browser-test-overlay.xul. Instead of relying on the load and MozAfterPaint events to call testInit(), I'm calling it directly -- that seems to work okay.

In testInit(), we wait for "contentEvent", normally sent from redirect.html; that's never received, presumably because of e10s. I've hacked around that, invoking that with a setTimeout instead.

That allows me to load the harness, and start to parse the test list, but then I start hitting various js errors. All of those problems seem to relate back to SpecialPowers being undefined. I wouldn't expect SpecialPowers to be needed, but it turns out there are various references: http://mxr.mozilla.org/mozilla-central/source/testing/mochitest/tests/SimpleTest/setup.js#105.

I can't quite account for why SpecialPowers is undefined in this environment while it doesn't seem to cause problems anywhere else.
I don't know if you included the ChromePowers.js script? You might have to need it, it's sort of a mock object for specialpowers in chrome:// scope.
+  loader.loadSubScript("chrome://mochikit/content/tests/SimpleTest/specialpowersAPI.js");
+  loader.loadSubScript("chrome://mochikit/content/tests/SimpleTest/SpecialPowersObserverAPI.js");
+  loader.loadSubScript("chrome://mochikit/content/tests/SimpleTest/ChromePowers.js");
+  loader.loadSubScript("chrome://mochikit/content/chrome-harness.js");
+  loader.loadSubScript("chrome://mochikit/content/browser-test.js");
+  loader.loadSubScript("chrome://mochikit/content/cc-analyzer.js");
+  loader.loadSubScript("chrome://mochikit/content/mochitest-e10s-utils.js");

12-18 00:10:05.645 I/GeckoDump(  562): ⰲ겿{"action":"log","time":1418861405646,"thread":"","js_source":"TestRunner","pid":null,"source":"mochitest","level":"INFO","message":"SimpleTest START"}ⰲ겿
12-18 00:10:05.655 I/GeckoDump(  562): TEST-UNEXPECTED-FAIL: setup.js | error parsing http://mochi.test:8888/tests.json (ReferenceError: SpecialPowers is not defined)
12-18 00:10:05.655 W/GeckoConsole(  562): [JavaScript Error: "ReferenceError: SpecialPowers is not defined" {file: "chrome://mochikit/content/tests/SimpleTest/TestRunner.js" line: 530}]
Don't you also need to load specialpowers.js?
Shouldn't you perhaps use loader.loadSubScript("chrome://specialpowers/content/specialpowersAPI.js")
instead of chrome://mochikit/content/tests/SimpleTest/?
Thanks much for the ideas, but that seems neither better nor worse:

+  loader.loadSubScript("chrome://specialpowers/content/specialpowersAPI.js");
+  loader.loadSubScript("chrome://specialpowers/content/SpecialPowersObserverAPI.js");
+  loader.loadSubScript("chrome://specialpowers/content/specialpowers.js");
+  loader.loadSubScript("chrome://specialpowers/content/ChromePowers.js");
+  loader.loadSubScript("chrome://mochikit/content/chrome-harness.js");
+  loader.loadSubScript("chrome://mochikit/content/browser-test.js");
+  loader.loadSubScript("chrome://mochikit/content/cc-analyzer.js");
+  loader.loadSubScript("chrome://mochikit/content/mochitest-e10s-utils.js");

12-18 00:33:36.404 I/GeckoDump(  563): ⰲ겿{"action":"log","time":1418862816407,"thread":"","js_source":"TestRunner","pid":null,"source":"mochitest","level":"INFO","message":"SimpleTest START"}ⰲ겿
12-18 00:33:36.414 I/GeckoDump(  563): TEST-UNEXPECTED-FAIL: setup.js | error parsing http://mochi.test:8888/tests.json (ReferenceError: SpecialPowers is not defined)
12-18 00:33:36.414 W/GeckoConsole(  563): [JavaScript Error: "ReferenceError: SpecialPowers is not defined" {file: "chrome://mochikit/content/tests/SimpleTest/TestRunner.js" line: 530}]
So, SpecialPowers is already loaded by Marionette, so you may be double-loading it, which could be bad.  To make use of SpecialPowers in a Marionette call, you have to use special_powers = True, e.g., http://dxr.mozilla.org/mozilla-central/source/testing/marionette/client/marionette/tests/unit/test_import_script.py#30
(In reply to Jonathan Griffin (:jgriffin) from comment #49)
> Marionette call, you have to use special_powers = True, e.g.,

I tried that too -- no change.

Then I realized that the code complaining about SpecialPowers was being loaded from http://hg.mozilla.org/mozilla-central/annotate/5c7a6294b82a/testing/mochitest/harness.xul#l24; loading special powers from the harness made a big difference: tests are running now!

There are still some troubling errors being reported:

0 INFO SimpleTest START
1 INFO TEST-START | chrome://mochitests/content/chrome/caps/tests/mochitest/test_bug995943.xul
TEST-INFO | specialpowers.js |  Failed to attach specialpowers to window exception: TypeError: SpecialPowers is not a constructor
TEST-INFO | specialpowers.js |  Failed to attach specialpowers to window exception: TypeError: SpecialPowers is not a constructor
2 INFO Invoking go for window with id: 10
3 INFO TEST-PASS | chrome://mochitests/content/chrome/caps/tests/mochitest/test_bug995943.xul | Should only call go once! 
4 INFO Invoking checkLoadFileURI with domain: http://example.com, shouldLoad: false
5 INFO Navigating iframe to http://example.com/tests/js/xpconnect/tests/mochitest/file_empty.html
JavaScript error: chrome://specialpowers/content/SpecialPowersObserver.js, line 103: NS_ERROR_NOT_INITIALIZED: Component returned failure code: 0xc1f30001 (NS_ERROR_NOT_INITIALIZED) [nsIMessageSender.sendAsyncMessage]
###################################### forms.js loaded
############################### browserElementPanning.js loaded
######################## BrowserElementChildPreload.js loaded
6 INFO TEST-UNEXPECTED-FAIL | chrome://mochitests/content/chrome/caps/tests/mochitest/test_bug995943.xul | Test timed out. - expected PASS
7 INFO TEST-OK | chrome://mochitests/content/chrome/caps/tests/mochitest/test_bug995943.xul | took 301138ms
TEST-INFO | specialpowers.js |  Failed to attach specialpowers to window exception: TypeError: SpecialPowers is not a constructor

but this is a lot better than I have seen previously on b2g.
Note that we don't expect any tests with .xul extensions to run on B2G; the only tests that will work are .js tests.
Nice! But I'm still confused as to why mochitest-chrome even needs specialpowers in the first place? Would you like help investigating this? If so, do you mind uploading what you have so far?
(In reply to Jonathan Griffin (:jgriffin) from comment #51)
> Note that we don't expect any tests with .xul extensions to run on B2G; the
> only tests that will work are .js tests.

.js tests? Don't you mean .html tests?

(In reply to Andrew Halberstadt [:ahal] from comment #52)
> Nice! But I'm still confused as to why mochitest-chrome even needs
> specialpowers in the first place?

Because there are certain utility functions built into SpecialPowers (like pushPrefEnv) which are useful to use, even in privileged code.
special powers has a sister call chromepowers() which mimics specialpowers functionality for chrome tests.  This way we can author tests and share libraries which depend on special powers.
(In reply to Bobby Holley (Busy with media, don't ask for DOM/JS/XPConnect things) from comment #53)
> (In reply to Jonathan Griffin (:jgriffin) from comment #51)
> > Note that we don't expect any tests with .xul extensions to run on B2G; the
> > only tests that will work are .js tests.
> 
> .js tests? Don't you mean .html tests?
> 
Yes, sorry...I mean .html tests are the only ones we expect to be able to run on B2G.
This is how I'm running mochitest-chrome on b2g. It basically sets options.chrome and then does everything the same as "mochitest-remote".
As noted in the comment, there's a hack here; I need to re-work getChromeTestDir to make this cleaner.
Attached patch wip - harness hacks (obsolete) — — Splinter Review 
These are the changes I used to kick off the harness load and get the results I showed in Comment 50. Obviously this needs work!

When I updated my repo today and re-tested these changes, they stopped working -- tests hang and I don't see any errors. I don't know where this is going wrong.
(In reply to Andrew Halberstadt [:ahal] from comment #52)
> Nice! But I'm still confused as to why mochitest-chrome even needs
> specialpowers in the first place? Would you like help investigating this? If
> so, do you mind uploading what you have so far?

I agree with the sentiments of Bobby and Joel: We're sharing code between plain mochitests and chrome -- specialpowers dependencies are a natural consequence. I'm not very concerned about that.

I *am* concerned that I need to load specialpowers when I don't on other platforms -- I think that's a sign that tests are accidentally running in a different environment, and I think that's a consequence of the terrible hacks in my 4th patch. Suggestions for how to avoid these hacks and accomplish the harness load in a reasonable way are more than welcome!
(In reply to Geoff Brown [:gbrown] from comment #59)
> When I updated my repo today and re-tested these changes, they stopped
> working -- tests hang and I don't see any errors. I don't know where this is
> going wrong.

Actually, I think there is just a long delay, rather than an infinite hang. 

$ ./mach mochitest-chrome-remote layout/base/tests/chrome/test_bug370436.html
...
0 INFO SimpleTest START
1 INFO TEST-START | layout/base/tests/chrome/test_bug370436.html
TEST-INFO | specialpowers.js |  Failed to attach specialpowers to window exception: TypeError: SpecialPowers is not a constructor
TEST-INFO | specialpowers.js |  Failed to attach specialpowers to window exception: TypeError: SpecialPowers is not a constructor
2 INFO must wait for load
3 INFO TEST-PASS | layout/base/tests/chrome/test_bug370436.html | Shouldn't have SOWs in chrome 
4 INFO TEST-PASS | layout/base/tests/chrome/test_bug370436.html | Shouldn't have SOWs in chrome 
5 INFO TEST-PASS | layout/base/tests/chrome/test_bug370436.html | Shouldn't have SOWs in chrome 
6 INFO TEST-PASS | layout/base/tests/chrome/test_bug370436.html | Shouldn't have SOWs in chrome 
7 INFO TEST-PASS | layout/base/tests/chrome/test_bug370436.html | Shouldn't have SOWs in chrome 
8 INFO TEST-PASS | layout/base/tests/chrome/test_bug370436.html | Shouldn't have SOWs in chrome 
9 INFO TEST-PASS | layout/base/tests/chrome/test_bug370436.html | Shouldn't have SOWs in chrome 
10 INFO TEST-PASS | layout/base/tests/chrome/test_bug370436.html | Word 1 selected correctly 
11 INFO TEST-PASS | layout/base/tests/chrome/test_bug370436.html | Word 2 selected correctly 
12 INFO TEST-PASS | layout/base/tests/chrome/test_bug370436.html | Word 3 selected correctly 
13 INFO TEST-PASS | layout/base/tests/chrome/test_bug370436.html | Word 4 selected correctly 
14 INFO TEST-PASS | layout/base/tests/chrome/test_bug370436.html | Word 5 selected correctly 
15 INFO TEST-PASS | layout/base/tests/chrome/test_bug370436.html | Word 6 selected correctly 
16 INFO TEST-PASS | layout/base/tests/chrome/test_bug370436.html | Word 7 selected correctly 
17 INFO MEMORY STAT vsize after test: 150593536
18 INFO MEMORY STAT vsizeMaxContiguous not supported in this build configuration.
19 INFO MEMORY STAT residentFast after test: 55218176
20 INFO MEMORY STAT heapAllocated after test: 25556628
21 INFO TEST-OK | layout/base/tests/chrome/test_bug370436.html | took 1722ms
22 INFO TEST-START | Shutdown
23 INFO Passed:  14
24 INFO Failed:  0
25 INFO Todo:    0
26 INFO Slowest: 1721ms - chrome://mochitests/content/chrome/layout/base/tests/chrome/test_bug370436.html
27 INFO SimpleTest FINISHED
28 INFO TEST-INFO | Ran 1 Loops
29 INFO SimpleTest FINISHED
-*- NetworkService: NetworkService shutdown
[Parent 352] WARNING: waitpid failed pid:359 errno:10: file /home/gbrown/B2G2/gecko/ipc/chromium/src/base/process_util_posix.cc, line 267
JavaScript error: resource://gre/modules/IndexedDBHelper.jsm, line 121: UnknownError: The operation failed for reasons unrelated to the database itself and not covered by any other error code.
TEST-INFO | leakcheck | default process: leak threshold set at 5180 bytes
TEST-INFO | leakcheck | plugin process: leak threshold set at 0 bytes
TEST-INFO | leakcheck | tab process: leak threshold set at 2000000 bytes
TEST-INFO | leakcheck | geckomediaplugin process: leak threshold set at 20000 bytes
TEST-INFO | leakcheck | default process: ignoring missing output line for total leaks
Stopping web server
Stopping web socket server
Stopping ssltunnel
runtestsb2g.py | Running tests: end.
SUITE-END | took 164s
Adds the mochitest-chrome-remote mach command and avoids a check that prevents specifying a test path for chrome.
Attachment #8538800 - Attachment is obsolete: true
Comment on attachment 8538866 [details] [diff] [review]
wip - add "mochitest-chrome-remote" mach command

Review of attachment 8538866 [details] [diff] [review]:
-----------------------------------------------------------------

::: testing/mochitest/mach_commands.py
@@ +843,5 @@
>                  xre_path=self.xre_path, test_paths=test_paths, **kwargs)
>  
> +    @Command('mochitest-chrome-remote', category='testing',
> +        description='Run a remote mochitest-chrome.',
> +        conditions=[conditions.is_b2g, is_emulator])

This is fine for now since mochitest-plain does the same thing, but I regret starting this pattern and filed bug 1046992 to move away from it. You have enough on your plate so no need to worry about it now, we can fix it as part of the other bug.
(In reply to Geoff Brown [:gbrown] from comment #61)
> (In reply to Geoff Brown [:gbrown] from comment #59)
> > When I updated my repo today and re-tested these changes, they stopped
> > working -- tests hang and I don't see any errors. I don't know where this is
> > going wrong.
> 
> Actually, I think there is just a long delay, rather than an infinite hang. 

Actually, I was just anxious and not reading logs well. It takes a while to start the emulator and push files, but the test execution is fine.
Aha! 

I was getting the nsIWebNavigation reference from the wrong window, when loading the harness. If instead I use:

  let homescreen = document.getElementById('systemapp');
  var webNav = homescreen.contentWindow.QueryInterface(Ci.nsIInterfaceRequestor)
                                   .getInterface(Ci.nsIWebNavigation);
  webNav.loadURI(url, null, null, null, null);

then things work better: 1. The harness and test are visible in the emulator; 2. The specialpowers hacks are no longer necessary. Yeah!
As noted earlier, this is probably not the right long-term solution, but bug 1046992 should clear that up.

For now, I just want *some* simple way to start mochitest-chrome tests on B2G.

My default test command line is:

./mach mochitest-chrome-remote layout/base/tests/chrome/test_bug370436.html
Attachment #8538866 - Attachment is obsolete: true
Attachment #8540760 - Flags: review?(ahalberstadt)
This pushes the chrome test files to the remote device.

An unfortunate complication arises because the location of the chrome test files is written to the profile. The profile is passed to Marionette, so the profile must be built before using the device manager initialized in the Marionette runner. But the device manager should be queried to determine the remote test root used for the location of the chrome test files. In this patch, I provide a dummy location for the chrome test files until self.app_ctx is initialized, and update the profile with the correct location after Marionette is initialized.
Attachment #8538807 - Attachment is obsolete: true
Attachment #8540765 - Flags: review?(ahalberstadt)
Simplified.
Attachment #8538811 - Attachment is obsolete: true
Attachment #8538813 - Attachment is obsolete: true
Comment on attachment 8540760 [details] [diff] [review]
add "mochitest-chrome-remote" mach command

Review of attachment 8540760 [details] [diff] [review]:
-----------------------------------------------------------------

Cool, lgtm!
Attachment #8540760 - Flags: review?(ahalberstadt) → review+
I've tried to keep this simple and avoid impacting plain mochitests on b2g or mochitest-chrome on other platforms: Pass chrome option to b2g_start_script; if b2g_start_script called with chrome option, load browser-test.js and call b2gStart() to load harness and start chrome tests.
Attachment #8540801 - Attachment is obsolete: true
Attachment #8540866 - Flags: review?(ahalberstadt)
Comment on attachment 8540765 [details] [diff] [review]
push chrome test files to device

Review of attachment 8540765 [details] [diff] [review]:
-----------------------------------------------------------------

Looks good. I'm assuming this isn't the whole thing needed to get it working.. There should be some changes to b2g_start_script.js or a new script that is similar, right?

::: testing/mochitest/runtestsb2g.py
@@ +190,5 @@
>                  Services.io.offline = false;
>                  """)
>  
> +            if options.chrome:
> +                self.app_ctx.dm.removeDir(self.remote_chrome_test_dir)

Shouldn't self.remote_chrome_test_dir get defined and set to None in __init__ or in the class namespace? Otherwise this will raise if there have been no calls to getChromeTestDir() yet.

@@ +248,5 @@
> +        # writing the dummy.
> +        if hasattr(self, 'app_ctx'):
> +            self.remote_chrome_test_dir = posixpath.join(self.app_ctx.remote_test_root, 'chrome');
> +            return self.remote_chrome_test_dir
> +        return 'dummy-chrome-test-dir'

I haven't looked into this problem yet, so maybe this is the best way forward, but modifications to mozrunner and/or marionette are totally on the table if there's a cleaner fix involving changes to those.
Attachment #8540765 - Flags: review?(ahalberstadt) → review+
Comment on attachment 8540866 [details] [diff] [review]
instead of loading via browser-test-overlay.xul, simply load harness on b2g

Review of attachment 8540866 [details] [diff] [review]:
-----------------------------------------------------------------

::: testing/mochitest/b2g_start_script.js
@@ +89,5 @@
>  
> +if (chrome) {
> +  let loader = Cc["@mozilla.org/moz/jssubscript-loader;1"].getService(Ci.mozIJSSubScriptLoader);
> +  loader.loadSubScript("chrome://mochikit/content/browser-test.js");
> +  b2gStart();

Is b2gStart what actually kicks off the tests? If so, should we avoid setting container.src = mochitestUrl below?
Attachment #8540866 - Flags: review?(ahalberstadt) → review+
(In reply to Andrew Halberstadt [:ahal] from comment #71)
> Shouldn't self.remote_chrome_test_dir get defined and set to None in
> __init__ or in the class namespace? Otherwise this will raise if there have
> been no calls to getChromeTestDir() yet.

Good idea.

> @@ +248,5 @@
> > +        # writing the dummy.
> > +        if hasattr(self, 'app_ctx'):
> > +            self.remote_chrome_test_dir = posixpath.join(self.app_ctx.remote_test_root, 'chrome');
> > +            return self.remote_chrome_test_dir
> > +        return 'dummy-chrome-test-dir'
> 
> I haven't looked into this problem yet, so maybe this is the best way
> forward, but modifications to mozrunner and/or marionette are totally on the
> table if there's a cleaner fix involving changes to those.

I thought about trying to call down to get_dm in b2g.py, but that's really buried under several layers. I also considered pulling the devicemanager access out of the runner and into some separate class, but I didn't think that effort was warranted when I could work around it.
(In reply to Andrew Halberstadt [:ahal] from comment #72)
> Is b2gStart what actually kicks off the tests? If so, should we avoid
> setting container.src = mochitestUrl below?

Yes, b2gStart kicks off the tests. Setting container.src has not caused me any trouble, but I agree it seems unsafe...I'll avoid it in the final patch.
Thank you for doing this!

Does this mean that any .html mochitest-chrome test will now run by default on b2g? If so, that's worth an announcement on dev-platform.
Flags: needinfo?(gbrown)
Once a job is created and scheduled on the CI system.  These changes allow for mochitest-chrome tests to be run on b2g.  I know this was developed/tested on an emulator, so all b2g builds/devices we test on could take an extra fix or two.
Blocks: 1116187
(In reply to Bobby Holley (Busy with media, don't ask for DOM/JS/XPConnect things) from comment #78)
> Does this mean that any .html mochitest-chrome test will now run by default on b2g?

With these patches, my claim is that we can run mochitest-chrome tests on b2g emulator via 

mach mochitest-chrome-remote

I've spot-tested a few .html tests with good results, but I have not tested extensively. I would expect some (many?) tests to fail -- it seems like every test suite needs some TLC before it will run reliably on a new platform.

I think :jgriffin has the right idea with bug 1116187: Let's see this running on try next.
Flags: needinfo?(gbrown)
No longer blocks: 1116187
Depends on: 1116187
Depends on: 1142641
Assigning to myself for the final scheduling bits.
Assignee: gbrown → jgriffin
https://groups.google.com/forum/#!topic/mozilla.dev.platform/mF9wyTQqA7U
Status: REOPENED → RESOLVED
Closed: 12 years ago → 9 years ago
Resolution: --- → FIXED
Keywords: leave-open
Depends on: 1146973
On bug 1171917 I'm adding a new chrome test for B2G. If I push to try and use the following try syntax 'try: -b o -p emulator -u mochitest-chrome -t none' the results show up on try. On the other hand when using 'try: -b do -p all -u all -t none' before landing I do not see the chrome tests results. Are these b2g chrome test results checked when landing (I mean checking in)? Thanks!
Flags: needinfo?(jgriffin)
Sorry, I see them. I didn't notice the results show up when landing as well. Thanks!
Flags: needinfo?(jgriffin)
Component: Mochitest Chrome → Mochitest
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: