1016030 - gaia marionette tests fail to run on Mulet

Status: RESOLVED FIXED

(Firefox OS Graveyard :: Runtime, defect)

Description

[Alexandre Poirot [:ochameau]]

Today, if you try to run gaia marionette/integration tests you will have many errors as the test runner expects to find shell.html as top level window, whereas it gets browser.xul when running the mulet. And even before that, I haven't found a way to tell the test runner to use another binary other than gaia/b2g/b2g-bin.

Let's do whatever is necessary to be able to run these tests on Mulet.

Comment 1

[Alexandre Poirot [:ochameau]]

1) Do I miss something or there is no way to specify a custom b2g binary to use to bin/gaia-marionette and/or marionette-mocha?

I tried to tweak marionette-js-runner/bin/marionette-mocha to accept a new --runtime argument,
but I ended being lost in abstraction between marionette-mocha and mozilla-runner/lib/detectbinary.js which end up selecting the actual binary we end up using.

From marionette-mocha options object I stopped walking through the call tree to marionette-js-runner/lib/childrunner.js: profileOptions which seems to filter the "hostOptions" object being passed to marionette-b2gdesktop-host. "seems" at it looks like something else is filtering after that.

2) Otherwise, I think I will keep it simple for the first step and start with running tests as on b2g desktop. i.e.  with shell.html being the top level window. In order to do that I have to pass an additional argument to the mulet binary. Would you be ok to add a mulet specific in marionette-b2gdesktop-host/index.js
Something like:
  if (mulet)
    userOptions.chrome = 'chrome://b2g/content/shell.html';
(and then patch mozilla-runner/lib/run.js to support 'chrome' attribute)

Then I would most likely pass some option from marionette-mocha to the host in order to pass a mulet option of something.


Otherwise with these few tweaks in nodejs code and the few blocking patches in blocking bugs, I was able to see gaia tests running on mulet!

Comment 2

[James Lal [:lightsofapollo]]

I would do this by adding another host (like this https://github.com/mozilla-b2g/marionette-b2gdesktop-host) then you can use it by passing:

```js
marionette-mocha --host npmmodulename
```

Copy pasting the b2g desktop host probably is a reasonable starting point... I think for what your trying to do the code should be pretty easy.

Comment 3

[Alexandre Poirot [:ochameau]]

It doesn't reply to the first point. How can I specify a custom runtime path? If we can't, how can we implement such marionette-mocha argument?

Comment 4

[James Lal [:lightsofapollo]]

The host sets up the logic to find the "product" if you use something similar https://github.com/mozilla-b2g/marionette-b2gdesktop-host/blob/master/index.js#L80 this will search for the binaries named $product-bin and $product in the two locations we place them on osx/linux.

Again I don't think adding options to the marionette-mocha thing is what you want to do adding this as another host will give you better control and allow you to set your own defaults... I will try to catch you on irc.

Comment 5

[Alexandre Poirot [:ochameau]]

me on irc, "I don't want to set a new kind of default location for mulet binaries, I just want to pass a one time argument to specify the mulet binary living in my mozilla-central objdir. marionette code can't automagically find it unless I give a path to it."

Also is there anyone in europe or east US that I can ping about this code?

Comment 6

[James Lal [:lightsofapollo]]

I don't think so... Even in Taipei knows quite a bit though. I can hack together something that bypasses the path for b2g I think that is more or less.

Comment 7

[James Lal [:lightsofapollo]]

Evan** as in evanxd my mistake above!

Comment 8

[Alexandre Poirot [:ochameau]]

Attachment: b2gdesktop-host patch

I'm really not sure it is worth creating yet another project/repo, just for one line difference. May be later, if we have to introduce more mulet specifics, we can introduce one. Also keep in mind mulet is meant to replace b2g desktop, so we are either going to deprecate this host in favor of a new one, or migrate this one over to mulet.

Also, this patch isn't enough. I have another mozilla-central patch for server side marionette code in order to be able to run them on mulet.

Comment 9

[Alexandre Poirot [:ochameau]]

(also as noted on the pull request, this patch is based on top of bug 1019152 patch)

Comment 10

[Gareth Aye [:gaye] (back from PTO)]

I went ahead and looked at your work over in https://bugzilla.mozilla.org/show_bug.cgi?id=1019152. Does this still need attention?

Comment 11

[Alexandre Poirot [:ochameau]]

(In reply to Gareth Aye [:gaye] from comment #10)
> I went ahead and looked at your work over in
> https://bugzilla.mozilla.org/show_bug.cgi?id=1019152. Does this still need
> attention?

Yes! bug 1019152 doesn't make marionette tests to work with mulet. It just help running tests on a custom gecko binary. attachment 8436976 [details] [review] is a first step toward supporting mulet. As said in comment 8 I also need a patch against marionette-server.js from mozilla-central.

Comment 12

[Gareth Aye [:gaye] (back from PTO)]

https://github.com/mozilla-b2g/marionette-b2gdesktop-host/commit/1d6966f59b8611d7ffcf6f673e641e5770c5198b landed. There are some test issues both here and in marionette-js-runner that I would like to track down before we publish new versions and update gaia-node-modules. Will try to get to it ASAP. Thanks :ochameau!

Comment 13

[Alexandre Poirot [:ochameau]]

Attachment: Tune marionette-server for mulet support

Jonathan, We made it so that various test suites on Mulet, like gaia integration, gaia ui tests
are running Mulet with shell.html as top level window.
It makes it way easier to support the mulet,
as the mulet looks like b2g desktop.

We made that by passing "-chrome chrome://b2g/content/shell.html" as firefox-bin argument.

Now in order to make it works, I have to convinced marionette-server.js
that we are running on B2G and expect shell.html as top level window.

Here is a patch to do so. I'm bit sad to make this JS be preprocessed just for that.
One alternative could be to check for some runtime value...
We may be able to do:
  var appName = (isMulet() ? "B2G" : Services.appinfo.name);

Comment 14

[Jonathan Griffin (:jgriffin)]

I'd rather not add preprocessing here, because in some contexts (like Firefox OS Certification Suites), we install Marionette as an extension, and it would be nice not to have B2G and non-B2G versions of that.

How easy is it to do:

  var appName = (isMulet() ? "B2G" : Services.appinfo.name);

Comment 15

[Jonathan Griffin (:jgriffin)]

Comment on attachment 8453081 [details] [diff] [review]
Tune marionette-server for mulet support

Review of attachment 8453081 [details] [diff] [review]:
-----------------------------------------------------------------

See comment #14.

Comment 16

[Alexandre Poirot [:ochameau]]

Attachment: Set MOZ_APP_BASENAME=mulet to ease identifying mulet builds

Unfortunately, it ends up breaking the build itself and I can't get why...
  https://tbpl.mozilla.org/?tree=Try&rev=f96f35e92dfe
12:18:14    ERROR -  mozinstall.mozinstall.InvalidBinary: "/builds/slave/test/build/application/mulet" does not contain a valid binary.
12:18:14    ERROR - Return code: 1
12:18:14    FATAL - Halting on failure while running ['/builds/slave/test/build/venv/bin/mozinstall', '/builds/slave/test/build/installer.tar.bz2', '--destination', '/builds/slave/test/build/application']

(I get a mulet and mulet-bin binaries in my dist/bin...)

Comment 17

[Alexandre Poirot [:ochameau]]

Attachment: Identify mulet via a pref instead.

This patch looks more like a workaround, but at least it doesn't rely on preprocessing.

https://tbpl.mozilla.org/?tree=Try&rev=75c0bd6d8c01

Comment 18

[Alexandre Poirot [:ochameau]]

Attachment: Identify mulet via a pref instead.

Comment 19

[Jonathan Griffin (:jgriffin)]

If you want to change the binary name to mulet, we'll have to modify the mozharness scripts to expect that; currently they expect a binary name of firefox or b2g, and mozinstall fails when it doesn't find that.

Comment 20

[Alexandre Poirot [:ochameau]]

Do you think that would be easy to rename the binary?
If we just have to tweak a few build files that could be worth to differentiate mulet a bit more.

I'm bit lost in mozharness. Also it looks like it would require some coordination between gecko and build script landings?

Would it be only about tweaking stage_product and/or product_name over here?
http://mxr.mozilla.org/build/source/buildbot-configs/mozilla/config.py#699

Comment 21

[Jonathan Griffin (:jgriffin)]

Changing the name wouldn't necessarily be easy, but it might be worth the reduced confusion in the future.

It would require landings in mozinstall, mozharness, and buildbot, at least, and there may be places that would need to change I'm not aware of.  It's not possible to coordinate landings in all of these things precisely, so there would be some amount of breakage while the change was rolled out in different systems.

Comment 22

[Jonathan Griffin (:jgriffin)]

Is this patch actually still needed?  I fixed this a different way, here:

http://hg.mozilla.org/mozilla-central/diff/e1601237530e/testing/marionette/marionette-server.js

The problem is that a Mulet build can act like a Firefox build (if you're not running a Gaia profile) or like a B2G build (if you are) from Marionette's perspective.  Your patch would prevent Marionette from operating correctly on the build in "Firefox mode", but if we don't care about "Firefox mode", then your patch is cleaner.

Comment 23

[Alexandre Poirot [:ochameau]]

Oh right, I didn't knew you already fixed that!
I think we should sync up about what you did while I was away.

Comment 24

[Jonathan Griffin (:jgriffin)]

Comment on attachment 8468529 [details] [diff] [review]
Identify mulet via a pref instead.

Removing review flag since I think there's no need to land this; ping me if I'm wrong.

Comment 25

[Alexandre Poirot [:ochameau]]

Attachment: Fix gaia's devtools test on mulet

I have no idea why it does work on b2g desktop... but dbg-server.jsm has a wrong URL in this gaia test... We always use resource://gre/modules/devtools/dbg-server.jsm URL in gecko codebase.

Comment 26

[Alexandre Poirot [:ochameau]]

Comment on attachment 8468529 [details] [diff] [review]
Identify mulet via a pref instead.

Patch no longer needed!

Comment 27

[J. Ryan Stinnett [:jryans] (Use needinfo, replies may be slow)]

Comment on attachment 8474546 [details] [review]
Fix gaia's devtools test on mulet

That is perplexing!  Thanks for fixing.  Hopefully Gaia-Try will confirm that it works for b2g as well.

Comment 28

[Alexandre Poirot [:ochameau]]

Attachment: marionette report

With the gaia patch apply (to fix devtools test) I got only 7 failures:
 265 passing (50m)
  45 pending
   7 failing

1) Find My Device lock > Lock the screen through the test app:
2) Vertical - Collection Pin Bookmark "before each" hook:
3) Vertical - Collection Pin Bookmark "after each" hook:
4) Vertical - Collection Pin Bookmark "after all" hook:
5) Vertical Home - Packaged App Pending app is displayed until determined role=homescreen:
6) Desktop Notifications should be cleared after opening app:
7) Dialer > Keypad Entering a 3 digits number with the keypad:

Comment 29

[Alexandre Poirot [:ochameau]]

Ok so there is some intermittent in this list, a second run gave me:

265 passing (41m)
  45 pending
  5 failing

1) Find My Device lock > Lock the screen through the test app:
2) Share as ringtone Share activity Create new ringtone:
3) Vertical - Collection Pin Bookmark pins the bookmarked result:
4) Desktop Notifications should be cleared after opening app:
5) Dialer > Keypad Entering a 3 digits number with the keypad:

The number of passing are odd as there is the same number of passing tests but less failing!!

So here is the list of tests that failed on both runs:
1) Find My Device lock > Lock the screen through the test app:
3) Vertical - Collection Pin Bookmark pins the bookmarked result: (not 100% sure that's the exact same test)
4) Desktop Notifications should be cleared after opening app:
5) Dialer > Keypad Entering a 3 digits number with the keypad:

Comment 30

[Alexandre Poirot [:ochameau]]

Comment on attachment 8474546 [details] [review]
Fix gaia's devtools test on mulet

master:
https://github.com/mozilla-b2g/gaia/commit/af25e2e9ee6c563e4768ebc7c1da1c5a4c661927