1019152 - Add an option to run integration test on a custom gecko

Status: RESOLVED FIXED

(Firefox OS Graveyard :: Gaia::TestAgent, defect)

Description

[Alexandre Poirot [:ochameau]]

Unless I miss something, I haven't found any way to run gaia-integration-tests on a custom b2g. It seems to always use gaia/b2g/b2g-bin.
It's not handy for platform developers, having various b2g in objdirs.
Nor for starting working on mulet which isn't integrated yet into gaia build/test scripts.

It would be super handy to have an additional argument on bin/gaia-marionette, like "--runtime ~/obdir/dist/bin/b2g-bin" in order to run on the given binary.

Comment 1

[Alexandre Poirot [:ochameau]]

James, do you think you can work on this or mentor me on this story?

Comment 2

[Julien Wajsberg [:julienw]]

Gareth can probably help too.

Comment 3

[Gareth Aye [:gaye] (back from PTO)]

I can definitely help with this... will add some info later today

Comment 4

[Gareth Aye [:gaye] (back from PTO)]

So I think that this is already possible on a per-test basis without any modifications to the marionette-js-runner code via

```js
marionette.client({
  hostOptions: {
    runtime: '/path/to/your/b2g/binary'
  }
});
```

If you want to do this on a suite-wide basis, you can monkey-patch this line https://github.com/mozilla-b2g/marionette-b2gdesktop-host/blob/master/index.js#L8.

If you don't like the idea of monkey-patching, you might need to do a bit more hacking to fill in this TODO https://github.com/mozilla-b2g/marionette-js-runner/blob/master/lib/childrunner.js#L314.

Our marionette-b2gdesktop-host https://github.com/mozilla-b2g/marionette-b2gdesktop-host/blob/master/index.js#L35 can also take global options with a path it will use by default if a given call to marionette.client() does not specify. Essentially, you would add another option like `--runtime` to the ones that get parsed here https://github.com/mozilla-b2g/marionette-js-runner/blob/master/bin/marionette-mocha#L95 and then pass it down from marionette-mocha to lib/parentrunner.js to lib/childrunner.js and then to the call to the Host constructor.

Does that help?

Comment 5

[Gareth Aye [:gaye] (back from PTO)]

That being said (and I don't know too much about mulet), but you may want to take a look at https://github.com/mozilla-b2g/mozilla-runner which is a node wrapper for shelling out to some gecko.

Comment 6

[Alexandre Poirot [:ochameau]]

(In reply to Gareth Aye [:gaye] from comment #4)
> So I think that this is already possible on a per-test basis without any
> modifications to the marionette-js-runner code via
> 
> If you want to do this on a suite-wide basis, you can monkey-patch this line
> https://github.com/mozilla-b2g/marionette-b2gdesktop-host/blob/master/index.
> js#L8.

It won't help gecko developers. We need a way to run any gaia test on a custom gecko,
typically, a binary coming from mozilla-central objdir.
This feature request isn't much related to mulet, it will just help me test it!

> 
> If you don't like the idea of monkey-patching, you might need to do a bit
> more hacking to fill in this TODO
> https://github.com/mozilla-b2g/marionette-js-runner/blob/master/lib/
> childrunner.js#L314.
> 
> Our marionette-b2gdesktop-host
> https://github.com/mozilla-b2g/marionette-b2gdesktop-host/blob/master/index.
> js#L35 can also take global options with a path it will use by default if a
> given call to marionette.client() does not specify. Essentially, you would
> add another option like `--runtime` to the ones that get parsed here
> https://github.com/mozilla-b2g/marionette-js-runner/blob/master/bin/
> marionette-mocha#L95 and then pass it down from marionette-mocha to
> lib/parentrunner.js to lib/childrunner.js and then to the call to the Host
> constructor.

That's more or less what I've identified in bug 1016030 comment 1.
I ended up being lost in abstraction. There is "options" objects everywhere
and it is pretty challenging to trace who is who!
But your description helped me figure out my way.

I ended up hacking that:
  https://github.com/mozilla-b2g/gaia-node-modules/pull/39

I would like to first submit patches for this particular bug, the --runtime argument.
Does that look good to you?
Would you accept these patches if I submit them to marionette-js-runner, marionette-b2gdesktop-host, mozilla-runner?
(are your reviewer for all these projects?)
It is quite painfull to have to submit such small patches, all tightly related to different repo :/

Then, in bug 1016030, I'll discuss with you guys about introducing a new host or not. I'm not convinced we should introduce a new one. The sole modification made to b2g host is this line https://github.com/mozilla-b2g/gaia-node-modules/pull/39/files#diff-83e4cb07e17ada66312f55e26dcf6057R82

Comment 7

[Gareth Aye [:gaye] (back from PTO)]

Yep either James or I can review. The reason we have so many repos is actually because we wanted to have very loose coupling between the different components. It does make things a bit of a hassle when you want to make changes though. Send pull requests to each of those repos, and then let's continue on GH. Thanks!

Comment 8

[Alexandre Poirot [:ochameau]]

Attachment: marionette-js-runner patch

Comment 9

[Alexandre Poirot [:ochameau]]

Attachment: b2gdesktop-host patch

Comment 10

[Alexandre Poirot [:ochameau]]

Attachment: mozilla-runner

Comment 11

[Gareth Aye [:gaye] (back from PTO)]

Aside from a few nits, this all looks great to me. Thanks :ochameau!

Comment 12

[Alexandre Poirot [:ochameau]]

I updated the pull requests but I see some failure on travis for marionette-js-runner and b2gdesktop-host patches.
I'm not sure it is related to my patch, is it some intermittents?

Comment 13

[Gareth Aye [:gaye] (back from PTO)]

I just landed

marionette-b2gdesktop-host:
https://github.com/mozilla-b2g/marionette-b2gdesktop-host/commit/e9ceead931e80e58d212c0d3e6774a1b1f8a3cb6

mozilla-runner:
https://github.com/mozilla-b2g/mozilla-runner/commit/5198e043e52445759668b7f45a6a8a56370f46f9

The marionette-b2gdesktop-host failure was definitely unrelated to your patch. I'll take a look to see what's going on there later today. I do think that the marionette-js-runner failure might be related to your patch, but interesting that the tests passed for one version of node and not another. I'll take a look as well and when I'm finished I'll publish new versions to npm and update gaia-node-modules.

Thanks :ochameau!

Comment 14

[Gareth Aye [:gaye] (back from PTO)]

https://github.com/mozilla-b2g/marionette-js-runner/commit/9f0b0338fd2fea5a7293b37f50e0213cdddc6ec8 landed on master.

Comment 15

[Gareth Aye [:gaye] (back from PTO)]

Attachment: Link to Github pull-request: https://github.com/mozilla-b2g/gaia/pull/20867

Patch to upgrade marionette-js-runner in gaia to downstream these changes

Comment 16

[Gareth Aye [:gaye] (back from PTO)]

https://github.com/mozilla-b2g/gaia/commit/d499090b7cddd4c7a00f439174221f17eee95d64 landed on gaia master. Thanks again for the work here :)