1033143 - Has a server to run JSMarionette tests on real device

Status: RESOLVED DUPLICATE of bug 994764

(Testing Graveyard :: JSMarionette, defect)

Description

[Evan Tseng [:evanxd]]

It would be great we have a server(maybe try server?) to run JSMarionette tests on real device.

Currently, we already have some JSMarionette tests, like Camera tests[1], some Settings tests[2][3], or more do not run in anywhere regularly. This kind of situation lets the tests useless, and can not protect some features of apps.

We should improve this. So is it possible we could have a server to run the tests on device regularly?

[1] https://github.com/mozilla-b2g/gaia/tree/master/apps/camera/test/marionette
[2] https://github.com/mozilla-b2g/gaia/tree/master/apps/settings/test/marionette/tests/airplane_mode_settings_test.js
[3] https://github.com/mozilla-b2g/gaia/tree/master/apps/settings/test/marionette/tests/message_settings_test.js

Comment 1

[Kevin Grandon :kgrandon]

Maybe I am a crazy person, but I feel that we could have these running in b2g desktop if we wanted. We could implement the APIs, or at worst case implement the necessary shims in extensions. (We used to have a fully working camera shim that used getUserMedia and canvas for example).

I would definitely love to see these on a device, I would also very much like to see *all* tests running equally as well on desktop as a device.

Comment 2

[Evan Tseng [:evanxd]]

Hi Jason,

I heard that we will have a plan to setup a machine to run JSMarionette and Python Marionette tests on real device form Brian Huang. It is great!

Currently, not all JSMarionette test could run on device directly. There are some issues for that, and I think I could help this.

And I have some questions for the plan. Could you have time to reply me?
Do we already have a schedule for getting it done? Do we run the tests on device for each pull request or build, or other practice? Do the tests also run in try-server? So we could see the result in the TBPL?

Thanks.

Comment 3

[Jason Smith [:jsmith]]

I think Stephen is the person you want here. We do have a goal for Q3 to standup per commit JSMarionette tests (https://wiki.mozilla.org/QA/Goals/2014q3#Firefox_OS), but for only critical sanity tests that can be ran within a 5 - 8 minute timeframe.

Comment 4

[Stephen Donner [:stephend] Not actively reading bugmail]

(In reply to Evan Tseng [:evanxd][:愛聞插低] from comment #2)

> And I have some questions for the plan. Could you have time to reply me?
> Do we already have a schedule for getting it done? Do we run the tests on
> device for each pull request or build, or other practice? Do the tests also
> run in try-server? So we could see the result in the TBPL?
> 
> Thanks.

Hi Evan -

As Jason pointed out, we have a Q3 goal to get as close to per-commit, on-device (Flame) runs of a yet-to-be-defined, (but quick-running) acceptance-test suite (5-8 min, as Jason said).  The goal for those is to get as many as them running on device in JS, where possible, so developers can better-support them than the current Python tests (a few of which we'll likely include in this quick-running, on-device suite, until we get capabilities and tests ported).

Bug 994764 and its blocking bugs support the goal of being able to have a unified runner for both the JS and Python-based Marionette tests, on device.

(We have a lot of work on various teams to get there, but Brian's referring to the lab buildout that's progressing really well -- bug 1019792 tracks that effort.)

And finally, I'm sure we'd all appreciate your help towards these efforts (esp. as we try to port over some Python tests into JavaScript), so thanks in advance!

Comment 5

[Evan Tseng [:evanxd]]

Hi Stephen,

Thanks for the reply.

So is it possible we could run the some JSMarionette tests that need to run on real device after we have the the environment?

Just like the bug said, currently we have some JSMarionette tests in gaia project do NOT run in anywhere. As a gaia developer, I deeply hope we could have an environment to run these tests to protect code.

Thanks.

Comment 6

[Jonathan Griffin (:jgriffin)]

Yes, once ahal is done with bug 994764 and related issues, we will start running MarionetteJS tests on Flame devices; our goal is to have this running no later than July 21 (start of 2.1).

Comment 7

[Zac C (:zac)]

(In reply to Jonathan Griffin (:jgriffin) from comment #6)
> Yes, once ahal is done with bug 994764 and related issues, we will start
> running MarionetteJS tests on Flame devices; our goal is to have this
> running no later than July 21 (start of 2.1).

I've got a list of other things that we need higher up in the stack to get stability; Shall I file them as bugs?

Comment 8

[Jonathan Griffin (:jgriffin)]

(In reply to Zac C (:zac) from comment #7)
> (In reply to Jonathan Griffin (:jgriffin) from comment #6)
> > Yes, once ahal is done with bug 994764 and related issues, we will start
> > running MarionetteJS tests on Flame devices; our goal is to have this
> > running no later than July 21 (start of 2.1).
> 
> I've got a list of other things that we need higher up in the stack to get
> stability; Shall I file them as bugs?

Yes, please!

Comment 9

[Evan Tseng [:evanxd]]

(In reply to Jonathan Griffin (:jgriffin) from comment #6)
> Yes, once ahal is done with bug 994764 and related issues, we will start
> running MarionetteJS tests on Flame devices; our goal is to have this
> running no later than July 21 (start of 2.1).

Hi Jonathan,

Did you mean that we will run ALL JSMarionette tests in Gaia on Flame devices?
Oh, it is really great! :D

Comment 10

[Jonathan Griffin (:jgriffin)]

Yes, that's the goal, with a subset of tests being run very frequently (per-commit) and the others a little less frequently.

Comment 11

[Evan Tseng [:evanxd]]

Nice, great! :D