831830 - Run Marionette unit tests before WebAPI tests, not after

Status: RESOLVED FIXED

(Remote Protocol :: Marionette, defect)

Description

[Jonathan Griffin (:jgriffin)]

On marionette-webapi test runs, we end up running all of the WebAPI tests, and then the Marionette unit tests.  We should run these the other way around.

Comment 1

[Andrew Halberstadt [:ahal]]

I always thought that these should be two separate jobs entirely. It's kind of weird that on desktop we have only Mn tests but on B2G Mn means "marionette+webapi".

* Is there any particular reason we run them together?
* How hard would it be to run them separately?

Comment 2

[Jonathan Griffin (:jgriffin)]

Attachment: Run Marionette unit tests before webapi tests,

Going to run this on try to make sure we don't get any weird behavior.

Comment 3

[Jonathan Griffin (:jgriffin)]

I don't think we need to run them separately.  For mochitests, there are a set of mochitest 'unit tests' (harness sanity, special powers, etc) that we run before the rest of the tests.  It's just a way of catching problems in the harness that may prevent later tests from failing.

Comment 4

[Jonathan Griffin (:jgriffin)]

https://tbpl.mozilla.org/?tree=Try&rev=09cd068fe89c

Comment 5

[Robert Wood [:rwood]]

Comment on attachment 703393 [details] [diff] [review]
Run Marionette unit tests before webapi tests,

Review of attachment 703393 [details] [diff] [review]:
-----------------------------------------------------------------

Lgtm and ran fine on my local VM.

Comment 6

[Henrik Skupin [:whimboo][⌚️UTC+1]]

(In reply to Jonathan Griffin (:jgriffin) from comment #3)
> I don't think we need to run them separately.  For mochitests, there are a
> set of mochitest 'unit tests' (harness sanity, special powers, etc) that we
> run before the rest of the tests.  It's just a way of catching problems in
> the harness that may prevent later tests from failing.

When are those tests run? For each Mochitest chunk or already in the Build phase? We could save a lot of time if it would happen in the latter.

Comment 7

[Jonathan Griffin (:jgriffin)]

They're not run once per chunk, but just once per set, usually as part of chunk 1.

Comment 8

[Henrik Skupin [:whimboo][⌚️UTC+1]]

But shouldn't we run this in the build step? If something is wrong with Marionette we do not have to execute all the referenced tests which will save a lot of time on the testing machines.

Comment 9

[Jonathan Griffin (:jgriffin)]

It isn't possible to run them in the build step.  The only kinds of tests we can run in the build step are the C++ tests.  Marionette has a set of dependencies which currently are not met on the build slaves.

Comment 10

[Jonathan Griffin (:jgriffin)]

https://hg.mozilla.org/integration/mozilla-inbound/rev/5e5b67c6c031
https://hg.mozilla.org/releases/mozilla-b2g18/rev/4dd8cb0b17a1

Comment 11

[:Ms2ger (he/him; ⌚ UTC+1/+2)]

https://hg.mozilla.org/mozilla-central/rev/5e5b67c6c031