1644235 - Run Marionette unit tests with Fission enabled in CI

Status: RESOLVED FIXED

(Remote Protocol :: Marionette, task, P1)

Description

[Henrik Skupin [:whimboo][⌚️UTC+2] (away 10/03 - 10/13)]

As noticed on bug 1640141 we do not run the Marionette tests with Fission enabled in CI. To make Marionette Fission compatible it would help a lot to get these scheduled regularly on autoland and mozilla-central.

Note that Marionette is also in use for web-platform-tests (which run with Fission enabled) and partly for setting up the Mochitests. Both test suites look fine so far. But to make sure that the Mn test suite is not completely broken here a try build as triggered with fuzzy --full:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=75f6efd00ec0f3a0b5cf403d0327bce1777606f0

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+2] (away 10/03 - 10/13)]

Those tests are all green. Joel, would it be ok to get them running regularly?

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

thanks for working on this and getting it green. I did some retriggers (5x) to see how stable this is.

I think the only one unstable is osx.

for the rest I am happy to see them get added to the autoland queue and worked into our scheduling algorithm.

Comment 3

[Henrik Skupin [:whimboo][⌚️UTC+2] (away 10/03 - 10/13)]

Looks like bug 1625410 is the one blocker here. Maja, maybe you wanna have a look at this bug?

Comment 4

[Chris Peterson [:cpeterson]]

Tracking for Fission Nightly M6c (work expected to land in Q3). The blocker bug 1625410 is being tracked for Fission Nightly M6a (work expected to land in June)

Comment 5

[Neha Kochar [:neha]]

Henrik, if bug 1625410 is the only intermittent failure left, can we skip this test for now and start running these tests?

Comment 6

[Henrik Skupin [:whimboo][⌚️UTC+2] (away 10/03 - 10/13)]

Ok, so lets see how it looks like:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=fe2127f0495adf939bcf3025cd4f96dffc7bec5b

Comment 7

[Henrik Skupin [:whimboo][⌚️UTC+2] (away 10/03 - 10/13)]

Note that right now we do not run WebRender jobs for Marionette. As such I wonder if we should add those, and where we would have to run fission jobs. Chris, do you have feedback for us? Maybe we should do it like mochitests do?

https://searchfox.org/mozilla-central/source/taskcluster/ci/test/mochitest.yml#19-25

Comment 8

[Chris Peterson [:cpeterson]]

(In reply to Henrik Skupin (:whimboo) [⌚️UTC+2] from comment #7)

Note that right now we do not run WebRender jobs for Marionette. As such I wonder if we should add those, and where we would have to run fission jobs. Chris, do you have feedback for us? Maybe we should do it like mochitests do?

WebRender is not a hard requirement for Fission. So as long as we can run the Mn tests with Fission somewhere (with or without WebRender), then the Fission team is happy.

We can run Mn on the non-QR platforms. We want to test both Linux64 and Windows 10 and both opt and debug. (Some of our other Fission tests are opt-only, but really should have included debug. I don't want to overlook that again here.) We're not running Fission mochitests on macOS yet, so we shouldn't try running Fission Mn on macOS yet.

Given the platforms that currently run Fission mochitests:

https://searchfox.org/mozilla-central/rev/3434a9df60373a997263107e6f124fb164ddebf2/taskcluster/ci/test/mochitest.yml#19-25

I suggest we run Fission Mn on:

            linux.*64/.*: ['autoland']
            linux.*64-shippable/.*: ['mozilla-central']
            windows10-64/.*: ['autoland']
            windows10-64-shippable/.*: ['mozilla-central']

These Mn tests can all be tier 2.

Comment 9

[Henrik Skupin [:whimboo][⌚️UTC+2] (away 10/03 - 10/13)]

Sounds good. I will get that added.

Comment 10

[Henrik Skupin [:whimboo][⌚️UTC+2] (away 10/03 - 10/13)]

Attachment: Bug 1644235 - [marionette] Enable Marionette unit tests with Fission in CI.

Comment 11

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/9bf594aab9d0
[marionette] Enable Marionette unit tests with Fission in CI. r=cpeterson,marionette-reviewers,maja_zf

Comment 12

[Dorel Luca [:dluca]]

https://hg.mozilla.org/mozilla-central/rev/9bf594aab9d0