1921411 - Split Mn job on Treeherder into unit and integration tests

Status: RESOLVED FIXED

(Testing :: Marionette Client and Harness, task, P2)

Description

[Henrik Skupin [:whimboo][⌚️UTC+2]]

On bug 1916456 we added the capability to split the Mn job into multiple chunks. It was necessary because a lot of new tests were added over the past couple months exercising restart scenarios for Firefox and causing task timeouts for the Mn jobs for ASAN builds.

But as noticed this doesn't seem to be enough given that there are some tests like the windows enrollment tests that take about 15 minutes for ASAN and still cause the Mn2 job to fail with a task timeout:

https://treeherder.mozilla.org/logviewer?job_id=475926892&repo=autoland&lineNumber=105162

As such we should split the Marionette jobs into unit tests that only run the tests at:
https://searchfox.org/mozilla-central/source/testing/marionette/harness/marionette_harness/tests/unit/

And as well the integration (e2e) tests that run the tests from all the other locations. Only those we may have to chunk.

CC'ing Joel for further input.

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

this is a good idea, we should be able to do this via the .yml files, creating a job for the unittests by pointing directly at the manifest and adjusting the moz.build to not include the unittests as part of MARIONETTE_MANFIESTS so the existing tasks run faster.

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: Bug 1921411 - Split out marionette self tests into own job. r=whimboo

Comment 3

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/57a2af12c61e
Split out marionette self tests into own job. r=webdriver-reviewers,jdescottes

Comment 4

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: Bug 1921411 - adjust marionette-unit to be 60 minute timeout on asan. r=aryx

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/7a6329ddf6b7
adjust marionette-unit to be 60 minute timeout on asan. r=aryx. CLOSED TREE

Comment 6

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/57a2af12c61e
https://hg.mozilla.org/mozilla-central/rev/7a6329ddf6b7

Comment 7

[Pulsebot]

https://hg.mozilla.org/releases/mozilla-beta/rev/ca24ea1bb02f
https://hg.mozilla.org/releases/mozilla-beta/rev/edce6d3ed725

Comment 8

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Hi Joel, thank you for the work on this bug and getting the changes landed. I had a look and I actually wonder if we should put both of the jobs into a Mn group instead. That would give a better separation on Treeherder. What do you think?

Comment 9

[Joel Maher ( :jmaher ) (UTC -8)]

this sounds like a good idea. Would we have: Mn(Mn Mn-swr unit unit-swr) ? I ask as the Mn(Mn) seems repetitive, maybe there is a better symbol to use inside the Mn() ?

Comment 10

[Henrik Skupin [:whimboo][⌚️UTC+2]]

(In reply to Joel Maher ( :jmaher ) (UTC -8) from comment #9)

this sounds like a good idea. Would we have: Mn(Mn Mn-swr unit unit-swr) ? I ask as the Mn(Mn) seems repetitive, maybe there is a better symbol to use inside the Mn() ?

What if we do it similar to Mochitests and just use numbers for integration tests? That would result in Mn(1, 2, unit1, unit2). Also would it be possible to have the swr jobs in their own group like Mn-swr(1, 2, unit1, unit2)?