979512 - Add a command line option to split the tests and run a specified chunk

Status: RESOLVED FIXED

(Remote Protocol :: Marionette, defect)

Description

[Dave Hunt [:davehunt] [he/him] ⌚BST]

It looks like we're going to need to split the manifest when running Gaia UI tests against emulators on TBPL due to us hitting the 2 hour build limit. Rather than split up the manifest files manually and have to maintain these separate files, it would be good to control this via a command line option.

I'm thinking something like --chunk=1:4 where the tests selected to run would be split into four groups, as equal in number as possible, and the first group would be run. Subsequent (or parallel) jobs using --chunk=2:4 etc would run the remaining groups.

So long as all jobs are running on the same revision then the groups can be trusted not to overlap or omit tests.

Comment 1

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Attachment: Add a command line option to split the tests and run a specified chunk. v1.0

Comment 2

[Jonathan Griffin (:jgriffin)]

Comment on attachment 8388459 [details] [diff] [review]
Add a command line option to split the tests and run a specified chunk. v1.0

Review of attachment 8388459 [details] [diff] [review]:
-----------------------------------------------------------------

::: testing/marionette/client/marionette/runner/base.py
@@ +975,5 @@
>              if self.marionette.check_for_crash():
>                  break
>  
>      def run_test_sets(self):
> +        run_chunk, total_chunks = [int(s) for s in self.chunk.split(':')]

Can we use this_chunk instead of run_chunk to be consistent with mochitest and other harnesses?

Comment 3

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Attachment: Add a command line option to split the tests and run a specified chunk. v1.1

Carried r+

Comment 4

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Before landing this, I notice that mochitest uses --this-chunk and --total-chunks command line options. Do we want to simply use that rather than introduce a strictly formatted string? My initial thinking was that it would be better to keep the number of command line options to a minimum, but the two arguments would be less error prone and would be consistent with mochitests.

Comment 5

[Jonathan Griffin (:jgriffin)]

dave and I discussed and agreed it made sense to use the same argument names as mochitest, for consistency's sake.

Comment 6

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Attachment: Add a command line option to split the tests and run a specified chunk. v2.0

Comment 7

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/integration/fx-team/rev/387c7e65c6ef

Comment 8

[Ryan VanderMeulen [:RyanVM]]

Backed out for Gu bustage.
https://hg.mozilla.org/integration/fx-team/rev/8b9f265aae5a

https://tbpl.mozilla.org/php/getParsedLog.php?id=36389340&tree=Fx-Team

Comment 9

[Ryan VanderMeulen [:RyanVM]]

And other bustage:
https://tbpl.mozilla.org/php/getParsedLog.php?id=36389402&tree=Fx-Team

Please make sure this passes on Try before re-requesting checkin.

Comment 10

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Attachment: Add a command line option to split the tests and run a specified chunk. v2.1

Carrying r+ running on try: https://tbpl.mozilla.org/?tree=Try&rev=7fb1e0c74d74

Comment 11

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Sorry for that silly mistake Ryan, try looks good now.

Comment 12

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/2cd13f9fe11e

Comment 13

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/2cd13f9fe11e