818156 - Improve test chunking

Status: RESOLVED FIXED

(Testing :: Reftest, defect)

Description

[Geoff Brown [:gbrown]]

mochitests and reftests are split into "chunks" lovingly known as R1, R2, R3, etc.

Chunking splits the total set of tests into N same-sized chunks, but the chunk size is based on the total number of tests, not the total number of tests run (on Android for instance, hundreds of tests are skipped).

In bug 816574, we see that Android R3 is running longer than R1, R2, and R4 and note that test are split (approximately):

R1: 974/0/1243
R2: 672/0/1544
R3: 1846/0/369
R4: 2059/0/157

If actually-run tests were split evenly, it might help make run times more even.

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: evenly distribute the tests when chunking for reftests (1.0)

Comment 2

[Geoff Brown [:gbrown]]

Comment on attachment 688914 [details] [diff] [review]
evenly distribute the tests when chunking for reftests (1.0)

Review of attachment 688914 [details] [diff] [review]:
-----------------------------------------------------------------

LGTM.

I note the new (Android 2.2 opt) distribution from a try run:

R1: 1301/0/203  0:34:24.177045
R2: 1426/0/77   0:36:42.323001
R3: 1407/0/95   0:49:26.699267
R4: 1417/0/86   0:32:21.953126

and compare against a typical current run:

R1: 974/0/1243  0:24:44.394815
R2: 672/0/1544  0:16:52.424199
R3: 1846/0/369  1:02:01.818326
R4: 2059/0/157  0:46:29.893534

The time distribution looks much better!

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

inbound:
https://hg.mozilla.org/integration/mozilla-inbound/rev/dccbc4f3d092

Comment 4

[Andrew Halberstadt [:ahal]]

This is awesome, thanks! I'll need this on aurora and beta too for b2g. I can land once it gets merged over.

Comment 5

[Andrew Halberstadt [:ahal]]

So this patch hides skipped tests from the results (e.g from comment 2 the known problems in R1 go from 1243 to 203). Also prior to this patch you would see something along the lines of:
> REFTEST TEST-KNOWN-FAIL | http://10.250.48.201:30063/tests/layout/reftests/bugs/379349-1c.xhtml | (SKIP)

in the log files. After this patch these are no longer being output. I don't think this is information we want to lose. We should modify the patch such that the chunking algorithm ignore skipped tests, but they aren't removed from the actual list of tests that get processed.

Comment 6

[Joel Maher ( :jmaher ) (UTC -8)]

so we could put this information in the log file at the time we are pruning the list.

Comment 7

[Ed Morley [:emorley]]

https://hg.mozilla.org/mozilla-central/rev/dccbc4f3d092

Comment 8

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-aurora/rev/da9d0e35d176
https://hg.mozilla.org/releases/mozilla-beta/rev/76cfdb1796e7