487689 - Divide Mochitest suite into smaller suites

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

(couldnt find another bug on this, but please DUP if I missed something).

Of all the suites in "Unittests", the one that takes the most time is mochitest. Can we break up the one big suite into a set of smaller mochitest suites (mochitest1, mochitest2... mochitestN) which:
- could all be run concurrently
- would each take approx the same time as the other unitsuites being run concurrently 
- could be run independently of each other if a developer wanted to reproduce a problem.

This is not actually blocking us in bug#452861, but we'll need this around the time as we enable concurrent suites running in production pool-of-slaves.  

(I know that Joel has been working on running "chunks" of Mochitest for mobile testing. But aiui, that is taking the big suite, and dynamically picking a subset to run. )

Comment 1

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Instead of worrying about breaking out mochitest into subsuites, each with identical runtimes, would it be simpler to just break out mochitest by each different toplevel subdirectory?

Each different subsuites would likely take different times, so it wouldnt be a perfect optimization, but it might be good enough?!

Comment 2

[(not currently active) Ted Mielczarek]

Yeah, I think that would be fine. I think to make something like this work well we might have to get some sort of test manifest like reftest, although maybe we could hack something useful.

Comment 3

[Chris AtLee [:catlee]]

Going to grab this as the RelEng part of splitting up mochitests.

Just did some sanity checking.  grep 'INFO Running' | wc -l from a recent regular mochitest run gave 2586 lines.

Splitting up into 5 chunks, and grabbing each of the logs, and applying the same grep gives:
wc -l {1..5}running.txt
   440 1running.txt
   151 2running.txt
  1566 3running.txt
   260 4running.txt
   169 5running.txt
  2586 total

yay!

Comment 4

[Chris AtLee [:catlee]]

Attachment: buildbotcustom to enable split mochitests

Comment 5

[Chris AtLee [:catlee]]

Attachment: buildbot-configs to enable split mochitests

Comment 6

[Chris AtLee [:catlee]]

Attachment: change builder name to "mochitest-i/j" instead of just "mochitest-i"

Same as before, just updating builder names to include the total number of chunks.  Looks like this works on the MozillaTest waterfall.

Comment 7

[Lukas Blakk [:lsblakk] use ?needinfo]

Comment on attachment 403481 [details] [diff] [review]
buildbot-configs to enable split mochitests

http://hg.mozilla.org/build/buildbot-configs/rev/f6da3efcd371
http://hg.mozilla.org/build/buildbotcustom/rev/470033925ebe

Comment 8

[Lukas Blakk [:lsblakk] use ?needinfo]

Backed out changes:
http://hg.mozilla.org/build/buildbot-configs/rev/36537acf3bf6
http://hg.mozilla.org/build/buildbotcustom/rev/6f1a6b9e9338

Comment 9

[Chris AtLee [:catlee]]

Attachment: Un-bitrotted patch for buildbotcustom

Comment 10

[Chris AtLee [:catlee]]

Attachment: buildbot-configs to enable split mochitests

Just freshened this one up a bit

Comment 11

[Chris AtLee [:catlee]]

Running for the better part of a week without issues with the splitting process itself.