Closed Bug 487689 Opened 12 years ago Closed 11 years ago

Divide Mochitest suite into smaller suites

Categories

(Release Engineering :: General, defect, P2)

defect

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: joduinn, Assigned: catlee)

References

Details

Attachments

(2 files, 3 obsolete files)

(couldnt find another bug on this, but please DUP if I missed something).

Of all the suites in "Unittests", the one that takes the most time is mochitest. Can we break up the one big suite into a set of smaller mochitest suites (mochitest1, mochitest2... mochitestN) which:
- could all be run concurrently
- would each take approx the same time as the other unitsuites being run concurrently 
- could be run independently of each other if a developer wanted to reproduce a problem.

This is not actually blocking us in bug#452861, but we'll need this around the time as we enable concurrent suites running in production pool-of-slaves.  

(I know that Joel has been working on running "chunks" of Mochitest for mobile testing. But aiui, that is taking the big suite, and dynamically picking a subset to run. )
Instead of worrying about breaking out mochitest into subsuites, each with identical runtimes, would it be simpler to just break out mochitest by each different toplevel subdirectory?

Each different subsuites would likely take different times, so it wouldnt be a perfect optimization, but it might be good enough?!
Yeah, I think that would be fine. I think to make something like this work well we might have to get some sort of test manifest like reftest, although maybe we could hack something useful.
Going to grab this as the RelEng part of splitting up mochitests.

Just did some sanity checking.  grep 'INFO Running' | wc -l from a recent regular mochitest run gave 2586 lines.

Splitting up into 5 chunks, and grabbing each of the logs, and applying the same grep gives:
wc -l {1..5}running.txt
   440 1running.txt
   151 2running.txt
  1566 3running.txt
   260 4running.txt
   169 5running.txt
  2586 total

yay!
Assignee: nobody → catlee
Component: Mochitest → Release Engineering
Product: Testing → mozilla.org
QA Contact: mochitest → release
Version: unspecified → other
OS: Mac OS X → All
Hardware: x86 → All
Attachment #403477 - Flags: review?(bhearsum) → review+
Attachment #403481 - Flags: review?(bhearsum) → review+
Same as before, just updating builder names to include the total number of chunks.  Looks like this works on the MozillaTest waterfall.
Attachment #403477 - Attachment is obsolete: true
Attachment #403614 - Flags: review?(bhearsum)
Attachment #403614 - Flags: review?(bhearsum) → review+
Priority: -- → P2
Attachment #403614 - Attachment is obsolete: true
Attachment #407593 - Flags: review?(bhearsum)
Just freshened this one up a bit
Attachment #403481 - Attachment is obsolete: true
Attachment #407594 - Flags: review?(bhearsum)
Attachment #407593 - Flags: review?(bhearsum) → review+
Attachment #407594 - Flags: review?(bhearsum) → review+
Running for the better part of a week without issues with the splitting process itself.
Status: NEW → RESOLVED
Closed: 11 years ago
Resolution: --- → FIXED
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.