Closed Bug 725510 Opened 10 years ago Closed 10 years ago

xpcshell harness support for random execution order

Categories

(Testing :: XPCShell Harness, defect)

defect
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED
mozilla13

People

(Reporter: gps, Assigned: gps)

References

Details

Attachments

(1 file)

It is generally accepted that executing unit tests in random order can be a good thing (tm). I propose adding a --random flag to the xpcshell harness to randomize execution of individual test files.

Basically, testing/xpcshell/runxpcshelltests.py should be updated to support random execution.

This should be a pretty straight-forward change. I would put [good first bug] but I'm not sure who the mentor would be. (I'd do it, but I'm not an owner of this module.)

An obvious follow-up bug will be to support random test execution order in the build system. A follow-up to that will be a (possibly controversial and bug-revealing) change to the official build machines to execute tests in random order. But, let's start small and implement optional functionality first.
I'd suspect that xpcshell is the least likely test harness to have these kinds of inter-test dependent bugs, actually. The xpcshell test harness runs a new xpcshell process for each test, and it creates a fresh temporary directory for each test to use as a profile directory. The other test harnesses (mochitest, reftest) run all their tests in one browser session, which is much more likely to have those sorts of problems. (In fact, we hit a number of weird bugs like that when we split Mochitest into parts.)
This is a pretty straight-forward patch.

Random execution is off by default.
Assignee: nobody → gps
Status: NEW → ASSIGNED
Attachment #595589 - Flags: review?(ted.mielczarek)
Comment on attachment 595589 [details] [diff] [review]
Add --shuffle to xpcshell test runner

Review of attachment 595589 [details] [diff] [review]:
-----------------------------------------------------------------

Simple, I like it! It might be nice to add a test for this to selftest.py:
http://mxr.mozilla.org/mozilla-central/source/testing/xpcshell/selftest.py

I'm not sure what you want to assert, other than "passing shuffle doesn't cause anything to explode", but we have a bad habit of breaking little-used features of our test harnesses.
Attachment #595589 - Flags: review?(ted.mielczarek) → review+
https://hg.mozilla.org/integration/mozilla-inbound/rev/e20072b2802c

Patch committed with test additions after IRC review from Ted.
Blocks: 727189
https://hg.mozilla.org/mozilla-central/rev/e20072b2802c
Status: ASSIGNED → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla13
You need to log in before you can comment on or make changes to this bug.