Closed Bug 725510 Opened 11 years ago Closed 11 years ago
xpcshell harness support for random execution order
It is generally accepted that executing unit tests in random order can be a good thing (tm). I propose adding a --random flag to the xpcshell harness to randomize execution of individual test files. Basically, testing/xpcshell/runxpcshelltests.py should be updated to support random execution. This should be a pretty straight-forward change. I would put [good first bug] but I'm not sure who the mentor would be. (I'd do it, but I'm not an owner of this module.) An obvious follow-up bug will be to support random test execution order in the build system. A follow-up to that will be a (possibly controversial and bug-revealing) change to the official build machines to execute tests in random order. But, let's start small and implement optional functionality first.
I'd suspect that xpcshell is the least likely test harness to have these kinds of inter-test dependent bugs, actually. The xpcshell test harness runs a new xpcshell process for each test, and it creates a fresh temporary directory for each test to use as a profile directory. The other test harnesses (mochitest, reftest) run all their tests in one browser session, which is much more likely to have those sorts of problems. (In fact, we hit a number of weird bugs like that when we split Mochitest into parts.)
This is a pretty straight-forward patch. Random execution is off by default.
Assignee: nobody → gps
Status: NEW → ASSIGNED
Attachment #595589 - Flags: review?(ted.mielczarek)
Comment on attachment 595589 [details] [diff] [review] Add --shuffle to xpcshell test runner Review of attachment 595589 [details] [diff] [review]: ----------------------------------------------------------------- Simple, I like it! It might be nice to add a test for this to selftest.py: http://mxr.mozilla.org/mozilla-central/source/testing/xpcshell/selftest.py I'm not sure what you want to assert, other than "passing shuffle doesn't cause anything to explode", but we have a bad habit of breaking little-used features of our test harnesses.
Attachment #595589 - Flags: review?(ted.mielczarek) → review+
https://hg.mozilla.org/integration/mozilla-inbound/rev/e20072b2802c Patch committed with test additions after IRC review from Ted.
Status: ASSIGNED → RESOLVED
Closed: 11 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla13
You need to log in before you can comment on or make changes to this bug.