Closed Bug 725510 Opened 11 years ago Closed 11 years ago

xpcshell harness support for random execution order


(Testing :: XPCShell Harness, defect)

Not set


(Not tracked)



(Reporter: gps, Assigned: gps)




(1 file)

It is generally accepted that executing unit tests in random order can be a good thing (tm). I propose adding a --random flag to the xpcshell harness to randomize execution of individual test files.

Basically, testing/xpcshell/ should be updated to support random execution.

This should be a pretty straight-forward change. I would put [good first bug] but I'm not sure who the mentor would be. (I'd do it, but I'm not an owner of this module.)

An obvious follow-up bug will be to support random test execution order in the build system. A follow-up to that will be a (possibly controversial and bug-revealing) change to the official build machines to execute tests in random order. But, let's start small and implement optional functionality first.
I'd suspect that xpcshell is the least likely test harness to have these kinds of inter-test dependent bugs, actually. The xpcshell test harness runs a new xpcshell process for each test, and it creates a fresh temporary directory for each test to use as a profile directory. The other test harnesses (mochitest, reftest) run all their tests in one browser session, which is much more likely to have those sorts of problems. (In fact, we hit a number of weird bugs like that when we split Mochitest into parts.)
This is a pretty straight-forward patch.

Random execution is off by default.
Assignee: nobody → gps
Attachment #595589 - Flags: review?(ted.mielczarek)
Comment on attachment 595589 [details] [diff] [review]
Add --shuffle to xpcshell test runner

Review of attachment 595589 [details] [diff] [review]:

Simple, I like it! It might be nice to add a test for this to

I'm not sure what you want to assert, other than "passing shuffle doesn't cause anything to explode", but we have a bad habit of breaking little-used features of our test harnesses.
Attachment #595589 - Flags: review?(ted.mielczarek) → review+

Patch committed with test additions after IRC review from Ted.
Blocks: 727189
Closed: 11 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla13
You need to log in before you can comment on or make changes to this bug.