725510 - xpcshell harness support for random execution order

Status: RESOLVED FIXED

(Testing :: XPCShell Harness, defect)

Description

[Gregory Szorc [:gps]]

It is generally accepted that executing unit tests in random order can be a good thing (tm). I propose adding a --random flag to the xpcshell harness to randomize execution of individual test files.

Basically, testing/xpcshell/runxpcshelltests.py should be updated to support random execution.

This should be a pretty straight-forward change. I would put [good first bug] but I'm not sure who the mentor would be. (I'd do it, but I'm not an owner of this module.)

An obvious follow-up bug will be to support random test execution order in the build system. A follow-up to that will be a (possibly controversial and bug-revealing) change to the official build machines to execute tests in random order. But, let's start small and implement optional functionality first.

Comment 1

[(not currently active) Ted Mielczarek]

I'd suspect that xpcshell is the least likely test harness to have these kinds of inter-test dependent bugs, actually. The xpcshell test harness runs a new xpcshell process for each test, and it creates a fresh temporary directory for each test to use as a profile directory. The other test harnesses (mochitest, reftest) run all their tests in one browser session, which is much more likely to have those sorts of problems. (In fact, we hit a number of weird bugs like that when we split Mochitest into parts.)

Comment 2

[Gregory Szorc [:gps]]

Attachment: Add --shuffle to xpcshell test runner

This is a pretty straight-forward patch.

Random execution is off by default.

Comment 3

[(not currently active) Ted Mielczarek]

Comment on attachment 595589 [details] [diff] [review]
Add --shuffle to xpcshell test runner

Review of attachment 595589 [details] [diff] [review]:
-----------------------------------------------------------------

Simple, I like it! It might be nice to add a test for this to selftest.py:
http://mxr.mozilla.org/mozilla-central/source/testing/xpcshell/selftest.py

I'm not sure what you want to assert, other than "passing shuffle doesn't cause anything to explode", but we have a bad habit of breaking little-used features of our test harnesses.

Comment 4

[Gregory Szorc [:gps]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/e20072b2802c

Patch committed with test additions after IRC review from Ted.

Comment 5

[Marco Bonardo [:mak] (Away Apr 25 - May 5)]

https://hg.mozilla.org/mozilla-central/rev/e20072b2802c