1273986 - Run reftests in random order in automation

Status: NEW

(Testing :: Reftest, defect)

Description

[Gregory Szorc [:gps]]

Trivial patch coming up shortly.

Comment 1

[Gregory Szorc [:gps]]

Attachment: MozReview Request: Bug 1273986 - Execute reftests in random order; r?dbaron

When tests are executed in the same order every time, undetected
cross-test dependencies can be introduced, leading to unexpected
passes or failures.

This commit randomizes the execution of reftests in automation.
This should hopefully lead to more test failures due to previously
undetected ordering dependencies and possibly even race conditions.

While this may upset people who care about seeing fewer test
failures, running tests in random order is more indicative of real
life browser behavior and thus could lead to higher browser quality
and stability.

Review commit: https://reviewboard.mozilla.org/r/53628/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/53628/

Comment 2

[Gregory Szorc [:gps]]

The Try push at https://treeherder.mozilla.org/#/jobs?repo=try&revision=42bf784f43b25bf3d03ce6d5ccdcfa384c792a57 has possibly uncovered some new failures on Windows as a result of this.

Comment 3

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

How does shuffle deal with the chunking of the reftests?  Does it just shuffle within the chunk?

Is there a mechanism for replaying an order in which a failure was observed?

Comment 4

[Gregory Szorc [:gps]]

Chunking occurs before shuffling.

We'll want the ability to seed the RNG so shuffle order can be specified and reproducible. Alternatively, we could invent a mechanism to save the execution order to a file and provide a mechanism to execute from that. RND seeding feels slightly easier.

Comment 5

[Gregory Szorc [:gps]]

It looks like the shuffle is done in JavaScript. I, uh, don't think JavaScript offers a way to seed a random number generator. Perhaps we have something in XPCOM to do this. I suppose we could change the Fisher-Yates shuffle in reftest.jsm to something else (that only needs a single random value which we can set in a pref).

Comment 6

[Gregory Szorc [:gps]]

I'm not actively working on this. I would encourage someone to pick it up because executing tests in random order is a good way to flush out bugs.

Comment 7

[Kartikaya Gupta (email:[e-mail])]

Comment on attachment 8753979 [details]
MozReview Request: Bug 1273986 - Execute reftests in random order; r?dbaron

Dropping stale review flag to get this bug out of the triage dashboard.