984911 - Look into sorting of mochitest manifests

Status: NEW

(Testing :: Mochitest, defect)

Description

[Bill McCloskey [inactive unless it's an emergency] (:billm)]

In bug 938019, we tried running tests in the order they're specified in the .ini files. That caused a bunch of tests to go orange because it's a different order than we currently run them, and tests don't like that. I think we have several options here.

1. Track down the sources of the test failures and fix the bugs. This would be best, but the most time consuming.

2. Require that tests appear in manifest files in order. I think we already do this for moz.build files.

3. Move the sorting from runtests.py to manifestparser.py.

Comment 1

[(not currently active) Ted Mielczarek]

What I want is sort of 2a -- Just sort the existing manifests into alphabetical order, which should preserve the existing behavior, then stop sorting in the harness. New tests will get run in whatever order they're listed in the manifest. (The only issue here is bad interactions between tests, which we have a long history of. For new tests the burden is on the test author.)