1815447 - [meta] List tests in mochitest test manifest files in alphabetical order

Status: RESOLVED INVALID

(Testing :: Mochitest, task)

Description

[Mike Conley (:mconley) (:⚙️)]

Not all of our mochitest test manifests are in alphabetical order. Normally, that'd be annoying because order is nice and chaos is not - however, the bigger issue is that the mochitest test runners run the tests in alphabetical order anyways. This means that the listing inside of the mochitest ini files does not necessarily reflect execution order.

This makes using ./mach mochitest features like --start-at and --end-at difficult because it's unclear how to set the starting and ending tests when the test manifest is out of order (unless you do the work of determining the actual execution order).

So we should really just update our mochitest manifest files to reflect reality. I landed a linter in bug 1756878 to add a warning for out-of-order manifest files, and this is a meta-bug for a broader effort to put all of our mochitest test files in alphabetical order and then enforcing this rule at the error level.

Comment 1

[Rob Wu [:robwu]]

The linter in https://searchfox.org/mozilla-central/rev/3a3965bc78cf76f23ac0569c7f3c98b15f1dca5c/tools/lint/test-manifest-alpha.yml only checks ini files, but all test manifests have moved to toml. Is this bug and the linter still relevant?

The alphabetical order is already enforced by the test-manifest-toml linter, of which you can see the result at https://phabricator.services.mozilla.com/D240948

I still have a pending query in bug 1953004 about why the linter did not automatically activate for recent changes, but that could be a follow-up.

Comment 2

[Mike Conley (:mconley) (:⚙️)]

I think with the change to TOML, most of the dependencies here and this meta can be closed as INVALID now. I can do that.

Comment 3

[Mike Conley (:mconley) (:⚙️)]

With bug 1821199, we've switched to TOML for our testing manifests, and its linter should be maintaining the alphabetical ordering invariant.