1645867 - Separate manifests by suite in tests-by-manifest.json

Status: RESOLVED WONTFIX

(Firefox Build System :: Task Configuration, task)

Description

[Edwin Takahashi (:egao | infrequent contributor)]

Currently, all test manifests are thrown into one large JSON file without consideration given to the suite.

Bug 1634554 and one of its constituent patch, https://phabricator.services.mozilla.com/D78318 shows the need to be able to filter out manifests that belong to a certain suite. this would be much easier (and reliable) if manifests were grouped by the suite.

Explore if this is possible.

Comment 1

[Edwin Takahashi (:egao | infrequent contributor)]

See also: https://phabricator.services.mozilla.com/D77839#change-lnnLH5ZxXdXf as this could be useful.

Comment 2

[Edwin Takahashi (:egao | infrequent contributor)]

:ahal - this came up as part of :jgraham's review on the --test-groups file patch stack.

Would any suites see benefits from this change?

Comment 3

[Edwin Takahashi (:egao | infrequent contributor)]

Actually set ni.

Comment 4

[Andrew Halberstadt [:ahal]]

This artifact is only used by treeherder afaik, so probably not. If we change the format we'd need to coordinate with Armen on the treeherder side of things.

We could also potentially keep this in a single file, but key the manifests-by-task by suite (similar to what you did to the runtimes files).

Comment 5

[Edwin Takahashi (:egao | infrequent contributor)]

(In reply to Andrew Halberstadt [:ahal] from comment #4)

This artifact is only used by treeherder afaik, so probably not. If we change the format we'd need to coordinate with Armen on the treeherder side of things.

Interesting, I did not know this was only used by treeherder (other than WPT).

We could also potentially keep this in a single file, but key the manifests-by-task by suite (similar to what you did to the runtimes files).

That's what I was suggesting, by the way. A structure would look like:

{
  "web-platform-tests": {
     '/dom/some/test": [
        'test_path'
     ]
  }
}

But I'm thinking personally that being able to write the artifact from the chunking process would permit me to customize the file structure without impacting others.