1634551 - Investigate adding WPT's "groups" to Mozilla's test metadata

Status: RESOLVED FIXED

(Testing :: web-platform-tests, task, P3)

Description

[Andrew Halberstadt [:ahal]]

The WPT harness creates test "groups" here:
https://searchfox.org/mozilla-central/rev/7fd1c1c34923ece7ad8c822bee062dd0491d64dc/testing/web-platform/tests/tools/wptrunner/wptrunner/testloader.py#347

These "groups" are the smallest unit of tests that will run together in a single browser instance (i.e no restart between them). It's also the value that mozlog reports in its suite_start log action. Consequently, they're what get stored in ActiveData and used by bugbug to determine what to schedule.

It would be very helpful, if these group values could somehow be propagated to the metadata that manifestupdate returns, as called by moztest.TestResolver here:
https://searchfox.org/mozilla-central/rev/7fd1c1c34923ece7ad8c822bee062dd0491d64dc/testing/mozbase/moztest/moztest/resolve.py#719

Maybe we would create a new wpttest.Test.group attribute or something.

This would allow us to store the group in each test's manifest_relpath key just like we have for all the other test harnesses. That value could then be used to by the taskgraph to make sure we don't accidentally split up groups when chunking.

I'm unsure if this is feasible / desirable from WPT's perspective though.

Comment 1

[Andrew Halberstadt [:ahal]]

James, can you comment on the last sentence? Any tips on how we can accomplish this?

Comment 2

[James Graham [:jgraham]]

So the problem is that the "groups" from the point of view of wpt aren't a constant, but depend on the run-by-dir setting. And as :egao found in the other bug it's a little more complex than that even because foo/ can be a group for all the tests directly under foo but not the tests under foo/bar. So you need to know what run-by-dir is to make sense of the groups later (e.g. because if it's set to 2 then foo/bar/ will include foo/bar/baz but if it's set to 3 it won't).

So we could certainly make manifestupdate.run take a run_by_dir setting and return a mapping of groups to test ids using PathGroupLoader.tests_by_group. But I think maybe if we want this grouping abstraction to work with wpt we need a way to specify that the command line arguments are to be treated as test groups rather than as paths, or something (so that ./mach wpt --run-by-dir 3 --paths-as-groups foo/barwould excludefoo/bar/baz`).

Comment 3

[Andrew Halberstadt [:ahal]]

That all sounds fine to me!

But I think maybe if we want this grouping abstraction to work with wpt we need a way to specify that the command line arguments are to be treated as test groups rather than as paths, or something (so that ./mach wpt --run-by-dir 3 --paths-as-groups foo/barwould excludefoo/bar/baz`).

This is bug 1634554 :).

However, there's benefit to solving this bug even without that other bug. Edwin's current chunking algorithm creates some arbitrary groups that are different than the ones that WPT creates. He tries to make them less granular so that we don't accidentally split WPT-created groups across chunks. This has some downsides though:

1. The groups that ActiveData and bugbug (aka the ML algorithm) recognize come from WPT, and the groups that the taskgraph recognizes are different. We need to translate them.
2. The groups that the taskgraph will ultimately end up running will be larger than necessary (bad from a selection efficiency standpoint).

If instead the groups that the taskgraph uses to run its chunking algorithm are the same ones that bugbug knows about, the above problems disappear.

I don't think Edwin's work needs to block on this though. The group translation should be easy enough and we can live with less than ideal scheduling for now. This and bug 1634554 are follow-ups to his work in bug 1608837.

Comment 4

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1634551 - [taskgraph] Move WPT group assignment into the TestResolver, r?egao

This moves the creation of WPT groups into the TestResolver, where all kinds of
other consumers will be able to access them.

Comment 5

[Phabricator Automation]

Comment on attachment 9150460 [details]
Bug 1634551 - [taskgraph] Move WPT group assignment into the TestResolver, r?egao

Revision D76085 was moved to bug 1633866. Setting attachment 9150460 [details] to obsolete.

Comment 6

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1634551 - [runtimes] Update 'writeruntimes' script to use 'manifest' value from TestResolver, r?egao

This shouldn't materially change the keys in the runtimes files.

Depends on D76085

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/c271cdf24dd5
[taskgraph] Move WPT group assignment into the TestResolver, r=egao
https://hg.mozilla.org/integration/autoland/rev/0356e5fff183
[runtimes] Update 'writeruntimes' script to use 'manifest' value from TestResolver, r=egao

Comment 8

[Noemi Erli[:noemi_erli]]

https://hg.mozilla.org/mozilla-central/rev/c271cdf24dd5
https://hg.mozilla.org/mozilla-central/rev/0356e5fff183