1634554 - Investigate adding a command line flag to run "groups"

Status: RESOLVED FIXED

(Testing :: web-platform-tests, task)

Description

[Andrew Halberstadt [:ahal]]

The WPT harness creates test "groups" here:
https://searchfox.org/mozilla-central/rev/7fd1c1c34923ece7ad8c822bee062dd0491d64dc/testing/web-platform/tests/tools/wptrunner/wptrunner/testloader.py#347

These "groups" are the smallest unit of tests that will run together in a single browser instance (i.e no restart between them). It's also the value that mozlog reports in its suite_start log action. Consequently, they're what get stored in ActiveData and used by bugbug to determine what to schedule.

It would be helpful if we had a command line flag to run only these groups. I.e:

$ ./mach web-platform-tests --group /html/semantics

which would run only tests that are actually in that group, rather than treating it like a file-system path. This would help avoid a double-scheduling problem we've had when trying to enable taskgraph chunking.

I'm not sure if this is feasible / worth it. But would like to investigate a bit further.

Comment 1

[Andrew Halberstadt [:ahal]]

This doesn't technically depend on 1608837, but from the smart-scheduling standpoint it doesn't make a ton of sense to implement it before that other one is done.

Comment 2

[James Graham [:jgraham]]

So I think there is a different approach here which has the advantage of putting the test resolver/decision task/scheduling infrastructure in full control and giving us a single source of truth, without having to do things like match the assumptions about groups between the harness and the scheduling.

Instead of passing in a list of groups and hoping that we can reconstruct the same list of tests inside the harness, we should pass in an explicit list of tests, split into named groups e.g.

{
  "/dom": ["/dom/test.html"],
  "/dom/foo/": ["/dom/foo/test1.html"]
}

Note that the group names need not be path prefixes since they are now opaque strings. But path prefixes are the obvious choice.

Then the calling code gets to fully define the test groups that it wants and which tests they each contain. This is basically like constructing a test manifest, but at runtime. In terms of the harness, we'd make a new command line option to read this e.g. from a file. Then we'd have another GroupedSource like PathGroupedSource which would just use the passed-in test groups. The command line parsing would check we weren't also supplying --run-by-dir or similar options. There would also need to be a test filter to only select the tests in the supplied group; this is a little different from the current --include mechanism so it might be worth just having a new filter. It might be possible to make this compose with the other options (--include, --exclude, --include-manifest, and positional arguments which are like --include) but it would probably also be OK to make them mutually exclusive.

I think from a decision task point of view the main question with this approach is how to pass the relevant data down to the harness.

Comment 3

[James Graham [:jgraham]]

Attachment: Bug 1634554 - Add support for --test-groups to wpt

This adds a --test-groups command line argument which points at a JSON
file containing lists of tests divided into explicit groups like:

{"/dom": ["/dom/historical.html", ...],
"/dom/events/": [...]}

This is for situations where the division of tests into groups is
performed by an external process (in the case of gecko: by the
decision task).

Group names must be a path prefix, as this metadata is reused as the
test "scope" which is passed down into the output and can be used by
automatic metadata update for per-group properties like the LSAN allow
list.

--test-groups is incompatible with --run-by-dir but composes with
passing an explicit include list by running the intersection of the
supplied tests.

Comment 4

[Andrew Halberstadt [:ahal]]

Nice, thanks!

So in mozilla-central we'd need to somehow propagate the groups from the taskgraph down to the workers. Currently we use MOZHARNESS_TEST_PATHS to do this, but this only includes manifests. Including all tests would be several orders of magnitude larger, so I'm not sure if that's a problem or not.

Alternatively, the Decision task already uploads an artifact that maps tests -> test-manifests. Maybe mozharness could just download this artifact from TC, strip out non-WPT things and then forward the file to the WPT harness. Here's an example artifact:
https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/JYQQJXz_RmuBbsm0um68-Q/runs/0/artifacts/public/tests-by-manifest.json.gz

We could also upload a specialized artifact from the Decision task just for this purpose if it made sense.

Comment 5

[Edwin Takahashi (:egao | infrequent contributor)]

I can begin initial investigation into this once the web-platform-tests-reftest patch looks like it is in pretty good shape.

Comment 6

[Andrew Halberstadt [:ahal]]

On Matrix :jgraham wrote:

When I wrote that patch I was imagining we'd produce one artifact per task with a list of tests/groups to run in that task. So yeah, it doesn't quite work if we want to provide a single artifact for all tasks and then command line arguments that select specific groups from in that larger artifact

I agree that passing in the full list of test groups for the entire suite in addition to the list of groups to run is pretty heavy handed. Personally I think the grouping algorithm is pretty simple and the risk of divergence is small, so I'd be inclined to instead:

1. Double check that they both produce identical output. Maybe move the algorithm to a place that both moztest and WPT can import from easily (I think :egao tried to import it from WPT but it wasn't available in all contexts). If that's not feasible, an integration test that makes sure they both match could also work.
2. Implement a CLI flag to run groups instead of a directories.
3. Update mozharness to pass this flag in.

Comment 7

[Edwin Takahashi (:egao | infrequent contributor)]

Attachment: Bug 1634554 - Part 1. add support for loading externally computed WPT groups

Comment 8

[Edwin Takahashi (:egao | infrequent contributor)]

Attachment: Bug 1634554 - Part 1: download decision task artifact and strip out non-WPT tests in mozharness/web_platform_tests.py

Comment 9

[Edwin Takahashi (:egao | infrequent contributor)]

Attachment: Bug 1634554 - Part 3. simplify chunking of web-platform-tests suites

Changes:

• stop using runtime information for the chunking process of web-platform-tests
• remove the wpt_group_translation workaround as it is no longer necessary
• simplify chunking process to iterate and assign test paths to chunks in order

Comment 10

[Edwin Takahashi (:egao | infrequent contributor)]

Attachment: Bug 1634554 - Part 2: pass file containing MOZHARNESS_TEST_PATHS to web-platform-tests harness

Changes:

• if MOZHARNESS_TEST_PATHS is defined, write the contents of that directory to a JSON file.
• adjust where --run-by-dir is defined to not conflict with --test-groups` flag.

Comment 11

[Edwin Takahashi (:egao | infrequent contributor)]

Attachment: Bug 1634554 - Part 0: TestResolver to use the path to web-platform-tests up to maximum depth of 3

Changes:

• when generating the groups in TestResolver for web-platform-tests, use the paths to the test files up to a maximum depth of 3 by default.

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/71aa576ada8b
Add support for --test-groups to wpt r=ahal,egao
https://hg.mozilla.org/integration/autoland/rev/3240acbd85b6
Part 0: TestResolver to use the path to web-platform-tests up to maximum depth of 3 r=ahal
https://hg.mozilla.org/integration/autoland/rev/177cc940d57f
Part 1: download decision task artifact and strip out non-WPT tests in mozharness/web_platform_tests.py r=ahal,jgraham
https://hg.mozilla.org/integration/autoland/rev/ae24c9902708
Part 2: pass file containing MOZHARNESS_TEST_PATHS to web-platform-tests harness r=ahal,jgraham
https://hg.mozilla.org/integration/autoland/rev/e8db70067dbf
Part 3. simplify chunking of web-platform-tests suites r=ahal

Comment 13

[Web Platform Test Sync Bot [:wpt-sync] (Matrix: #interop:mozilla.org)]

Created web-platform-tests PR https://github.com/web-platform-tests/wpt/pull/24153 for changes under testing/web-platform/tests

Comment 14

[Web Platform Test Sync Bot [:wpt-sync] (Matrix: #interop:mozilla.org)]

Can't merge web-platform-tests PR due to failing upstream checks:
Github PR https://github.com/web-platform-tests/wpt/pull/24153
* Community-TC (pull_request) (https://community-tc.services.mozilla.com/tasks/groups/XfgjNihkRKmBJREEWq6cKQ)

Comment 15

[Bogdan Tara[:bogdan_tara | bogdant]]

Backed out 5 changesets (bug 1634554) for lint failure.

Push with failures: https://treeherder.mozilla.org/#/jobs?repo=autoland&selectedTaskRun=HAdjz3dDReuFG0YsYZgVKQ.0&resultStatus=testfailed%2Cbusted%2Cexception%2Cretry%2Cusercancel%2Crunning%2Cpending%2Crunnable&fromchange=e8db70067dbfbc4cb63964597ba596ba4093cd43&tochange=0edf10e6c6d6fc5651945243c33b2553ca5887e3

Backout link: https://hg.mozilla.org/integration/autoland/rev/0edf10e6c6d6fc5651945243c33b2553ca5887e3

Failures logs:
Lint: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=306399192&repo=autoland&lineNumber=483

[task 2020-06-15T19:07:06.009Z] 19:07:06.9 flake8 (98) | Command: --config /builds/worker/checkouts/gecko/.flake8 --output-file /tmp/tmplapxcxat --format {"path":"%(path)s","lineno":%(row)s,"column":%(col)s,"rule":"%(code)s","message":"%(text)s"} --filename *.configure,*.py
[task 2020-06-15T19:07:10.914Z] 19:07:10.914 flake8 (101) | Finished in 5.28 seconds
[task 2020-06-15T19:08:33.308Z] 19:08:33.307 flake8 (99) | Finished in 87.68 seconds
[task 2020-06-15T19:08:45.258Z] 19:08:45.258 flake8 (100) | Finished in 99.63 seconds
[task 2020-06-15T19:08:48.152Z] 19:08:48.152 flake8 (98) | Finished in 102.53 seconds
[task 2020-06-15T19:08:48.159Z] TEST-UNEXPECTED-ERROR | /builds/worker/checkouts/gecko/taskcluster/taskgraph/util/chunking.py:12:1 | 'collections.OrderedDict' imported but unused (F401)
[taskcluster 2020-06-15 19:08:48.592Z] === Task Finished ===

TVw: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=306401306&repo=autoland&lineNumber=230

[task 2020-06-15T19:27:00.145Z] 19:27:00  WARNING - Prefer load_and_update instead
[task 2020-06-15T19:27:00.146Z] 19:27:00     INFO - Per-test run updated with manifest Z:\task_1592248067\build\tests\web-platform\mozilla\meta\MANIFEST.json
[task 2020-06-15T19:27:00.150Z] 19:27:00  WARNING - No tests to verify.
[task 2020-06-15T19:27:00.150Z] 19:27:00     INFO - Running post-action listener: set_extra_try_arguments
[task 2020-06-15T19:27:00.150Z] 19:27:00     INFO - Running post-action listener: setup_coverage_tools
[task 2020-06-15T19:27:00.150Z] 19:27:00     INFO - [mozharness: 2020-06-15 19:27:00.150000Z] Finished download-and-extract step (success)
[task 2020-06-15T19:27:00.150Z] 19:27:00     INFO - [mozharness: 2020-06-15 19:27:00.150000Z] Running download-and-process-manifest step.
[task 2020-06-15T19:27:00.150Z] 19:27:00     INFO - Running pre-action listener: _resource_record_pre_action
[task 2020-06-15T19:27:00.150Z] 19:27:00     INFO - Running main action method: download_and_process_manifest
[task 2020-06-15T19:27:00.150Z] 19:27:00    FATAL - TESTS_BY_MANIFEST_URL not defined.
[task 2020-06-15T19:27:00.150Z] 19:27:00    FATAL - Running post_fatal callback...
[task 2020-06-15T19:27:00.150Z] 19:27:00    FATAL - Exiting -1
[task 2020-06-15T19:27:00.150Z] 19:27:00     INFO - Running post-action listener: _resource_record_post_action
[task 2020-06-15T19:27:00.150Z] 19:27:00     INFO - [mozharness: 2020-06-15 19:27:00.150000Z] Finished download-and-process-manifest step (failed)
[task 2020-06-15T19:27:00.151Z] 19:27:00     INFO - Running post-run listener: _resource_record_post_run
[fetches 2020-06-15T19:27:00.238Z] removing Z:/task_1592248067/fetches
[fetches 2020-06-15T19:27:00.242Z] finished
[taskcluster 2020-06-15T19:27:00.264Z]    Exit Code: -1
[taskcluster 2020-06-15T19:27:00.264Z]    User Time: 15.6001ms
[taskcluster 2020-06-15T19:27:00.264Z]  Kernel Time: 0s
[taskcluster 2020-06-15T19:27:00.264Z]    Wall Time: 5m44.2528368s
[taskcluster 2020-06-15T19:27:00.264Z]       Result: FAILED
[taskcluster 2020-06-15T19:27:00.264Z] === Task Finished ===
[taskcluster 2020-06-15T19:27:00.264Z] Task Duration: 5m44.2578378s
[taskcluster 2020-06-15T19:27:00.340Z] Uploading artifact public/logs/localconfig.json from file logs\localconfig.json with content encoding "gzip", mime type "application/octet-stream" and expiry 2021-06-15T18:57:07.179Z
[taskcluster 2020-06-15T19:27:00.583Z] Uploading redirect artifact public/logs/live.log to URL https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/Lt75MxftRuGsDLry5f9R0w/runs/0/artifacts/public/logs/live_backing.log with mime type "text/plain; charset=utf-8" and expiry 2021-06-15T18:57:07.179Z
[taskcluster:error] exit status -1

Comment 16

[Web Platform Test Sync Bot [:wpt-sync] (Matrix: #interop:mozilla.org)]

Upstream PR was closed without merging

Comment 17

[Edwin Takahashi (:egao | infrequent contributor)]

Fixed the issue with TVw.

Comment 19

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/997be00caadb
Add support for --test-groups to wpt r=ahal,egao
https://hg.mozilla.org/integration/autoland/rev/647a0f8be8e6
Part 0: TestResolver to use the path to web-platform-tests up to maximum depth of 3 r=ahal
https://hg.mozilla.org/integration/autoland/rev/087aa04fd31f
Part 1: download decision task artifact and strip out non-WPT tests in mozharness/web_platform_tests.py r=ahal,jgraham
https://hg.mozilla.org/integration/autoland/rev/6ebb50488afe
Part 2: pass file containing MOZHARNESS_TEST_PATHS to web-platform-tests harness r=ahal,jgraham
https://hg.mozilla.org/integration/autoland/rev/a6c0a79565ef
Part 3. simplify chunking of web-platform-tests suites r=ahal

Comment 20

[Web Platform Test Sync Bot [:wpt-sync] (Matrix: #interop:mozilla.org)]

Upstream web-platform-tests status checks passed, PR will merge once commit reaches central.

Comment 21

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/997be00caadb
https://hg.mozilla.org/mozilla-central/rev/647a0f8be8e6
https://hg.mozilla.org/mozilla-central/rev/087aa04fd31f
https://hg.mozilla.org/mozilla-central/rev/6ebb50488afe
https://hg.mozilla.org/mozilla-central/rev/a6c0a79565ef

Comment 22

[Web Platform Test Sync Bot [:wpt-sync] (Matrix: #interop:mozilla.org)]

Upstream PR merged by moz-wptsync-bot