1631515 - Skipped web-platform-tests aren't assigned to a group in logger.suite_start

Status: REOPENED

(Testing :: web-platform-tests, defect, P5)

Description

[Kyle Lahnakoski [:ekyle]]

Structured logs are supposed to generate a JSON line like (added formatting for clarity):

{"tests": {
    groupName1: [test11, test12, ...],
    groupName2: [test21, test22, ...],
    ....
}}

This does not always happen. For example, in

https://firefoxci.taskcluster-artifacts.net/FKJFB_HET0q1gTy-Pzow_w/0/public/test_info//wpt_raw.log

the following test can not be found in any group

/content-security-policy/media-src/media-src-7_1.html

Comment 1

[Kyle Lahnakoski [:ekyle]]

ActiveData can provide fresh instances of the same problem:

https://activedata.allizom.org/tools/query.html#query_id=47cp3IPY

Comment 2

[Edwin Takahashi (:egao | infrequent contributor)]

Requesting clarification of the problem scope.

Example pulled from the activedata query above:
test: /2dcontext/compositing/2d.composite.canvas.copy.html
task: https://firefox-ci-tc.services.mozilla.com/tasks/PcVMnSSwR_au9vJIjzP3pQ
treeherder: https://treeherder.mozilla.org/#/jobs?repo=try&revision=68e8ca06e5c0d174e5744aaf588d730459552102&searchStr=test-android-em-7.0-x86_64%2Fdebug-geckoview-web-platform-tests-backlog-e10s-4
raw log: https://firefoxci.taskcluster-artifacts.net/PcVMnSSwR_au9vJIjzP3pQ/0/public/test_info//wpt_raw.log
live log: https://firefoxci.taskcluster-artifacts.net/PcVMnSSwR_au9vJIjzP3pQ/0/public/logs/live_backing.log

This is what I see in the raw logs:

{"source": "web-platform-tests", "test": "/2dcontext/compositing/2d.composite.canvas.copy.html", "thread": "MainThread", "time": 1587383012642, "action": "test_start", "pid": 1113}
{"status": "SKIP", "source": "web-platform-tests", "test": "/2dcontext/compositing/2d.composite.canvas.copy.html", "thread": "MainThread", "time": 1587383012642, "action": "test_end", "pid": 1113}

I am not certain what the I am expected to see vs what I actually see recorded in the raw logs.

Comment 3

[Edwin Takahashi (:egao | infrequent contributor)]

Addendum to the above message: the portion that handles the construction of the log message is https://searchfox.org/mozilla-central/source/testing/mozbase/mozlog/mozlog/structuredlog.py#281-290 which is a component of mozlog, not web-platform-tests.

But :ahal changed the component back to web-platform-tests, so the assumption held here is that this issue is due to how web-platform-tests is sending the components to be assembled into a log message, hence it is an issue with the web-platform-test harness.

Is this assumption correct?

Comment 4

[Andrew Halberstadt [:ahal]]

Yes, I believe this is an issue in WPT.

Though if this is only happening to SKIP tests, then it likely isn't a huge deal. Possibly even WONTFIX. We should verify if there are any instances of non-skip tests where there is no group.

Comment 5

[Andrew Halberstadt [:ahal]]

Note the example test from comment 0 is also skipped:
https://searchfox.org/mozilla-central/source/testing/web-platform/meta/content-security-policy/media-src/media-src-7_1.html.ini#2

Comment 6

[Edwin Takahashi (:egao | infrequent contributor)]

(In reply to Andrew Halberstadt [:ahal] from comment #5)

Note the example test from comment 0 is also skipped:
https://searchfox.org/mozilla-central/source/testing/web-platform/meta/content-security-policy/media-src/media-src-7_1.html.ini#2

My observations so far confirm what you've seen - the query only returns tests that appear to have been skipped on web-platform-tests.

Comment 7

[Kyle Lahnakoski [:ekyle]]

:marco I assume this s OK: After removing the skips, and a variety of crashes, all tests have a group.

https://activedata.allizom.org/tools/query.html#query_id=bTD8OPxQ

Comment 8

[Andrew Halberstadt [:ahal]]

I think the bug is still valid (ideally skipped tests still have a group), I just assume it's not super important for the smart scheduling project.

Comment 9

[Marco Castelluccio [:marco]]

There's only one problem that we need to handle: currently mozci would consider a skipped group as a passing group.

Comment 10

[BugBot [:suhaib / :marco/ :calixte]]

The severity field is not set for this bug.
:jgraham, could you have a look please?

For more information, please visit auto_nag documentation.

Comment 11

[Marco Castelluccio [:marco]]

(In reply to Marco Castelluccio [:marco] from comment #9)

There's only one problem that we need to handle: currently mozci would consider a skipped group as a passing group.

I thought this meant the skipped groups would be considered as passing in ActiveData, but it isn't actually the case, so this bug is not needed at all for smart scheduling (bug 1632242 is, but not for WPT since WPT currently doesn't output skipped groups in the "groups" section of errorsummary.log).

Comment 12

[Marco Castelluccio [:marco]]

:jgraham on Matrix [edited to use permalinks]:

https://searchfox.org/mozilla-central/rev/61fceb7c0729773f544a9656f474e36cd636e5ea/testing/web-platform/tests/tools/wptrunner/wptrunner/wptrunner.py#242 we pass in tests and at https://searchfox.org/mozilla-central/rev/61fceb7c0729773f544a9656f474e36cd636e5ea/testing/web-platform/tests/tools/wptrunner/wptrunner/wptrunner.py#276 we use disabled_tests
If you want to change things, just make disabled_tests = test_loader.disabled_tests[test_type] as a variable, make the first call suite_start(tests + disabled_tests) and change the second to use the new variable
Or something like that