Skipped web-platform-tests aren't assigned to a group in logger.suite_start
Categories
(Testing :: web-platform-tests, defect, P5)
Tracking
(Not tracked)
People
(Reporter: ekyle, Unassigned)
References
(Blocks 1 open bug)
Details
Structured logs are supposed to generate a JSON line like (added formatting for clarity):
{"tests": {
groupName1: [test11, test12, ...],
groupName2: [test21, test22, ...],
....
}}
This does not always happen. For example, in
https://firefoxci.taskcluster-artifacts.net/FKJFB_HET0q1gTy-Pzow_w/0/public/test_info//wpt_raw.log
the following test can not be found in any group
/content-security-policy/media-src/media-src-7_1.html
Reporter | ||
Updated•5 years ago
|
Reporter | ||
Comment 1•5 years ago
|
||
ActiveData can provide fresh instances of the same problem:
https://activedata.allizom.org/tools/query.html#query_id=47cp3IPY
Reporter | ||
Updated•5 years ago
|
Updated•5 years ago
|
Comment 2•5 years ago
|
||
Requesting clarification of the problem scope.
Example pulled from the activedata query above:
test: /2dcontext/compositing/2d.composite.canvas.copy.html
task: https://firefox-ci-tc.services.mozilla.com/tasks/PcVMnSSwR_au9vJIjzP3pQ
treeherder: https://treeherder.mozilla.org/#/jobs?repo=try&revision=68e8ca06e5c0d174e5744aaf588d730459552102&searchStr=test-android-em-7.0-x86_64%2Fdebug-geckoview-web-platform-tests-backlog-e10s-4
raw log: https://firefoxci.taskcluster-artifacts.net/PcVMnSSwR_au9vJIjzP3pQ/0/public/test_info//wpt_raw.log
live log: https://firefoxci.taskcluster-artifacts.net/PcVMnSSwR_au9vJIjzP3pQ/0/public/logs/live_backing.log
This is what I see in the raw logs:
{"source": "web-platform-tests", "test": "/2dcontext/compositing/2d.composite.canvas.copy.html", "thread": "MainThread", "time": 1587383012642, "action": "test_start", "pid": 1113}
{"status": "SKIP", "source": "web-platform-tests", "test": "/2dcontext/compositing/2d.composite.canvas.copy.html", "thread": "MainThread", "time": 1587383012642, "action": "test_end", "pid": 1113}
I am not certain what the I am expected to see vs what I actually see recorded in the raw logs.
Comment 3•5 years ago
|
||
Addendum to the above message: the portion that handles the construction of the log message is https://searchfox.org/mozilla-central/source/testing/mozbase/mozlog/mozlog/structuredlog.py#281-290 which is a component of mozlog
, not web-platform-tests
.
But :ahal changed the component back to web-platform-tests
, so the assumption held here is that this issue is due to how web-platform-tests is sending the components to be assembled into a log message, hence it is an issue with the web-platform-test harness.
Is this assumption correct?
Comment 4•5 years ago
|
||
Yes, I believe this is an issue in WPT.
Though if this is only happening to SKIP
tests, then it likely isn't a huge deal. Possibly even WONTFIX. We should verify if there are any instances of non-skip tests where there is no group.
Comment 5•5 years ago
|
||
Note the example test from comment 0 is also skipped:
https://searchfox.org/mozilla-central/source/testing/web-platform/meta/content-security-policy/media-src/media-src-7_1.html.ini#2
Comment 6•5 years ago
|
||
(In reply to Andrew Halberstadt [:ahal] from comment #5)
Note the example test from comment 0 is also skipped:
https://searchfox.org/mozilla-central/source/testing/web-platform/meta/content-security-policy/media-src/media-src-7_1.html.ini#2
My observations so far confirm what you've seen - the query only returns tests that appear to have been skipped on web-platform-tests.
Reporter | ||
Comment 7•5 years ago
|
||
:marco I assume this s OK: After removing the skips, and a variety of crashes, all tests have a group.
https://activedata.allizom.org/tools/query.html#query_id=bTD8OPxQ
Comment 8•5 years ago
|
||
I think the bug is still valid (ideally skipped tests still have a group), I just assume it's not super important for the smart scheduling project.
Updated•5 years ago
|
Updated•5 years ago
|
Updated•5 years ago
|
Comment 9•5 years ago
|
||
There's only one problem that we need to handle: currently mozci would consider a skipped group as a passing group.
Comment 10•4 years ago
|
||
The severity field is not set for this bug.
:jgraham, could you have a look please?
For more information, please visit auto_nag documentation.
Comment 11•4 years ago
|
||
(In reply to Marco Castelluccio [:marco] from comment #9)
There's only one problem that we need to handle: currently mozci would consider a skipped group as a passing group.
I thought this meant the skipped groups would be considered as passing in ActiveData, but it isn't actually the case, so this bug is not needed at all for smart scheduling (bug 1632242 is, but not for WPT since WPT currently doesn't output skipped groups in the "groups" section of errorsummary.log).
Updated•4 years ago
|
Comment 12•4 years ago
|
||
:jgraham on Matrix [edited to use permalinks]:
https://searchfox.org/mozilla-central/rev/61fceb7c0729773f544a9656f474e36cd636e5ea/testing/web-platform/tests/tools/wptrunner/wptrunner/wptrunner.py#242 we pass in
tests
and at https://searchfox.org/mozilla-central/rev/61fceb7c0729773f544a9656f474e36cd636e5ea/testing/web-platform/tests/tools/wptrunner/wptrunner/wptrunner.py#276 we usedisabled_tests
If you want to change things, just makedisabled_tests = test_loader.disabled_tests[test_type]
as a variable, make the first callsuite_start(tests + disabled_tests)
and change the second to use the new variable
Or something like that
Description
•