test-verify-wpt fails to verify crashtest-style web platform tests
Categories
(Testing :: web-platform-tests, defect)
Tracking
(firefox128 fixed)
| Tracking | Status | |
|---|---|---|
| firefox128 | --- | fixed |
People
(Reporter: dholbert, Assigned: jmaher)
Details
Attachments
(1 file)
WPT crashtests don't seem to be included in test-verify runs.
Here's a Try run that I did, adding 3 intended-to-fail trivial WPTs: reftest-style, crashtest-style, and testharness.js-style:
https://treeherder.mozilla.org/jobs?repo=try&revision=867e244f0cae0e8135568978e67426efd4cb02e7
Here it is filtered for the test-verify runs:
https://treeherder.mozilla.org/jobs?repo=try&revision=867e244f0cae0e8135568978e67426efd4cb02e7&searchStr=tv
The reftest (test-reftest.html) and testharness.js test (test-testharness.html) from my Try run do get included in the test-verify-wpt task, but the crashtest (test-crash.html) is not included. It is in the commit that I pushed, though, and it does fail (via timeout) as you can see by the Wc oranges on that push.
(I initially had a theory that this issue was specific to the mozilla WPT directory, but it's not; I tried adding the test in the shared testing/web-platform/tests/ directory, too, and test-verify still fails to pick it up.)
|
Reporter | |
Comment 1•8 months ago
•
|
||
Here's a simpler try run where the push is just adding a single crashtest in the shared WPT directory, and it fails to get picked up by test-verify, and so the test-verify tasks are green despite the fact that the test intentionally fails:
https://treeherder.mozilla.org/jobs?repo=try&revision=cce74c078a260b7fe4145cff7bfe8c1f66a48f70
|
|
Reporter | |
Comment 2•8 months ago
|
||
jmaher, I think you know about the inner workings of test-verify -- maybe you could take a look, or know who could?
|
Assignee | |
Updated•4 months ago
|
|
|
Assignee | |
Comment 3•4 months ago
|
||
|
||
Updated•4 months ago
|
|
||
Comment 5•4 months ago
|
||
| bugherder | ||
Description
•