1991412 - wpt-update generates incorrect metadata for subtests

Status: NEW

(Testing :: web-platform-tests, defect)

Description

[Yoshi Cheng-Hao Huang [:allstars.chh][:allstarschh][:yoshi]]

I notice that mach wpt-update doesn't generate the correct metadata for the test expectation.

Take the file /import-maps/not-overridden/failed-resolution.html for example.

The ini was added in bug 1929548,
Three expected results for the 3 subtests were added
https://hg-edge.mozilla.org/integration/autoland/diff/78c5d10d5326700b89632a77d4e323e03c2b5dd8/testing/web-platform/meta/import-maps/not-overridden/failed-resolution.html.ini

But later in Bug 1979197, two entries in the ini file were removed
https://hg-edge.mozilla.org/mozilla-central/diff/d7ee16f25653563beb6da0b04409603a2273a9bf/testing/web-platform/meta/import-maps/not-overridden/failed-resolution.html.ini

But the problem is that Firefox still fails in those two subtests whose entries got removed in bug 1979197.
"[Resolution after import map should not be redefined for bare prefixes or exact matches]"
"[Resolution after import map should be redefined for non-prefixes]"

It's also weird that the WPT doesn't catch this error, but this looks like another bug.

Back to the failed test, failed-resolution.html
Running mach wpt-update will get the same result
Only the first subtest is marked as "FAIL"

wpt.fyi also has this problem
https://wpt.fyi/results/import-maps/not-overridden/failed-resolution.html?label=experimental&label=master&aligned
Only the first subtest is shown

According to the documentation
https://web-platform-tests.org/writing-tests/testharness-api.html#Test

 class Test(name)
    A single subtest. A Test is not constructed directly but via the test(), 

So the those test() calls in failed-resolution.html should be counted as subtests as well

https://searchfox.org/firefox-main/rev/955360e8e3a773601c56f08069449d8d9261b873/testing/web-platform/tests/import-maps/not-overridden/failed-resolution.html#42
https://searchfox.org/firefox-main/rev/955360e8e3a773601c56f08069449d8d9261b873/testing/web-platform/tests/import-maps/not-overridden/failed-resolution.html#46
https://searchfox.org/firefox-main/rev/955360e8e3a773601c56f08069449d8d9261b873/testing/web-platform/tests/import-maps/not-overridden/failed-resolution.html#50

Comment 1

[BugBot [:suhaib / :marco/ :calixte]]

The severity field is not set for this bug.
:jgraham, could you have a look please?

For more information, please visit BugBot documentation.

Comment 2

[James Graham [:jgraham]]

In this case it's the test that's broken, not the infrastructure.

The problem is that test() is only useful for things that run synchronously, and the module import adds asynchronous behaviour. In particular we end the test when the load event fires. In this case the top-level await presumably means we end up spinning the event loop inside the first test(), allowing load to fire before we've reached the second test, so only the first test is counted.

Converting the test functions to promise_test fixes the issue.