Closed Bug 1569559 Opened 1 year ago Closed 10 months ago

Metadata update still removing release_or_beta conditions

Categories

(Testing :: web-platform-tests, defect, P1)

Version 3
defect

Tracking

(firefox-esr68 unaffected, firefox71 wontfix, firefox72 wontfix, firefox73 fixed)

RESOLVED FIXED
mozilla73
Tracking Status
firefox-esr68 --- unaffected
firefox71 --- wontfix
firefox72 --- wontfix
firefox73 --- fixed

People

(Reporter: jgraham, Assigned: jgraham)

References

(Regression)

Details

(Keywords: regression)

Attachments

(1 file)

Bugbug thinks this bug is a task, but please change it back in case of error.

Type: defect → task
Type: task → defect
Regressed by: 1545143
See Also: → 1564917

The priority flag is not set for this bug.
:jgraham, could you have a look please?

For more information, please visit auto_nag documentation.

Flags: needinfo?(james)

As best as I can tell this is now working; I only get release_or_beta conditions removed when the value of the condition matches the new default (e.g. if we have if release_or_beta: PASS and then the test starts passing everywhere). I wonder if the previous incident was caused by running an update with an older gecko base not including the fixes. Please reopen if the bug reappears.

Status: NEW → RESOLVED
Closed: 1 year ago
Flags: needinfo?(james)
Resolution: --- → WORKSFORME
Status: RESOLVED → REOPENED
Resolution: WORKSFORME → ---

The problem here is that we are ending up with different subtest names on beta and nightly. The logic to remove stale subtests doesn't account for that possibility. Ideally we wouldn't have different test names in the PASS vs FAIL case. Can we look at the specific webaudio test that's causing problems and try to figure out why we're seeing that? If possible we should fix the test, otherwise we might need to make the cleanup code ignore conditions that it couldn't itself have created. That does mean that we may end up with stale conditions if there are release_or_beta conditions on subtests that are later removed.

Thank you for the analysis.

audioworklet-addmodule-resolution.https.html is failing on release_or_beta before any subtests have been created.
The harness then assumes the test is a single page test and creates an imaginary subtest name based on the title for the file.

This test is not using setup() to wrap the code that is failing. Wrapping in setup() changes the failure to a toplevel ERROR instead of a subtest FAIL, which the avoids the problem of unexpected subtest names. I've proposed this change in https://phabricator.services.mozilla.com/D46685.

In general, can a failure during set-up cause a different number of subtests to run? We often may not notice fallout from removing stale state because hopefully at least as many subtests would run on nightly as on beta, but the reverse seems quite possible.

It would seem reasonable that release_or_beta conditions would be considered stale only if there were data to support that, but addressing that may be less urgent with https://phabricator.services.mozilla.com/D46685.

See Also: → 1458068

As observed in https://bugzilla.mozilla.org/show_bug.cgi?id=1585946#c2, and reported in https://github.com/web-platform-tests/wpt/issues/10201 and https://bugzilla.mozilla.org/show_bug.cgi?id=1458068 there are still a bunch of imported webaudio tests that do not have consistent subtest names and so are continuing to cause problems.

https://phabricator.services.mozilla.com/D47025 is one part of changes required to make test names consistent.

Even if tests are fixed to aim to produce consistent subtest names on failure, there is still a chance of differing sets of names if there is a crash. I guess that situation can be special cased, if desired. I haven't checked whether or not it already is.

Depends on: 1592503

I"m going to look at this either this week or at the start of next year.

Priority: -- → P1

Sometimes we see subtests that only appear in certain conditions e.g. if we
get an early error on nightly but the tests run as expected in stable. In this
case the wpt sync bot will update the metadata with --full and remove the
missing subtests. To work around this, check if subtests contain any
conditions that aren't part of the update set, and if so never
remove them

Assignee: nobody → james
Pushed by james@hoppipolla.co.uk:
https://hg.mozilla.org/integration/autoland/rev/256f8dffafec
Don't remove subtests where a condition is outside the updatable set, r=maja_zf
Created web-platform-tests PR https://github.com/web-platform-tests/wpt/pull/20894 for changes under testing/web-platform/tests
Upstream web-platform-tests status checks passed, PR will merge once commit reaches central.
Upstream PR merged by moz-wptsync-bot
Status: REOPENED → RESOLVED
Closed: 1 year ago10 months ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla73
You need to log in before you can comment on or make changes to this bug.