Open Bug 1569559 Opened 3 months ago Updated 15 days ago

Metadata update still removing release_or_beta conditions

Categories

(Testing :: web-platform-tests, defect)

Version 3
defect
Not set

Tracking

(Not tracked)

REOPENED

People

(Reporter: jgraham, Unassigned)

References

(Regression)

Details

(Keywords: regression)

Bugbug thinks this bug is a task, but please change it back in case of error.

Type: defect → task
Type: task → defect
Regressed by: 1545143
See Also: → 1564917

The priority flag is not set for this bug.
:jgraham, could you have a look please?

For more information, please visit auto_nag documentation.

Flags: needinfo?(james)

As best as I can tell this is now working; I only get release_or_beta conditions removed when the value of the condition matches the new default (e.g. if we have if release_or_beta: PASS and then the test starts passing everywhere). I wonder if the previous incident was caused by running an update with an older gecko base not including the fixes. Please reopen if the bug reappears.

Status: NEW → RESOLVED
Closed: 2 months ago
Flags: needinfo?(james)
Resolution: --- → WORKSFORME
Blocks: 1581108
Status: RESOLVED → REOPENED
Resolution: WORKSFORME → ---

The problem here is that we are ending up with different subtest names on beta and nightly. The logic to remove stale subtests doesn't account for that possibility. Ideally we wouldn't have different test names in the PASS vs FAIL case. Can we look at the specific webaudio test that's causing problems and try to figure out why we're seeing that? If possible we should fix the test, otherwise we might need to make the cleanup code ignore conditions that it couldn't itself have created. That does mean that we may end up with stale conditions if there are release_or_beta conditions on subtests that are later removed.

Thank you for the analysis.

audioworklet-addmodule-resolution.https.html is failing on release_or_beta before any subtests have been created.
The harness then assumes the test is a single page test and creates an imaginary subtest name based on the title for the file.

This test is not using setup() to wrap the code that is failing. Wrapping in setup() changes the failure to a toplevel ERROR instead of a subtest FAIL, which the avoids the problem of unexpected subtest names. I've proposed this change in https://phabricator.services.mozilla.com/D46685.

In general, can a failure during set-up cause a different number of subtests to run? We often may not notice fallout from removing stale state because hopefully at least as many subtests would run on nightly as on beta, but the reverse seems quite possible.

It would seem reasonable that release_or_beta conditions would be considered stale only if there were data to support that, but addressing that may be less urgent with https://phabricator.services.mozilla.com/D46685.

Blocks: 1585946
See Also: → 1458068
You need to log in before you can comment on or make changes to this bug.