1796603 - Metadata update still sometimes makes unnecessary updates

Status: RESOLVED FIXED

(Testing :: web-platform-tests, defect)

Description

[James Graham [:jgraham]]

Despite the changes in 1579168 we're still seeing unnecessary changes in metadata updates including e.g. changing [OK, TIMEOUT] to [TIMEOUT, OK], or reordering unchanged fields.

For intermittents in particular, we'd like the update to enforce the invariant that we don't make automatic changes as long as the new field value is a subset of the old field value (i.e. if something matches a rule with expected: [PASS, FAIL, TIMEOUT] and the new data suggests a rule of [FAIL, PASS] we shouldn't make an update).

Comment 1

[BugBot [:suhaib / :marco/ :calixte]]

The severity field is not set for this bug.
:jgraham, could you have a look please?

For more information, please visit auto_nag documentation.

Comment 2

[James Graham [:jgraham]]

Attachment: Bug 1796603 - Don't mark test for update if all statuses are in known intermittents,

Previously we had the idea that if we had a test marked e.g. [OK,
TIMEOUT] but we saw more timeouts that fails we might want to update
this to [TIMEOUT, OK]. In practice this adds a lot of unnecessary
churn, and not much value.

The most basic thing we can do here is simply not run metadata updates
at all for a test when all of the results we get are either expected
or part of the known intermittents.

Comment 3

[James Graham [:jgraham]]

Attachment: Bug 1796603 - Reuse existing conditions when all values are subsets of the existing value,

If we have to update some of the metadata for a test (e.g. because one
subtest got different results), for the subtests that didn't change we
want to avoid creating new conditions where possible.

Previously we reused existing conditions in the case that all the new
values for run info matching that condition were the same, but didn't
handle the case that the values are different, but subsets of the
existing value.

This adds additional handling for the subset case, so that when we
have a condition like [TIMEOUT, OK] and various runs give a mix of
those existing values, we just reuse the existing conditional rather
than creating more specific ones.

Comment 4

[BugBot [:suhaib / :marco/ :calixte]]

There are some r+ patches which didn't land and no activity in this bug for 2 weeks.
:jgraham, could you have a look please?
If you still have some work to do, you can add an action "Plan Changes" in Phabricator.
For more information, please visit BugBot documentation.

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/24074bb9dbe1
Don't mark test for update if all statuses are in known intermittents, r=Sasha
https://hg.mozilla.org/integration/autoland/rev/40cf739f51d9
Reuse existing conditions when all values are subsets of the existing value, r=Sasha

Comment 6

[Web Platform Test Sync Bot [:wpt-sync] (Matrix: #interop:mozilla.org)]

Created web-platform-tests PR https://github.com/web-platform-tests/wpt/pull/39961 for changes under testing/web-platform/tests

Comment 7

[Sandor Molnar[:smolnar]]

https://hg.mozilla.org/mozilla-central/rev/24074bb9dbe1
https://hg.mozilla.org/mozilla-central/rev/40cf739f51d9

Comment 8

[Web Platform Test Sync Bot [:wpt-sync] (Matrix: #interop:mozilla.org)]

Upstream PR merged by moz-wptsync-bot