1568497 - Combination of --update-intermittent and --full is surprising

Status: RESOLVED FIXED

(Testing :: web-platform-tests, defect, P1)

Description

[James Graham [:jgraham]]

--update-intermittent should add new intermittents to the existing list, or create a list if there isn't one. --full should remove annotations for configurations that don't exist any longer. But --update-intermittent with --full is currently removing all the existing annotations and then adding only intermittents seen in the new results, causing unexpected data loss e.g.

[foo.html]
   expected: [TIMEOUT, OK]

with a new set of results showing ERROR and OK will end up as

[foo.html]
   expected: [ERROR, OK]

rather than as e.g.

[foo.html]
   expected: [ERROR, TIMEOUT, OK]

Comment 1

[Nikki Sharpley [:nikkis]]

I've tried to reproduce this bug locally but everything seems to be working as it should when --update-intermittent and --full are used together. May I see which logs/tests you used?

I suspect here is where we will need to make adjustments. https://searchfox.org/mozilla-central/source/testing/web-platform/tests/tools/wptrunner/wptrunner/manifestupdate.py#563
Although, new_value = self.updated_value(current_value, nodes[0].result_values) should already take --update-intermittent into account...

Comment 2

[Nikki Sharpley [:nikkis]]

Attachment: Bug 1568497 - Maintain update-intermittent for tests with conditions during a full update,

When update_intermittent is True and remove_intermittent is False,
ensure full_update does not remove any existing intermittent statuses
from tests with conditions.

Comment 3

[BugBot [:suhaib / :marco/ :calixte]]

The priority flag is not set for this bug.
:jgraham, could you have a look please?

For more information, please visit auto_nag documentation.

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/c267aa48b4df
Maintain update-intermittent for tests with conditions during a full update, r=jgraham

Comment 5

[Narcis Beleuzu [:NarcisB]]

https://hg.mozilla.org/mozilla-central/rev/c267aa48b4df

Comment 6

[Web Platform Test Sync Bot [:wpt-sync] (Matrix: #interop:mozilla.org)]

Can't merge web-platform-tests PR due to failing upstream checks:
Github PR https://github.com/web-platform-tests/wpt/pull/18505
* Taskcluster (pull_request) (https://tools.taskcluster.net/task-group-inspector/#/HVJAuQKBSrOOhiD5X-VLzg)

Comment 7

[Web Platform Test Sync Bot [:wpt-sync] (Matrix: #interop:mozilla.org)]

Can't merge web-platform-tests PR due to failing upstream checks:
Github PR https://github.com/web-platform-tests/wpt/pull/18505
* Taskcluster (pull_request) (https://tools.taskcluster.net/task-group-inspector/#/POoJBjRERpKxjy-ASwM4XA)

Comment 8

[Web Platform Test Sync Bot [:wpt-sync] (Matrix: #interop:mozilla.org)]

Upstream PR merged by moz-wptsync-bot

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/f98fe564945f
[wpt PR 18505] - [Gecko Bug 1568497] Maintain update-intermittent for tests with conditions during a full update, a=testonly

Comment 10

[Arthur Iakab [arthur_iakab]]

https://hg.mozilla.org/mozilla-central/rev/f98fe564945f