1401684 - Add option to disable unstable tests with wpt-update

Status: RESOLVED FIXED

(Testing :: web-platform-tests, enhancement)

Description

[Maja Frydrychowicz :impossibus (was :maja_zf) (needinfo me)]

Given a set of log files from multiple runs of the same tests, wpt-update should update metadata to disable tests that have inconsistent results.

Comment 1

[Maja Frydrychowicz :impossibus (was :maja_zf) (needinfo me)]

Attachment: Bug 1401684 - Add option to disable unstable tests with wpt-update;

Add a --stability option to wpt-update. If test results are inconsistent
across many runs, the test is disabled for the condition(s) indicated by
the corresponding log files. The --stability option is used to specify
a reason for disabling the tests (typically a bug number), which is
recorded in metadata.

Review commit: https://reviewboard.mozilla.org/r/181874/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/181874/

Comment 2

[James Graham [:jgraham]]

Comment on attachment 8910439 [details]
Bug 1401684 - Add option to disable unstable tests with wpt-update;

https://reviewboard.mozilla.org/r/181874/#review187554

::: testing/web-platform/tests/tools/wptrunner/wptrunner/manifestupdate.py:249
(Diff revision 1)
>                          self.new_expected,
>                          property_order=self.root.property_order,
>                          boolean_properties=self.root.boolean_properties)
> -                except ConditionError:
> +                except ConditionError as e:
> +                    if stability is not None:
> +                       self.set("disabled", stability or "unstable", e.cond.children[0])

Would be good to get a list of disabled tests here and output it somehow so that we can tell which tests were disabled.

Comment 3

[Maja Frydrychowicz :impossibus (was :maja_zf) (needinfo me)]

Comment on attachment 8910439 [details]
Bug 1401684 - Add option to disable unstable tests with wpt-update;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/181874/diff/1-2/

Comment 4

[Maja Frydrychowicz :impossibus (was :maja_zf) (needinfo me)]

Comment on attachment 8910439 [details]
Bug 1401684 - Add option to disable unstable tests with wpt-update;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/181874/diff/2-3/

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/3f6b16e6955a
Add option to disable unstable tests with wpt-update; r=jgraham

Comment 6

[Maja Frydrychowicz :impossibus (was :maja_zf) (needinfo me)]

Comment on attachment 8910439 [details]
Bug 1401684 - Add option to disable unstable tests with wpt-update;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/181874/diff/3-4/

Comment 7

[Mozilla Autoland]

We're sorry - something has gone wrong while rewriting or rebasing your commits. The commits being pushed no longer match what was requested. Please file a bug.

Comment 8

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/3f6b16e6955a