1348833 - Add support for retriggering web platform tests in automation with extra options

Status: NEW

(Testing :: web-platform-tests, enhancement)

Description

[William Lachance (:wlach)]

Much as we've done already for reftests and mochitests, it would be nice to add support for retriggering web platform tests w/ extra debugging enabled (and a reduced set of test cases).

For the most part, it looks like we could just use the existing action to run them:

https://hg.mozilla.org/mozilla-central/file/tip/taskcluster/actions/test-retrigger-action.py

The only difference in command-line invocation seems to be that instead of `--run-until-fail`, web platform test uses `--repeat-until-unexpected`.

A few options here:

1. Add an argument --repeat-until-unexpected argument to web platform tests that is a synonym of --run-until-fail
2. Replace --repeat-until-unexpected with --run-until-fail
3. Keep command argument as-is but conditionally change logic in test-retrigger-action.py

James, do you have any preference? Once we've decided I can just go ahead and implement.

Comment 1

[James Graham [:jgraham]]

wpt uses --repeat-until-unexpected because it has to deal with the notion of expected fails; --run-until-fail then doesn't quite make sense because a test that's expected to fail shouldn't cause the repetition to stop. So I guess my first preference would be conditional logic in the caller, but if that's problematic I would accept a (hidden?) --run-until-fail alias for --repeat-until-unexpected.

Comment 2

[William Lachance (:wlach)]

Ok, I think a bit of conditional logic here should be ok. I'll take a crack at solving this.

Comment 3

[William Lachance (:wlach)]

Attachment: Bug 1348833 - Bound wpt's --repeat-until-unexpected

Instead of repeating the test an unlimited number of times, only
repeat in the number of times specified by --repeat (or 30, if
not specified)

Review commit: https://reviewboard.mozilla.org/r/123910/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/123910/

Comment 4

[William Lachance (:wlach)]

Attachment: Bug 1348833 - Add `test-type` tag for web platform tests

Review commit: https://reviewboard.mozilla.org/r/123912/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/123912/

Comment 5

[William Lachance (:wlach)]

Attachment: Bug 1348833 - Add ability to retrigger web-platform-tests via taskcluster

Review commit: https://reviewboard.mozilla.org/r/123914/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/123914/

Comment 6

[James Graham [:jgraham]]

Comment on attachment 8851641 [details]
Bug 1348833 - Add ability to retrigger web-platform-tests via taskcluster

https://reviewboard.mozilla.org/r/123914/#review126392

Comment 7

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8851640 [details]
Bug 1348833 - Add `test-type` tag for web platform tests

https://reviewboard.mozilla.org/r/123912/#review126444

Comment 8

[William Lachance (:wlach)]

Comment on attachment 8851639 [details]
Bug 1348833 - Bound wpt's --repeat-until-unexpected

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/123910/diff/1-2/

Comment 9

[William Lachance (:wlach)]

Comment on attachment 8851640 [details]
Bug 1348833 - Add `test-type` tag for web platform tests

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/123912/diff/1-2/

Comment 10

[William Lachance (:wlach)]

Comment on attachment 8851641 [details]
Bug 1348833 - Add ability to retrigger web-platform-tests via taskcluster

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/123914/diff/1-2/

Comment 11

[James Graham [:jgraham]]

Comment on attachment 8851639 [details]
Bug 1348833 - Bound wpt's --repeat-until-unexpected

https://reviewboard.mozilla.org/r/123910/#review126776

::: testing/web-platform/harness/wptrunner/wptrunner.py:173
(Diff revision 2)
>  
> -            while repeat_count < repeat or repeat_until_unexpected:
> +            # If repeat is None, that implies repeat_until_unexpected is
> +            # true, and we should repeat indefinitely until an unexpected
> +            # result is found. Otherwise just repeat the test the specified
> +            # number of times
> +            while repeat is None or repeat_count < repeat:

Per out irc converstaion repeat-until-unexpected should not impose an arbitary maximum number of repetitions, but using the option in concert with --repeat should.

Comment 12

[William Lachance (:wlach)]

Comment on attachment 8851639 [details]
Bug 1348833 - Bound wpt's --repeat-until-unexpected

https://reviewboard.mozilla.org/r/123910/#review126776

> Per out irc converstaion repeat-until-unexpected should not impose an arbitary maximum number of repetitions, but using the option in concert with --repeat should.

That's exactly the behaviour I attempted implement. Am I missing something?

Comment 13

[James Graham [:jgraham]]

Comment on attachment 8851639 [details]
Bug 1348833 - Bound wpt's --repeat-until-unexpected

https://reviewboard.mozilla.org/r/123910/#review126796

Comment 14

[William Lachance (:wlach)]

So I built the above patchset on the assumption that wpt worked with as a one-click loaner (which retriggerable actions piggies). I was almost certain I had gotten this working before, but apparently I was hallucinating. So this will most likely have to wait until bug 1294820 is fixed.

Comment 15

[William Lachance (:wlach)]

Not sure if anyone took up this work elsewhere, regardless it's not something I'm going to get to anytime soon!