Add support for retriggering web platform tests in automation with extra options

NEW
Assigned to

Status

Testing
web-platform-tests
a month ago
a month ago

People

(Reporter: wlach, Assigned: wlach)

Tracking

(Depends on: 1 bug, Blocks: 1 bug)

Firefox Tracking Flags

(Not tracked)

Details

MozReview Requests

Submitter Diff Changes Open Issues Last Updated
Loading...
Error loading review requests:

Attachments

(3 attachments)

Much as we've done already for reftests and mochitests, it would be nice to add support for retriggering web platform tests w/ extra debugging enabled (and a reduced set of test cases).

For the most part, it looks like we could just use the existing action to run them:

https://hg.mozilla.org/mozilla-central/file/tip/taskcluster/actions/test-retrigger-action.py

The only difference in command-line invocation seems to be that instead of `--run-until-fail`, web platform test uses `--repeat-until-unexpected`.

A few options here:

1. Add an argument --repeat-until-unexpected argument to web platform tests that is a synonym of --run-until-fail
2. Replace --repeat-until-unexpected with --run-until-fail
3. Keep command argument as-is but conditionally change logic in test-retrigger-action.py

James, do you have any preference? Once we've decided I can just go ahead and implement.
Flags: needinfo?(james)
wpt uses --repeat-until-unexpected because it has to deal with the notion of expected fails; --run-until-fail then doesn't quite make sense because a test that's expected to fail shouldn't cause the repetition to stop. So I guess my first preference would be conditional logic in the caller, but if that's problematic I would accept a (hidden?) --run-until-fail alias for --repeat-until-unexpected.
Flags: needinfo?(james)
Ok, I think a bit of conditional logic here should be ok. I'll take a crack at solving this.
Assignee: nobody → wlachance
Depends on: 1350905
Comment hidden (mozreview-request)
Comment hidden (mozreview-request)
Comment hidden (mozreview-request)

Comment 6

a month ago
mozreview-review
Comment on attachment 8851641 [details]
Bug 1348833 - Add ability to retrigger web-platform-tests via taskcluster

https://reviewboard.mozilla.org/r/123914/#review126392
Attachment #8851641 - Flags: review?(james) → review+

Comment 7

a month ago
mozreview-review
Comment on attachment 8851640 [details]
Bug 1348833 - Add `test-type` tag for web platform tests

https://reviewboard.mozilla.org/r/123912/#review126444
Attachment #8851640 - Flags: review?(dustin) → review+
Comment hidden (mozreview-request)
Comment hidden (mozreview-request)
Comment hidden (mozreview-request)

Comment 11

a month ago
mozreview-review
Comment on attachment 8851639 [details]
Bug 1348833 - Bound wpt's --repeat-until-unexpected

https://reviewboard.mozilla.org/r/123910/#review126776

::: testing/web-platform/harness/wptrunner/wptrunner.py:173
(Diff revision 2)
>  
> -            while repeat_count < repeat or repeat_until_unexpected:
> +            # If repeat is None, that implies repeat_until_unexpected is
> +            # true, and we should repeat indefinitely until an unexpected
> +            # result is found. Otherwise just repeat the test the specified
> +            # number of times
> +            while repeat is None or repeat_count < repeat:

Per out irc converstaion repeat-until-unexpected should not impose an arbitary maximum number of repetitions, but using the option in concert with --repeat should.
Attachment #8851639 - Flags: review?(james) → review-
(Assignee)

Comment 12

a month ago
mozreview-review-reply
Comment on attachment 8851639 [details]
Bug 1348833 - Bound wpt's --repeat-until-unexpected

https://reviewboard.mozilla.org/r/123910/#review126776

> Per out irc converstaion repeat-until-unexpected should not impose an arbitary maximum number of repetitions, but using the option in concert with --repeat should.

That's exactly the behaviour I attempted implement. Am I missing something?

Comment 13

a month ago
mozreview-review
Comment on attachment 8851639 [details]
Bug 1348833 - Bound wpt's --repeat-until-unexpected

https://reviewboard.mozilla.org/r/123910/#review126796
Attachment #8851639 - Flags: review- → review+
So I built the above patchset on the assumption that wpt worked with as a one-click loaner (which retriggerable actions piggies). I was almost certain I had gotten this working before, but apparently I was hallucinating. So this will most likely have to wait until bug 1294820 is fixed.
Depends on: 1294820
You need to log in before you can comment on or make changes to this bug.