1490761 - [wpt-sync] Sync PR 12657 - Validate pull requests in TaskCluster

Status: RESOLVED FIXED

(Testing :: web-platform-tests, enhancement, P4)

Description

[Web Platform Test Sync Bot [:wpt-sync] (Matrix: #interop:mozilla.org)]

Sync web-platform-tests PR 12657 into mozilla-central (this bug is closed when the sync is complete).

PR: https://github.com/web-platform-tests/wpt/pull/12657
Details from upstream follow.

Mike Pennisi <[e-mail]> wrote:
>  Validate pull requests in TaskCluster
>  
>  Extend the configuration for the TaskCluster service to generate tasks in
>  response to GitHub pull requests. Each pull request should create one cluster
>  containing four tasks, all of which operate on the affected tests as identified
>  by `wpt tests-affected`:
>  
>  - run the tests in Chrome and publish the results as a gzipped artifact
>  - run the tests in Firefox and publish the results as a gzipped artifact
>  - verify the stability of the tests in Chrome
>  - verify the stability of the tests in Firefox
>  
>  The former two tasks are intended for future use in comparing test results with
>  data available on https://wpt.fyi. This will be informative only. The latter
>  two tasks are intended to verify that the affected tests are stable. This will
>  influence whether or not the pull request may be merged.
>  
>  A demonstration of expected functionality is available on Bocoup's fork of WPT:
>  
>  - `master` branch (this functionality was implemented previously, and this pull
>    request is not intended to influence the behavior we see today)
>    - [`master` branch of WPT fork](https://github.com/bocoup/wpt/commits/master)
>    - [Task cluster](https://tools.taskcluster.net/groups/dSGgOEa3S9y5ebTLBKIVMw) 
>  - Pull request validation
>    - [pull request](https://github.com/bocoup/wpt/pull/4)
>    - [Task cluster](https://tools.taskcluster.net/groups/ZhRXqeYgTZS3C4TifTRzFg)
>  
>  It seems wise to vet this in WPT for a bit before allowing the results to
>  control whether pull requests may be merged. The configuration proposed here
>  will permit unstable pull requests, but it stores the status of the check in a
>  dedicated artifact. After some time, we can use this to compare the behavior
>  between the current TravisCI-powered checks and this new one. If they align, we
>  can revert the second commit.
>  
>  Due to some changes to indentation, using `-w` or the [`?w=1` query string
>  parameter](?w=1) may make this change set easier to review.
>  
>  [fixes #10503]

Comment 1

[Web Platform Test Sync Bot [:wpt-sync] (Matrix: #interop:mozilla.org)]

The PR was not expected to affect any tests, but the try push wasn't a success. Check the try results for infrastructure issues

Comment 2

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/85c26a94f8a0
[wpt PR 12657] - Validate pull requests in TaskCluster, a=testonly

Comment 3

[Web Platform Test Sync Bot [:wpt-sync] (Matrix: #interop:mozilla.org)]

Result changes from PR not available.

Comment 4

[Dorel Luca [:dluca]]

https://hg.mozilla.org/mozilla-central/rev/85c26a94f8a0

Comment 5

[Dorel Luca [:dluca]]

Backed out for WPT mass failure at /fullscreen/idlharness.window.html

Backout: https://hg.mozilla.org/mozilla-central/rev/e1a681345945ab8c8b8270afcd2f62bf042226c2

Failure push: 
https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&resultStatus=testfailed,busted,exception&classifiedState=unclassified&revision=be6801cdeb537afa655bbd67d12ead8dc01c6c38

https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=3f3f67d1c0bf5359ac39ddddec5999ceb928e7ca

Failure log: https://treeherder.mozilla.org/logviewer.html#?job_id=199470643&repo=mozilla-central&lineNumber=1487

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/31d722ce7495
[wpt PR 12657] - Validate pull requests in TaskCluster, a=testonly

Comment 7

[Andrei Ciure[:aciure]]

https://hg.mozilla.org/mozilla-central/rev/31d722ce7495