1364561 - (wpt-sync) [meta] enable two-way sync between w3c/web-platform-tests and Mozilla repositories

Status: RESOLVED FIXED

(Testing :: web-platform-tests, enhancement)

Description

[Maja Frydrychowicz :impossibus (was :maja_zf) (needinfo me)]

(Pasting summary of initial ideas from our planning doc.)

Goal: Fast, automated, two-way sync between web-platform-tests (wpt) on GitHub and mozilla-central. Preserving or matching repository history is not essential.

The current two-way sync for wpt is a batch process that is run by hand every couple of weeks. This setup works OK, but it has significant issues

Proposed Setup: For upstreaming and downstreaming, we start synchronisation as early as possible in the development cycle i.e. as soon as a patch is posted in m-c or a PR is opened upstream.

# Downstreaming Tests (github to m-c)

PR is opened upstream
Open a bug in bugzilla assigned to a component determined from the files changed in the upstream PR
Create a try job based on Mozilla-central + the upstream changes which just runs the changed tests (or tests that are expected to change if there's a change to a support file, harness, fixtures).
Based on the results of the try run, update the expectation data, and schedule a new  run with the updated expectations, set to rebuild N times to check for stability issues.
In case any test is found to be unstable, update the metadata to disable the test(?)  and comment on the upstream PR if it is still open.
If the PR is updated restart the process, using the existing branch as the basis for the new one, excluding any disabled tests.
Once the PR merges upstream and we have a green try push, merge the patch into autoland (or inbound) and update the Bugzilla bug with details about tests which were previously in the repository and which changed status.


# Upstreaming Tests (m-c to github)

Patch is posted to bugzilla or mozreview that touches testing/web-platform/tests.
Open a corresponding PR upstream with the changes applied.
Link the PR in the bugzilla bug.
If that failed because of merge conflicts or so on comment in the bugzilla bug.
When the travis status is reported mirror that in the bugzilla bug.
When the patch is updated, update the corresponding PR.
When the patch merges to inbound, do nothing in case there's a backout.
Once the patch reaches central, merge the upstream PR.
If there is now a backout, also backout upstream(?)

Comment 1

[Maja Frydrychowicz :impossibus (was :maja_zf) (needinfo me)]

Intent to implement: https://groups.google.com/forum/#!topic/mozilla.tools/H10nWFXknKU

Comment 2

[Geoff Brown [:gbrown]]

This sounds great!

Minor request: Please include links to the try pushes in the bugzilla bug -- they may be useful for verifying when intermittent failures were introduced (inevitably, some will slip by the verification procedure).

Comment 3

[Maja Frydrychowicz :impossibus (was :maja_zf) (needinfo me)]

This has been in production since Feb. 2018, with automatic landings to mozilla-central disabled: https://groups.google.com/forum/#!topic/mozilla.dev.platform/YpfD8ftYthM

We'll be ready to turn on automatic landings in the next weeks.

Please file issues at https://github.com/mozilla/wpt-sync/issues

Comment 4

[Kagami Rosylight [:saschanaz] (they/them)]

Is there anything remaining here?