1367041 - Add support for running other browers from mach wpt.

Status: RESOLVED FIXED

(Testing :: web-platform-tests, enhancement)

Description

[James Graham [:jgraham]]

This is helpful for comaparing tests results, and can be based on the upstream runwpt insfrastructure.

Comment 1

[James Graham [:jgraham]]

Attachment: Bug 1367041 - Add |mach wpt| support for Chrome, Edge and Servo,

Using the wptrun infrastructure from upstream, it is now posible to
make it easy to run web-platform-tests in other browsers. The syntax
used is

mach wpt --product [chrome|servo|edge] [tests]

This will try to use the selected product; possibly prompting to
install dependencies like the WebDriver implementation. For servo if
the install isn't on the PATH then --binary can be used to point to
the actual location.

Because manifest metadata is kept in the same directory as expectation
data and we don't want to reuse Firefox expectation data for other
browsers, a new products subdirectory is introduced and added to the
ignore files. This will contain a subdirectory for each product into
which a copy of the test manifest is placed. It may also be used to
store any expectation data for the other products, in the same way as
testing/web-platform/meta.

Review commit: https://reviewboard.mozilla.org/r/141858/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/141858/

Comment 2

[David Burns :automatedtester]

Comment on attachment 8870405 [details]
Bug 1367041 - Add |mach wpt| support for Chrome, Edge and Servo,

https://reviewboard.mozilla.org/r/141858/#review145508

::: .gitignore:93
(Diff revision 1)
>  GPATH
>  
>  # Git clone directory for updating web-platform-tests
>  testing/web-platform/sync/
>  
> +# Third party metadata for web-platform-tests

Should there be a .hgignore version too?

::: .gitignore:94
(Diff revision 1)
>  
>  # Git clone directory for updating web-platform-tests
>  testing/web-platform/sync/
>  
> +# Third party metadata for web-platform-tests
> +testing/web-platform/products/

Should this be reviewed by a build peer?

::: testing/web-platform/README.md:248
(Diff revision 1)
> -   e.g. `~/bin`
> -
> -2. In your gecko source tree activate the virtualenv created by mach,
> -   since this has most dependencies already installed. This is typically
> -   in objdir/_virtualenv and is activated via e.g.
> -
> +
> +By default these browsers run without expectation metadata, but it can
> +be added in the `testing/web-platform/products/<product>`
> +directory. To run with the same metadata as for Firefox (so that
> +differences are reported as unexpected results), pass `--meta
> +testing/web-platform/meta` to the machc command.

typo `machc` should be `mach`

Comment 3

[James Graham [:jgraham]]

Comment on attachment 8870405 [details]
Bug 1367041 - Add |mach wpt| support for Chrome, Edge and Servo,

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/141858/diff/1-2/

Comment 4

[(not currently active) Ted Mielczarek]

Comment on attachment 8870405 [details]
Bug 1367041 - Add |mach wpt| support for Chrome, Edge and Servo,

https://reviewboard.mozilla.org/r/141858/#review147804

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/448b7c4ed09e
Add |mach wpt| support for Chrome, Edge and Servo, r=AutomatedTester

Comment 6

[Carsten Book [:Tomcat]]

backed this out for bustage in https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=3ae791d519fd85a6b7479f0e8898b062633afda7 because this or the other bug caused https://hg.mozilla.org/integration/mozilla-inbound/rev/34d32278afbfa8a25c6a68e9b62b5efa3b4aeef2

Comment 7

[Pulsebot]

Backout by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/ab34e949f80e
Backed out changeset 448b7c4ed09e

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/64ff9b546c6e
Add |mach wpt| support for Chrome, Edge and Servo, r=AutomatedTester

Comment 9

[Carsten Book [:Tomcat]]

backed out in https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=cef7da731412df2175b2f28db7ad0660a3ecc313 for wpt runner failures from this or the other change like https://treeherder.mozilla.org/logviewer.html#?job_id=103371942&repo=mozilla-inbound

Comment 10

[Pulsebot]

Backout by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/f7553df81f14
Backed out changeset 64ff9b546c6e

Comment 11

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/ecb85cc1ee3e
Add |mach wpt| support for Chrome, Edge and Servo, r=AutomatedTester

Comment 12

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/ecb85cc1ee3e