1383922 - Allow try-test-paths syntax to refer to wpt subtypes

Status: RESOLVED FIXED

(Testing :: web-platform-tests, enhancement)

Description

[Maja Frydrychowicz :impossibus (was :maja_zf) (needinfo me)]

It's possible to use --try-test-paths syntax to schedule wdspec tests on platforms where they aren't configured to run, for example: try: -b do -p all -u web-platform-tests-1 --try-test-paths web-platform-tests:testing/web-platform/tests/webdriver

This may occur when the wpt-sync service pushes to try to check the tests affected by a change that we want to downstream from the web-platform-tests repository.

Comment 1

[Maja Frydrychowicz :impossibus (was :maja_zf) (needinfo me)]

Attachment: Bug 1383922 - Allow try-test-paths syntax to refer to wpt subtypes

We want to accommodate the wpt-sync service, which
pushes to try with --try-test-paths to check tests affected by an
upstream web-platform-tests change. The try push is against all
available builds/platforms, which means it may request wdspec tests to
run in a context where we don't usually run them and/or no geckodriver
binary is available.

Rather than erroring out with non-linux64 builds, remove wdspec
tests from try_test_paths to allow all other requested tests to run.

Outside of wpt-sync, wdspec tests are requested on try by a small
set of developers who are familiar with the CI infrastructure, so
quietly omitting these tests from "invalid" try_test_paths pushes is
unlikely to hide any critical bugs. The catch is that we have to
remember to change this if we decide to enable wdspec tests on more
platforms.

Review commit: https://reviewboard.mozilla.org/r/160700/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/160700/

Comment 2

[Maja Frydrychowicz :impossibus (was :maja_zf) (needinfo me)]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=b663a2bdc5a13bb95df0932451846547801ea035

Comment 3

[Maja Frydrychowicz :impossibus (was :maja_zf) (needinfo me)]

Comment on attachment 8889657 [details]
Bug 1383922 - Allow try-test-paths syntax to refer to wpt subtypes

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/160700/diff/1-2/

Comment 4

[James Graham [:jgraham]]

Comment on attachment 8889657 [details]
Bug 1383922 - Allow try-test-paths syntax to refer to wpt subtypes

https://reviewboard.mozilla.org/r/160700/#review166186

I'm a little worried about the special-casing here. For example I'm thinking of enabling web-paltform-tests-reftests on QuantumRender builds, and I don't want try to error out because it tries to run testharness tests on that platform. I think we have access to the taskgraph, somehow? Can we use that to make a more generic solution here?

Comment 5

[Maja Frydrychowicz :impossibus (was :maja_zf) (needinfo me)]

In that case, I think the best option is to generate try syntax that refers to the different kinds of wpt jobs. Either teach |mach try| or implement it directly in the wpt-sync service.

So, given [testing/web-platform/tests/2dcontext, testing/web-platform/tests/cookies], generate this:

try: -b do -p linux -u web-platform-tests-1,web-platform-tests-e10s-1,web-platform-tests-reftests-1,web-platform-tests-reftests-e10s-1 --try-test-paths web-platform-tests-reftests:testing/web-platform/tests/2dcontext web-platform-tests:testing/web-platform/tests/cookies 

Instead of this, which lumps everything under "web-platform-tests" -- this is what ./mach try does now:

try: -b do -p linux -u web-platform-tests-1,web-platform-tests-e10s-1 --try-test-paths web-platform-tests:testing/web-platform/tests/2dcontext web-platform-tests:testing/web-platform/tests/cookies web-platform-tests:testing/web-platform/tests/webdriver

Comment 6

[James Graham [:jgraham]]

Generating the right jobs for each wpt test type seems like a good idea to me.

Comment 7

[Maja Frydrychowicz :impossibus (was :maja_zf) (needinfo me)]

Comment on attachment 8889657 [details]
Bug 1383922 - Allow try-test-paths syntax to refer to wpt subtypes

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/160700/diff/2-3/

Comment 8

[Maja Frydrychowicz :impossibus (was :maja_zf) (needinfo me)]

With --try-test-paths:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=359fd9d5f5e1856032083d46239a8ada67981346

Comment 9

[James Graham [:jgraham]]

Comment on attachment 8889657 [details]
Bug 1383922 - Allow try-test-paths syntax to refer to wpt subtypes

https://reviewboard.mozilla.org/r/160700/#review168704

I admit I'm not sure where this works out which web-platform-tests are of which type, but based on the try push it apparently works, and I at least understand why it works given that it manages that, so it seems fine to me :)

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/4e799d2c0555
Allow try-test-paths syntax to refer to wpt subtypes r=jgraham

Comment 11

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/4e799d2c0555