1574864 - mach try fuzzy with paths is producing a mismatch of suites and tests

Status: RESOLVED FIXED

(Developer Infrastructure :: Try, defect)

Description

[James Graham [:jgraham]]

Running mach try fuzzy q='web-platform-tests !macosx !shippable' testing/web-platform/tests/infrastructure/ produces the following try_task_config.json:

{
    "tasks": [
        "test-android-em-7.0-x86_64/debug-geckoview-web-platform-tests-e10s-1",
        "test-android-em-7.0-x86_64/debug-geckoview-web-platform-tests-reftests-e10s-1",
        "test-android-em-7.0-x86_64/opt-geckoview-web-platform-tests-e10s-1",
        "test-android-em-7.0-x86_64/opt-geckoview-web-platform-tests-reftests-e10s-1",
        "test-linux64-asan/opt-web-platform-tests-e10s-1",
        "test-linux64-asan/opt-web-platform-tests-reftests-e10s-1",
        "test-linux64-asan/opt-web-platform-tests-wdspec-e10s-1",
        "test-linux64-qr/debug-web-platform-tests-e10s-1",
        "test-linux64-qr/debug-web-platform-tests-reftests-e10s-1",
        "test-linux64-qr/debug-web-platform-tests-wdspec-e10s-1",
        "test-linux64-qr/opt-web-platform-tests-e10s-1",
        "test-linux64-qr/opt-web-platform-tests-reftests-e10s-1",
        "test-linux64-qr/opt-web-platform-tests-wdspec-e10s-1",
        "test-linux64/debug-web-platform-tests-e10s-1",
        "test-linux64/debug-web-platform-tests-reftests-e10s-1",
        "test-linux64/debug-web-platform-tests-reftests-sw-e10s-1",
        "test-linux64/debug-web-platform-tests-sw-e10s-1",
        "test-linux64/debug-web-platform-tests-wdspec-e10s-1",
        "test-linux64/debug-web-platform-tests-wdspec-sw-e10s-1",
        "test-linux64/opt-web-platform-tests-e10s-1",
        "test-linux64/opt-web-platform-tests-reftests-e10s-1",
        "test-linux64/opt-web-platform-tests-wdspec-e10s-1",
        "test-windows10-64-qr/debug-web-platform-tests-e10s-1",
        "test-windows10-64-qr/debug-web-platform-tests-reftests-e10s-1",
        "test-windows10-64-qr/debug-web-platform-tests-wdspec-e10s-1",
        "test-windows10-64-qr/opt-web-platform-tests-e10s-1",
        "test-windows10-64-qr/opt-web-platform-tests-reftests-e10s-1",
        "test-windows10-64-qr/opt-web-platform-tests-wdspec-e10s-1",
        "test-windows10-64/debug-web-platform-tests-e10s-1",
        "test-windows10-64/debug-web-platform-tests-reftests-e10s-1",
        "test-windows10-64/debug-web-platform-tests-wdspec-e10s-1",
        "test-windows10-64/opt-web-platform-tests-e10s-1",
        "test-windows10-64/opt-web-platform-tests-reftests-e10s-1",
        "test-windows10-64/opt-web-platform-tests-wdspec-e10s-1",
        "test-windows7-32/debug-web-platform-tests-e10s-1",
        "test-windows7-32/debug-web-platform-tests-reftests-e10s-1",
        "test-windows7-32/debug-web-platform-tests-wdspec-e10s-1",
        "test-windows7-32/opt-web-platform-tests-e10s-1",
        "test-windows7-32/opt-web-platform-tests-reftests-e10s-1",
        "test-windows7-32/opt-web-platform-tests-wdspec-e10s-1"
    ],
    "templates": {
        "artifact": {
            "enabled": "1"
        },
        "env": {
            "MOZHARNESS_TEST_PATHS": "{\"web-platform-tests-reftest\": [\"testing/web-platform/tests/infrastructure\"]}",
            "TRY_SELECTOR": "fuzzy"
        }
    },
    "version": 1
}

Note that this (correctly) selects all the wpt jobs but only applies the path filter to reftests, creating a nonsense try job that tries to run all the non-reftests in a single chunk per type and ends up just wasting resources.

It seem the behaviour stems from resolve_tests_by_suite doing an exact match on the passed in paths, whereas I'm expecting a prefix match (and presumably we are doing a prefix match to select the tasks in the first place, since passing in a directory with only wpt reftests correctly selects only reftest jobs). In the wpt case we get reftests because there happens to be a reftest directly in the passed in directory, but not other tests because they are exclusively in subdirectories.

One thing that's unclear to me is whether it's acceptable for that function to simply return a mapping from sutie to input path, assuming that the harness can cope with finding all the tests in subdirectories under that path so that when the input is {"testing/web-platform/infrastructure/"} we can just return {"web-platform-tests-reftests": ["testing/web-platform/infrastructure"], "web-platform-tests-testharness": ["testing/web-platform/infrastructure"], "web-platform-tests-wdspec": ["testing/web-platform/infrastructure"]} rather than the paths to actual (directories containing) tests.

Comment 1

[James Graham [:jgraham]]

Attachment: Bug 1574864 - Make resolve_tests_by_suite prefix match paths,

The current behaviour of resolve_tests_by suite is to look for exact
matches between the passed in path and the paths associated with each
test. But that doesn't work for cases where you want to path in an
ancestor path and run all the tests under that path.

This changeset makes resolve_tests_by_suite return a list of the input
paths that prefix-match one or more tests in a given suite, sorted by
suite. It is assumed that the harness itself will be able to handle
such inputs even when they don't correspond to a directory that
actualy contains tests.

Comment 2

[James Graham [:jgraham]]

Created a patch, but I'm not convinced it's correct; particularly the part about returning input values rather than resolved values, since this may not match the behaviour of all harnesses.

Comment 3

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/cb2287a46423
Make resolve_tests_by_suite prefix match paths, r=ahal

Comment 4

[Alexandru Michis [:malexandru]]

https://hg.mozilla.org/mozilla-central/rev/cb2287a46423