1509045 - Some mochitest test suites are wrongly selected by filter_tasks_by_paths because the mochitest task_regex is too loose

Status: NEW

(Testing :: General, defect, P3)

Description

[Marco Castelluccio [:marco]]

For example for the test "dom/html/test/forms/xbl/test_step_attribute.html", which has flavor "mochitest" and no subsuite, mochitest-webgl1-core is selected because the regex is "mochitest(?!-a11y|-browser|-chrome|-clip|-devtools|-gpu|-media|-screen)($|.*(-1|[^0-9])$)".

Comment 1

[Marco Castelluccio [:marco]]

It would be good, after fixing this bug, to add assertions to desktop_unittest.py (and the equivalents for marionette and wpt) to make sure that, when MOZHARNESS_TEST_PATHS is defined and has a value and if the suite is supported by MOZHARNESS_TEST_PATHS, MOZHARNESS_TEST_PATHS[suite] is not empty (so we avoid this problem reoccuring).

Comment 2

[Andrew Halberstadt [:ahal]]

Now that bug 1509048 is fixed, |mach try| selectors (which are currently the only consumers of the "task_regex"), have access to the full taskgraph, rather than just the labels.

That means that we should be able to replace all these task_regexes with `filter` functions. E.g:

    'marionette': {
        'aliases': ('mn',),
        'mach_command': 'marionette-test',
        'kwargs': {'tests': None},
        'task_filter': lambda t: t.attr['kind'] == 'test' && t.attr['unittest_suite'] == 'marionette',
    },

This would be much less fragile than relying on regexes.