1598776 - test-verify-e10s (TV) jobs fail when a directory path is specified: "Per-test run could not find requested test: '<path>'"

Status: RESOLVED FIXED

(Developer Infrastructure :: Try, defect, P3)

Description

[Matthew N. [:MattN]]

Attachment: Example commit with try_task_config.json

I want to run test-verification on my push for just a directory but it ends up failing instead

Fuzzy query=!android !shippable !gpu !wpt&paths=toolkit/components/passwordmgr/

Logs:

[task 2019-11-22T21:54:28.004Z] 21:54:28 INFO - Per-test run found explicit request in MOZHARNESS_TEST_PATHS:
[task 2019-11-22T21:54:28.005Z] 21:54:28 INFO - set([u'toolkit/components/passwordmgr'])
[task 2019-11-22T21:54:28.513Z] 21:54:28 INFO - Per-test run updated with manifest /builds/worker/workspace/build/tests/mochitest/tests/mochitest.ini (9991 active, 489 skipped)
[task 2019-11-22T21:54:28.556Z] 21:54:28 INFO - Per-test run updated with manifest /builds/worker/workspace/build/tests/mochitest/chrome/chrome.ini (949 active, 89 skipped)
[task 2019-11-22T21:54:28.681Z] 21:54:28 INFO - Per-test run updated with manifest /builds/worker/workspace/build/tests/mochitest/browser/browser-chrome.ini (4443 active, 434 skipped)
[task 2019-11-22T21:54:28.690Z] 21:54:28 INFO - Per-test run updated with manifest /builds/worker/workspace/build/tests/mochitest/a11y/a11y.ini (290 active, 4 skipped)
[task 2019-11-22T21:54:28.793Z] 21:54:28 INFO - Per-test run updated with manifest /builds/worker/workspace/build/tests/xpcshell/tests/xpcshell.ini (3065 active, 303 skipped)
[task 2019-11-22T21:54:29.738Z] 21:54:29 INFO - Per-test run updated with manifest /builds/worker/workspace/build/tests/reftest/tests/layout/reftests/reftest.list (23885 tests)
[task 2019-11-22T21:54:29.870Z] 21:54:29 INFO - Per-test run updated with manifest /builds/worker/workspace/build/tests/reftest/tests/testing/crashtest/crashtests.list (3619 tests)
[task 2019-11-22T21:54:31.041Z] 21:54:31 INFO - Per-test run updated with manifest /builds/worker/workspace/build/tests/jsreftest/tests/jstests.list (40615 tests)
[task 2019-11-22T21:54:31.042Z] 21:54:31 FATAL - Per-test run could not find requested test 'toolkit/components/passwordmgr'
[task 2019-11-22T21:54:31.042Z] 21:54:31 FATAL - Running post_fatal callback...
[task 2019-11-22T21:54:31.042Z] 21:54:31 FATAL - Exiting -1

Comment 1

[Matthew N. [:MattN]]

This also affects "Try Chooser Enhanced" pushes with paths too:

     "env": {
         "MOZHARNESS_TEST_PATHS": "{\"mochitest-browser-chrome\": [\"toolkit/components/passwordmgr\"], \"mochitest-plain\": [\"toolkit/components/passwordmgr\"], \"xpcshell\": [\"toolkit/components/passwordmgr\"]}",
         "TRY_SELECTOR": "chooser"
     }

https://treeherder.mozilla.org/#/jobs?repo=try&revision=24cbad64a2d83a0aeae893ffb1b784b5f399b7d3

Comment 2

[Simon Giesecke [:sg] [he/him]]

A similar thing now also happens for the web-platform crashtests: https://treeherder.mozilla.org/#/jobs?repo=try&selectedJob=285782953&revision=8e4f7c5ec145d2871797afe87bdf85c6eaf65cd8

Comment 3

[Matthew N. [:MattN]]

If we want to save resources in automation this seems like it would be a good bug to fix… then more people could specify paths with their try pushes and reduce jobs runs. bdanforth just hit this issue yesterday.

Comment 5

[Intermittent Failures Robot]

3 failures in 4823 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 3

Platform breakdown:

• windows10-64: 1
• linux1804-64: 1
• windows7-32: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1598776&startday=2020-04-06&endday=2020-04-12&tree=all

Comment 6

[Geoff Brown [:gbrown]]

Specifying the path to an individual test is supported:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=92b1ddff86d49ffa770ade2d38b1439e3d61db67

but there has never been an attempt to support test verification of a directory.

Comment 7

[:Gijs (out for now; he/him)]

How hard would it be to fix this? AIUI, if you push without a path, running a verify job will do what happens on m-c, ie check tests modified in the "new" commits. That's what I would expect here with the directory...

Comment 8

[Andrew Halberstadt [:ahal]]

Not sure, I've never looked at the test-verify logic closely.. But yeah, can't imagine it would be terribly hard especially if there is already a case that does it. Not sure if this belongs here or in Task Configuration (might need changes from both). Odds of me having time to look at this anytime soon are slim.

Comment 9

[Anny Gakhokidze [:annyG]]

I've ran into this many times and the behaviour I'd like to have is to run test verify for all the newly added tests in a directory I specified.

Comment 10

[Geoff Brown [:gbrown]]

(In reply to Anny Gakhokidze [:annyG] from comment #9)

I've ran into this many times and the behaviour I'd like to have is to run test verify for all the newly added tests in a directory I specified.

You can get most of that behavior by not specifying a directory at all: If you have newly added tests in a changeset and push to try running test-verify, TV will find the new (or modified) tests in your push and run all of those. For instance, in this push, I added 3 new tests, ran 'mach try fuzzy' (without specifying any path) and selected a test-verify task:

https://treeherder.mozilla.org/jobs?repo=try&revision=64817eab06d50afd02fb56ccd21e2e85b2335177

...and the 3 new tests were verified.

Comment 11

[:Gijs (out for now; he/him)]

(In reply to Geoff Brown [:gbrown] from comment #10)

(In reply to Anny Gakhokidze [:annyG] from comment #9)

I've ran into this many times and the behaviour I'd like to have is to run test verify for all the newly added tests in a directory I specified.

You can get most of that behavior by not specifying a directory at all: If you have newly added tests in a changeset and push to try running test-verify, TV will find the new (or modified) tests in your push and run all of those. For instance, in this push, I added 3 new tests, ran 'mach try fuzzy' (without specifying any path) and selected a test-verify task:

https://treeherder.mozilla.org/jobs?repo=try&revision=64817eab06d50afd02fb56ccd21e2e85b2335177

...and the 3 new tests were verified.

Yes, but then you can't run the test in their "normal" job without running all such jobs (ie all chunks), which means using a lot more resources for all the extra jobs, plus the other directories that run in the 1 job per suite you do need. There's no way to say "run all the mochitest-bc tests in this directory, and run TV on the ones I changed".

Comment 12

[Geoff Brown [:gbrown]]

Thanks much Anny and Gijs! Your comments clarified the needs here.

Comment 13

[Geoff Brown [:gbrown]]

Attachment: Bug 1598776 - Support directory specification for test-verify; r=

When test-verify is run and a try test path is specified, as with 'mach try ... <path>',
and the path is a directory, run test-verify on every changed file in the push that
also resides in the specified directory.

Comment 14

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/4badfe252982
Support directory specification for test-verify; r=jmaher

Comment 15

[Andreea Pavel [:apavel]]

https://hg.mozilla.org/mozilla-central/rev/4badfe252982

Comment 16

[:Gijs (out for now; he/him)]

This was marked as fixed, but I just did a trypush: https://treeherder.mozilla.org/jobs?repo=try&revision=5e026b0a8293c9add8e6fa1cae4db6aaad336f3c&selectedTaskRun=MkpcC5a2Rbu4bReuArAaAQ.0 and none of the verify jobs ran successfully, erroring out with:

Per-test run could not find requested test 'uriloader/exthandler/tests/mochitest/head.js'

The push had &paths=uriloader/exthandler/tests/mochitest/.

Maybe we're not filtering out support files correctly?

Comment 17

[Geoff Brown [:gbrown]]

Yes, I neglected this case: If a path is specified, and there are changes to files in that path which are not test files, that fatal error will be triggered.

I'll push a follow-up for this case.

Comment 18

[Geoff Brown [:gbrown]]

Attachment: Bug 1598776 - Do not fail if test-verify changed file does not match specified try path; r=

Relax the check for path mismatch in test-verify. This check was previously fatal so
that cases like 'mach try ... path/to/test_<some-typo>.js' would be obvious. Now that
the specified path can use a directory to match any changed file in the directory,
relaxing the check allows for changed non-test files in the specified directory.

Comment 19

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/01f842fc69c7
Do not fail if test-verify changed file does not match specified try path; r=jmaher

Comment 20

[Cristian Brindusan [:cbrindusan]]

https://hg.mozilla.org/mozilla-central/rev/01f842fc69c7