1482413 - Improve diagnostics for TV-bf

Status: RESOLVED FIXED

(Testing :: General, enhancement, P1)

Description

[Geoff Brown [:gbrown]]

If a bad path is passed to TV-bf, it runs green and does nothing. I think it should fail and report the bad path.

Comment 1

[Geoff Brown [:gbrown]]

Attachment: log requested path, fail if path invalid/not found

Some examples of failures and passes at

https://treeherder.mozilla.org/#/jobs?repo=try&revision=8ba248923bfe1c04406d76653122e224c04a2caf

(I have fixed the lint failure.)


Do you have ideas for related logging improvements?

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8999199 [details] [diff] [review]
log requested path, fail if path invalid/not found

Review of attachment 8999199 [details] [diff] [review]:
-----------------------------------------------------------------

overall this is a positive thing.  In terms of what else we can do- I would like to consider turning the job orange or maybe red if we cannot find the path specified.  There are 2 cases here:
1) the file is not found
2) we tried to run the file (i.e. launched the browser) and no tests were run

If we were to turn the job orange, that would need to account for tests that have duplicate names which we skip, likewise some other random exceptions with names ( like test-oop-extensions: https://searchfox.org/mozilla-central/source/browser/components/extensions/test/browser/browser-remote.ini )

for tv-bf we should be more specific- we are running this with the intention of a specified test and the job should fail.  Overall I think you caught the main case (#1 above) and we will have to sort out some details.

What other logging can we do- maybe details about total files found, total files not able to run (ex: skip-if), total chunks, paths for finding files.  Getting this in sooner will help out.

Comment 3

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/71af60e17c8b
Improve mozharness diagnostics in per-test runs, especially for TV-bf; r=jmaher

Comment 4

[Andreea Pavel [:apavel]]

https://hg.mozilla.org/mozilla-central/rev/71af60e17c8b

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/a71e4a20a0a6
More mozharness logging updates for test-verify; r=me,a=test-only

Comment 6

[Daniel Varga [:dvarga]]

https://hg.mozilla.org/mozilla-central/rev/a71e4a20a0a6

Comment 7

[Geoff Brown [:gbrown]]

Attachment: log some manifest entry counts, identify skipped test when possible

A few more log statements, inspired by Joel's feedback. There's no reasonably convenient way to identify skipped reftests, or wpt. I also looked at logging in the taskcluster code, but I think it is better not to clutter the decision task.

Comment 8

[Geoff Brown [:gbrown]]

https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&selectedJob=201290342&revision=615776a8b2ef9e06b39b498cc51bf888657d84c2

[task 2018-09-24T22:55:35.085Z] 22:55:35     INFO - Per-test run updated with manifest /builds/worker/workspace/build/tests/mochitest/tests/mochitest.ini (9801 active, 472 skipped)
[task 2018-09-24T22:55:35.149Z] 22:55:35     INFO - Per-test run updated with manifest /builds/worker/workspace/build/tests/mochitest/chrome/chrome.ini (980 active, 104 skipped)
[task 2018-09-24T22:55:35.354Z] 22:55:35     INFO - Per-test run updated with manifest /builds/worker/workspace/build/tests/mochitest/browser/browser-chrome.ini (4122 active, 432 skipped)
[task 2018-09-24T22:55:35.367Z] 22:55:35     INFO - Per-test run updated with manifest /builds/worker/workspace/build/tests/mochitest/a11y/a11y.ini (298 active, 7 skipped)
[task 2018-09-24T22:55:35.505Z] 22:55:35     INFO - Per-test run updated with manifest /builds/worker/workspace/build/tests/xpcshell/tests/xpcshell.ini (2988 active, 123 skipped)
[task 2018-09-24T22:55:36.958Z] 22:55:36     INFO - Per-test run updated with manifest /builds/worker/workspace/build/tests/reftest/tests/layout/reftests/reftest.list (23848 tests)
[task 2018-09-24T22:55:37.157Z] 22:55:37     INFO - Per-test run updated with manifest /builds/worker/workspace/build/tests/reftest/tests/testing/crashtest/crashtests.list (3603 tests)
[task 2018-09-24T22:55:38.570Z] 22:55:38     INFO - Per-test run updated with manifest /builds/worker/workspace/build/tests/jsreftest/tests/jstests.list (33989 tests)
[task 2018-09-24T22:55:38.571Z] 22:55:38     INFO - Per-test run found test testing/mochitest/tests/Harness_sanity/test_sanity.html (plain/None)
[task 2018-09-24T22:55:38.572Z] 22:55:38     INFO - 'testing/mochitest/tests/Harness_sanity/test_importInMainProcess.html' has been skipped on this platform.

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/d66dcdb73140
Improve diagnostics for test-verify and per-test coverage; r=me,a=test-only

Comment 10

[Eliza Balazs [:ebalazs_]]

https://hg.mozilla.org/mozilla-central/rev/d66dcdb73140