Closed Bug 1482413 Opened 3 years ago Closed 3 years ago

Improve diagnostics for TV-bf


(Testing :: General, enhancement, P1)

Version 3


(firefox64 fixed)

Tracking Status
firefox64 --- fixed


(Reporter: gbrown, Assigned: gbrown)


(Blocks 1 open bug)



(2 files)

If a bad path is passed to TV-bf, it runs green and does nothing. I think it should fail and report the bad path.
Some examples of failures and passes at

(I have fixed the lint failure.)

Do you have ideas for related logging improvements?
Attachment #8999199 - Flags: review?(jmaher)
Comment on attachment 8999199 [details] [diff] [review]
log requested path, fail if path invalid/not found

Review of attachment 8999199 [details] [diff] [review]:

overall this is a positive thing.  In terms of what else we can do- I would like to consider turning the job orange or maybe red if we cannot find the path specified.  There are 2 cases here:
1) the file is not found
2) we tried to run the file (i.e. launched the browser) and no tests were run

If we were to turn the job orange, that would need to account for tests that have duplicate names which we skip, likewise some other random exceptions with names ( like test-oop-extensions: )

for tv-bf we should be more specific- we are running this with the intention of a specified test and the job should fail.  Overall I think you caught the main case (#1 above) and we will have to sort out some details.

What other logging can we do- maybe details about total files found, total files not able to run (ex: skip-if), total chunks, paths for finding files.  Getting this in sooner will help out.
Attachment #8999199 - Flags: review?(jmaher) → review+
Pushed by
Improve mozharness diagnostics in per-test runs, especially for TV-bf; r=jmaher
Keywords: leave-open
Priority: -- → P1
Blocks: 1483292
Depends on: 1483421
Blocks: test-verify
Pushed by
More mozharness logging updates for test-verify; r=me,a=test-only
Keywords: leave-open
A few more log statements, inspired by Joel's feedback. There's no reasonably convenient way to identify skipped reftests, or wpt. I also looked at logging in the taskcluster code, but I think it is better not to clutter the decision task.
Attachment #9011631 - Flags: review+

[task 2018-09-24T22:55:35.085Z] 22:55:35     INFO - Per-test run updated with manifest /builds/worker/workspace/build/tests/mochitest/tests/mochitest.ini (9801 active, 472 skipped)
[task 2018-09-24T22:55:35.149Z] 22:55:35     INFO - Per-test run updated with manifest /builds/worker/workspace/build/tests/mochitest/chrome/chrome.ini (980 active, 104 skipped)
[task 2018-09-24T22:55:35.354Z] 22:55:35     INFO - Per-test run updated with manifest /builds/worker/workspace/build/tests/mochitest/browser/browser-chrome.ini (4122 active, 432 skipped)
[task 2018-09-24T22:55:35.367Z] 22:55:35     INFO - Per-test run updated with manifest /builds/worker/workspace/build/tests/mochitest/a11y/a11y.ini (298 active, 7 skipped)
[task 2018-09-24T22:55:35.505Z] 22:55:35     INFO - Per-test run updated with manifest /builds/worker/workspace/build/tests/xpcshell/tests/xpcshell.ini (2988 active, 123 skipped)
[task 2018-09-24T22:55:36.958Z] 22:55:36     INFO - Per-test run updated with manifest /builds/worker/workspace/build/tests/reftest/tests/layout/reftests/reftest.list (23848 tests)
[task 2018-09-24T22:55:37.157Z] 22:55:37     INFO - Per-test run updated with manifest /builds/worker/workspace/build/tests/reftest/tests/testing/crashtest/crashtests.list (3603 tests)
[task 2018-09-24T22:55:38.570Z] 22:55:38     INFO - Per-test run updated with manifest /builds/worker/workspace/build/tests/jsreftest/tests/jstests.list (33989 tests)
[task 2018-09-24T22:55:38.571Z] 22:55:38     INFO - Per-test run found test testing/mochitest/tests/Harness_sanity/test_sanity.html (plain/None)
[task 2018-09-24T22:55:38.572Z] 22:55:38     INFO - 'testing/mochitest/tests/Harness_sanity/test_importInMainProcess.html' has been skipped on this platform.
Pushed by
Improve diagnostics for test-verify and per-test coverage; r=me,a=test-only
Closed: 3 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla64
Blocks: 1499208
No longer blocks: 1499208
You need to log in before you can comment on or make changes to this bug.