Closed Bug 1482413 Opened Last year Closed Last year

Improve diagnostics for TV-bf

Categories

(Testing :: General, enhancement, P1)

Version 3
enhancement

Tracking

(firefox64 fixed)

RESOLVED FIXED
mozilla64
Tracking Status
firefox64 --- fixed

People

(Reporter: gbrown, Assigned: gbrown)

References

(Blocks 2 open bugs)

Details

Attachments

(2 files)

If a bad path is passed to TV-bf, it runs green and does nothing. I think it should fail and report the bad path.
Some examples of failures and passes at

https://treeherder.mozilla.org/#/jobs?repo=try&revision=8ba248923bfe1c04406d76653122e224c04a2caf

(I have fixed the lint failure.)


Do you have ideas for related logging improvements?
Attachment #8999199 - Flags: review?(jmaher)
Comment on attachment 8999199 [details] [diff] [review]
log requested path, fail if path invalid/not found

Review of attachment 8999199 [details] [diff] [review]:
-----------------------------------------------------------------

overall this is a positive thing.  In terms of what else we can do- I would like to consider turning the job orange or maybe red if we cannot find the path specified.  There are 2 cases here:
1) the file is not found
2) we tried to run the file (i.e. launched the browser) and no tests were run

If we were to turn the job orange, that would need to account for tests that have duplicate names which we skip, likewise some other random exceptions with names ( like test-oop-extensions: https://searchfox.org/mozilla-central/source/browser/components/extensions/test/browser/browser-remote.ini )

for tv-bf we should be more specific- we are running this with the intention of a specified test and the job should fail.  Overall I think you caught the main case (#1 above) and we will have to sort out some details.

What other logging can we do- maybe details about total files found, total files not able to run (ex: skip-if), total chunks, paths for finding files.  Getting this in sooner will help out.
Attachment #8999199 - Flags: review?(jmaher) → review+
Pushed by gbrown@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/71af60e17c8b
Improve mozharness diagnostics in per-test runs, especially for TV-bf; r=jmaher
Keywords: leave-open
Priority: -- → P1
Blocks: 1483292
Depends on: 1483421
Blocks: test-verify
Pushed by gbrown@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/a71e4a20a0a6
More mozharness logging updates for test-verify; r=me,a=test-only
Keywords: leave-open
A few more log statements, inspired by Joel's feedback. There's no reasonably convenient way to identify skipped reftests, or wpt. I also looked at logging in the taskcluster code, but I think it is better not to clutter the decision task.
Attachment #9011631 - Flags: review+
https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&selectedJob=201290342&revision=615776a8b2ef9e06b39b498cc51bf888657d84c2

[task 2018-09-24T22:55:35.085Z] 22:55:35     INFO - Per-test run updated with manifest /builds/worker/workspace/build/tests/mochitest/tests/mochitest.ini (9801 active, 472 skipped)
[task 2018-09-24T22:55:35.149Z] 22:55:35     INFO - Per-test run updated with manifest /builds/worker/workspace/build/tests/mochitest/chrome/chrome.ini (980 active, 104 skipped)
[task 2018-09-24T22:55:35.354Z] 22:55:35     INFO - Per-test run updated with manifest /builds/worker/workspace/build/tests/mochitest/browser/browser-chrome.ini (4122 active, 432 skipped)
[task 2018-09-24T22:55:35.367Z] 22:55:35     INFO - Per-test run updated with manifest /builds/worker/workspace/build/tests/mochitest/a11y/a11y.ini (298 active, 7 skipped)
[task 2018-09-24T22:55:35.505Z] 22:55:35     INFO - Per-test run updated with manifest /builds/worker/workspace/build/tests/xpcshell/tests/xpcshell.ini (2988 active, 123 skipped)
[task 2018-09-24T22:55:36.958Z] 22:55:36     INFO - Per-test run updated with manifest /builds/worker/workspace/build/tests/reftest/tests/layout/reftests/reftest.list (23848 tests)
[task 2018-09-24T22:55:37.157Z] 22:55:37     INFO - Per-test run updated with manifest /builds/worker/workspace/build/tests/reftest/tests/testing/crashtest/crashtests.list (3603 tests)
[task 2018-09-24T22:55:38.570Z] 22:55:38     INFO - Per-test run updated with manifest /builds/worker/workspace/build/tests/jsreftest/tests/jstests.list (33989 tests)
[task 2018-09-24T22:55:38.571Z] 22:55:38     INFO - Per-test run found test testing/mochitest/tests/Harness_sanity/test_sanity.html (plain/None)
[task 2018-09-24T22:55:38.572Z] 22:55:38     INFO - 'testing/mochitest/tests/Harness_sanity/test_importInMainProcess.html' has been skipped on this platform.
Pushed by gbrown@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/d66dcdb73140
Improve diagnostics for test-verify and per-test coverage; r=me,a=test-only
https://hg.mozilla.org/mozilla-central/rev/d66dcdb73140
Status: NEW → RESOLVED
Closed: Last year
Resolution: --- → FIXED
Target Milestone: --- → mozilla64
Blocks: 1499208
No longer blocks: 1499208
You need to log in before you can comment on or make changes to this bug.