Closed Bug 1313265 Opened 3 years ago Closed 3 years ago

mochitest run with unexpected failures gets reported as 'success'

Categories

(Testing :: Mochitest, defect)

defect
Not set

Tracking

(firefox52 fixed)

RESOLVED FIXED
mozilla52
Tracking Status
firefox52 --- fixed

People

(Reporter: maja_zf, Assigned: ahal)

References

(Blocks 1 open bug)

Details

Attachments

(2 files, 1 obsolete file)

In this try run, I noticed that a green mochitest job is actually full of unexpected failures: https://treeherder.mozilla.org/#/jobs?repo=try&author=mjzffr@gmail.com&exclusion_profile=false&filter-tier=1&filter-tier=2&filter-tier=3.com&selectedJob=29966891

This may be a problem with either the mochitest harness or the parser used in the DesktopUnittest mozharness script.

Log excerpt:
16:05:35     INFO -  mozprofile.addons WARNING | Could not install c:\slave\test\build\tests\mochitest\extensions\mozscreenshots: [Errno 2] No such file or directory: 'c:\\slave\\test\\build\\tests\\mochitest\\extensions\\mozscreenshots\\install.rdf'
16:05:35     INFO - TEST-UNEXPECTED-FAIL | runtests.py | Certificate integration failed
16:05:35     INFO -  dir: dom/canvas/test/crossorigin
16:05:36     INFO -  mozprofile.addons WARNING | Could not install c:\slave\test\build\tests\mochitest\extensions\mozscreenshots: [Errno 2] No such file or directory: 'c:\\slave\\test\\build\\tests\\mochitest\\extensions\\mozscreenshots\\install.rdf'
16:05:37     INFO - TEST-UNEXPECTED-FAIL | runtests.py | Certificate integration failed
16:05:37     INFO -  dir: dom/canvas/test
16:05:37     INFO -  mozprofile.addons WARNING | Could not install c:\slave\test\build\tests\mochitest\extensions\mozscreenshots: [Errno 2] No such file or directory: 'c:\\slave\\test\\build\\tests\\mochitest\\extensions\\mozscreenshots\\install.rdf'
16:05:38     INFO - TEST-UNEXPECTED-FAIL | runtests.py | Certificate integration failed
16:05:38     INFO -  dir: dom/console/tests
16:05:38     INFO -  mozprofile.addons WARNING | Could not install c:\slave\test\build\tests\mochitest\extensions\mozscreenshots: [Errno 2] No such file or directory: 'c:\\slave\\test\\build\\tests\\mochitest\\extensions\\mozscreenshots\\install.rdf'
16:05:39     INFO - TEST-UNEXPECTED-FAIL | runtests.py | Certificate integration failed
16:05:39     INFO -  0 INFO TEST-START | Shutdown
16:05:39     INFO -  1 INFO Passed:  0
16:05:39     INFO -  2 INFO Failed:  0
16:05:39     INFO -  3 INFO Todo:    0
16:05:39     INFO -  4 INFO Mode:    non-e10s
16:05:39     INFO -  5 INFO SimpleTest FINISHED
16:05:39     INFO - SUITE-END | took 33s
16:05:40    ERROR - Return code: 1
16:05:40     INFO - TinderboxPrint: mochitest-plain-chunked<br/>106/0/0
16:05:40     INFO - # TBPL SUCCESS #
16:05:40     INFO - The mochitest suite: plain-chunked ran with return status: SUCCESS
16:05:40     INFO - Running post-action listener: _package_coverage_data
16:05:40     INFO - Running post-action listener: _resource_record_post_action
16:05:40     INFO - [mozharness: 2016-10-26 23:05:40.103000Z] Finished run-tests step (success)
This is a possible regression from bug 1261194.
Yikes, this could be bad. Glad you caught this relatively quickly! Hopefully no bustage slipped through and I can land the fix for this ASAP.
Assignee: nobody → ahalberstadt
Status: NEW → ASSIGNED
Ok, there's something else going on here. I did some unit testing in isolation and as far as I can see, the StructuredOutputParser is returning FAILURE when parsing that specific log. But TBPL SUCCESS is clearly being printed, so I'm confused. At any rate, the thing that I thought was the problem didn't end up fixing this in my try run.. so perhaps bug 1261194 didn't regress this after all? I'll need to resort to try debugging to get more info.
Ah, so it is regressed by bug 1261194, but the problem is more benign then I originally thought. This happening because in runtests.py we do `self.log.info('TEST-UNEXPECTED-FAIL | ... ') in a bunch of places. We used to need to do this to get the old DesktopUnittestOutputParser to pick these up, but the StructuredOutputParser will (correctly) not flag those as errors.
The above patches seem to fix the issue:
https://treeherder.mozilla.org/#/jobs?repo=try&author=ahalberstadt@mozilla.com&fromchange=016d7dc82a78a5152c6d40b6646354d1b2079173&tochange=3189ca9b68796706869eb7e29ebf99eb62ad2cca

Bottom push is based on top of Maja's try push and shows the job failing like it's supposed to. The top push is to make sure no hidden bustage slipped through, results still pending.
Oh duh, android still needs the special MochitestFormatter.. I'll just remove that middle commit.
Attachment #8805245 - Attachment is obsolete: true
Attachment #8805245 - Flags: review?(james)
Comment on attachment 8805246 [details]
Bug 1313265 - Ensure unstructured logs containing TEST-UNEXPECTED-FAIL fail the job desktop_unittest.py,

https://reviewboard.mozilla.org/r/89026/#review88418
Attachment #8805246 - Flags: review?(james) → review+
Comment on attachment 8805244 [details]
Bug 1313265 - Log messages containing 'TEST-UNEXPECTED-FAIL' at the error level in mochitest,

https://reviewboard.mozilla.org/r/89022/#review88416
Attachment #8805244 - Flags: review?(james) → review+
Pushed by ahalberstadt@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/28afa97fa70d
Log messages containing 'TEST-UNEXPECTED-FAIL' at the error level in mochitest, r=jgraham
https://hg.mozilla.org/integration/autoland/rev/b96b2b49be98
Ensure unstructured logs containing TEST-UNEXPECTED-FAIL fail the job desktop_unittest.py, r=jgraham
https://hg.mozilla.org/mozilla-central/rev/28afa97fa70d
https://hg.mozilla.org/mozilla-central/rev/b96b2b49be98
Status: ASSIGNED → RESOLVED
Closed: 3 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla52
You need to log in before you can comment on or make changes to this bug.