"leaked 1 window(s) until shutdown" doesn't show up as a failure on the test summary when run as standalone

RESOLVED FIXED in Firefox 59

Status

Testing
Mochitest
--
critical
RESOLVED FIXED
4 months ago
23 days ago

People

(Reporter: standard8, Assigned: gbrown)

Tracking

Version 3
mozilla59
Points:
---

Firefox Tracking Flags

(firefox59 fixed)

Details

(Whiteboard: [PI:November])

Attachments

(1 attachment)

(Reporter)

Description

4 months ago
I'm attempting to debug some leaks, I've been running a debug build (latest master) and running commands like:

./mach mochitest browser/components/places/tests/browser/browser_bookmarkProperties_bookmarkAllTabs.js

The test runs, and at the bottom of the output I see:

TEST-INFO | checking window state
Browser Chrome Test Summary
	Passed: 6
	Failed: 0
	Todo: 0
	Mode: e10s

however, all is not fine. Scrolling back up the summary, there is:

TEST-INFO | Main app process: exit 0
31 ERROR TEST-UNEXPECTED-FAIL | browser/components/places/tests/browser/browser_bookmarkProperties_bookmarkAllTabs.js | leaked 1 window(s) until shutdown [url = chrome://browser/content/places/bookmarkProperties2.xul]
32 ERROR TEST-UNEXPECTED-FAIL | browser/components/places/tests/browser/browser_bookmarkProperties_bookmarkAllTabs.js | leaked 1 window(s) until shutdown [url = about:blank]

Those failures should be shown in the test summary.

Marking as critical, as this can lead to developers attempting to land bugs, but then getting backed out again due to having missed the leaks because the summary said it was fine.
:gbrown, is this something you could pick up in the near future?
Whiteboard: [PI:November]
(Assignee)

Updated

24 days ago
Assignee: nobody → gbrown
(Assignee)

Comment 2

24 days ago
Created attachment 8931131 [details] [diff] [review]
count leak errors and include in mochitest fail count

I verified in a local run that the summary has a non-zero fail count when a leak is introduced.

https://treeherder.mozilla.org/#/jobs?repo=try&revision=8770f0e82bcc0dae6b3e822869d35025eef32771 verifies no adverse effects.
Attachment #8931131 - Flags: review?(ahalberstadt)
Comment on attachment 8931131 [details] [diff] [review]
count leak errors and include in mochitest fail count

Review of attachment 8931131 [details] [diff] [review]:
-----------------------------------------------------------------

Thanks, lgtm!
Attachment #8931131 - Flags: review?(ahalberstadt) → review+

Comment 4

23 days ago
Pushed by gbrown@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/f6d80e35ce4f
Include leaks in mochitest summary fail count; r=ahal

Comment 5

23 days ago
bugherder
https://hg.mozilla.org/mozilla-central/rev/f6d80e35ce4f
Status: NEW → RESOLVED
Last Resolved: 23 days ago
status-firefox59: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla59
You need to log in before you can comment on or make changes to this bug.