1531291 - TEST-PASS lines are output in mochitests when the test is referenced from more than one ini file

Status: NEW

(Testing :: Mochitest, defect, P3)

Description

[Mark Banner (:standard8)]

For some mochitests, we don't get to see TEST-PASS indications. For me, this is annoying since it means I don't get an easy indication of where a failure has happened in a test, nor do I get an indication of the previous pass values so that I can have them as an easy reference of what they were (and confidence the test is working correctly).

For example if I run ./mach mochitest browser/components/urlbar/tests/browser/browser_moz_action_link.js then the output is:

 0:12.13 TEST_START: browser/components/urlbar/tests/browser/browser_moz_action_link.js
 0:12.21 INFO Entering test bound 
 0:12.57 INFO Console message: [JavaScript Error: "The character encoding of the HTML document was not declared. The document will render with garbled text in some browser configurations if the document contains characters from outside the US-ASCII range. The character encoding of the page must be declared in the document or in the transfer protocol." {file: "data:text/html,<a id=a href="moz-action:foo," target=_blank rel="opener">Link</a>" line: 0}]
 0:12.58 INFO Console message: [JavaScript Error: "Security Error: Content at about:newtab may not load or link to resource://search-plugins/images/google.ico."]
 0:12.58 INFO Console message: [JavaScript Error: "Security Error: Content at about:newtab may not load or link to resource://search-plugins/images/google.ico."]
 0:12.62 INFO Console message: [JavaScript Error: "Security Error: Content at about:newtab may not load or link to resource://search-plugins/images/google.ico."]
 0:12.62 INFO Console message: [JavaScript Error: "Security Error: Content at about:newtab may not load or link to resource://search-plugins/images/google.ico."]
 0:12.65 GECKO(52792) JavaScript error: resource:///actors/NetErrorChild.jsm, line 724: TypeError: docShell.failedChannel is null
 0:12.73 INFO Console message: [JavaScript Error: "TypeError: docShell.failedChannel is null" {file: "resource:///actors/NetErrorChild.jsm" line: 724}]
 0:13.60 INFO Console message: [JavaScript Error: "The character encoding of the HTML document was not declared. The document will render with garbled text in some browser configurations if the document contains characters from outside the US-ASCII range. The character encoding of the page must be declared in the document or in the transfer protocol." {file: "data:text/html,<a id=a href="moz-action:foo" target=_blank rel="opener">Link</a>" line: 0}]
 0:13.73 GECKO(52792) JavaScript error: resource:///actors/NetErrorChild.jsm, line 724: TypeError: docShell.failedChannel is null
 0:13.75 INFO Console message: [JavaScript Error: "TypeError: docShell.failedChannel is null" {file: "resource:///actors/NetErrorChild.jsm" line: 724}]
 0:14.55 INFO Leaving test bound 
 0:14.57 GECKO(52792) MEMORY STAT vsizeMaxContiguous not supported in this build configuration.
 0:14.57 GECKO(52792) MEMORY STAT | vsize 6363MB | residentFast 454MB | heapAllocated 183MB
 0:14.57 TEST_END: Test OK. Subtests passed 10/10. Unexpected 0
 0:12.13 TEST_START: browser/components/urlbar/tests/browser/browser_moz_action_link.js
 0:12.21 INFO Entering test bound 
 0:12.57 INFO Console message: [JavaScript Error: "The character encoding of the HTML document was not declared. The document will render with garbled text in some browser configurations if the document contains characters from outside the US-ASCII range. The character encoding of the page must be declared in the document or in the transfer protocol." {file: "data:text/html,<a id=a href="moz-action:foo," target=_blank rel="opener">Link</a>" line: 0}]
 0:12.58 INFO Console message: [JavaScript Error: "Security Error: Content at about:newtab may not load or link to resource://search-plugins/images/google.ico."]
 0:12.58 INFO Console message: [JavaScript Error: "Security Error: Content at about:newtab may not load or link to resource://search-plugins/images/google.ico."]
 0:12.62 INFO Console message: [JavaScript Error: "Security Error: Content at about:newtab may not load or link to resource://search-plugins/images/google.ico."]
 0:12.62 INFO Console message: [JavaScript Error: "Security Error: Content at about:newtab may not load or link to resource://search-plugins/images/google.ico."]
 0:12.65 GECKO(52792) JavaScript error: resource:///actors/NetErrorChild.jsm, line 724: TypeError: docShell.failedChannel is null
 0:12.73 INFO Console message: [JavaScript Error: "TypeError: docShell.failedChannel is null" {file: "resource:///actors/NetErrorChild.jsm" line: 724}]
 0:13.60 INFO Console message: [JavaScript Error: "The character encoding of the HTML document was not declared. The document will render with garbled text in some browser configurations if the document contains characters from outside the US-ASCII range. The character encoding of the page must be declared in the document or in the transfer protocol." {file: "data:text/html,<a id=a href="moz-action:foo" target=_blank rel="opener">Link</a>" line: 0}]
 0:13.73 GECKO(52792) JavaScript error: resource:///actors/NetErrorChild.jsm, line 724: TypeError: docShell.failedChannel is null
 0:13.75 INFO Console message: [JavaScript Error: "TypeError: docShell.failedChannel is null" {file: "resource:///actors/NetErrorChild.jsm" line: 724}]
 0:14.55 INFO Leaving test bound 
 0:14.57 GECKO(52792) MEMORY STAT vsizeMaxContiguous not supported in this build configuration.
 0:14.57 GECKO(52792) MEMORY STAT | vsize 6363MB | residentFast 454MB | heapAllocated 183MB
 0:14.57 TEST_END: Test OK. Subtests passed 10/10. Unexpected 0

However, if I run something like ./mach mochitest testing/mochitest/tests/browser/browser_async.js, then I get:

 0:04.73 TEST_START: testing/mochitest/tests/browser/browser_async.js
 0:04.73 PASS I'm a test in head file - 
 0:05.08 INFO Console message: [JavaScript Error: "Security Error: Content at about:newtab may not load or link to resource://search-plugins/images/google.ico."]
 0:05.08 INFO Console message: [JavaScript Error: "Security Error: Content at about:newtab may not load or link to resource://search-plugins/images/google.ico."]
 0:05.08 INFO Console message: [JavaScript Error: "Security Error: Content at about:newtab may not load or link to resource://search-plugins/images/google.ico."]
 0:05.08 INFO Console message: [JavaScript Error: "Security Error: Content at about:newtab may not load or link to resource://search-plugins/images/google.ico."]
 0:05.24 PASS timeout ran - 
 0:05.24 PASS I'm a cleanup function in head file - 
 0:05.24 PASS Head cleanup function scope is correct - 
 0:05.25 GECKO(53076) MEMORY STAT vsizeMaxContiguous not supported in this build configuration.
 0:05.25 GECKO(53076) MEMORY STAT | vsize 6088MB | residentFast 405MB | heapAllocated 173MB
 0:05.25 TEST_END: Test OK. Subtests passed 4/4. Unexpected 0

I'm not yet sure what the circumstances are that cause this (if I did, I might be able to fix it).

Pretty sure this is a regression, and it didn't always used to be like this.

Comment 1

[Mark Banner (:standard8)]

So rather strangely, this occurs when a test is referenced from more than one directory. We're currently doing this with the urlbar tests, as we're running them with and without the new QuantumBar enabled.

However I've also been playing with the extension tests recently and they do the same.