563681 - warnings shouldn't increment number of tests failed

Status: RESOLVED DUPLICATE of bug 562762

(Add-on SDK Graveyard :: General, defect)

Description

[Dietrich Ayala (:dietrich)]

See bug 562762, wherein 1 of 924 tests fails, but there's no way to tell which test it was.

Comment 1

[Dietrich Ayala (:dietrich)]

morphin. see https://bugzilla.mozilla.org/show_bug.cgi?id=562762#c4

Comment 2

[Drew Willcoxon :adw]

Attachment: logs failing test filenames and function names

This patch is at the bottom of my queue, and it prints

  error: TEST FAILED: testFilename.functionName (reason)

whenever at least one assertion (or timeout, etc.) fails in a test function.  Is that what you're looking for?  It wouldn't actually help bug 562762 though, as I note in comment 4 there.

Comment 3

[Dietrich Ayala (:dietrich)]

that would definitely be helpful, thanks!

Comment 4

[Atul Varma [:atul]]

Comment on attachment 443378 [details] [diff] [review]
logs failing test filenames and function names

Looks good! Do you think any modifications should be made to packages/test-harness/lib/harness.js to use this new functionality, so that the # of tests stays a bit more consistent, and the final code checks to see if either any tests failed or if the new 'testFailed' property of the test is true?

Comment 5

[Drew Willcoxon :adw]

(In reply to comment #4)
> Looks good! Do you think any modifications should be made to
> packages/test-harness/lib/harness.js to use this new functionality, so that the
> # of tests stays a bit more consistent

Do you mean that the errors-logged case shouldn't increase the number of failed tests?

> and the final code checks to see if
> either any tests failed or if the new 'testFailed' property of the test is
> true?

|failed| is always incremented whenever the new log function is called, so IMO that's not necessary.  Really testFailed is only needed to ensure that the new error message is printed only on the first failure in a test function.  Maybe it should be renamed to better reflect that, like "failureLogged"?

BTW, this patch doesn't match the bug anymore, and in bug 560626 comment 5 Atul said he's "cautious about disabling the 'fail if any errors are logged' criteria," which seems to be what this bug is now about.

Comment 6

[Dietrich Ayala (:dietrich)]

duping against the original report of this.

Comment 7

[Drew Willcoxon :adw]

Spun out the patch into bug 565067.

Comment 8

[Matěj Cepl]

Hmm, is this really fixed?

johanka:jetpack-sdk$ source bin/activate
Welcome to the Jetpack SDK. Run 'cfx docs' for assistance.
(jetpack-sdk)johanka:jetpack-sdk$ cfx -a firefox testall
Testing all available packages: nsjetpack, test-harness, development-mode, jetpack-core.
++ date +%j
+ odd=1
+ '[' 1 -eq 1 ']'
+ setarch i686 /usr/bin/firefox -profile /tmp/tmpNk90d3.mozrunner
.....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................console: 
console: 
....console: 
..console: 
console: 
..console: 
...........console: 
console: 
...console: 
console: 
.console: 
....console: 
.........console: 
.............................................................error: TEST FAILED: test-securable-module.testSecurableModule (fail)
error: fail: modules shouldn't have chrome privileges by default. ("V <http://www.mozilla.org> bylo odepYeno \r\xEDst vlastnost XPCComponents.classes" != "Permission denied for <http://www.mozilla.org> to get property XPCComponents.classes")
info: Traceback (most recent call last):
  File "resource://testall-jetpack-core-lib/timer.js", line 57, in notify
    this._callback.apply(null, []);
  File "resource://testall-jetpack-core-lib/unit-test.js", line 224, in anonymous
    timer.setTimeout(function() { onDone(self); }, 0);
  File "resource://testall-jetpack-core-lib/unit-test.js", line 248, in runNextTest
    self.start({test: test, onDone: runNextTest});
  File "resource://testall-jetpack-core-lib/unit-test.js", line 266, in start
    this.test(this);
  File "resource://testall-jetpack-core-lib/unit-test.js", line 57, in runTest
    test(runner);
  File "resource://testall-jetpack-core-tests/test-securable-module.js", line 49, in anonymous
    file);
  File "resource://testall-jetpack-core-tests/test-securable-module.js", line 160, in run
    "modules shouldn't have chrome privileges by default."
  File "resource://testall-jetpack-core-tests/test-securable-module.js", line 33, in anonymous
    test.assertEqual(a, b, msg);
  File "resource://testall-jetpack-core-lib/unit-test.js", line 202, in assertEqual
    this.fail(message);
  File "resource://testall-jetpack-core-lib/unit-test.js", line 122, in fail
    console.trace();
.........................console: [JavaScript Error: "[Exception... "'JavaScript component does not have a method named: "notify"' when calling method: [nsITimerCallback::notify]"  nsresult: "0x80570030 (NS_ERROR_XPC_JSOBJECT_HAS_NO_FUNCTION_NAMED)"  location: "<unknown>"  data: no]"]
.console: [JavaScript Error: "[Exception... "'JavaScript component does not have a method named: "notify"' when calling method: [nsITimerCallback::notify]"  nsresult: "0x80570030 (NS_ERROR_XPC_JSOBJECT_HAS_NO_FUNCTION_NAMED)"  location: "<unknown>"  data: no]"]
.console: [JavaScript Error: "[Exception... "'JavaScript component does not have a method named: "notify"' when calling method: [nsITimerCallback::notify]"  nsresult: "0x80570030 (NS_ERROR_XPC_JSOBJECT_HAS_NO_FUNCTION_NAMED)"  location: "<unknown>"  data: no]"]
.................................................console: [JavaScript Error: "[Exception... "'JavaScript component does not have a method named: "notify"' when calling method: [nsITimerCallback::notify]"  nsresult: "0x80570030 (NS_ERROR_XPC_JSOBJECT_HAS_NO_FUNCTION_NAMED)"  location: "<unknown>"  data: no]"]
...............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................console: [JavaScript Error: "[Exception... "'JavaScript component does not have a method named: "notify"' when calling method: [nsITimerCallback::notify]"  nsresult: "0x80570030 (NS_ERROR_XPC_JSOBJECT_HAS_NO_FUNCTION_NAMED)"  location: "<unknown>"  data: no]"]
........................................................................................................................................................................................................
object count is 7993 in 65 modules (34384 across entire JS runtime)
Tracked memory objects in testing sandbox: 5
warning: LEAK [object Object]
warning: LEAK resource://testall-jetpack-core-lib/memory.js
warning: LEAK resource://testall-jetpack-core-lib/unload.js
warning: LEAK resource://testall-jetpack-core-lib/unit-test.js
warning: LEAK resource://testall-jetpack-core-lib/timer.js
warning: LEAK resource://testall-jetpack-core-lib/xpcom.js
warning: LEAK resource://testall-jetpack-core-lib/file.js
warning: LEAK resource://testall-jetpack-core-lib/byte-streams.js
warning: LEAK resource://testall-jetpack-core-lib/text-streams.js
warning: LEAK resource://testall-nsjetpack-tests/test-nsjetpack.js
warning: LEAK resource://testall-nsjetpack-tests/test-legacy-nsjetpack.js
warning: LEAK resource://testall-jetpack-core-tests/test-xul-app.js
warning: LEAK resource://testall-jetpack-core-lib/xul-app.js
warning: LEAK resource://testall-jetpack-core-tests/test-plain-text-console.js
warning: LEAK resource://testall-jetpack-core-tests/test-timer.js
warning: LEAK resource://testall-jetpack-core-tests/test-xpcom.js
warning: LEAK resource://testall-jetpack-core-lib/traceback.js
warning: LEAK resource://testall-jetpack-core-tests/test-unload.js
warning: LEAK resource://testall-jetpack-core-tests/test-collection.js
warning: LEAK resource://testall-jetpack-core-lib/collection.js
warning: LEAK resource://testall-jetpack-core-tests/test-traceback.js
warning: LEAK resource://testall-jetpack-core-tests/test-text-streams.js
warning: LEAK resource://testall-jetpack-core-lib/url.js
warning: LEAK resource://testall-jetpack-core-tests/test-globals.js
warning: LEAK resource://testall-jetpack-core-tests/test-preferences-service.js
warning: LEAK resource://testall-jetpack-core-lib/preferences-service.js
warning: LEAK resource://testall-jetpack-core-tests/test-url.js
warning: LEAK resource://testall-jetpack-core-tests/test-cuddlefish.js
warning: LEAK resource://testall-jetpack-core-tests/test-private-browsing.js
warning: LEAK resource://testall-jetpack-core-lib/private-browsing.js
warning: LEAK resource://testall-jetpack-core-lib/observer-service.js
warning: LEAK resource://testall-jetpack-core-lib/errors.js
warning: LEAK resource://testall-jetpack-core-tests/test-observer-service.js
warning: LEAK resource://testall-jetpack-core-tests/test-errors.js
warning: LEAK resource://testall-jetpack-core-tests/test-file.js
warning: LEAK resource://testall-jetpack-core-tests/test-window-utils.js
warning: LEAK resource://testall-jetpack-core-lib/window-utils.js
warning: LEAK resource://testall-jetpack-core-tests/test-securable-module.js
warning: LEAK resource://testall-jetpack-core-tests/test-tab-browser.js
warning: LEAK resource://testall-jetpack-core-lib/tab-browser.js
warning: LEAK resource://testall-jetpack-core-tests/test-api-utils.js
warning: LEAK resource://testall-jetpack-core-lib/api-utils.js
warning: LEAK resource://testall-jetpack-core-tests/test-memory.js
warning: LEAK resource://testall-jetpack-core-tests/test-xhr.js
warning: LEAK resource://testall-jetpack-core-lib/xhr.js
warning: LEAK resource://testall-jetpack-core-tests/test-context-menu.js
warning: LEAK resource://testall-jetpack-core-lib/context-menu.js
warning: LEAK resource://testall-test-harness-tests/test-packaging.js
warning: LEAK resource://testall-test-harness-tests/test-self.js
warning: LEAK resource://testall-jetpack-core-lib/self.js
warning: LEAK resource://testall-nsjetpack-lib/nsjetpack.js
warning: LEAK resource://testall-jetpack-core-lib/plain-text-console.js
warning: LEAK resource://testall-jetpack-core-lib/securable-module.js
warning: LEAK [object Object]

1769 of 1770 tests passed.
FAIL
Total time: 16.352805 seconds
Program terminated unsuccessfully.
(jetpack-sdk)johanka:jetpack-sdk$ 
(jetpack-sdk)johanka:jetpack-sdk$ chmod -x ~/.bin/firefox 
(jetpack-sdk)johanka:jetpack-sdk$ 
(jetpack-sdk)johanka:jetpack-sdk$ cfx -a firefox testall
Testing all available packages: nsjetpack, test-harness, development-mode, jetpack-core.
/home/matej/projekty/jetpack-sdk/python-lib/cuddlefish/__init__.py:484: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6
  if e.message.startswith(MOZRUNNER_BIN_NOT_FOUND):
Traceback (most recent call last):
  File "/home/matej/projekty/jetpack-sdk/bin/cfx", line 6, in <module>
    cuddlefish.run()
  File "/home/matej/projekty/jetpack-sdk/python-lib/cuddlefish/__init__.py", line 260, in run
    test_all_packages(env_root, defaults=options.__dict__)
  File "/home/matej/projekty/jetpack-sdk/python-lib/cuddlefish/__init__.py", line 188, in test_all_packages
    defaults=defaults)
  File "/home/matej/projekty/jetpack-sdk/python-lib/cuddlefish/__init__.py", line 482, in run
    logfile=options.logfile)
  File "/home/matej/projekty/jetpack-sdk/python-lib/cuddlefish/runner.py", line 290, in run_app
    runner.start()
  File "/home/matej/projekty/jetpack-sdk/python-lib/mozrunner/__init__.py", line 397, in start
    self.process_handler = run_command(self.command+self.cmdargs, self.env, **self.kp_kwargs)
  File "/home/matej/projekty/jetpack-sdk/python-lib/mozrunner/__init__.py", line 90, in run_command
    env=env, **killable_kwargs)
  File "/usr/lib64/python2.6/subprocess.py", line 595, in __init__
    errread, errwrite)
  File "/usr/lib64/python2.6/subprocess.py", line 1092, in _execute_child
    raise child_exception
OSError: [Errno 13] Permission denied
(jetpack-sdk)johanka:jetpack-sdk$

Comment 9

[Drew Willcoxon :adw]

Whoops, didn't mean to mark this fixed.  Reverting back to dupe per comment 6.

Comment 10

[Myk Melez [:myk] [@mykmelez]]

The Add-on SDK is no longer a Mozilla Labs experiment and has become a big enough project to warrant its own Bugzilla product, so the "Add-on SDK" product has been created for it, and I am moving its bugs to that product.

To filter bugmail related to this change, filter on the word "looptid".