1030062 - Loop Marinonette tests need clearer failure points

Status: RESOLVED FIXED

(Hello (Loop) :: Client, defect, P3)

Description

[Mark Banner (:standard8)]

We just had the following failure on tbpl, unfortunately the only information highlighted by tbpl was the Assertion error lines, there wasn't enough info for tbpl to find the actual error.

We need to make this clearer, so that we can easily identify different failures.

03:16:46     INFO -  AssertionError: 1 failure(s) encountered:
03:16:46     INFO -  should call connect
03:16:46     INFO -  expected connect to be called with exact arguments typeOf("string"), typeOf("string"), undefined
03:16:46     INFO -      connect(apiKey, sessionToken, function () {})
03:16:46     INFO -  fail@http://localhost:49677/build/tests/marionette/tests/browser/components/loop/test/shared/vendor/sinon-1.9.0.js:4687:25
03:16:46     INFO -  failAssertion@http://localhost:49677/build/tests/marionette/tests/browser/components/loop/test/shared/vendor/sinon-1.9.0.js:4648:9
03:16:46     INFO -  mirrorPropAsAssertion/assert[name]@http://localhost:49677/build/tests/marionette/tests/browser/components/loop/test/shared/vendor/sinon-1.9.0.js:4671:17
03:16:46     INFO -  @http://localhost:49677/build/tests/marionette/tests/browser/components/loop/test/shared/models_test.js:174:1
03:16:46     INFO -  Runnable.prototype.run@http://localhost:49677/build/tests/marionette/tests/browser/components/loop/test/shared/vendor/mocha-1.17.1.js:4336:24
03:16:46     INFO -  Runner.prototype.runTest@http://localhost:49677/build/tests/marionette/tests/browser/components/loop/test/shared/vendor/mocha-1.17.1.js:4724:5
03:16:46     INFO -  next/<@http://localhost:49677/build/tests/marionette/tests/browser/components/loop/test/shared/vendor/mocha-1.17.1.js:4802:7
03:16:46     INFO -  next@http://localhost:49677/build/tests/marionette/tests/browser/components/loop/test/shared/vendor/mocha-1.17.1.js:4649:1
03:16:46     INFO -  next/<@http://localhost:49677/build/tests/marionette/tests/browser/components/loop/test/shared/vendor/mocha-1.17.1.js:4659:7
03:16:46     INFO -  next@http://localhost:49677/build/tests/marionette/tests/browser/components/loop/test/shared/vendor/mocha-1.17.1.js:4597:1
03:16:46     INFO -  next/<@http://localhost:49677/build/tests/marionette/tests/browser/components/loop/test/shared/vendor/mocha-1.17.1.js:4621:7
03:16:46     INFO -  Runnable.prototype.run@http://localhost:49677/build/tests/marionette/tests/browser/components/loop/test/shared/vendor/mocha-1.17.1.js:4338:5
03:16:46     INFO -  next@http://localhost:49677/build/tests/marionette/tests/browser/components/loop/test/shared/vendor/mocha-1.17.1.js:4609:5
03:16:46     INFO -  Runner.prototype.hook/<@http://localhost:49677/build/tests/marionette/tests/browser/components/loop/test/shared/vendor/mocha-1.17.1.js:4626:5
03:16:46    ERROR -  TEST-UNEXPECTED-FAIL | test_shared_all.py test_shared_all.TestSharedUnits.test_units | AssertionError: 1 failure(s) encountered:

Comment 1

[Mark Banner (:standard8)]

Attachment: Loop Marionette tests need clearer failure points when displayed on tbpl

This improves the output a bit, so that we'll get additional information highlighted on tbpl, and it also clarifies which html file the failure has occurred in.

I've shown this to the sheriffs and they are happier with this format as it is more consistent with the rest of our harnesses. I think this is the best I can do in a short time, we can always improve further later on.

Typical output is now:

AssertionError: browser/components/loop/test/shared/index.html: 1 failure(s) encountered:
TEST-UNEXPECTED-FAIL | browser/components/loop/test/shared/index.html | should call connect - expected connect to be called with exact arguments typeOf("string"), typeOf("string"), undefined
expected connect to be called with exact arguments typeOf("string"), typeOf("string"), undefined
    connect(apiKey, sessionToken, function () {})
fail@http://localhost:52573/browser/components/loop/test/shared/vendor/sinon-1.9.0.js:4687:25
failAssertion@http://localhost:52573/browser/components/loop/test/shared/vendor/sinon-1.9.0.js:4648:9
mirrorPropAsAssertion/assert[name]@http://localhost:52573/browser/components/loop/test/shared/vendor/sinon-1.9.0.js:4671:17
@http://localhost:52573/browser/components/loop/test/shared/models_test.js:174:1
Runnable.prototype.run@http://localhost:52573/browser/components/loop/test/shared/vendor/mocha-1.17.1.js:4336:24
Runner.prototype.runTest@http://localhost:52573/browser/components/loop/test/shared/vendor/mocha-1.17.1.js:4724:5
next/<@http://localhost:52573/browser/components/loop/test/shared/vendor/mocha-1.17.1.js:4802:7
next@http://localhost:52573/browser/components/loop/test/shared/vendor/mocha-1.17.1.js:4649:1
next/<@http://localhost:52573/browser/components/loop/test/shared/vendor/mocha-1.17.1.js:4659:7
next@http://localhost:52573/browser/components/loop/test/shared/vendor/mocha-1.17.1.js:4597:1
next/<@http://localhost:52573/browser/components/loop/test/shared/vendor/mocha-1.17.1.js:4621:7
Runnable.prototype.run@http://localhost:52573/browser/components/loop/test/shared/vendor/mocha-1.17.1.js:4338:5
next@http://localhost:52573/browser/components/loop/test/shared/vendor/mocha-1.17.1.js:4609:5
Runner.prototype.hook/<@http://localhost:52573/browser/components/loop/test/shared/vendor/mocha-1.17.1.js:4626:5
timeslice@http://localhost:52573/browser/components/loop/test/shared/vendor/mocha-1.17.1.js:5733:5
TEST-UNEXPECTED-FAIL | test_shared_all.py test_shared_all.TestSharedUnits.test_units | AssertionError: browser/components/loop/test/shared/index.html: 1 failure(s) encountered:

Comment 2

[Dan Mosedale (:dmosedale, :dmose)]

Comment on attachment 8445836 [details] [diff] [review]
Loop Marionette tests need clearer failure points when displayed on tbpl

I'm not a reviewer for this code; redirecting to :AutomatedTester.

Comment 3

[Mark Banner (:standard8)]

https://hg.mozilla.org/integration/fx-team/rev/067286c21d06

Comment 4

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/067286c21d06

Comment 5

[u279076]

Untracking for QA. Please needinfo me to request testing.