Loop Marinonette tests need clearer failure points

RESOLVED FIXED in mozilla33

Status

Hello (Loop)
Client
P3
normal
RESOLVED FIXED
4 years ago
4 years ago

People

(Reporter: standard8, Assigned: standard8)

Tracking

(Blocks: 1 bug)

unspecified
mozilla33
Points:
---
Bug Flags:
qe-verify -

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(1 attachment)

(Assignee)

Description

4 years ago
We just had the following failure on tbpl, unfortunately the only information highlighted by tbpl was the Assertion error lines, there wasn't enough info for tbpl to find the actual error.

We need to make this clearer, so that we can easily identify different failures.

03:16:46     INFO -  AssertionError: 1 failure(s) encountered:
03:16:46     INFO -  should call connect
03:16:46     INFO -  expected connect to be called with exact arguments typeOf("string"), typeOf("string"), undefined
03:16:46     INFO -      connect(apiKey, sessionToken, function () {})
03:16:46     INFO -  fail@http://localhost:49677/build/tests/marionette/tests/browser/components/loop/test/shared/vendor/sinon-1.9.0.js:4687:25
03:16:46     INFO -  failAssertion@http://localhost:49677/build/tests/marionette/tests/browser/components/loop/test/shared/vendor/sinon-1.9.0.js:4648:9
03:16:46     INFO -  mirrorPropAsAssertion/assert[name]@http://localhost:49677/build/tests/marionette/tests/browser/components/loop/test/shared/vendor/sinon-1.9.0.js:4671:17
03:16:46     INFO -  @http://localhost:49677/build/tests/marionette/tests/browser/components/loop/test/shared/models_test.js:174:1
03:16:46     INFO -  Runnable.prototype.run@http://localhost:49677/build/tests/marionette/tests/browser/components/loop/test/shared/vendor/mocha-1.17.1.js:4336:24
03:16:46     INFO -  Runner.prototype.runTest@http://localhost:49677/build/tests/marionette/tests/browser/components/loop/test/shared/vendor/mocha-1.17.1.js:4724:5
03:16:46     INFO -  next/<@http://localhost:49677/build/tests/marionette/tests/browser/components/loop/test/shared/vendor/mocha-1.17.1.js:4802:7
03:16:46     INFO -  next@http://localhost:49677/build/tests/marionette/tests/browser/components/loop/test/shared/vendor/mocha-1.17.1.js:4649:1
03:16:46     INFO -  next/<@http://localhost:49677/build/tests/marionette/tests/browser/components/loop/test/shared/vendor/mocha-1.17.1.js:4659:7
03:16:46     INFO -  next@http://localhost:49677/build/tests/marionette/tests/browser/components/loop/test/shared/vendor/mocha-1.17.1.js:4597:1
03:16:46     INFO -  next/<@http://localhost:49677/build/tests/marionette/tests/browser/components/loop/test/shared/vendor/mocha-1.17.1.js:4621:7
03:16:46     INFO -  Runnable.prototype.run@http://localhost:49677/build/tests/marionette/tests/browser/components/loop/test/shared/vendor/mocha-1.17.1.js:4338:5
03:16:46     INFO -  next@http://localhost:49677/build/tests/marionette/tests/browser/components/loop/test/shared/vendor/mocha-1.17.1.js:4609:5
03:16:46     INFO -  Runner.prototype.hook/<@http://localhost:49677/build/tests/marionette/tests/browser/components/loop/test/shared/vendor/mocha-1.17.1.js:4626:5
03:16:46    ERROR -  TEST-UNEXPECTED-FAIL | test_shared_all.py test_shared_all.TestSharedUnits.test_units | AssertionError: 1 failure(s) encountered:

Updated

4 years ago
Blocks: 778688
(Assignee)

Comment 1

4 years ago
Created attachment 8445836 [details] [diff] [review]
Loop Marionette tests need clearer failure points when displayed on tbpl

This improves the output a bit, so that we'll get additional information highlighted on tbpl, and it also clarifies which html file the failure has occurred in.

I've shown this to the sheriffs and they are happier with this format as it is more consistent with the rest of our harnesses. I think this is the best I can do in a short time, we can always improve further later on.

Typical output is now:

AssertionError: browser/components/loop/test/shared/index.html: 1 failure(s) encountered:
TEST-UNEXPECTED-FAIL | browser/components/loop/test/shared/index.html | should call connect - expected connect to be called with exact arguments typeOf("string"), typeOf("string"), undefined
expected connect to be called with exact arguments typeOf("string"), typeOf("string"), undefined
    connect(apiKey, sessionToken, function () {})
fail@http://localhost:52573/browser/components/loop/test/shared/vendor/sinon-1.9.0.js:4687:25
failAssertion@http://localhost:52573/browser/components/loop/test/shared/vendor/sinon-1.9.0.js:4648:9
mirrorPropAsAssertion/assert[name]@http://localhost:52573/browser/components/loop/test/shared/vendor/sinon-1.9.0.js:4671:17
@http://localhost:52573/browser/components/loop/test/shared/models_test.js:174:1
Runnable.prototype.run@http://localhost:52573/browser/components/loop/test/shared/vendor/mocha-1.17.1.js:4336:24
Runner.prototype.runTest@http://localhost:52573/browser/components/loop/test/shared/vendor/mocha-1.17.1.js:4724:5
next/<@http://localhost:52573/browser/components/loop/test/shared/vendor/mocha-1.17.1.js:4802:7
next@http://localhost:52573/browser/components/loop/test/shared/vendor/mocha-1.17.1.js:4649:1
next/<@http://localhost:52573/browser/components/loop/test/shared/vendor/mocha-1.17.1.js:4659:7
next@http://localhost:52573/browser/components/loop/test/shared/vendor/mocha-1.17.1.js:4597:1
next/<@http://localhost:52573/browser/components/loop/test/shared/vendor/mocha-1.17.1.js:4621:7
Runnable.prototype.run@http://localhost:52573/browser/components/loop/test/shared/vendor/mocha-1.17.1.js:4338:5
next@http://localhost:52573/browser/components/loop/test/shared/vendor/mocha-1.17.1.js:4609:5
Runner.prototype.hook/<@http://localhost:52573/browser/components/loop/test/shared/vendor/mocha-1.17.1.js:4626:5
timeslice@http://localhost:52573/browser/components/loop/test/shared/vendor/mocha-1.17.1.js:5733:5
TEST-UNEXPECTED-FAIL | test_shared_all.py test_shared_all.TestSharedUnits.test_units | AssertionError: browser/components/loop/test/shared/index.html: 1 failure(s) encountered:
Attachment #8445836 - Flags: review?(dmose)
(Assignee)

Updated

4 years ago
Assignee: nobody → standard8
Priority: -- → P3
Comment on attachment 8445836 [details] [diff] [review]
Loop Marionette tests need clearer failure points when displayed on tbpl

I'm not a reviewer for this code; redirecting to :AutomatedTester.
Attachment #8445836 - Flags: review?(dmose) → review?(dburns)
Attachment #8445836 - Flags: review?(dburns) → review+
(Assignee)

Comment 3

4 years ago
https://hg.mozilla.org/integration/fx-team/rev/067286c21d06
Target Milestone: --- → mozilla33
https://hg.mozilla.org/mozilla-central/rev/067286c21d06
Status: NEW → RESOLVED
Last Resolved: 4 years ago
Resolution: --- → FIXED
Untracking for QA. Please needinfo me to request testing.
Flags: qe-verify-
You need to log in before you can comment on or make changes to this bug.