903536 - Gaia unit tests need to report test filenames on failure

Status: RESOLVED FIXED

(Firefox OS Graveyard :: Gaia, defect)

Description

[Jonathan Griffin (:jgriffin)]

The gaia unit tests were briefly visible in TBPL but were hidden again due to:

https://wiki.mozilla.org/Sheriffing/Job_Visibility_Policy#6.29_Outputs_failures_in_a_TBPL-starrable_format

Specifically, the error messages need to include the test filename, which is currently not the case.

Comment 1

[Jonathan Griffin (:jgriffin)]

Probably the "right" place to fix this is in the gaia harness, but we could possibly also address this in the Python testrunner, by running the tests one at a time, instead of in a single large batch.  This way, the runner would always know what test is being executed.

Comment 2

[Dylan Oliver [:doliver]]

Julien -- can you take a look at this one?

Comment 3

[Julien Wajsberg [:julienw]]

The test-runner just takes all the js files and run them, therefore it doesn't know the file name.

However, we do output the suite name in errors. Maybe we can try to have a standardized top-level suite name, containing the file being tested, eg "thread_ui.js" (afaik the test-agent adds the app currently tested). I don't know if that's possible for all test files.

Another possibility is to just get the meaningful part of the stacktrace, eg :

 1) [communications-contacts] Render contact Render name with name:
TypeError: searchBox is null
at cd_renderPhones (http://communications.gaiamobile.org:8080/contacts/js/contacts_details.js:385)
at doReloadContactDetails (http://communications.gaiamobile.org:8080/contacts/js/contacts_details.js:169)
at cd_render (http://communications.gaiamobile.org:8080/contacts/js/contacts_details.js:148)
at (anonymous) (http://communications.gaiamobile.org:8080/contacts/test/unit/contacts_details_test.js:143)

The last non-test-agent file is the test file.

But sometimes we don't have stack trace (I don't go into details):

 1) [system] system/UpdateManager "before each" hook:
Error: TypeError: self.toaster is null (http://system.gaiamobile.org:8080/js/update_manager.js?time=1376479005777:405)
at onerror (http://test-agent.gaiamobile.org:8080/common/vendor/mocha/mocha.js:4959)

In this case we don't have the test file but we still have the tested file. The part "system/UpdateManager" is the suite name and this can be standardized too if necessary.

Mocha can output a JSON output too, but the stacktrace is still a big string, so not sure this is useful.

James, do you know if the test-agent could somehow add the current test file name beside the app name, between the brackets ? Eg [system/update_manager_test.js] ?

Comment 4

[Gareth Aye [:gaye] (back from PTO)]

So it's also possible that the mocha community would be happy with a patch to add filename awareness to the https://github.com/visionmedia/mocha/blob/master/lib/test.js struct. I went ahead and filed a bug https://github.com/visionmedia/mocha/issues/950 and will update here.

Comment 5

[Jonathan Griffin (:jgriffin)]

Our runner already is aware of the stack trace, but as you pointed out, that isn't a reliable way to get a filename.  We actually need the test file and not the file that the exception occurs in, so e.g., we know what to disable when that is necessary.

Comment 6

[James Lal [:lightsofapollo]]

So its really easy to get the file name by throwing and catching an error as part of the framework. 

Going to koi? this as its really important to get this working in TBPL in a way that makes the sheriffs happy.

Comment 7

[Dylan Oliver [:doliver]]

Triage: This bug has been added to the v1.2 backlog but is not blocking release

Comment 8

[Jonathan Griffin (:jgriffin)]

Attachment: Pointer to Github pull request: https://github.com/mozilla-b2g/gaia/pull/12032#attch-to-bugzilla

Pointer to Github pull-request

Comment 9

[Jonathan Griffin (:jgriffin)]

Comment on attachment 801536 [details]
Pointer to Github pull request: https://github.com/mozilla-b2g/gaia/pull/12032#attch-to-bugzilla

This fixes the problem in the Python testrunner.  It may be better to fix it in the JS harness instead, but I'll attach this patch as an option.

Note that with this patch, the tests runs a bit slower, and as a result, suffer a few more failures...possibly because the screen has time to redraw between tests, which is currently not the case.

Comment 10

[James Lal [:lightsofapollo]]

Comment on attachment 801536 [details]
Pointer to Github pull request: https://github.com/mozilla-b2g/gaia/pull/12032#attch-to-bugzilla

https://github.com/mozilla-b2g/gaia/commit/cb46f8a8ef312cc9ce0afb80b89023bd99d10bbd