1040056 - SUITE-START log line is incorrect

Status: RESOLVED FIXED

(Remote Protocol :: Marionette, defect)

Description

[Malini Das [:mdas] - Away, not checking bugmail]

The gaiatests run with confusing messages like:

04:38:01 SUITE-START | Running 0 tests
04:38:01 TEST-START | test_camera_capture_video.py TestCamera.test_capture_a_video
04:38:47 TEST-END PASS | test_camera_capture_video.py TestCamera.test_capture_a_video | took 45845ms
04:38:47 SUITE-END | took 45s
04:38:48 SUITE-START | Running 0 tests
04:38:48 TEST-START | test_camera_multiple_shots.py TestCameraMultipleShots.test_capture_multiple_shots
04:39:33 TEST-END PASS | test_camera_multiple_shots.py TestCameraMultipleShots.test_capture_multiple_shots | took 45160ms
04:39:33 SUITE-END | took 45s

from https://jenkins1.qa.scl3.mozilla.com/job/flame.mozilla-central.ui.adhoc/2/console

Where "suites" claim they run 0 tests, but then the tests actually get run. This should be fixed in the output so we won't have confusing logs.

Chris, can you take this one?

Comment 1

[Zac C (:zac)]

I looked into this briefly and the first thing that stood out was that the device test is using moztest 0.5 from pypi whereas the in-tree version is 0.6.

Comment 2

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Could be a dupe of bug 1020625. We might just need a moztest version bump to fix.

Comment 3

[James Graham [:jgraham]]

Th problem is that we don't know the tests upfront, so we pass a zero-length list to suite_start. The formatter interprets that as 0 tests.

The two possible fixes are to make the formatter deal with the empty list case more gracefully, and to figure out the list of tests before they are run (I don't know if this is possible in unittest).

Comment 4

[Chris Manchester (limited bugmail, email directly)]

I suspect davehunt is exactly right in comment 2, but I'll take a closer look at this today.

Comment 5

[James Graham [:jgraham]]

There are two problems here. One is indeed comment 2, the other is comment 3

Comment 6

[Chris Manchester (limited bugmail, email directly)]

Ok, one of my blunders here was not uploading a new moztest to pypi with bug 1020625, I've done that just now.

Comment 7

[Zac C (:zac)]

You should make marionette_client require moztest>=0.6 too.

Comment 8

[Chris Manchester (limited bugmail, email directly)]

Attachment: Increment version of moztest required by marionette to include structured logging modification.

When I run the gaia ui tests locally with moztest 0.6 and marionette 0.8, the output looks ok (no extra suite_start messages).

Comment 9

[Malini Das [:mdas] - Away, not checking bugmail]

Comment on attachment 8458290 [details] [diff] [review]
Increment version of moztest required by marionette to include structured logging modification.

Review of attachment 8458290 [details] [diff] [review]:
-----------------------------------------------------------------

lgtm, thanks!

Comment 10

[Chris Manchester (limited bugmail, email directly)]

https://hg.mozilla.org/integration/mozilla-inbound/rev/fa5713f17e8f

Comment 11

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/fa5713f17e8f