Closed Bug 1040056 Opened 7 years ago Closed 7 years ago

SUITE-START log line is incorrect

Categories

(Testing :: Marionette, defect)

defect
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED
mozilla34

People

(Reporter: mdas, Assigned: chmanchester)

Details

Attachments

(1 file)

The gaiatests run with confusing messages like:

04:38:01 SUITE-START | Running 0 tests
04:38:01 TEST-START | test_camera_capture_video.py TestCamera.test_capture_a_video
04:38:47 TEST-END PASS | test_camera_capture_video.py TestCamera.test_capture_a_video | took 45845ms
04:38:47 SUITE-END | took 45s
04:38:48 SUITE-START | Running 0 tests
04:38:48 TEST-START | test_camera_multiple_shots.py TestCameraMultipleShots.test_capture_multiple_shots
04:39:33 TEST-END PASS | test_camera_multiple_shots.py TestCameraMultipleShots.test_capture_multiple_shots | took 45160ms
04:39:33 SUITE-END | took 45s

from https://jenkins1.qa.scl3.mozilla.com/job/flame.mozilla-central.ui.adhoc/2/console

Where "suites" claim they run 0 tests, but then the tests actually get run. This should be fixed in the output so we won't have confusing logs.

Chris, can you take this one?
Flags: needinfo?(cmanchester)
I looked into this briefly and the first thing that stood out was that the device test is using moztest 0.5 from pypi whereas the in-tree version is 0.6.
Could be a dupe of bug 1020625. We might just need a moztest version bump to fix.
Th problem is that we don't know the tests upfront, so we pass a zero-length list to suite_start. The formatter interprets that as 0 tests.

The two possible fixes are to make the formatter deal with the empty list case more gracefully, and to figure out the list of tests before they are run (I don't know if this is possible in unittest).
I suspect davehunt is exactly right in comment 2, but I'll take a closer look at this today.
Assignee: nobody → cmanchester
Flags: needinfo?(cmanchester)
There are two problems here. One is indeed comment 2, the other is comment 3
Ok, one of my blunders here was not uploading a new moztest to pypi with bug 1020625, I've done that just now.
You should make marionette_client require moztest>=0.6 too.
When I run the gaia ui tests locally with moztest 0.6 and marionette 0.8, the output looks ok (no extra suite_start messages).
Attachment #8458290 - Flags: review?(mdas)
Comment on attachment 8458290 [details] [diff] [review]
Increment version of moztest required by marionette to include structured logging modification.

Review of attachment 8458290 [details] [diff] [review]:
-----------------------------------------------------------------

lgtm, thanks!
Attachment #8458290 - Flags: review?(mdas) → review+
https://hg.mozilla.org/mozilla-central/rev/fa5713f17e8f
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla34
You need to log in before you can comment on or make changes to this bug.