Closed Bug 1050148 Opened 5 years ago Closed 5 years ago

--run-by-dir fails to get proper test summarization for tbpl

Categories

(Testing :: Mochitest, defect)

defect
Not set

Tracking

(firefox33 fixed, firefox34 fixed)

RESOLVED FIXED
mozilla34
Tracking Status
firefox33 --- fixed
firefox34 --- fixed

People

(Reporter: jmaher, Assigned: akachkach)

Details

Attachments

(1 file)

2 weeks ago we had this working, now it is failing, what appears to be happening is that our summarization here:
http://dxr.mozilla.org/mozilla-central/source/testing/mochitest/runtests.py#1639

seems to be printed before the output from the browser (https://tbpl.mozilla.org/php/getParsedLog.php?id=45382543&tree=Try&full=1):
19:09:08     INFO -  nsTraceRefcnt::DumpStatistics: 1091 entries
19:09:08     INFO -  TEST-PASS | leakcheck | no leaks detected!
19:09:08     INFO -  runtests.py | Running tests: end.
19:09:08     INFO -  0 INFO TEST-START | Shutdown
19:09:08     INFO -  1 INFO Passed:  258288
19:09:08     INFO -  2 INFO Failed:  0
19:09:08     INFO -  3 INFO Todo:    23956
19:09:08     INFO -  4 INFO SimpleTest FINISHED
19:09:08     INFO -  62409 INFO TEST-START | Shutdown
19:09:08     INFO -  62410 INFO Passed:  375
19:09:08     INFO -  62411 INFO Failed:  0
19:09:08     INFO -  62412 INFO Todo:    0
19:09:08     INFO -  62413 INFO Slowest: 1816ms - /tests/dom/imptests/html/html/webappapis/atob/test_base64.html
19:09:08     INFO -  62414 INFO SimpleTest FINISHED
19:09:08     INFO -  62415 INFO TEST-INFO | Ran 1 Loops
19:09:08     INFO -  62416 INFO SimpleTest FINISHED
19:09:08     INFO -  62417 INFO SUITE-END | took 1090s
19:09:08     INFO -  0
19:09:08    ERROR - Return code: 1

^ unrelated this return code 1 might be related to: http://dxr.mozilla.org/mozilla-central/source/testing/mochitest/runtests.py#1706

Is it possible that the structured logging changes have changed the order we output data?
:akachkach, you might be able to help us confirm or deny the relationship to structured logging.
Flags: needinfo?(akachkach)
Since we changed the buffering code, this might be related to structured logging. I have a patch that would pass the summary generation to the Python side, if that makes things easier.

Did this happen before the structured logging patch (= printing the summary generated from the browser side too)? Also, do you have an idea what causes the big difference between the Python-calculated summary and the one from JS?

(code that calculates the summary: http://dxr.mozilla.org/mozilla-central/source/testing/mochitest/runtests.py#1933)
Flags: needinfo?(akachkach)
this worked before (3 weeks ago) and it picked up the summary of tests from the python side.

As for the differences in calculation we are running the entire mochitest harness over and over again and we summarize the results of the entire job instead of the browser instance.

I would be interested in the patch that sends the summary generation to python.
This should do it (the issue was just that the log buffer was only dumped after sending the suite_end message, and not after each iteration of that loop).

Is there a way to try this on tbpl? (I can't run mochitests locally for now)
Attachment #8469427 - Flags: review?(jmaher)
Comment on attachment 8469427 [details] [diff] [review]
0001-Bug-1050148-dump-the-log-buffer-after-each-run-with-.patch

Review of attachment 8469427 [details] [diff] [review]:
-----------------------------------------------------------------

oh, this is cool!  I can test this on try easily, let me add it to my queue.
Attachment #8469427 - Flags: review?(jmaher) → review+
this is working on try server, thanks for writing this!
Thanks for testing it Joel!
Keywords: checkin-needed
https://hg.mozilla.org/mozilla-central/rev/2c6e6c9b37a1
Status: NEW → RESOLVED
Closed: 5 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla34
You need to log in before you can comment on or make changes to this bug.