505765 - Redirect code coverage output to local log files

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Chris AtLee [:catlee]]

Recent code coverage test runs have been generating many gigabyte log files, which is overloading the buildbot master.

The log files should instead be logged locally and compressed, then uploaded along with the code coverage results.

They compress *extremely* well.  One example of a 6GB log file compressed down to around 6MB using bzip2.

Comment 1

[Chris AtLee [:catlee]]

Attachment: [WIP] Run tests with output locally compressed

This patch factors out the running of tests in the UnittestFactory into a new method, addTestSteps.

For code coverage runs, we completely replace addTestSteps, and run all the various make commands piped through bzip2 to save the logs locally.  The logs are tarred up and uploaded at the end.

The patch also tars up and uploads the js coverage report (bug 504642).

Initial testing looks good.  Timeouts may have to be adjusted since the commands no produce _no_ output for buildbot.

Comment 2

[Chris AtLee [:catlee]]

Comment on attachment 392030 [details] [diff] [review]
[WIP] Run tests with output locally compressed

Oh, this patch also splits up the mochitest-plain test by directory, until some better solution can be deployed.

Comment 3

[Chris AtLee [:catlee]]

Attachment: Run tests with output locally compressed

Same as before, except adding different timeouts for the different test types.

Comment 4

[[e-mail] (:bhearsum)]

Comment on attachment 392802 [details] [diff] [review]
Run tests with output locally compressed

>diff --git a/process/factory.py b/process/factory.py

>+        for mochitest in mochitests:
>+            commands.append(
>+                ('mochitest-plain-%s' % mochitest,
>+                 ['make', 'TEST_PATH=%s' % mochitest, 'mochitest-plain']),
>+                )

I think this reads better if you do 'for dir in mochitests' or 'for dir in mochitest_dirs'. r=bhearsum with that change.

Comment 5

[Chris AtLee [:catlee]]

Attachment: Run tests with output locally compressed

Same as before, with nit addressed.  Will carry over r+.

Comment 6

[Chris AtLee [:catlee]]

Comment on attachment 393224 [details] [diff] [review]
Run tests with output locally compressed

actually, it's not the same as the previous patch...I think I uploaded the wrong one before!

Comment 7

[Chris AtLee [:catlee]]

Comment on attachment 393224 [details] [diff] [review]
Run tests with output locally compressed

changeset:   380:3f153532ad8f

Comment 8

[Chris AtLee [:catlee]]

This should be all done now, but the job we kicked off this morning was killed off in a case of mistaken identity.

Back in the pool for now, until a green run happens.

Comment 9

[Murali Nandigama [:murali]]

That is why!!!

Thanks for the update.

Comment 10

[Armen [:armenzg]]

Is there anything left to be done in this bug or can it be resolved fixed?

Comment 11

[Murali Nandigama [:murali]]

Can please check !! the tar ball for logs  mochitest-browser-chrome.log.bz2 seem to be corrupt.

Comment 12

[Armen [:armenzg]]

I have confirmed what you are saying by downloading this http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/experimental/codecoverage/codecoverage-mozilla-central-logs.tar

Can we try to fix/investigate that in the blocking bug (bug 510489) and close this one? I get the feeling that what was being asked in this bug has been accomplished and the follow up work should be done in the blocking bug.

Comment 13

[Murali Nandigama [:murali]]

I agree. You can close this as completed and please pursue the issue in  the bug 510489