583183 - Use gzip compression for tinderbox logs

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Chris AtLee [:catlee]]

Attachment: Fix gzip format

Using gzip compression is significantly faster than using bzip2.

Needs some buildbot patches first, then buildbotcustom work.

The buildbot patch can be applied to both 0.7.10 masters and 0.8.0.

Comment 1

[Chris AtLee [:catlee]]

Attachment: Use logCompression='gzip' for TinderboxMailNotifiers (0.8.0 branch)

Comment 2

[Chris AtLee [:catlee]]

Attachment: Use logCompression='gzip' for TinderboxMailNotifiers (default branch)

Comment 3

[Chris AtLee [:catlee]]

Working for realz: http://tinderbox.mozilla.org/showlog.cgi?log=MozillaTest/1280467514.1280467528.25109.gz&fulltext=1

Comment 4

[Chris AtLee [:catlee]]

Speeding up this step is important because compressing the logs happens in the mail thread, and can block everything else from happening for a significant amount of time:

2010-07-29 22:44:57-0700 [Broker,4666,10.250.49.96]  <Build Rev3 Fedora 12x64 tryserver debug test mochitests-1/5>: build finished
2010-07-29 22:45:47-0700 [Broker,4666,10.250.49.96] sending mail (864601 bytes) to ['[e-mail]']
2010-07-29 22:45:47-0700 [Broker,4666,10.250.49.96] Starting factory <twisted.mail.smtp.SMTPSenderFactory instance at 0x1a6f8e8c>

Here the master process was blocked for 50 seconds compressing this log file.

Comment 5

[[e-mail] (:bhearsum)]

Comment on attachment 461469 [details] [diff] [review]
Fix gzip format

Aha, I thought one of these was busted.

Comment 6

[[e-mail] (:bhearsum)]

Comment on attachment 461471 [details] [diff] [review]
Use logCompression='gzip' for TinderboxMailNotifiers (default branch)

Looks fine. If you're planning to update Buildbot on the 0.7.x masters can you update mozilla2{,-staging}/*, too? Probably not necessary to update those masters, though.

Comment 7

[Chris AtLee [:catlee]]

Comment on attachment 461469 [details] [diff] [review]
Fix gzip format

changeset:   76:bbb93feea7a2

Comment 8

[Chris AtLee [:catlee]]

Comment on attachment 461470 [details] [diff] [review]
Use logCompression='gzip' for TinderboxMailNotifiers (0.8.0 branch)

changeset:   851:edba7275dec1

Comment 9

[Chris AtLee [:catlee]]

Comment on attachment 461471 [details] [diff] [review]
Use logCompression='gzip' for TinderboxMailNotifiers (default branch)

changeset:   853:d94b2b27d220

Comment 10

[Chris AtLee [:catlee]]

e.g.

http://tinderbox.mozilla.org/showlog.cgi?log=TraceMonkey/1280928709.1280928900.13614.gz&fulltext=1