bugzilla.mozilla.org will be intermittently unavailable on Saturday, March 24th, from 16:00 until 20:00 UTC.

tweak gz parameters in processbuild.pl -> compress_log_file()



Webtools Graveyard
7 years ago
4 years ago


(Reporter: glob, Assigned: glob)




(1 attachment)



7 years ago
Created attachment 556778 [details] [diff] [review]
patch v1

there's a few simple tweaks we can do to the gzip part of the compress_log_file() in processbuild.pl function which should speed it up.

first, here's the timings of runs using the same batch of 100 test messages:


real	0m10.062s
user	0m9.386s
sys	0m0.164s

changing the read buffer from 4k to 64k speeds things up:

real	0m8.000s
user	0m7.521s
sys	0m0.146s

and on top of this i figure we may as well change the compression level to Z_BEST_SPEED:

real	0m6.241s
user	0m5.789s
sys	0m0.164s
Attachment #556778 - Flags: review?(bear)


7 years ago
Attachment #556778 - Flags: review?(bear) → review+
A ~40% improvement in log compression time would be great, but we should check what happens to the disk requirements. We're often space limited on the server, and have shortened the retention period as more branches were added. It'd be a trade-off between processing speed and retention time.

Comment 2

7 years ago
using default compression results in 15m of logs, switching to best_speed increases this to 20m.

here's some more timings:

comp time size
0    6.32 244m
1    6.30 20m
2    6.38 19m
3    6.38 18m
4    7.30 17m
5    7.43 16m
6    8.12 15m
9    13.7 15m

perhaps "4" would be a reasonable mid-ground?

i assume getting more disk on the server isn't an option given the short life expectancy of tinderbox.
So bear, how much space is free, and what is the current retention period ? I thought it was as short as 10 or 14 days, but there was a comment recently that it was month.

Comment 4

5 years ago
given the moves away from tinderbox, there isn't much value in resolving this issue.
Last Resolved: 5 years ago
Resolution: --- → WONTFIX
Product: Webtools → Webtools Graveyard
You need to log in before you can comment on or make changes to this bug.