715987 - Evaluate using parallel compression tools

Status: RESOLVED FIXED

(Release Engineering :: General, enhancement)

Description

[John Ford [:jhford] CET/CEST Berlin Time]

There exists:

http://compression.ca/pbzip2/ -- parallel BZIP2
http://www.zlib.net/pigz/     -- parallel GZ
https://github.com/vasi/pixz  -- parallel XZ

Lets evaluate whether they:
1) are actually faster
2) compress as well
3) produce output that is acceptable for tinderbox builds
4) produce output that is acceptable for release artifacts

It would also be interesting to find out if compression/decompression is taking enough time during our builds to warrant any changes.

Comment 1

[Chris AtLee [:catlee]]

(In reply to John Ford [:jhford] from comment #0)
> It would also be interesting to find out if compression/decompression is
> taking enough time during our builds to warrant any changes.

IMO the think we should optimize here first is network bandwidth, and then disk storage - both of which are achieved by better compression ratios (which xz/7zip/lzma provide - see bug 733530).

for bzip2:
time bzip2 -c -9 firefox-13.0a1.en-US.linux-i686.tests.tar > /dev/null
4.40s user 0.05s system 99% cpu 24.473 total

time pbzip2 -c -9 firefox-13.0a1.en-US.linux-i686.tests.tar > /dev/null
51.68s user 0.30s system 375% cpu 13.845 total

xz does not yet support multiple threads:
time xz -c -9 firefox-13.0a1.en-US.linux-i686.tests.tar > /dev/null
61.90s user 0.41s system 99% cpu 1:02.62 total

nor does lzma:
time lzma -c -9 firefox-13.0a1.en-US.linux-i686.tests.tar > /dev/null
104.85s user 0.43s system 99% cpu 1:45.50 total

7zip does:
time 7z a -so -txz -mx=9 dummy firefox-13.0a1.en-US.linux-i686.tests.tar > /dev/null
69.90s user 0.91s system 153% cpu 46.263 total

I would conclude from this that the time savings from using pbzip2 are probably not worthwhile given the effort required to deploy the new tool. Switching to xz or 7zip would increase our compression times slightly but I think the disk/network tradeoffs are worth it.