1392370 - Investigate and potentially use multi-threaded xz compression

Status: RESOLVED FIXED

(Firefox Build System :: General, enhancement)

Description

[Gregory Szorc [:gps]]

Toolchain tasks can spend >7 minutes in xz compression. Modern versions of xz-utils support multi-threaded compression via -T. Archives produced with multi-threaded compression can be read by legacy xz tools. Essentially, it splits the input archive into segments and compresses each independently. You will lose some compression ratio. How much depends on block sizes and the input to the compressor.

Given that we spent minutes in xz, let's investigate parallel compression. If the compression ratio loss is minimal, we should probably use `xz -T 0` everywhere so we automatically scale to the number of available threads. Or at least something like `xz -T 4` to get some compression speed-up.

Comment 1

[Mike Hommey [:glandium]]

Why not use zstd?

Comment 2

[Gregory Szorc [:gps]]

I'm not opposed to zstd. We already use zstd (parallel compression even) for Docker images. zstd isn't packaged as widely yet. So the main downside is we'd have to install binaries in various places. Not the hardest thing to do. Also, tooltool may need to be taught up zstd for cases where we use "unpack": true.

Comment 3

[Mike Hommey [:glandium]]

Ah, gah, it would make life miserable for mach bootstrap. For instance, Ubuntu 16.04 (LTS and released a few months ago) comes with an old version of zstd, so even if it's installed, it can't read files produced by zstd 1.x.

Comment 4

[Mike Hommey [:glandium]]

While the xz-utils provided in Debian 7 is not modern enough, I added pxz to the toolchain-build docker image in bug 1427326.

Comparing the time spent compression an archive of GCC 6:

$ time tar -Jcf gcc.tar.xz gcc
real    6m17.828s
user    6m17.500s
sys     0m1.964s
$ time tar -cf - gcc | pxz --compress -T $(nproc) > gcc.tar.pxz
real    0m55.424s
user    9m50.416s
sys     0m1.824s
$ ls -l gcc.tar.xz gcc.tar.pxz
-rw-r--r-- 1 root root 160037852 Dec 29 08:52 gcc.tar.pxz
-rw-r--r-- 1 root root 156507952 Dec 29 08:50 gcc.tar.xz

Marginally larger, significantly faster.

Comment 5

[Mike Hommey [:glandium]]

FWIW:

# time tar -cf - gcc | zstd -o gcc.tar.zst
/*stdin*\            : 29.60%   (860518400 => 254714054 bytes, gcc.tar.zst)   
real    0m7.734s
user    0m7.380s
sys     0m0.728s

Even with max compression and max time, we don't get near xz/pxz in terms of size:
# time tar -cf - gcc | zstd -19 -o gcc.tar.zst
/*stdin*\            : 21.21%   (860518400 => 182553689 bytes, gcc.tar.zst2)   
real    4m1.778s
user    4m1.168s
sys     0m1.176s

Comment 6

[Mike Hommey [:glandium]]

# time tar -cf - gcc | zstd -19 -T$(nproc) -o gcc.tar.zst
/*stdin*\            : 21.55%   (860518400 => 185462606 bytes, gcc.tar.zst2)   
real    0m26.783s
user    5m29.768s
sys     0m1.016s

Comment 7

[Gregory Szorc [:gps]]

zstd level 21 or 22 should get pretty close to xz. But probably still a bit larger. Those modes allocate a very large window and thus require a lot of memory to decompress. We actually had to stop using the higher levels for Mercurial bundles because 32-bit Python processes couldn't allocate enough memory. See bug 1344790.

Comment 8

[Mike Hommey [:glandium]]

Attachment: Bug 1392370 - Enable xz parallel compression on Debian-based docker images.

This makes the XZ_OPT environment variable set for every task that runs
on those docker images, effectively enabling parallel compression for
all of them.

For e.g. the gcc 4.9 toolchain build task, this makes the final
compression step take 23s instead of > 3 minutes.

Review commit: https://reviewboard.mozilla.org/r/213852/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/213852/

Comment 9

[Gregory Szorc [:gps]]

Comment on attachment 8943513 [details]
Bug 1392370 - Enable xz parallel compression on Debian-based docker images.

https://reviewboard.mozilla.org/r/213852/#review220054

I'll likely autoland this later once the trees are reopened. Don't want to land it now in case tons of stuff piles up behind it.

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/78e74514176f
Enable xz parallel compression on Debian-based docker images. r=gps

Comment 11

[Bogdan Tara[:bogdan_tara | bogdant]]

https://hg.mozilla.org/mozilla-central/rev/78e74514176f