1637381 - Use zst for toolchain archives

Status: RESOLVED FIXED

(Firefox Build System :: Toolchains, enhancement)

Description

[Chris AtLee [:catlee] (away until at least Aug 11)]

Most of the time using zst gives far better decompression performance than xz, at the cost of slightly worse compression performance.

For toolchains that are used in CI, we should prefer zst.

Comment 1

[Chris AtLee [:catlee] (away until at least Aug 11)]

Attachment: Bug 1637381: add zstd backport for deb8 r=#build

Comment 2

[Chris AtLee [:catlee] (away until at least Aug 11)]

Attachment: Bug 1637381: Install zstd to the base debian images r=#build

Depends on D74925

Comment 3

[Chris AtLee [:catlee] (away until at least Aug 11)]

Attachment: Bug 1637381: Install zstd for test images r=#build

Depends on D74926

Comment 4

[Chris AtLee [:catlee] (away until at least Aug 11)]

Attachment: Bug 1637381: Use zstd for android toolchains r=glandium

Depends on D74927

Comment 5

[Chris AtLee [:catlee] (away until at least Aug 11)]

Attachment: Bug 1637381: Use zstd for rust toolchains r=glandium

Depends on D74928

Comment 6

[Chris AtLee [:catlee] (away until at least Aug 11)]

Attachment: Bug 1637381: Use zstd for clang toolchains r=glandium

Depends on D74929

Comment 7

[dmajor (away)]

Just to check: MozillaBuild's version of tar can handle zstd when we download a clang in boostrap?

Comment 8

[Chris AtLee [:catlee] (away until at least Aug 11)]

(In reply to :dmajor from comment #7)

Just to check: MozillaBuild's version of tar can handle zstd when we download a clang in boostrap?

Probably not, but I'm planning on adjusting bootstrap to handle them using the python-zstandard module.

Comment 9

[dmajor (away)]

(In reply to Chris AtLee [:catlee] from comment #8)

(In reply to :dmajor from comment #7)

Just to check: MozillaBuild's version of tar can handle zstd when we download a clang in boostrap?

Probably not, but I'm planning on adjusting bootstrap to handle them using the python-zstandard module.

Seems like we'd need to be set as a blocking bug of this.

Comment 10

[Chris AtLee [:catlee] (away until at least Aug 11)]

(In reply to :dmajor from comment #9)

(In reply to Chris AtLee [:catlee] from comment #8)

(In reply to :dmajor from comment #7)

Just to check: MozillaBuild's version of tar can handle zstd when we download a clang in boostrap?

Probably not, but I'm planning on adjusting bootstrap to handle them using the python-zstandard module.

Seems like we'd need to be set as a blocking bug of this.

I'll attach a patch to this series

Comment 11

[Chris AtLee [:catlee] (away until at least Aug 11)]

FWIW, MozillaBuild doesn't include zstd, nor does its version of tar support --zstd.

If we want bootstrapping to support this, we will need to ensure that the python-zstandard module is available, or maybe we can shell out to zstd if it exists. We don't have any prior art for ensuring a python package is available from bootstrap, so I'm not sure what the best way forward here is.

Comment 12

[dmajor (away)]

Maybe the path of least resistance is to leave win64-clang-cl unchanged. Since many Windows build flavors are cross-compiled now, it's really the Linux toolchains that are the most important piece of this.

Comment 13

[Chris AtLee [:catlee] (away until at least Aug 11)]

Attachment: Bug 1637381: Add support for extracting zst files to mozbuild tooltool r=glandium

Depends on D74931

Comment 14

[Chris AtLee [:catlee] (away until at least Aug 11)]

Attachment: Bug 1637381: Add python-zstandard to ubuntu test images so they can unpack toolchains r=glandium

Comment 15

[Chris AtLee [:catlee] (away until at least Aug 11)]

Attachment: Bug 1637381: Add zstdpy script r=glandium

This is a simple command line utility for doing zstd compression/decompression
of stdin to stdout. It depends on python's zstandard module being available.

Comment 16

[Chris AtLee [:catlee] (away until at least Aug 11)]

Attachment: Bug 1637381: Use python3 for repack_rust.py r=glandium

Depends on D74928

Comment 17

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/4aae9dc67310
Add support for extracting zst files to mozbuild tooltool r=glandium
https://hg.mozilla.org/integration/autoland/rev/aab5979b57e2
Add python-zstandard to ubuntu test images so they can unpack toolchains r=glandium
https://hg.mozilla.org/integration/autoland/rev/245952ad63ad
Add zstdpy script r=glandium
https://hg.mozilla.org/integration/autoland/rev/193de423be34
Use zstd for android toolchains r=firefox-build-system-reviewers,glandium
https://hg.mozilla.org/integration/autoland/rev/fecf02b98dd7
Use python3 for repack_rust.py r=glandium
https://hg.mozilla.org/integration/autoland/rev/e81e04151467
Use zstd for rust toolchains r=glandium
https://hg.mozilla.org/integration/autoland/rev/923ff4b6bda7
Use zstd for clang toolchains r=glandium

Comment 18

[Chris AtLee [:catlee] (away until at least Aug 11)]

Attachment: Bug 1637381: Install zstandard on linting images on a CLOSED TREE

Comment 19

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/fcd7cb8b4cdf
Install zstandard on linting images on a CLOSED TREE

Comment 20

[Bogdan Tara[:bogdan_tara | bogdant]]

https://hg.mozilla.org/mozilla-central/rev/4aae9dc67310
https://hg.mozilla.org/mozilla-central/rev/aab5979b57e2
https://hg.mozilla.org/mozilla-central/rev/245952ad63ad
https://hg.mozilla.org/mozilla-central/rev/193de423be34
https://hg.mozilla.org/mozilla-central/rev/fecf02b98dd7
https://hg.mozilla.org/mozilla-central/rev/e81e04151467
https://hg.mozilla.org/mozilla-central/rev/923ff4b6bda7
https://hg.mozilla.org/mozilla-central/rev/fcd7cb8b4cdf

Comment 21

[Chris AtLee [:catlee] (away until at least Aug 11)]

Attachment: Bug 1637381: Fix reference to old tarball for win64-clang-tidy r=glandium

Comment 22

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/63b7f29fb954
Fix reference to old tarball for win64-clang-tidy r=glandium

Comment 23

[Atila Butkovits]

https://hg.mozilla.org/mozilla-central/rev/63b7f29fb954