Closed Bug 1637381 Opened 3 months ago Closed 3 months ago

Use zst for toolchain archives

Categories

(Firefox Build System :: Toolchains, enhancement)

enhancement

Tracking

(firefox78 fixed)

RESOLVED FIXED
mozilla78
Tracking Status
firefox78 --- fixed

People

(Reporter: catlee, Assigned: catlee)

References

(Blocks 1 open bug)

Details

Attachments

(9 files, 3 obsolete files)

47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review

Most of the time using zst gives far better decompression performance than xz, at the cost of slightly worse compression performance.

For toolchains that are used in CI, we should prefer zst.

Just to check: MozillaBuild's version of tar can handle zstd when we download a clang in boostrap?

(In reply to :dmajor from comment #7)

Just to check: MozillaBuild's version of tar can handle zstd when we download a clang in boostrap?

Probably not, but I'm planning on adjusting bootstrap to handle them using the python-zstandard module.

(In reply to Chris AtLee [:catlee] from comment #8)

(In reply to :dmajor from comment #7)

Just to check: MozillaBuild's version of tar can handle zstd when we download a clang in boostrap?

Probably not, but I'm planning on adjusting bootstrap to handle them using the python-zstandard module.

Seems like we'd need to be set as a blocking bug of this.

(In reply to :dmajor from comment #9)

(In reply to Chris AtLee [:catlee] from comment #8)

(In reply to :dmajor from comment #7)

Just to check: MozillaBuild's version of tar can handle zstd when we download a clang in boostrap?

Probably not, but I'm planning on adjusting bootstrap to handle them using the python-zstandard module.

Seems like we'd need to be set as a blocking bug of this.

I'll attach a patch to this series

FWIW, MozillaBuild doesn't include zstd, nor does its version of tar support --zstd.

If we want bootstrapping to support this, we will need to ensure that the python-zstandard module is available, or maybe we can shell out to zstd if it exists. We don't have any prior art for ensuring a python package is available from bootstrap, so I'm not sure what the best way forward here is.

Maybe the path of least resistance is to leave win64-clang-cl unchanged. Since many Windows build flavors are cross-compiled now, it's really the Linux toolchains that are the most important piece of this.

Attachment #9147734 - Attachment description: Bug 1637381: add zstd backport for deb8 → Bug 1637381: add zstd backport for deb8 r=#build
Attachment #9147735 - Attachment description: Bug 1637381: Install zstd to the base debian images → Bug 1637381: Install zstd to the base debian images r=#build
Attachment #9147736 - Attachment description: Bug 1637381: Install zstd for test images → Bug 1637381: Install zstd for test images r=#build
Attachment #9147737 - Attachment description: Bug 1637381: Use zstd for android toolchains → Bug 1637381: Use zstd for android toolchains r=#build
Attachment #9147738 - Attachment description: Bug 1637381: Use zstd for rust toolchains → Bug 1637381: Use zstd for rust toolchains r=#build
Attachment #9147739 - Attachment description: Bug 1637381: Use zstd for clang toolchains → Bug 1637381: Use zstd for clang toolchains r=#build
Attachment #9148165 - Attachment description: Bug 1637381: Add support for extracting zst files to mozbuild tooltool r=#build → Bug 1637381: Add support for extracting zst files to mozbuild tooltool r=glandium

This is a simple command line utility for doing zstd compression/decompression
of stdin to stdout. It depends on python's zstandard module being available.

Attachment #9147737 - Attachment description: Bug 1637381: Use zstd for android toolchains r=#build → Bug 1637381: Use zstd for android toolchains r=glandium
Attachment #9147738 - Attachment description: Bug 1637381: Use zstd for rust toolchains r=#build → Bug 1637381: Use zstd for rust toolchains r=glandium
Attachment #9147739 - Attachment description: Bug 1637381: Use zstd for clang toolchains r=#build → Bug 1637381: Use zstd for clang toolchains r=glandium
Attachment #9147735 - Attachment is obsolete: true
Attachment #9147734 - Attachment is obsolete: true
Attachment #9147736 - Attachment is obsolete: true
Pushed by catlee@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/4aae9dc67310
Add support for extracting zst files to mozbuild tooltool r=glandium
https://hg.mozilla.org/integration/autoland/rev/aab5979b57e2
Add python-zstandard to ubuntu test images so they can unpack toolchains r=glandium
https://hg.mozilla.org/integration/autoland/rev/245952ad63ad
Add zstdpy script r=glandium
https://hg.mozilla.org/integration/autoland/rev/193de423be34
Use zstd for android toolchains r=firefox-build-system-reviewers,glandium
https://hg.mozilla.org/integration/autoland/rev/fecf02b98dd7
Use python3 for repack_rust.py r=glandium
https://hg.mozilla.org/integration/autoland/rev/e81e04151467
Use zstd for rust toolchains r=glandium
https://hg.mozilla.org/integration/autoland/rev/923ff4b6bda7
Use zstd for clang toolchains r=glandium
Pushed by btara@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/fcd7cb8b4cdf
Install zstandard on linting images on a CLOSED TREE
Regressions: 1640031
No longer regressions: 1640031
Regressions: 1640031
Pushed by catlee@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/63b7f29fb954
Fix reference to old tarball for win64-clang-tidy r=glandium
Regressions: 1641242
You need to log in before you can comment on or make changes to this bug.