Use zst for toolchain archives
Categories
(Firefox Build System :: Toolchains, enhancement)
Tracking
(firefox78 fixed)
Tracking | Status | |
---|---|---|
firefox78 | --- | fixed |
People
(Reporter: catlee, Assigned: catlee)
References
(Blocks 1 open bug)
Details
Attachments
(9 files, 3 obsolete files)
47 bytes,
text/x-phabricator-request
|
Details | Review | |
47 bytes,
text/x-phabricator-request
|
Details | Review | |
47 bytes,
text/x-phabricator-request
|
Details | Review | |
47 bytes,
text/x-phabricator-request
|
Details | Review | |
47 bytes,
text/x-phabricator-request
|
Details | Review | |
47 bytes,
text/x-phabricator-request
|
Details | Review | |
47 bytes,
text/x-phabricator-request
|
Details | Review | |
47 bytes,
text/x-phabricator-request
|
Details | Review | |
47 bytes,
text/x-phabricator-request
|
Details | Review |
Most of the time using zst gives far better decompression performance than xz, at the cost of slightly worse compression performance.
For toolchains that are used in CI, we should prefer zst.
Assignee | ||
Comment 1•11 months ago
|
||
Assignee | ||
Comment 2•11 months ago
|
||
Depends on D74925
Assignee | ||
Comment 3•11 months ago
|
||
Depends on D74926
Assignee | ||
Comment 4•11 months ago
|
||
Depends on D74927
Assignee | ||
Comment 5•11 months ago
|
||
Depends on D74928
Assignee | ||
Comment 6•11 months ago
|
||
Depends on D74929
![]() |
||
Comment 7•11 months ago
|
||
Just to check: MozillaBuild's version of tar can handle zstd when we download a clang in boostrap?
Assignee | ||
Comment 8•11 months ago
|
||
(In reply to :dmajor from comment #7)
Just to check: MozillaBuild's version of tar can handle zstd when we download a clang in boostrap?
Probably not, but I'm planning on adjusting bootstrap to handle them using the python-zstandard module.
![]() |
||
Comment 9•11 months ago
|
||
(In reply to Chris AtLee [:catlee] from comment #8)
(In reply to :dmajor from comment #7)
Just to check: MozillaBuild's version of tar can handle zstd when we download a clang in boostrap?
Probably not, but I'm planning on adjusting bootstrap to handle them using the python-zstandard module.
Seems like we'd need to be set as a blocking bug of this.
Assignee | ||
Comment 10•11 months ago
|
||
(In reply to :dmajor from comment #9)
(In reply to Chris AtLee [:catlee] from comment #8)
(In reply to :dmajor from comment #7)
Just to check: MozillaBuild's version of tar can handle zstd when we download a clang in boostrap?
Probably not, but I'm planning on adjusting bootstrap to handle them using the python-zstandard module.
Seems like we'd need to be set as a blocking bug of this.
I'll attach a patch to this series
Assignee | ||
Comment 11•11 months ago
|
||
FWIW, MozillaBuild doesn't include zstd
, nor does its version of tar
support --zstd.
If we want bootstrapping to support this, we will need to ensure that the python-zstandard module is available, or maybe we can shell out to zstd
if it exists. We don't have any prior art for ensuring a python package is available from bootstrap, so I'm not sure what the best way forward here is.
![]() |
||
Comment 12•11 months ago
|
||
Maybe the path of least resistance is to leave win64-clang-cl unchanged. Since many Windows build flavors are cross-compiled now, it's really the Linux toolchains that are the most important piece of this.
Assignee | ||
Comment 13•11 months ago
|
||
Depends on D74931
Updated•11 months ago
|
Updated•11 months ago
|
Updated•11 months ago
|
Updated•11 months ago
|
Updated•11 months ago
|
Updated•11 months ago
|
Updated•11 months ago
|
Assignee | ||
Comment 14•11 months ago
|
||
Assignee | ||
Comment 15•11 months ago
|
||
This is a simple command line utility for doing zstd compression/decompression
of stdin to stdout. It depends on python's zstandard module being available.
Updated•11 months ago
|
Updated•11 months ago
|
Updated•11 months ago
|
Updated•11 months ago
|
Updated•11 months ago
|
Updated•11 months ago
|
Assignee | ||
Comment 16•11 months ago
|
||
Depends on D74928
Comment 17•11 months ago
|
||
Pushed by catlee@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/4aae9dc67310 Add support for extracting zst files to mozbuild tooltool r=glandium https://hg.mozilla.org/integration/autoland/rev/aab5979b57e2 Add python-zstandard to ubuntu test images so they can unpack toolchains r=glandium https://hg.mozilla.org/integration/autoland/rev/245952ad63ad Add zstdpy script r=glandium https://hg.mozilla.org/integration/autoland/rev/193de423be34 Use zstd for android toolchains r=firefox-build-system-reviewers,glandium https://hg.mozilla.org/integration/autoland/rev/fecf02b98dd7 Use python3 for repack_rust.py r=glandium https://hg.mozilla.org/integration/autoland/rev/e81e04151467 Use zstd for rust toolchains r=glandium https://hg.mozilla.org/integration/autoland/rev/923ff4b6bda7 Use zstd for clang toolchains r=glandium
Assignee | ||
Comment 18•11 months ago
|
||
Comment 19•11 months ago
|
||
Pushed by btara@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/fcd7cb8b4cdf Install zstandard on linting images on a CLOSED TREE
Comment 20•11 months ago
|
||
bugherder |
https://hg.mozilla.org/mozilla-central/rev/4aae9dc67310
https://hg.mozilla.org/mozilla-central/rev/aab5979b57e2
https://hg.mozilla.org/mozilla-central/rev/245952ad63ad
https://hg.mozilla.org/mozilla-central/rev/193de423be34
https://hg.mozilla.org/mozilla-central/rev/fecf02b98dd7
https://hg.mozilla.org/mozilla-central/rev/e81e04151467
https://hg.mozilla.org/mozilla-central/rev/923ff4b6bda7
https://hg.mozilla.org/mozilla-central/rev/fcd7cb8b4cdf
Assignee | ||
Comment 21•11 months ago
|
||
Comment 22•11 months ago
|
||
Pushed by catlee@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/63b7f29fb954 Fix reference to old tarball for win64-clang-tidy r=glandium
Comment 23•11 months ago
|
||
bugherder |
Description
•