1579224 - Use zstd as the standard package format for test / build archives in CI

Status: NEW

(Firefox Build System :: Task Configuration, enhancement)

Description

[Chris AtLee [:catlee]]

zstd gives us much faster, and more efficient compression than zip, gz, or bz2.

We should standardize on using zstd for most packages used in CI. For example:

• tests.tar.gz should be tests.tar.zst
• target.tar.bz2 should be target.tar.bz2

There will of course be some exceptions (e.g. apk files), but in general we should standardize on a single archive format wherever possible.

Comment 1

[Mike Hommey [:glandium]]

(In reply to Chris AtLee [:catlee] from comment #0)

• target.tar.bz2 should be target.tar.bz2

You mean .zst. That being said, that should be compensated by beetmover or something recompressing as bz2 before publishing.

Comment 2

[Mike Hommey [:glandium]]

Also, something to keep in mind: this impacts running mach package locally, which should automatically install the module in the virtualenv when required (mach taskcluster-load-image does that, for example)

Comment 3

[Chris AtLee [:catlee]]

it also impacts doing artifact builds. mach needs to be able to use zstd to consume/generate zstd archives.

Comment 4

[Chris AtLee [:catlee]]

I've done some work to see what's involved in getting us moved over to zst everywhere.

Some things have been are pretty straightforward, like adjusting the build system, mozharness, mozfile, etc. to produce and consume tar.zst archives.

The hardest part has been ensuring that the python module is importable everywhere. In some cases it's been easier to have the command-line tools installed, and then to call out to zstd from our various harnesses to do the compression/decompression.

In particular, mozharness wants to download and unpack archives before it sets up its own virtualenv.

Patches can be seen on try, e.g.: https://treeherder.mozilla.org/#/jobs?repo=try&revision=84a458965a0417c061f2250a7a8e3aca3f0e84b8

Comment 5

[BugBot [:suhaib / :marco/ :calixte]]

Bugbug thinks this bug should belong to this component, but please revert this change in case of error.