1331084 - Generate zstd bundles with high compression

Status: RESOLVED FIXED

(Developer Services :: Mercurial: hg.mozilla.org, defect)

Description

[Gregory Szorc [:gps]]

Mercurial 4.1 introduces support for the zstd compression format.

Compared to zlib, zstd bundles for mozilla-unified can be several hundred megabytes smaller. See https://www.mercurial-scm.org/repo/hg/rev/41a8106789ca for some numbers.

This means that people cloning Firefox with clone bundles could potentially save hundreds of megabytes from the transfer, allowing them to acquire the Firefox source code faster and with lower risk of e.g. a connection drop.

I'd like to roll out generation of zstd bundles for certain repositories (namely mozilla-central and mozilla-unified) *before* Mercurial 4.1 is running on the SSH and HTTP endpoints. We would do this by introducing a new virtualenv with Mercurial 4.1 and then changing the bundle generation timer to run from that virtualenv. There should be little risk involved with doing this: our big reason for not adopting new Mercurial releases as soon as they come out is stability on the SSH and HTTP endpoints and compatibility with extensions. The bundle generation mechanism shouldn't have these concerns.

Comment 1

[Gregory Szorc [:gps]]

Attachment: ansible/hg-ssh: create a virtualenv with Mercurial 4.1 (bug 1331084);

Mercurial 4.1 supports zstd compression. We want to roll out zstd
bundles for clone bundles ASAP because it is a compelling performance
win. However, our policy is we don't roll out new releases of Mercurial
until a point release is available (usually after a month). This policy
stems from extra caution so we don't risk the stability of
hg.mozilla.org (we're mostly concerned about memory leaks and
performance regressions which may slip into the first major release).

These concerns don't apply as much to bundle generation. And IMO
the benefits of zstd bundles (drastically smaller sizes) outweigh
the marginal risk associated with using a .0 release before the point
release.

So, this commit introduces a virtualenv on the hgssh machines with
Mercurial 4.1. It is currently 4.1-rc. That can be upgraded to 4.1
final on February 1.

Review commit: https://reviewboard.mozilla.org/r/107938/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/107938/

Comment 2

[Gregory Szorc [:gps]]

Attachment: hgserver: use Mercurial 4.1 for generating clone bundles (bug 1331084);

Now that we have Mercurial 4.1 installed, let's use it for clone bundle
generation.

We only use Mercurial 4.1 for the invocation of the `hg bundle`
process and nothing more. The script itself still runs out of the
"tools" virtualenv.

Review commit: https://reviewboard.mozilla.org/r/107940/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/107940/

Comment 3

[Gregory Szorc [:gps]]

Attachment: hgserver: support generating zstd bundles (bug 1331084);

This commit introduces *optional* support for generating Mercurial
bundles compressed with the zstd compression engine.

Only "version 2" bundles support zstd compression. So, there is only
a "zstd-v2" bundle type defined. This required the introduction of a
new "all" option to allow generation of these bundles on both
generaldelta and non-generaldelta repos, since the only behavior was
either or.

There are 2 zstd bundle flavors defined: a default one and a "max"
setting that cranks up the compression as far as it can go. There are
separate options for both and they are both disabled by default. This
is because I wish to be conservative with the initial zstd rollout
and only enable it on specific repositories. Eventually, we'll enable
zstd bundle generation by default and only enable zstd-max for high
volume repos. This is because zstd level 22 takes ~45m to run against
mozilla-central compared to ~7m for the default level 3. We do want
to support both because the size difference between level 3 and level
22 for mozilla-central is ~494MB (1,217MB versus ~723MB)!

As the lack of changes to existing tests demonstrate, each repo needs
to opt in to producing zstd bundles. And, the added tests show that
zstd bundles are produced as expected.

Review commit: https://reviewboard.mozilla.org/r/107942/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/107942/

Comment 4

[Gregory Szorc [:gps]]

Attachment: hgserver: generate zstd bundles for mozilla-central-gd and v-c-t (bug 1331084);

Now that we have support for generating zstd bundles, let's turn it on!

We enable support for generating zstd bundles on the experimental
generaldelta conversion of mozilla-central and the
version-control-tools repo. This should prove out that zstd bundle
generation works. I chose these repos because they aren't used by any
tier-1 Firefox automation. However, mozilla-central-gd is used by
incubator/stylo, so we can trigger builds on that repo to prove that
zstd bundles don't interfere with CI (they shouldn't because Mercurial
clients should ignore unknown/unsupported bundle types).

We can follow up later with wide-spread enabling of zstd bundles and
usage of zstd-max on specific repos.

Review commit: https://reviewboard.mozilla.org/r/107944/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/107944/

Comment 5

[Gregory Szorc [:gps]]

I'm on PTO early next week and probably won't have time to deploy this until Wednesday at the earliest. At that point, we can just install Mercurial 4.1 final instead of RC. If you r+, I'll likely wait for 4.1 final before landing and deploying.

Comment 6

[:glob ✱]

Comment on attachment 8831375 [details]
ansible/hg-ssh: create a virtualenv with Mercurial 4.1 (bug 1331084);

https://reviewboard.mozilla.org/r/107938/#review109538

Comment 7

[:glob ✱]

Comment on attachment 8831376 [details]
hgserver: use Mercurial 4.1 for generating clone bundles (bug 1331084);

https://reviewboard.mozilla.org/r/107940/#review109540

Comment 8

[:glob ✱]

Comment on attachment 8831377 [details]
hgserver: support generating zstd bundles (bug 1331084);

https://reviewboard.mozilla.org/r/107942/#review109542

Comment 9

[:glob ✱]

Comment on attachment 8831378 [details]
hgserver: generate zstd bundles for mozilla-central-gd and v-c-t (bug 1331084);

https://reviewboard.mozilla.org/r/107944/#review109544

Comment 10

[Gregory Szorc [:gps]]

Comment on attachment 8831375 [details]
ansible/hg-ssh: create a virtualenv with Mercurial 4.1 (bug 1331084);

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/107938/diff/1-2/

Comment 11

[Gregory Szorc [:gps]]

Comment on attachment 8831376 [details]
hgserver: use Mercurial 4.1 for generating clone bundles (bug 1331084);

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/107940/diff/1-2/

Comment 12

[Gregory Szorc [:gps]]

Comment on attachment 8831377 [details]
hgserver: support generating zstd bundles (bug 1331084);

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/107942/diff/1-2/

Comment 13

[Gregory Szorc [:gps]]

Comment on attachment 8831378 [details]
hgserver: generate zstd bundles for mozilla-central-gd and v-c-t (bug 1331084);

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/107944/diff/1-2/

Comment 14

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/hgcustom/version-control-tools/rev/9da7ff5a4d0f
ansible/hg-ssh: create a virtualenv with Mercurial 4.1 ; r=glob
https://hg.mozilla.org/hgcustom/version-control-tools/rev/b7706f32bbe3
hgserver: use Mercurial 4.1 for generating clone bundles ; r=glob
https://hg.mozilla.org/hgcustom/version-control-tools/rev/2bd6a73c1f15
hgserver: support generating zstd bundles ; r=glob
https://hg.mozilla.org/hgcustom/version-control-tools/rev/6b36f450cadd
hgserver: generate zstd bundles for mozilla-central-gd and v-c-t ; r=glob