1354311 - Bundle generation cleanups

Status: RESOLVED FIXED

(Developer Services :: Mercurial: hg.mozilla.org, enhancement)

Description

[Gregory Szorc [:gps]]

I have some minor fixups to bundle generation. This will bring the code in a good state to be compatible with modern best practices.

Comment 1

[Gregory Szorc [:gps]]

Attachment: ansible/hg-ssh: stop generating bzip2 bundles (bug 1354311);

The bzip2 bundles take a long time to generate and don't yield as good
compression as zstd. The bzip2 bundles exist to allow users to download
a smaller bundle. However, they aren't widely used because it requires
an explicit client-side config to prioritize them over gzip and zstd.

Since zstd is a better smaller bundle, let's remove the generation of
bzip2 bundles.

FWIW, I've already made this change on the server because of the
conversion of repos to generaldelta. The bundle generation code
uses ``bzip-v1`` as the bundle type. Using this against a generaldelta
repo would require several hours of CPU, which is why I disabled it.

Review commit: https://reviewboard.mozilla.org/r/127378/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/127378/

Comment 2

[Gregory Szorc [:gps]]

Attachment: generate-hg-s3-bundles: remove support for bzip2 bundles (bug 1354311);

We no longer generate these bundles. So remove the feature
support for it.

The test change is a bit wonky. I did it this way to make the diff
easier to reason about. I'll remove this kludge in the next commit.

Review commit: https://reviewboard.mozilla.org/r/127380/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/127380/

Comment 3

[Gregory Szorc [:gps]]

Attachment: hgserver: remove extra commit in test (bug 1354311);

The previous commit didn't touch this commit in the test because it
didn't want to introduce extra test fallout.

We remove the commit here and see that hashes and only hashes change.

Review commit: https://reviewboard.mozilla.org/r/127382/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/127382/

Comment 4

[Gregory Szorc [:gps]]

Attachment: hgserver: invert tests to prefer generaldelta repos (bug 1354311);

All server-side repos are now using generaldelta. The tests should prefer
generaldelta usage.

The only changes in tests should be various "v1" becoming "v2" and "packed1"
becoming "packed1-gd" and vice-versa.

Review commit: https://reviewboard.mozilla.org/r/127384/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/127384/

Comment 5

[Gregory Szorc [:gps]]

Attachment: generate-hg-s3-bundles: simplify generaldelta checking (bug 1354311);

This is never a large file. We can use a one-liner.

Review commit: https://reviewboard.mozilla.org/r/127386/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/127386/

Comment 6

[Gregory Szorc [:gps]]

Attachment: generate-hg-s3-bundles: require generaldelta repos (bug 1354311);

All repos we generate bundles for have been converted to generaldelta.

The bundle generation code has code paths to treat generaldelta and
non-generaldelta repos separately. This is now effectively dead code.

This commit starts the process of removing the non-generaldelta code
by rejecting repos that don't have the generaldelta requirement.

Review commit: https://reviewboard.mozilla.org/r/127388/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/127388/

Comment 7

[Gregory Szorc [:gps]]

Attachment: generate-hg-s3-bundles: remove code generating non-generaldelta bundles (bug 1354311);

We no longer produce bundles that don't use generaldelta. So remove
code related to it.

The "all" option was only used by generaldelta code, which is why some
bundle type entries for generaldelta have been updated in this change.

Review commit: https://reviewboard.mozilla.org/r/127390/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/127390/

Comment 8

[Gregory Szorc [:gps]]

Attachment: generate-hg-s3-bundles: renamed packed1-gd to packed1 (bug 1354311);

We no longer have generaldelta and non-generaldelta versions of these
bundles. So drop the "-gd" suffix since it isn't needed.

When we deploy this, we'll want to remove existing packed repo bundles
from the server to prevent any confusion arising from the rename.

As far as Mercurial is concerned, the filename doesn't matter: the only
thing that matters is the bundle specification in the clone bundles
manifest. And that doesn't change.

Review commit: https://reviewboard.mozilla.org/r/127392/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/127392/

Comment 9

[Gregory Szorc [:gps]]

Attachment: generate-hg-s3-bundles: use tools virtualenv (bug 1354311);

In order to support zstd bundle generation before deploying Mercurial
4.1 on most server processes, we established an "hg_pre" virtualenv
running 4.1 and used it for bundle generation.

All virtualenvs are now 4.1.2. So we no longer need this workaround.

I'm going to keep the hg_pre virtualenv around because I think there
is value in having the ability to install a newer release of
Mercurial so ad-hoc processes can have access to it.

Review commit: https://reviewboard.mozilla.org/r/127394/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/127394/

Comment 10

[Gregory Szorc [:gps]]

Comment on attachment 8855543 [details]
ansible/hg-ssh: stop generating bzip2 bundles (bug 1354311);

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/127378/diff/1-2/

Comment 11

[Gregory Szorc [:gps]]

Comment on attachment 8855544 [details]
generate-hg-s3-bundles: remove support for bzip2 bundles (bug 1354311);

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/127380/diff/1-2/

Comment 12

[Gregory Szorc [:gps]]

Comment on attachment 8855545 [details]
hgserver: remove extra commit in test (bug 1354311);

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/127382/diff/1-2/

Comment 13

[Gregory Szorc [:gps]]

Comment on attachment 8855546 [details]
hgserver: invert tests to prefer generaldelta repos (bug 1354311);

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/127384/diff/1-2/

Comment 14

[Gregory Szorc [:gps]]

Comment on attachment 8855547 [details]
generate-hg-s3-bundles: simplify generaldelta checking (bug 1354311);

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/127386/diff/1-2/

Comment 15

[Gregory Szorc [:gps]]

Comment on attachment 8855548 [details]
generate-hg-s3-bundles: require generaldelta repos (bug 1354311);

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/127388/diff/1-2/

Comment 16

[Gregory Szorc [:gps]]

Comment on attachment 8855549 [details]
generate-hg-s3-bundles: remove code generating non-generaldelta bundles (bug 1354311);

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/127390/diff/1-2/

Comment 17

[Gregory Szorc [:gps]]

Comment on attachment 8855550 [details]
generate-hg-s3-bundles: renamed packed1-gd to packed1 (bug 1354311);

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/127392/diff/1-2/

Comment 18

[Gregory Szorc [:gps]]

Comment on attachment 8855551 [details]
generate-hg-s3-bundles: use tools virtualenv (bug 1354311);

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/127394/diff/1-2/

Comment 19

[:glob ✱]

Comment on attachment 8855543 [details]
ansible/hg-ssh: stop generating bzip2 bundles (bug 1354311);

https://reviewboard.mozilla.org/r/127378/#review130740

Comment 20

[:glob ✱]

Comment on attachment 8855544 [details]
generate-hg-s3-bundles: remove support for bzip2 bundles (bug 1354311);

https://reviewboard.mozilla.org/r/127380/#review130742

Comment 21

[:glob ✱]

Comment on attachment 8855545 [details]
hgserver: remove extra commit in test (bug 1354311);

https://reviewboard.mozilla.org/r/127382/#review130744

Comment 22

[:glob ✱]

Comment on attachment 8855546 [details]
hgserver: invert tests to prefer generaldelta repos (bug 1354311);

https://reviewboard.mozilla.org/r/127384/#review130746

Comment 23

[:glob ✱]

Comment on attachment 8855547 [details]
generate-hg-s3-bundles: simplify generaldelta checking (bug 1354311);

https://reviewboard.mozilla.org/r/127386/#review130748

assuming there's never

Comment 24

[:glob ✱]

Comment on attachment 8855548 [details]
generate-hg-s3-bundles: require generaldelta repos (bug 1354311);

https://reviewboard.mozilla.org/r/127388/#review130750

Comment 25

[:glob ✱]

Comment on attachment 8855549 [details]
generate-hg-s3-bundles: remove code generating non-generaldelta bundles (bug 1354311);

https://reviewboard.mozilla.org/r/127390/#review130752

Comment 26

[:glob ✱]

Comment on attachment 8855550 [details]
generate-hg-s3-bundles: renamed packed1-gd to packed1 (bug 1354311);

https://reviewboard.mozilla.org/r/127392/#review130754

Comment 27

[:glob ✱]

Comment on attachment 8855551 [details]
generate-hg-s3-bundles: use tools virtualenv (bug 1354311);

https://reviewboard.mozilla.org/r/127394/#review130756

Comment 28

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/hgcustom/version-control-tools/rev/75796fa6f7c4
ansible/hg-ssh: stop generating bzip2 bundles ; r=glob
https://hg.mozilla.org/hgcustom/version-control-tools/rev/bde4add4abb5
generate-hg-s3-bundles: remove support for bzip2 bundles ; r=glob
https://hg.mozilla.org/hgcustom/version-control-tools/rev/7ba4b6b49bb8
hgserver: remove extra commit in test ; r=glob
https://hg.mozilla.org/hgcustom/version-control-tools/rev/9d1fc54127e8
hgserver: invert tests to prefer generaldelta repos ; r=glob
https://hg.mozilla.org/hgcustom/version-control-tools/rev/0b3e482bc91d
generate-hg-s3-bundles: simplify generaldelta checking ; r=glob
https://hg.mozilla.org/hgcustom/version-control-tools/rev/9689c02cb88d
generate-hg-s3-bundles: require generaldelta repos ; r=glob
https://hg.mozilla.org/hgcustom/version-control-tools/rev/340cb5c7181f
generate-hg-s3-bundles: remove code generating non-generaldelta bundles ; r=glob
https://hg.mozilla.org/hgcustom/version-control-tools/rev/de72aef4e48b
generate-hg-s3-bundles: renamed packed1-gd to packed1 ; r=glob
https://hg.mozilla.org/hgcustom/version-control-tools/rev/90d75c8c4c35
generate-hg-s3-bundles: use tools virtualenv ; r=glob

Comment 29

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/hgcustom/version-control-tools/rev/fd0c73668d1a
generate-hg-s3-bundles: remove "all" keyword