1216216 - Make bundleclone compatible with Mercurial 3.6

Status: RESOLVED FIXED

(Developer Services :: Mercurial: bundleclone, defect)

Description

[Gregory Szorc [:gps]]

bundleclone's functionality has been added to Mercurial 3.6. We need to:

1) make the bundleclone extension compatible with code refactorings in Mercurial 3.6
2) make the bundleclone extension essentially no-op when the server supports the official Mercurial clone bundles feature
3) make hg.mozilla.org support the new 3.6 clone bundles feature via the bundleclone extension before the server is upgraded to 3.6 (so 3.6 clients can use the built-in clone bundles feature before we upgrade the server)

This work should be performed ASAP so we can start testing the 3.6 RC at Mozilla and against hg.mozilla.org

Comment 1

[Gregory Szorc [:gps]]

Attachment: MozReview Request: bundleclone: make test-sni.t work on older Mercurial versions; r?smacleod

bundleclone: make test-sni.t work on older Mercurial versions; r?smacleod

These tests were failing when executed on older Mercurial versions
because they assumed we were using a modern Mercurial. bundleclone
(unlike every other extension in version-control-tools) supports ancient
versions of Mercurial. So make the tests pass on old versions.

Comment 2

[Gregory Szorc [:gps]]

Attachment: MozReview Request: bundleclone: make tests pass on old Mercurial versions; r?smacleod

bundleclone: make tests pass on old Mercurial versions; r?smacleod

The tests for bundle manifest copying were failing on Mercurial releases
older than 3.0. We don't care about this functionality on these old
versions. So just don't run the tests there.

Since we added the first consumer of hghave, we needed to install a stub
file in the test directory to make the test harness happy.

Comment 3

[Gregory Szorc [:gps]]

Attachment: MozReview Request: bundleclone: import streamclone.py from upstream; r?smacleod

bundleclone: import streamclone.py from upstream; r?smacleod

We replace our copies of upstream code with the equivalent code from
Mercurial 3.6's streamclone.py. The majority of this change should be a
rubber stamp. There are a few changes in the extension code to support
renamed and refactored APIs.

Comment 4

[Gregory Szorc [:gps]]

Attachment: MozReview Request: bundleclone: make compatible with Mercurial 3.6 (bug 1216216); r?smacleod

bundleclone: make compatible with Mercurial 3.6 (bug 1216216); r?smacleod

Mercurial 3.6 changed the API for changegroup application slightly. We
detect support for it.

With this change, tests pass on all versions of Mercurial from 2.5 to
3.6.

Comment 5

[Gregory Szorc [:gps]]

Attachment: MozReview Request: bundleclone: refactor capabilities(); r?smacleod

bundleclone: refactor capabilities(); r?smacleod

This will make future refactoring slightly simpler.

Comment 6

[Gregory Szorc [:gps]]

Attachment: MozReview Request: bundleclone: only advertise bundles capability if manifest present; r?smacleod

bundleclone: only advertise bundles capability if manifest present; r?smacleod

This brings the behavior in-line with the feature in core. It also
prevents an extra round trip from client to server when cloning from a
repository that doesn't have a bundleclone.manifest file.

Comment 7

[Gregory Szorc [:gps]]

Comment on attachment 8675819 [details]
MozReview Request: bundleclone: make test-sni.t work on older Mercurial versions; r?smacleod

bundleclone: make test-sni.t work on older Mercurial versions; r?smacleod

These tests were failing when executed on older Mercurial versions
because they assumed we were using a modern Mercurial. bundleclone
(unlike every other extension in version-control-tools) supports ancient
versions of Mercurial. So make the tests pass on old versions.

Comment 8

[Gregory Szorc [:gps]]

Comment on attachment 8675820 [details]
MozReview Request: bundleclone: make tests pass on old Mercurial versions; r?smacleod

bundleclone: make tests pass on old Mercurial versions; r?smacleod

The tests for bundle manifest copying were failing on Mercurial releases
older than 3.0. We don't care about this functionality on these old
versions. So just don't run the tests there.

Since we added the first consumer of hghave, we needed to install a stub
file in the test directory to make the test harness happy.

Comment 9

[Gregory Szorc [:gps]]

Comment on attachment 8675821 [details]
MozReview Request: bundleclone: import streamclone.py from upstream; r?smacleod

bundleclone: import streamclone.py from upstream; r?smacleod

We replace our copies of upstream code with the equivalent code from
Mercurial 3.6's streamclone.py. The majority of this change should be a
rubber stamp. There are a few changes in the extension code to support
renamed and refactored APIs.

Comment 10

[Gregory Szorc [:gps]]

Comment on attachment 8675822 [details]
MozReview Request: bundleclone: make compatible with Mercurial 3.6 (bug 1216216); r?smacleod

bundleclone: make compatible with Mercurial 3.6 (bug 1216216); r?smacleod

Mercurial 3.6 changed the API for changegroup application slightly. We
detect support for it.

With this change, tests pass on all versions of Mercurial from 2.5 to
3.6.

Comment 11

[Gregory Szorc [:gps]]

Comment on attachment 8675823 [details]
MozReview Request: bundleclone: refactor capabilities(); r?smacleod

bundleclone: refactor capabilities(); r?smacleod

This will make future refactoring slightly simpler.

Comment 12

[Gregory Szorc [:gps]]

Comment on attachment 8675824 [details]
MozReview Request: bundleclone: only advertise bundles capability if manifest present; r?smacleod

bundleclone: only advertise bundles capability if manifest present; r?smacleod

This brings the behavior in-line with the feature in core. It also
prevents an extra round trip from client to server when cloning from a
repository that doesn't have a bundleclone.manifest file.

Comment 13

[Gregory Szorc [:gps]]

Attachment: MozReview Request: bundleclone: vendor clonebundles extension (bug 1216216); r?smacleod

bundleclone: vendor clonebundles extension (bug 1216216); r?smacleod

We import hgext/clonebundles.py unmodified from changeset
df1a29ec45bfaee30f974b06a0214a3359b369aa of Mercurial so we can run it
before we deploy Mercurial 3.6 to hg.mozilla.org.

Comment 14

[Gregory Szorc [:gps]]

Attachment: MozReview Request: bundleclone: disable bundleclone when clone bundles feature detected (bug 1216216); r?smacleod

bundleclone: disable bundleclone when clone bundles feature detected (bug 1216216); r?smacleod

Mercurial 3.6 has official support for the bundle clone feature set. The
bundleclone extension is thus redundant with that functionality (or at
least it should if the server operator is competent).

This commit adds detection of the clone bundles feature to the client
and will short-circuit the bundleclone functionality when the built-in
support for cloning from pre-generated bundles is sufficient.

Comment 15

[Gregory Szorc [:gps]]

Attachment: MozReview Request: bundleclone: support producing clone bundle compatible stream bundles (bug 1216216); r?smacleod

bundleclone: support producing clone bundle compatible stream bundles (bug 1216216); r?smacleod

When bundleclone was initially implemented, the concept of a "stream
bundle" was invented. It was essentially the wire protocol
representation of the existing streaming clone data saved to a file.

Mercurial 3.6 formalized the "stream bundle" format. It is slightly
different from and incompatible with the wire protocol format.

This patch teaches our `hg streambundle` command to produce the new,
formal format. We also add tests verifying a client requests the
appropriate file depending on whether clone bundles or bundleclone is
used.

Comment 16

[Gregory Szorc [:gps]]

Attachment: MozReview Request: bundleclone: support pulling clonebundles.manifest (bug 1216216); r?smacleod

bundleclone: support pulling clonebundles.manifest (bug 1216216); r?smacleod

We currently have support for `hg pull` copying the bundleclone.manifest
file. This is used for mirroring on hg.mozilla.org.

The core support for clone bundles doesn't (yet) support such a
mirroring ability. So, we implement support for clonebundles.manifest
mirroring.

Comment 17

[Gregory Szorc [:gps]]

Attachment: MozReview Request: scripts/generate-hg-s3-bundles: refactor to better support abstraction; r?smacleod

scripts/generate-hg-s3-bundles: refactor to better support abstraction; r?smacleod

There was inline code giving stream bundles special treatment. In
addition, we were mixing configuration for bundle generation and
manifest formatting into the same object.

We extract bundle generation and manifest settings into separate
objects. The bundle generation object now defines the ``hg`` arguments
to produce the bundle. The manifest object now defines the parameters
for that manifest.

Comment 18

[Gregory Szorc [:gps]]

Attachment: MozReview Request: scripts/generate-hg-s3-bundles: create clonebundles.manifest (bug 1216216); r?smacleod

scripts/generate-hg-s3-bundles: create clonebundles.manifest (bug 1216216); r?smacleod

The built-in Mercurial feature for serving pre-built bundles is slightly
different from bundleclone. It uses a separate manifest that has a
slightly different format (mostly cosmetic changes).

With this commit, we start writing this new manifest file when
generating bundles. As part of this, we generate the modern stream
bundle file format.

Comment 19

[Gregory Szorc [:gps]]

Attachment: MozReview Request: ansible: activate clonebundles extension on hg-web and hg-ssh (bug 1216216); r?smacleod

ansible: activate clonebundles extension on hg-web and hg-ssh (bug 1216216); r?smacleod

We are now generating clonebundles.manifest files. Activate our vendored
copy of the clonebundles extension from the Mercurial project so 3.6
clients can fetch bundles using the clone bundles mechanism built into
core Mercurial.

Comment 20

[Steven MacLeod [:smacleod]]

Comment on attachment 8675819 [details]
MozReview Request: bundleclone: make test-sni.t work on older Mercurial versions; r?smacleod

https://reviewboard.mozilla.org/r/22443/#review20309

Comment 21

[Steven MacLeod [:smacleod]]

Comment on attachment 8675820 [details]
MozReview Request: bundleclone: make tests pass on old Mercurial versions; r?smacleod

https://reviewboard.mozilla.org/r/22445/#review20311

Comment 22

[Steven MacLeod [:smacleod]]

Comment on attachment 8675821 [details]
MozReview Request: bundleclone: import streamclone.py from upstream; r?smacleod

https://reviewboard.mozilla.org/r/22447/#review20313

Comment 23

[Steven MacLeod [:smacleod]]

Comment on attachment 8675822 [details]
MozReview Request: bundleclone: make compatible with Mercurial 3.6 (bug 1216216); r?smacleod

https://reviewboard.mozilla.org/r/22449/#review20315

Comment 24

[Steven MacLeod [:smacleod]]

Comment on attachment 8675823 [details]
MozReview Request: bundleclone: refactor capabilities(); r?smacleod

https://reviewboard.mozilla.org/r/22451/#review20317

Comment 25

[Steven MacLeod [:smacleod]]

Comment on attachment 8675824 [details]
MozReview Request: bundleclone: only advertise bundles capability if manifest present; r?smacleod

https://reviewboard.mozilla.org/r/22453/#review20319

Comment 26

[Steven MacLeod [:smacleod]]

Comment on attachment 8675951 [details]
MozReview Request: bundleclone: vendor clonebundles extension (bug 1216216); r?smacleod

https://reviewboard.mozilla.org/r/22517/#review20321

Comment 27

[Steven MacLeod [:smacleod]]

Comment on attachment 8675952 [details]
MozReview Request: bundleclone: disable bundleclone when clone bundles feature detected (bug 1216216); r?smacleod

https://reviewboard.mozilla.org/r/22519/#review20325

Comment 28

[Steven MacLeod [:smacleod]]

Comment on attachment 8675953 [details]
MozReview Request: bundleclone: support producing clone bundle compatible stream bundles (bug 1216216); r?smacleod

https://reviewboard.mozilla.org/r/22521/#review20329

::: hgext/bundleclone/tests/test-clonebundles-compat.t:111
(Diff revision 1)
> +  $ hg -R server streambundle stream-legacy.hg
> +  writing stream-legacy.hg
> +  stream bundle file written successully.
> +  include the following in its manifest entry:
> +  stream=revlogv1
> +  $ hg -R server streambundle --type s1 stream-s1.hg
> +  writing 613 bytes for 4 files
> +  bundle requirements: revlogv1
> +
> +  $ cat > server/.hg/clonebundles.manifest << EOF
> +  > http://localhost:$HGPORT1/stream-s1.hg BUNDLESPEC=none-packed1;requirements%3Drevlogv1
> +  > EOF
> +
> +  $ cat > server/.hg/bundleclone.manifest << EOF
> +  > http://localhost:$HGPORT1/stream-legacy.hg stream=revlogv1
> +  > EOF

The ordering is a little strange here, can you stick the manifest creation with their bundle generation, or at least order `legacy-gen, s1-gen, legacy-manifest, s1-manifest`

Comment 29

[Steven MacLeod [:smacleod]]

Comment on attachment 8675954 [details]
MozReview Request: bundleclone: support pulling clonebundles.manifest (bug 1216216); r?smacleod

https://reviewboard.mozilla.org/r/22523/#review20331

Comment 30

[Steven MacLeod [:smacleod]]

Comment on attachment 8675955 [details]
MozReview Request: scripts/generate-hg-s3-bundles: refactor to better support abstraction; r?smacleod

https://reviewboard.mozilla.org/r/22525/#review20335

::: scripts/generate-hg-s3-bundles:120
(Diff revision 1)
> -def generate_bundle(repo, typ, temp_path, final_path):
> +def generate_bundle(repo, typ, temp_path, final_path, extra_args):
>      """Generate a single bundle of a type.
>  
>      Create a bundle of type ``typ`` from repo ``repo`` using temporary file
>      ``temp_path`` before atomically moving it to ``final_path``.

`typ` is no longer used, is there a reason you're keeping it as an argument?

Comment 31

[Steven MacLeod [:smacleod]]

Comment on attachment 8675956 [details]
MozReview Request: scripts/generate-hg-s3-bundles: create clonebundles.manifest (bug 1216216); r?smacleod

https://reviewboard.mozilla.org/r/22527/#review20337

Comment 32

[Steven MacLeod [:smacleod]]

Comment on attachment 8675957 [details]
MozReview Request: ansible: activate clonebundles extension on hg-web and hg-ssh (bug 1216216); r?smacleod

https://reviewboard.mozilla.org/r/22529/#review20339

Comment 33

[Gregory Szorc [:gps]]

https://reviewboard.mozilla.org/r/22525/#review20335

> `typ` is no longer used, is there a reason you're keeping it as an argument?

Good catch. Will fix in flight.

Comment 34

[Gregory Szorc [:gps]]

https://hg.mozilla.org/hgcustom/version-control-tools/rev/ca6db63ecd9282bd8720aa21c0a50fd976726386
bundleclone: make compatible with Mercurial 3.6 (bug 1216216); r=smacleod

https://hg.mozilla.org/hgcustom/version-control-tools/rev/145c83c1849464369e35ebd8a5532db73fa8470d
bundleclone: vendor clonebundles extension (bug 1216216); r=smacleod

https://hg.mozilla.org/hgcustom/version-control-tools/rev/e9fd484a51a08751dce8dccc1d94d2d25e5b8bcb
bundleclone: disable bundleclone when clone bundles feature detected (bug 1216216); r=smacleod

https://hg.mozilla.org/hgcustom/version-control-tools/rev/f9a129d2caec42fd4c6ffe78f6ddc87ee4173177
bundleclone: support producing clone bundle compatible stream bundles (bug 1216216); r=smacleod

https://hg.mozilla.org/hgcustom/version-control-tools/rev/8e374504c9ed457d2626b3502d5b8562195827ec
bundleclone: support pulling clonebundles.manifest (bug 1216216); r=smacleod

https://hg.mozilla.org/hgcustom/version-control-tools/rev/b061823cf73323b6ef725be2b60cf9523a7f0731
scripts/generate-hg-s3-bundles: create clonebundles.manifest (bug 1216216); r=smacleod

https://hg.mozilla.org/hgcustom/version-control-tools/rev/c0052bbdd6b85da4a879639ca994438f542c57bd
ansible: activate clonebundles extension on hg-web and hg-ssh (bug 1216216); r=smacleod

Comment 35

[Gregory Szorc [:gps]]

https://hg.mozilla.org/hgcustom/version-control-tools/rev/eda317f4cee4ac2ef6f80a1de0446edddd953c56
scripts/generate-hg-s3-bundles: fix errors in index formatting (bug 1216216)

Comment 36

[Gregory Szorc [:gps]]

https://hg.mozilla.org/hgcustom/version-control-tools/rev/800ccc3eb4356e8c3ccbffa554b962d1d8536fa5
scripts/generate-hg-s3-bundles: add entries to proper manifest (bug 1216216)

Comment 37

[Gregory Szorc [:gps]]

This is deployed, which a few follow-up fixups. I've manually regenerated bundles for mozilla-central and verified things appear to be working. I kicked off a regeneration of everything else. That should complete within the next few hours.