Closed Bug 1216216 Opened 5 years ago Closed 5 years ago

Make bundleclone compatible with Mercurial 3.6

Categories

(Developer Services :: Mercurial: bundleclone, defect)

defect
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: gps, Assigned: gps)

References

Details

Attachments

(13 files)

40 bytes, text/x-review-board-request
smacleod
: review+
Details
40 bytes, text/x-review-board-request
smacleod
: review+
Details
40 bytes, text/x-review-board-request
smacleod
: review+
Details
40 bytes, text/x-review-board-request
smacleod
: review+
Details
40 bytes, text/x-review-board-request
smacleod
: review+
Details
40 bytes, text/x-review-board-request
smacleod
: review+
Details
40 bytes, text/x-review-board-request
smacleod
: review+
Details
40 bytes, text/x-review-board-request
smacleod
: review+
Details
40 bytes, text/x-review-board-request
smacleod
: review+
Details
40 bytes, text/x-review-board-request
smacleod
: review+
Details
40 bytes, text/x-review-board-request
smacleod
: review+
Details
40 bytes, text/x-review-board-request
smacleod
: review+
Details
40 bytes, text/x-review-board-request
smacleod
: review+
Details
bundleclone's functionality has been added to Mercurial 3.6. We need to:

1) make the bundleclone extension compatible with code refactorings in Mercurial 3.6
2) make the bundleclone extension essentially no-op when the server supports the official Mercurial clone bundles feature
3) make hg.mozilla.org support the new 3.6 clone bundles feature via the bundleclone extension before the server is upgraded to 3.6 (so 3.6 clients can use the built-in clone bundles feature before we upgrade the server)

This work should be performed ASAP so we can start testing the 3.6 RC at Mozilla and against hg.mozilla.org
bundleclone: make test-sni.t work on older Mercurial versions; r?smacleod

These tests were failing when executed on older Mercurial versions
because they assumed we were using a modern Mercurial. bundleclone
(unlike every other extension in version-control-tools) supports ancient
versions of Mercurial. So make the tests pass on old versions.
Attachment #8675819 - Flags: review?(smacleod)
bundleclone: make tests pass on old Mercurial versions; r?smacleod

The tests for bundle manifest copying were failing on Mercurial releases
older than 3.0. We don't care about this functionality on these old
versions. So just don't run the tests there.

Since we added the first consumer of hghave, we needed to install a stub
file in the test directory to make the test harness happy.
Attachment #8675820 - Flags: review?(smacleod)
bundleclone: import streamclone.py from upstream; r?smacleod

We replace our copies of upstream code with the equivalent code from
Mercurial 3.6's streamclone.py. The majority of this change should be a
rubber stamp. There are a few changes in the extension code to support
renamed and refactored APIs.
Attachment #8675821 - Flags: review?(smacleod)
bundleclone: make compatible with Mercurial 3.6 (bug 1216216); r?smacleod

Mercurial 3.6 changed the API for changegroup application slightly. We
detect support for it.

With this change, tests pass on all versions of Mercurial from 2.5 to
3.6.
Attachment #8675822 - Flags: review?(smacleod)
bundleclone: refactor capabilities(); r?smacleod

This will make future refactoring slightly simpler.
Attachment #8675823 - Flags: review?(smacleod)
bundleclone: only advertise bundles capability if manifest present; r?smacleod

This brings the behavior in-line with the feature in core. It also
prevents an extra round trip from client to server when cloning from a
repository that doesn't have a bundleclone.manifest file.
Attachment #8675824 - Flags: review?(smacleod)
Comment on attachment 8675819 [details]
MozReview Request: bundleclone: make test-sni.t work on older Mercurial versions; r?smacleod

bundleclone: make test-sni.t work on older Mercurial versions; r?smacleod

These tests were failing when executed on older Mercurial versions
because they assumed we were using a modern Mercurial. bundleclone
(unlike every other extension in version-control-tools) supports ancient
versions of Mercurial. So make the tests pass on old versions.
Comment on attachment 8675820 [details]
MozReview Request: bundleclone: make tests pass on old Mercurial versions; r?smacleod

bundleclone: make tests pass on old Mercurial versions; r?smacleod

The tests for bundle manifest copying were failing on Mercurial releases
older than 3.0. We don't care about this functionality on these old
versions. So just don't run the tests there.

Since we added the first consumer of hghave, we needed to install a stub
file in the test directory to make the test harness happy.
Comment on attachment 8675821 [details]
MozReview Request: bundleclone: import streamclone.py from upstream; r?smacleod

bundleclone: import streamclone.py from upstream; r?smacleod

We replace our copies of upstream code with the equivalent code from
Mercurial 3.6's streamclone.py. The majority of this change should be a
rubber stamp. There are a few changes in the extension code to support
renamed and refactored APIs.
Comment on attachment 8675822 [details]
MozReview Request: bundleclone: make compatible with Mercurial 3.6 (bug 1216216); r?smacleod

bundleclone: make compatible with Mercurial 3.6 (bug 1216216); r?smacleod

Mercurial 3.6 changed the API for changegroup application slightly. We
detect support for it.

With this change, tests pass on all versions of Mercurial from 2.5 to
3.6.
Comment on attachment 8675823 [details]
MozReview Request: bundleclone: refactor capabilities(); r?smacleod

bundleclone: refactor capabilities(); r?smacleod

This will make future refactoring slightly simpler.
Comment on attachment 8675824 [details]
MozReview Request: bundleclone: only advertise bundles capability if manifest present; r?smacleod

bundleclone: only advertise bundles capability if manifest present; r?smacleod

This brings the behavior in-line with the feature in core. It also
prevents an extra round trip from client to server when cloning from a
repository that doesn't have a bundleclone.manifest file.
bundleclone: vendor clonebundles extension (bug 1216216); r?smacleod

We import hgext/clonebundles.py unmodified from changeset
df1a29ec45bfaee30f974b06a0214a3359b369aa of Mercurial so we can run it
before we deploy Mercurial 3.6 to hg.mozilla.org.
Attachment #8675951 - Flags: review?(smacleod)
bundleclone: disable bundleclone when clone bundles feature detected (bug 1216216); r?smacleod

Mercurial 3.6 has official support for the bundle clone feature set. The
bundleclone extension is thus redundant with that functionality (or at
least it should if the server operator is competent).

This commit adds detection of the clone bundles feature to the client
and will short-circuit the bundleclone functionality when the built-in
support for cloning from pre-generated bundles is sufficient.
Attachment #8675952 - Flags: review?(smacleod)
bundleclone: support producing clone bundle compatible stream bundles (bug 1216216); r?smacleod

When bundleclone was initially implemented, the concept of a "stream
bundle" was invented. It was essentially the wire protocol
representation of the existing streaming clone data saved to a file.

Mercurial 3.6 formalized the "stream bundle" format. It is slightly
different from and incompatible with the wire protocol format.

This patch teaches our `hg streambundle` command to produce the new,
formal format. We also add tests verifying a client requests the
appropriate file depending on whether clone bundles or bundleclone is
used.
Attachment #8675953 - Flags: review?(smacleod)
bundleclone: support pulling clonebundles.manifest (bug 1216216); r?smacleod

We currently have support for `hg pull` copying the bundleclone.manifest
file. This is used for mirroring on hg.mozilla.org.

The core support for clone bundles doesn't (yet) support such a
mirroring ability. So, we implement support for clonebundles.manifest
mirroring.
Attachment #8675954 - Flags: review?(smacleod)
scripts/generate-hg-s3-bundles: refactor to better support abstraction; r?smacleod

There was inline code giving stream bundles special treatment. In
addition, we were mixing configuration for bundle generation and
manifest formatting into the same object.

We extract bundle generation and manifest settings into separate
objects. The bundle generation object now defines the ``hg`` arguments
to produce the bundle. The manifest object now defines the parameters
for that manifest.
Attachment #8675955 - Flags: review?(smacleod)
scripts/generate-hg-s3-bundles: create clonebundles.manifest (bug 1216216); r?smacleod

The built-in Mercurial feature for serving pre-built bundles is slightly
different from bundleclone. It uses a separate manifest that has a
slightly different format (mostly cosmetic changes).

With this commit, we start writing this new manifest file when
generating bundles. As part of this, we generate the modern stream
bundle file format.
Attachment #8675956 - Flags: review?(smacleod)
ansible: activate clonebundles extension on hg-web and hg-ssh (bug 1216216); r?smacleod

We are now generating clonebundles.manifest files. Activate our vendored
copy of the clonebundles extension from the Mercurial project so 3.6
clients can fetch bundles using the clone bundles mechanism built into
core Mercurial.
Attachment #8675957 - Flags: review?(smacleod)
Comment on attachment 8675819 [details]
MozReview Request: bundleclone: make test-sni.t work on older Mercurial versions; r?smacleod

https://reviewboard.mozilla.org/r/22443/#review20309
Attachment #8675819 - Flags: review?(smacleod) → review+
Comment on attachment 8675820 [details]
MozReview Request: bundleclone: make tests pass on old Mercurial versions; r?smacleod

https://reviewboard.mozilla.org/r/22445/#review20311
Attachment #8675820 - Flags: review?(smacleod) → review+
Comment on attachment 8675821 [details]
MozReview Request: bundleclone: import streamclone.py from upstream; r?smacleod

https://reviewboard.mozilla.org/r/22447/#review20313
Attachment #8675821 - Flags: review?(smacleod) → review+
Comment on attachment 8675822 [details]
MozReview Request: bundleclone: make compatible with Mercurial 3.6 (bug 1216216); r?smacleod

https://reviewboard.mozilla.org/r/22449/#review20315
Attachment #8675822 - Flags: review?(smacleod) → review+
Comment on attachment 8675823 [details]
MozReview Request: bundleclone: refactor capabilities(); r?smacleod

https://reviewboard.mozilla.org/r/22451/#review20317
Attachment #8675823 - Flags: review?(smacleod) → review+
Comment on attachment 8675824 [details]
MozReview Request: bundleclone: only advertise bundles capability if manifest present; r?smacleod

https://reviewboard.mozilla.org/r/22453/#review20319
Attachment #8675824 - Flags: review?(smacleod) → review+
Attachment #8675951 - Flags: review?(smacleod) → review+
Comment on attachment 8675951 [details]
MozReview Request: bundleclone: vendor clonebundles extension (bug 1216216); r?smacleod

https://reviewboard.mozilla.org/r/22517/#review20321
Comment on attachment 8675952 [details]
MozReview Request: bundleclone: disable bundleclone when clone bundles feature detected (bug 1216216); r?smacleod

https://reviewboard.mozilla.org/r/22519/#review20325
Attachment #8675952 - Flags: review?(smacleod) → review+
Attachment #8675953 - Flags: review?(smacleod) → review+
Comment on attachment 8675953 [details]
MozReview Request: bundleclone: support producing clone bundle compatible stream bundles (bug 1216216); r?smacleod

https://reviewboard.mozilla.org/r/22521/#review20329

::: hgext/bundleclone/tests/test-clonebundles-compat.t:111
(Diff revision 1)
> +  $ hg -R server streambundle stream-legacy.hg
> +  writing stream-legacy.hg
> +  stream bundle file written successully.
> +  include the following in its manifest entry:
> +  stream=revlogv1
> +  $ hg -R server streambundle --type s1 stream-s1.hg
> +  writing 613 bytes for 4 files
> +  bundle requirements: revlogv1
> +
> +  $ cat > server/.hg/clonebundles.manifest << EOF
> +  > http://localhost:$HGPORT1/stream-s1.hg BUNDLESPEC=none-packed1;requirements%3Drevlogv1
> +  > EOF
> +
> +  $ cat > server/.hg/bundleclone.manifest << EOF
> +  > http://localhost:$HGPORT1/stream-legacy.hg stream=revlogv1
> +  > EOF

The ordering is a little strange here, can you stick the manifest creation with their bundle generation, or at least order `legacy-gen, s1-gen, legacy-manifest, s1-manifest`
Comment on attachment 8675954 [details]
MozReview Request: bundleclone: support pulling clonebundles.manifest (bug 1216216); r?smacleod

https://reviewboard.mozilla.org/r/22523/#review20331
Attachment #8675954 - Flags: review?(smacleod) → review+
Comment on attachment 8675955 [details]
MozReview Request: scripts/generate-hg-s3-bundles: refactor to better support abstraction; r?smacleod

https://reviewboard.mozilla.org/r/22525/#review20335

::: scripts/generate-hg-s3-bundles:120
(Diff revision 1)
> -def generate_bundle(repo, typ, temp_path, final_path):
> +def generate_bundle(repo, typ, temp_path, final_path, extra_args):
>      """Generate a single bundle of a type.
>  
>      Create a bundle of type ``typ`` from repo ``repo`` using temporary file
>      ``temp_path`` before atomically moving it to ``final_path``.

`typ` is no longer used, is there a reason you're keeping it as an argument?
Attachment #8675955 - Flags: review?(smacleod) → review+
Comment on attachment 8675956 [details]
MozReview Request: scripts/generate-hg-s3-bundles: create clonebundles.manifest (bug 1216216); r?smacleod

https://reviewboard.mozilla.org/r/22527/#review20337
Attachment #8675956 - Flags: review?(smacleod) → review+
Comment on attachment 8675957 [details]
MozReview Request: ansible: activate clonebundles extension on hg-web and hg-ssh (bug 1216216); r?smacleod

https://reviewboard.mozilla.org/r/22529/#review20339
Attachment #8675957 - Flags: review?(smacleod) → review+
https://reviewboard.mozilla.org/r/22525/#review20335

> `typ` is no longer used, is there a reason you're keeping it as an argument?

Good catch. Will fix in flight.
https://hg.mozilla.org/hgcustom/version-control-tools/rev/ca6db63ecd9282bd8720aa21c0a50fd976726386
bundleclone: make compatible with Mercurial 3.6 (bug 1216216); r=smacleod

https://hg.mozilla.org/hgcustom/version-control-tools/rev/145c83c1849464369e35ebd8a5532db73fa8470d
bundleclone: vendor clonebundles extension (bug 1216216); r=smacleod

https://hg.mozilla.org/hgcustom/version-control-tools/rev/e9fd484a51a08751dce8dccc1d94d2d25e5b8bcb
bundleclone: disable bundleclone when clone bundles feature detected (bug 1216216); r=smacleod

https://hg.mozilla.org/hgcustom/version-control-tools/rev/f9a129d2caec42fd4c6ffe78f6ddc87ee4173177
bundleclone: support producing clone bundle compatible stream bundles (bug 1216216); r=smacleod

https://hg.mozilla.org/hgcustom/version-control-tools/rev/8e374504c9ed457d2626b3502d5b8562195827ec
bundleclone: support pulling clonebundles.manifest (bug 1216216); r=smacleod

https://hg.mozilla.org/hgcustom/version-control-tools/rev/b061823cf73323b6ef725be2b60cf9523a7f0731
scripts/generate-hg-s3-bundles: create clonebundles.manifest (bug 1216216); r=smacleod

https://hg.mozilla.org/hgcustom/version-control-tools/rev/c0052bbdd6b85da4a879639ca994438f542c57bd
ansible: activate clonebundles extension on hg-web and hg-ssh (bug 1216216); r=smacleod
This is deployed, which a few follow-up fixups. I've manually regenerated bundles for mozilla-central and verified things appear to be working. I kicked off a regeneration of everything else. That should complete within the next few hours.
Status: ASSIGNED → RESOLVED
Closed: 5 years ago
Resolution: --- → FIXED
No longer blocks: 1217147
Blocks: 1217155
Blocks: 1221268
Blocks: 1352494
You need to log in before you can comment on or make changes to this bug.