Make bundleclone compatible with Mercurial 3.6

RESOLVED FIXED

Status

Developer Services
Mercurial: bundleclone
RESOLVED FIXED
2 years ago
3 months ago

People

(Reporter: gps, Assigned: gps)

Tracking

Details

MozReview Requests

Submitter Diff Changes Open Issues Last Updated
Loading...
Error loading review requests:

Attachments

(13 attachments)

40 bytes, text/x-review-board-request
smacleod
: review+
Details | Review
40 bytes, text/x-review-board-request
smacleod
: review+
Details | Review
40 bytes, text/x-review-board-request
smacleod
: review+
Details | Review
40 bytes, text/x-review-board-request
smacleod
: review+
Details | Review
40 bytes, text/x-review-board-request
smacleod
: review+
Details | Review
40 bytes, text/x-review-board-request
smacleod
: review+
Details | Review
40 bytes, text/x-review-board-request
smacleod
: review+
Details | Review
40 bytes, text/x-review-board-request
smacleod
: review+
Details | Review
40 bytes, text/x-review-board-request
smacleod
: review+
Details | Review
40 bytes, text/x-review-board-request
smacleod
: review+
Details | Review
40 bytes, text/x-review-board-request
smacleod
: review+
Details | Review
40 bytes, text/x-review-board-request
smacleod
: review+
Details | Review
40 bytes, text/x-review-board-request
smacleod
: review+
Details | Review
(Assignee)

Description

2 years ago
bundleclone's functionality has been added to Mercurial 3.6. We need to:

1) make the bundleclone extension compatible with code refactorings in Mercurial 3.6
2) make the bundleclone extension essentially no-op when the server supports the official Mercurial clone bundles feature
3) make hg.mozilla.org support the new 3.6 clone bundles feature via the bundleclone extension before the server is upgraded to 3.6 (so 3.6 clients can use the built-in clone bundles feature before we upgrade the server)

This work should be performed ASAP so we can start testing the 3.6 RC at Mozilla and against hg.mozilla.org
(Assignee)

Comment 1

2 years ago
Created attachment 8675819 [details]
MozReview Request: bundleclone: make test-sni.t work on older Mercurial versions; r?smacleod

bundleclone: make test-sni.t work on older Mercurial versions; r?smacleod

These tests were failing when executed on older Mercurial versions
because they assumed we were using a modern Mercurial. bundleclone
(unlike every other extension in version-control-tools) supports ancient
versions of Mercurial. So make the tests pass on old versions.
Attachment #8675819 - Flags: review?(smacleod)
(Assignee)

Comment 2

2 years ago
Created attachment 8675820 [details]
MozReview Request: bundleclone: make tests pass on old Mercurial versions; r?smacleod

bundleclone: make tests pass on old Mercurial versions; r?smacleod

The tests for bundle manifest copying were failing on Mercurial releases
older than 3.0. We don't care about this functionality on these old
versions. So just don't run the tests there.

Since we added the first consumer of hghave, we needed to install a stub
file in the test directory to make the test harness happy.
Attachment #8675820 - Flags: review?(smacleod)
(Assignee)

Comment 3

2 years ago
Created attachment 8675821 [details]
MozReview Request: bundleclone: import streamclone.py from upstream; r?smacleod

bundleclone: import streamclone.py from upstream; r?smacleod

We replace our copies of upstream code with the equivalent code from
Mercurial 3.6's streamclone.py. The majority of this change should be a
rubber stamp. There are a few changes in the extension code to support
renamed and refactored APIs.
Attachment #8675821 - Flags: review?(smacleod)
(Assignee)

Comment 4

2 years ago
Created attachment 8675822 [details]
MozReview Request: bundleclone: make compatible with Mercurial 3.6 (bug 1216216); r?smacleod

bundleclone: make compatible with Mercurial 3.6 (bug 1216216); r?smacleod

Mercurial 3.6 changed the API for changegroup application slightly. We
detect support for it.

With this change, tests pass on all versions of Mercurial from 2.5 to
3.6.
Attachment #8675822 - Flags: review?(smacleod)
(Assignee)

Comment 5

2 years ago
Created attachment 8675823 [details]
MozReview Request: bundleclone: refactor capabilities(); r?smacleod

bundleclone: refactor capabilities(); r?smacleod

This will make future refactoring slightly simpler.
Attachment #8675823 - Flags: review?(smacleod)
(Assignee)

Comment 6

2 years ago
Created attachment 8675824 [details]
MozReview Request: bundleclone: only advertise bundles capability if manifest present; r?smacleod

bundleclone: only advertise bundles capability if manifest present; r?smacleod

This brings the behavior in-line with the feature in core. It also
prevents an extra round trip from client to server when cloning from a
repository that doesn't have a bundleclone.manifest file.
Attachment #8675824 - Flags: review?(smacleod)
(Assignee)

Comment 7

2 years ago
Comment on attachment 8675819 [details]
MozReview Request: bundleclone: make test-sni.t work on older Mercurial versions; r?smacleod

bundleclone: make test-sni.t work on older Mercurial versions; r?smacleod

These tests were failing when executed on older Mercurial versions
because they assumed we were using a modern Mercurial. bundleclone
(unlike every other extension in version-control-tools) supports ancient
versions of Mercurial. So make the tests pass on old versions.
(Assignee)

Comment 8

2 years ago
Comment on attachment 8675820 [details]
MozReview Request: bundleclone: make tests pass on old Mercurial versions; r?smacleod

bundleclone: make tests pass on old Mercurial versions; r?smacleod

The tests for bundle manifest copying were failing on Mercurial releases
older than 3.0. We don't care about this functionality on these old
versions. So just don't run the tests there.

Since we added the first consumer of hghave, we needed to install a stub
file in the test directory to make the test harness happy.
(Assignee)

Comment 9

2 years ago
Comment on attachment 8675821 [details]
MozReview Request: bundleclone: import streamclone.py from upstream; r?smacleod

bundleclone: import streamclone.py from upstream; r?smacleod

We replace our copies of upstream code with the equivalent code from
Mercurial 3.6's streamclone.py. The majority of this change should be a
rubber stamp. There are a few changes in the extension code to support
renamed and refactored APIs.
(Assignee)

Comment 10

2 years ago
Comment on attachment 8675822 [details]
MozReview Request: bundleclone: make compatible with Mercurial 3.6 (bug 1216216); r?smacleod

bundleclone: make compatible with Mercurial 3.6 (bug 1216216); r?smacleod

Mercurial 3.6 changed the API for changegroup application slightly. We
detect support for it.

With this change, tests pass on all versions of Mercurial from 2.5 to
3.6.
(Assignee)

Comment 11

2 years ago
Comment on attachment 8675823 [details]
MozReview Request: bundleclone: refactor capabilities(); r?smacleod

bundleclone: refactor capabilities(); r?smacleod

This will make future refactoring slightly simpler.
(Assignee)

Comment 12

2 years ago
Comment on attachment 8675824 [details]
MozReview Request: bundleclone: only advertise bundles capability if manifest present; r?smacleod

bundleclone: only advertise bundles capability if manifest present; r?smacleod

This brings the behavior in-line with the feature in core. It also
prevents an extra round trip from client to server when cloning from a
repository that doesn't have a bundleclone.manifest file.
(Assignee)

Comment 13

2 years ago
Created attachment 8675951 [details]
MozReview Request: bundleclone: vendor clonebundles extension (bug 1216216); r?smacleod

bundleclone: vendor clonebundles extension (bug 1216216); r?smacleod

We import hgext/clonebundles.py unmodified from changeset
df1a29ec45bfaee30f974b06a0214a3359b369aa of Mercurial so we can run it
before we deploy Mercurial 3.6 to hg.mozilla.org.
Attachment #8675951 - Flags: review?(smacleod)
(Assignee)

Comment 14

2 years ago
Created attachment 8675952 [details]
MozReview Request: bundleclone: disable bundleclone when clone bundles feature detected (bug 1216216); r?smacleod

bundleclone: disable bundleclone when clone bundles feature detected (bug 1216216); r?smacleod

Mercurial 3.6 has official support for the bundle clone feature set. The
bundleclone extension is thus redundant with that functionality (or at
least it should if the server operator is competent).

This commit adds detection of the clone bundles feature to the client
and will short-circuit the bundleclone functionality when the built-in
support for cloning from pre-generated bundles is sufficient.
Attachment #8675952 - Flags: review?(smacleod)
(Assignee)

Comment 15

2 years ago
Created attachment 8675953 [details]
MozReview Request: bundleclone: support producing clone bundle compatible stream bundles (bug 1216216); r?smacleod

bundleclone: support producing clone bundle compatible stream bundles (bug 1216216); r?smacleod

When bundleclone was initially implemented, the concept of a "stream
bundle" was invented. It was essentially the wire protocol
representation of the existing streaming clone data saved to a file.

Mercurial 3.6 formalized the "stream bundle" format. It is slightly
different from and incompatible with the wire protocol format.

This patch teaches our `hg streambundle` command to produce the new,
formal format. We also add tests verifying a client requests the
appropriate file depending on whether clone bundles or bundleclone is
used.
Attachment #8675953 - Flags: review?(smacleod)
(Assignee)

Comment 16

2 years ago
Created attachment 8675954 [details]
MozReview Request: bundleclone: support pulling clonebundles.manifest (bug 1216216); r?smacleod

bundleclone: support pulling clonebundles.manifest (bug 1216216); r?smacleod

We currently have support for `hg pull` copying the bundleclone.manifest
file. This is used for mirroring on hg.mozilla.org.

The core support for clone bundles doesn't (yet) support such a
mirroring ability. So, we implement support for clonebundles.manifest
mirroring.
Attachment #8675954 - Flags: review?(smacleod)
(Assignee)

Comment 17

2 years ago
Created attachment 8675955 [details]
MozReview Request: scripts/generate-hg-s3-bundles: refactor to better support abstraction; r?smacleod

scripts/generate-hg-s3-bundles: refactor to better support abstraction; r?smacleod

There was inline code giving stream bundles special treatment. In
addition, we were mixing configuration for bundle generation and
manifest formatting into the same object.

We extract bundle generation and manifest settings into separate
objects. The bundle generation object now defines the ``hg`` arguments
to produce the bundle. The manifest object now defines the parameters
for that manifest.
Attachment #8675955 - Flags: review?(smacleod)
(Assignee)

Comment 18

2 years ago
Created attachment 8675956 [details]
MozReview Request: scripts/generate-hg-s3-bundles: create clonebundles.manifest (bug 1216216); r?smacleod

scripts/generate-hg-s3-bundles: create clonebundles.manifest (bug 1216216); r?smacleod

The built-in Mercurial feature for serving pre-built bundles is slightly
different from bundleclone. It uses a separate manifest that has a
slightly different format (mostly cosmetic changes).

With this commit, we start writing this new manifest file when
generating bundles. As part of this, we generate the modern stream
bundle file format.
Attachment #8675956 - Flags: review?(smacleod)
(Assignee)

Comment 19

2 years ago
Created attachment 8675957 [details]
MozReview Request: ansible: activate clonebundles extension on hg-web and hg-ssh (bug 1216216); r?smacleod

ansible: activate clonebundles extension on hg-web and hg-ssh (bug 1216216); r?smacleod

We are now generating clonebundles.manifest files. Activate our vendored
copy of the clonebundles extension from the Mercurial project so 3.6
clients can fetch bundles using the clone bundles mechanism built into
core Mercurial.
Attachment #8675957 - Flags: review?(smacleod)
Comment on attachment 8675819 [details]
MozReview Request: bundleclone: make test-sni.t work on older Mercurial versions; r?smacleod

https://reviewboard.mozilla.org/r/22443/#review20309
Attachment #8675819 - Flags: review?(smacleod) → review+
Comment on attachment 8675820 [details]
MozReview Request: bundleclone: make tests pass on old Mercurial versions; r?smacleod

https://reviewboard.mozilla.org/r/22445/#review20311
Attachment #8675820 - Flags: review?(smacleod) → review+
Comment on attachment 8675821 [details]
MozReview Request: bundleclone: import streamclone.py from upstream; r?smacleod

https://reviewboard.mozilla.org/r/22447/#review20313
Attachment #8675821 - Flags: review?(smacleod) → review+
Comment on attachment 8675822 [details]
MozReview Request: bundleclone: make compatible with Mercurial 3.6 (bug 1216216); r?smacleod

https://reviewboard.mozilla.org/r/22449/#review20315
Attachment #8675822 - Flags: review?(smacleod) → review+
Comment on attachment 8675823 [details]
MozReview Request: bundleclone: refactor capabilities(); r?smacleod

https://reviewboard.mozilla.org/r/22451/#review20317
Attachment #8675823 - Flags: review?(smacleod) → review+
Comment on attachment 8675824 [details]
MozReview Request: bundleclone: only advertise bundles capability if manifest present; r?smacleod

https://reviewboard.mozilla.org/r/22453/#review20319
Attachment #8675824 - Flags: review?(smacleod) → review+
Attachment #8675951 - Flags: review?(smacleod) → review+
Comment on attachment 8675951 [details]
MozReview Request: bundleclone: vendor clonebundles extension (bug 1216216); r?smacleod

https://reviewboard.mozilla.org/r/22517/#review20321
Comment on attachment 8675952 [details]
MozReview Request: bundleclone: disable bundleclone when clone bundles feature detected (bug 1216216); r?smacleod

https://reviewboard.mozilla.org/r/22519/#review20325
Attachment #8675952 - Flags: review?(smacleod) → review+
Attachment #8675953 - Flags: review?(smacleod) → review+
Comment on attachment 8675953 [details]
MozReview Request: bundleclone: support producing clone bundle compatible stream bundles (bug 1216216); r?smacleod

https://reviewboard.mozilla.org/r/22521/#review20329

::: hgext/bundleclone/tests/test-clonebundles-compat.t:111
(Diff revision 1)
> +  $ hg -R server streambundle stream-legacy.hg
> +  writing stream-legacy.hg
> +  stream bundle file written successully.
> +  include the following in its manifest entry:
> +  stream=revlogv1
> +  $ hg -R server streambundle --type s1 stream-s1.hg
> +  writing 613 bytes for 4 files
> +  bundle requirements: revlogv1
> +
> +  $ cat > server/.hg/clonebundles.manifest << EOF
> +  > http://localhost:$HGPORT1/stream-s1.hg BUNDLESPEC=none-packed1;requirements%3Drevlogv1
> +  > EOF
> +
> +  $ cat > server/.hg/bundleclone.manifest << EOF
> +  > http://localhost:$HGPORT1/stream-legacy.hg stream=revlogv1
> +  > EOF

The ordering is a little strange here, can you stick the manifest creation with their bundle generation, or at least order `legacy-gen, s1-gen, legacy-manifest, s1-manifest`
Comment on attachment 8675954 [details]
MozReview Request: bundleclone: support pulling clonebundles.manifest (bug 1216216); r?smacleod

https://reviewboard.mozilla.org/r/22523/#review20331
Attachment #8675954 - Flags: review?(smacleod) → review+
Comment on attachment 8675955 [details]
MozReview Request: scripts/generate-hg-s3-bundles: refactor to better support abstraction; r?smacleod

https://reviewboard.mozilla.org/r/22525/#review20335

::: scripts/generate-hg-s3-bundles:120
(Diff revision 1)
> -def generate_bundle(repo, typ, temp_path, final_path):
> +def generate_bundle(repo, typ, temp_path, final_path, extra_args):
>      """Generate a single bundle of a type.
>  
>      Create a bundle of type ``typ`` from repo ``repo`` using temporary file
>      ``temp_path`` before atomically moving it to ``final_path``.

`typ` is no longer used, is there a reason you're keeping it as an argument?
Attachment #8675955 - Flags: review?(smacleod) → review+
Comment on attachment 8675956 [details]
MozReview Request: scripts/generate-hg-s3-bundles: create clonebundles.manifest (bug 1216216); r?smacleod

https://reviewboard.mozilla.org/r/22527/#review20337
Attachment #8675956 - Flags: review?(smacleod) → review+
Comment on attachment 8675957 [details]
MozReview Request: ansible: activate clonebundles extension on hg-web and hg-ssh (bug 1216216); r?smacleod

https://reviewboard.mozilla.org/r/22529/#review20339
Attachment #8675957 - Flags: review?(smacleod) → review+
(Assignee)

Comment 33

2 years ago
https://reviewboard.mozilla.org/r/22525/#review20335

> `typ` is no longer used, is there a reason you're keeping it as an argument?

Good catch. Will fix in flight.
(Assignee)

Comment 34

2 years ago
https://hg.mozilla.org/hgcustom/version-control-tools/rev/ca6db63ecd9282bd8720aa21c0a50fd976726386
bundleclone: make compatible with Mercurial 3.6 (bug 1216216); r=smacleod

https://hg.mozilla.org/hgcustom/version-control-tools/rev/145c83c1849464369e35ebd8a5532db73fa8470d
bundleclone: vendor clonebundles extension (bug 1216216); r=smacleod

https://hg.mozilla.org/hgcustom/version-control-tools/rev/e9fd484a51a08751dce8dccc1d94d2d25e5b8bcb
bundleclone: disable bundleclone when clone bundles feature detected (bug 1216216); r=smacleod

https://hg.mozilla.org/hgcustom/version-control-tools/rev/f9a129d2caec42fd4c6ffe78f6ddc87ee4173177
bundleclone: support producing clone bundle compatible stream bundles (bug 1216216); r=smacleod

https://hg.mozilla.org/hgcustom/version-control-tools/rev/8e374504c9ed457d2626b3502d5b8562195827ec
bundleclone: support pulling clonebundles.manifest (bug 1216216); r=smacleod

https://hg.mozilla.org/hgcustom/version-control-tools/rev/b061823cf73323b6ef725be2b60cf9523a7f0731
scripts/generate-hg-s3-bundles: create clonebundles.manifest (bug 1216216); r=smacleod

https://hg.mozilla.org/hgcustom/version-control-tools/rev/c0052bbdd6b85da4a879639ca994438f542c57bd
ansible: activate clonebundles extension on hg-web and hg-ssh (bug 1216216); r=smacleod
(Assignee)

Comment 35

2 years ago
https://hg.mozilla.org/hgcustom/version-control-tools/rev/eda317f4cee4ac2ef6f80a1de0446edddd953c56
scripts/generate-hg-s3-bundles: fix errors in index formatting (bug 1216216)
(Assignee)

Comment 36

2 years ago
https://hg.mozilla.org/hgcustom/version-control-tools/rev/800ccc3eb4356e8c3ccbffa554b962d1d8536fa5
scripts/generate-hg-s3-bundles: add entries to proper manifest (bug 1216216)
(Assignee)

Comment 37

2 years ago
This is deployed, which a few follow-up fixups. I've manually regenerated bundles for mozilla-central and verified things appear to be working. I kicked off a regeneration of everything else. That should complete within the next few hours.
Status: ASSIGNED → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → FIXED
Blocks: 1217147
No longer blocks: 1217147
(Assignee)

Updated

2 years ago
Blocks: 1217155
(Assignee)

Updated

2 years ago
Blocks: 1221268
(Assignee)

Updated

3 months ago
Blocks: 1352494
You need to log in before you can comment on or make changes to this bug.