Closed
Bug 1216216
Opened 9 years ago
Closed 9 years ago
Make bundleclone compatible with Mercurial 3.6
Categories
(Developer Services :: Mercurial: bundleclone, defect)
Developer Services
Mercurial: bundleclone
Tracking
(Not tracked)
RESOLVED
FIXED
People
(Reporter: gps, Assigned: gps)
References
Details
Attachments
(13 files)
40 bytes,
text/x-review-board-request
|
smacleod
:
review+
|
Details |
40 bytes,
text/x-review-board-request
|
smacleod
:
review+
|
Details |
40 bytes,
text/x-review-board-request
|
smacleod
:
review+
|
Details |
40 bytes,
text/x-review-board-request
|
smacleod
:
review+
|
Details |
40 bytes,
text/x-review-board-request
|
smacleod
:
review+
|
Details |
40 bytes,
text/x-review-board-request
|
smacleod
:
review+
|
Details |
40 bytes,
text/x-review-board-request
|
smacleod
:
review+
|
Details |
40 bytes,
text/x-review-board-request
|
smacleod
:
review+
|
Details |
40 bytes,
text/x-review-board-request
|
smacleod
:
review+
|
Details |
40 bytes,
text/x-review-board-request
|
smacleod
:
review+
|
Details |
40 bytes,
text/x-review-board-request
|
smacleod
:
review+
|
Details |
40 bytes,
text/x-review-board-request
|
smacleod
:
review+
|
Details |
40 bytes,
text/x-review-board-request
|
smacleod
:
review+
|
Details |
bundleclone's functionality has been added to Mercurial 3.6. We need to: 1) make the bundleclone extension compatible with code refactorings in Mercurial 3.6 2) make the bundleclone extension essentially no-op when the server supports the official Mercurial clone bundles feature 3) make hg.mozilla.org support the new 3.6 clone bundles feature via the bundleclone extension before the server is upgraded to 3.6 (so 3.6 clients can use the built-in clone bundles feature before we upgrade the server) This work should be performed ASAP so we can start testing the 3.6 RC at Mozilla and against hg.mozilla.org
Assignee | ||
Comment 1•9 years ago
|
||
bundleclone: make test-sni.t work on older Mercurial versions; r?smacleod These tests were failing when executed on older Mercurial versions because they assumed we were using a modern Mercurial. bundleclone (unlike every other extension in version-control-tools) supports ancient versions of Mercurial. So make the tests pass on old versions.
Attachment #8675819 -
Flags: review?(smacleod)
Assignee | ||
Comment 2•9 years ago
|
||
bundleclone: make tests pass on old Mercurial versions; r?smacleod The tests for bundle manifest copying were failing on Mercurial releases older than 3.0. We don't care about this functionality on these old versions. So just don't run the tests there. Since we added the first consumer of hghave, we needed to install a stub file in the test directory to make the test harness happy.
Attachment #8675820 -
Flags: review?(smacleod)
Assignee | ||
Comment 3•9 years ago
|
||
bundleclone: import streamclone.py from upstream; r?smacleod We replace our copies of upstream code with the equivalent code from Mercurial 3.6's streamclone.py. The majority of this change should be a rubber stamp. There are a few changes in the extension code to support renamed and refactored APIs.
Attachment #8675821 -
Flags: review?(smacleod)
Assignee | ||
Comment 4•9 years ago
|
||
bundleclone: make compatible with Mercurial 3.6 (bug 1216216); r?smacleod Mercurial 3.6 changed the API for changegroup application slightly. We detect support for it. With this change, tests pass on all versions of Mercurial from 2.5 to 3.6.
Attachment #8675822 -
Flags: review?(smacleod)
Assignee | ||
Comment 5•9 years ago
|
||
bundleclone: refactor capabilities(); r?smacleod This will make future refactoring slightly simpler.
Attachment #8675823 -
Flags: review?(smacleod)
Assignee | ||
Comment 6•9 years ago
|
||
bundleclone: only advertise bundles capability if manifest present; r?smacleod This brings the behavior in-line with the feature in core. It also prevents an extra round trip from client to server when cloning from a repository that doesn't have a bundleclone.manifest file.
Attachment #8675824 -
Flags: review?(smacleod)
Assignee | ||
Comment 7•9 years ago
|
||
Comment on attachment 8675819 [details]
MozReview Request: bundleclone: make test-sni.t work on older Mercurial versions; r?smacleod
bundleclone: make test-sni.t work on older Mercurial versions; r?smacleod
These tests were failing when executed on older Mercurial versions
because they assumed we were using a modern Mercurial. bundleclone
(unlike every other extension in version-control-tools) supports ancient
versions of Mercurial. So make the tests pass on old versions.
Assignee | ||
Comment 8•9 years ago
|
||
Comment on attachment 8675820 [details]
MozReview Request: bundleclone: make tests pass on old Mercurial versions; r?smacleod
bundleclone: make tests pass on old Mercurial versions; r?smacleod
The tests for bundle manifest copying were failing on Mercurial releases
older than 3.0. We don't care about this functionality on these old
versions. So just don't run the tests there.
Since we added the first consumer of hghave, we needed to install a stub
file in the test directory to make the test harness happy.
Assignee | ||
Comment 9•9 years ago
|
||
Comment on attachment 8675821 [details]
MozReview Request: bundleclone: import streamclone.py from upstream; r?smacleod
bundleclone: import streamclone.py from upstream; r?smacleod
We replace our copies of upstream code with the equivalent code from
Mercurial 3.6's streamclone.py. The majority of this change should be a
rubber stamp. There are a few changes in the extension code to support
renamed and refactored APIs.
Assignee | ||
Comment 10•9 years ago
|
||
Comment on attachment 8675822 [details] MozReview Request: bundleclone: make compatible with Mercurial 3.6 (bug 1216216); r?smacleod bundleclone: make compatible with Mercurial 3.6 (bug 1216216); r?smacleod Mercurial 3.6 changed the API for changegroup application slightly. We detect support for it. With this change, tests pass on all versions of Mercurial from 2.5 to 3.6.
Assignee | ||
Comment 11•9 years ago
|
||
Comment on attachment 8675823 [details]
MozReview Request: bundleclone: refactor capabilities(); r?smacleod
bundleclone: refactor capabilities(); r?smacleod
This will make future refactoring slightly simpler.
Assignee | ||
Comment 12•9 years ago
|
||
Comment on attachment 8675824 [details]
MozReview Request: bundleclone: only advertise bundles capability if manifest present; r?smacleod
bundleclone: only advertise bundles capability if manifest present; r?smacleod
This brings the behavior in-line with the feature in core. It also
prevents an extra round trip from client to server when cloning from a
repository that doesn't have a bundleclone.manifest file.
Assignee | ||
Comment 13•9 years ago
|
||
bundleclone: vendor clonebundles extension (bug 1216216); r?smacleod We import hgext/clonebundles.py unmodified from changeset df1a29ec45bfaee30f974b06a0214a3359b369aa of Mercurial so we can run it before we deploy Mercurial 3.6 to hg.mozilla.org.
Attachment #8675951 -
Flags: review?(smacleod)
Assignee | ||
Comment 14•9 years ago
|
||
bundleclone: disable bundleclone when clone bundles feature detected (bug 1216216); r?smacleod Mercurial 3.6 has official support for the bundle clone feature set. The bundleclone extension is thus redundant with that functionality (or at least it should if the server operator is competent). This commit adds detection of the clone bundles feature to the client and will short-circuit the bundleclone functionality when the built-in support for cloning from pre-generated bundles is sufficient.
Attachment #8675952 -
Flags: review?(smacleod)
Assignee | ||
Comment 15•9 years ago
|
||
bundleclone: support producing clone bundle compatible stream bundles (bug 1216216); r?smacleod When bundleclone was initially implemented, the concept of a "stream bundle" was invented. It was essentially the wire protocol representation of the existing streaming clone data saved to a file. Mercurial 3.6 formalized the "stream bundle" format. It is slightly different from and incompatible with the wire protocol format. This patch teaches our `hg streambundle` command to produce the new, formal format. We also add tests verifying a client requests the appropriate file depending on whether clone bundles or bundleclone is used.
Attachment #8675953 -
Flags: review?(smacleod)
Assignee | ||
Comment 16•9 years ago
|
||
bundleclone: support pulling clonebundles.manifest (bug 1216216); r?smacleod We currently have support for `hg pull` copying the bundleclone.manifest file. This is used for mirroring on hg.mozilla.org. The core support for clone bundles doesn't (yet) support such a mirroring ability. So, we implement support for clonebundles.manifest mirroring.
Attachment #8675954 -
Flags: review?(smacleod)
Assignee | ||
Comment 17•9 years ago
|
||
scripts/generate-hg-s3-bundles: refactor to better support abstraction; r?smacleod There was inline code giving stream bundles special treatment. In addition, we were mixing configuration for bundle generation and manifest formatting into the same object. We extract bundle generation and manifest settings into separate objects. The bundle generation object now defines the ``hg`` arguments to produce the bundle. The manifest object now defines the parameters for that manifest.
Attachment #8675955 -
Flags: review?(smacleod)
Assignee | ||
Comment 18•9 years ago
|
||
scripts/generate-hg-s3-bundles: create clonebundles.manifest (bug 1216216); r?smacleod The built-in Mercurial feature for serving pre-built bundles is slightly different from bundleclone. It uses a separate manifest that has a slightly different format (mostly cosmetic changes). With this commit, we start writing this new manifest file when generating bundles. As part of this, we generate the modern stream bundle file format.
Attachment #8675956 -
Flags: review?(smacleod)
Assignee | ||
Comment 19•9 years ago
|
||
ansible: activate clonebundles extension on hg-web and hg-ssh (bug 1216216); r?smacleod We are now generating clonebundles.manifest files. Activate our vendored copy of the clonebundles extension from the Mercurial project so 3.6 clients can fetch bundles using the clone bundles mechanism built into core Mercurial.
Attachment #8675957 -
Flags: review?(smacleod)
Comment 20•9 years ago
|
||
Comment on attachment 8675819 [details] MozReview Request: bundleclone: make test-sni.t work on older Mercurial versions; r?smacleod https://reviewboard.mozilla.org/r/22443/#review20309
Attachment #8675819 -
Flags: review?(smacleod) → review+
Comment 21•9 years ago
|
||
Comment on attachment 8675820 [details] MozReview Request: bundleclone: make tests pass on old Mercurial versions; r?smacleod https://reviewboard.mozilla.org/r/22445/#review20311
Attachment #8675820 -
Flags: review?(smacleod) → review+
Comment 22•9 years ago
|
||
Comment on attachment 8675821 [details] MozReview Request: bundleclone: import streamclone.py from upstream; r?smacleod https://reviewboard.mozilla.org/r/22447/#review20313
Attachment #8675821 -
Flags: review?(smacleod) → review+
Comment 23•9 years ago
|
||
Comment on attachment 8675822 [details] MozReview Request: bundleclone: make compatible with Mercurial 3.6 (bug 1216216); r?smacleod https://reviewboard.mozilla.org/r/22449/#review20315
Attachment #8675822 -
Flags: review?(smacleod) → review+
Comment 24•9 years ago
|
||
Comment on attachment 8675823 [details] MozReview Request: bundleclone: refactor capabilities(); r?smacleod https://reviewboard.mozilla.org/r/22451/#review20317
Attachment #8675823 -
Flags: review?(smacleod) → review+
Comment 25•9 years ago
|
||
Comment on attachment 8675824 [details] MozReview Request: bundleclone: only advertise bundles capability if manifest present; r?smacleod https://reviewboard.mozilla.org/r/22453/#review20319
Attachment #8675824 -
Flags: review?(smacleod) → review+
Updated•9 years ago
|
Attachment #8675951 -
Flags: review?(smacleod) → review+
Comment 26•9 years ago
|
||
Comment on attachment 8675951 [details] MozReview Request: bundleclone: vendor clonebundles extension (bug 1216216); r?smacleod https://reviewboard.mozilla.org/r/22517/#review20321
Comment 27•9 years ago
|
||
Comment on attachment 8675952 [details] MozReview Request: bundleclone: disable bundleclone when clone bundles feature detected (bug 1216216); r?smacleod https://reviewboard.mozilla.org/r/22519/#review20325
Attachment #8675952 -
Flags: review?(smacleod) → review+
Updated•9 years ago
|
Attachment #8675953 -
Flags: review?(smacleod) → review+
Comment 28•9 years ago
|
||
Comment on attachment 8675953 [details] MozReview Request: bundleclone: support producing clone bundle compatible stream bundles (bug 1216216); r?smacleod https://reviewboard.mozilla.org/r/22521/#review20329 ::: hgext/bundleclone/tests/test-clonebundles-compat.t:111 (Diff revision 1) > + $ hg -R server streambundle stream-legacy.hg > + writing stream-legacy.hg > + stream bundle file written successully. > + include the following in its manifest entry: > + stream=revlogv1 > + $ hg -R server streambundle --type s1 stream-s1.hg > + writing 613 bytes for 4 files > + bundle requirements: revlogv1 > + > + $ cat > server/.hg/clonebundles.manifest << EOF > + > http://localhost:$HGPORT1/stream-s1.hg BUNDLESPEC=none-packed1;requirements%3Drevlogv1 > + > EOF > + > + $ cat > server/.hg/bundleclone.manifest << EOF > + > http://localhost:$HGPORT1/stream-legacy.hg stream=revlogv1 > + > EOF The ordering is a little strange here, can you stick the manifest creation with their bundle generation, or at least order `legacy-gen, s1-gen, legacy-manifest, s1-manifest`
Comment 29•9 years ago
|
||
Comment on attachment 8675954 [details] MozReview Request: bundleclone: support pulling clonebundles.manifest (bug 1216216); r?smacleod https://reviewboard.mozilla.org/r/22523/#review20331
Attachment #8675954 -
Flags: review?(smacleod) → review+
Comment 30•9 years ago
|
||
Comment on attachment 8675955 [details] MozReview Request: scripts/generate-hg-s3-bundles: refactor to better support abstraction; r?smacleod https://reviewboard.mozilla.org/r/22525/#review20335 ::: scripts/generate-hg-s3-bundles:120 (Diff revision 1) > -def generate_bundle(repo, typ, temp_path, final_path): > +def generate_bundle(repo, typ, temp_path, final_path, extra_args): > """Generate a single bundle of a type. > > Create a bundle of type ``typ`` from repo ``repo`` using temporary file > ``temp_path`` before atomically moving it to ``final_path``. `typ` is no longer used, is there a reason you're keeping it as an argument?
Attachment #8675955 -
Flags: review?(smacleod) → review+
Comment 31•9 years ago
|
||
Comment on attachment 8675956 [details] MozReview Request: scripts/generate-hg-s3-bundles: create clonebundles.manifest (bug 1216216); r?smacleod https://reviewboard.mozilla.org/r/22527/#review20337
Attachment #8675956 -
Flags: review?(smacleod) → review+
Comment 32•9 years ago
|
||
Comment on attachment 8675957 [details] MozReview Request: ansible: activate clonebundles extension on hg-web and hg-ssh (bug 1216216); r?smacleod https://reviewboard.mozilla.org/r/22529/#review20339
Attachment #8675957 -
Flags: review?(smacleod) → review+
Assignee | ||
Comment 33•9 years ago
|
||
https://reviewboard.mozilla.org/r/22525/#review20335 > `typ` is no longer used, is there a reason you're keeping it as an argument? Good catch. Will fix in flight.
Assignee | ||
Comment 34•9 years ago
|
||
https://hg.mozilla.org/hgcustom/version-control-tools/rev/ca6db63ecd9282bd8720aa21c0a50fd976726386 bundleclone: make compatible with Mercurial 3.6 (bug 1216216); r=smacleod https://hg.mozilla.org/hgcustom/version-control-tools/rev/145c83c1849464369e35ebd8a5532db73fa8470d bundleclone: vendor clonebundles extension (bug 1216216); r=smacleod https://hg.mozilla.org/hgcustom/version-control-tools/rev/e9fd484a51a08751dce8dccc1d94d2d25e5b8bcb bundleclone: disable bundleclone when clone bundles feature detected (bug 1216216); r=smacleod https://hg.mozilla.org/hgcustom/version-control-tools/rev/f9a129d2caec42fd4c6ffe78f6ddc87ee4173177 bundleclone: support producing clone bundle compatible stream bundles (bug 1216216); r=smacleod https://hg.mozilla.org/hgcustom/version-control-tools/rev/8e374504c9ed457d2626b3502d5b8562195827ec bundleclone: support pulling clonebundles.manifest (bug 1216216); r=smacleod https://hg.mozilla.org/hgcustom/version-control-tools/rev/b061823cf73323b6ef725be2b60cf9523a7f0731 scripts/generate-hg-s3-bundles: create clonebundles.manifest (bug 1216216); r=smacleod https://hg.mozilla.org/hgcustom/version-control-tools/rev/c0052bbdd6b85da4a879639ca994438f542c57bd ansible: activate clonebundles extension on hg-web and hg-ssh (bug 1216216); r=smacleod
Assignee | ||
Comment 35•9 years ago
|
||
https://hg.mozilla.org/hgcustom/version-control-tools/rev/eda317f4cee4ac2ef6f80a1de0446edddd953c56 scripts/generate-hg-s3-bundles: fix errors in index formatting (bug 1216216)
Assignee | ||
Comment 36•9 years ago
|
||
https://hg.mozilla.org/hgcustom/version-control-tools/rev/800ccc3eb4356e8c3ccbffa554b962d1d8536fa5 scripts/generate-hg-s3-bundles: add entries to proper manifest (bug 1216216)
Assignee | ||
Comment 37•9 years ago
|
||
This is deployed, which a few follow-up fixups. I've manually regenerated bundles for mozilla-central and verified things appear to be working. I kicked off a regeneration of everything else. That should complete within the next few hours.
Status: ASSIGNED → RESOLVED
Closed: 9 years ago
Resolution: --- → FIXED
You need to log in
before you can comment on or make changes to this bug.
Description
•