1607915 - switch to version 2 stream clone bundles

Status: RESOLVED FIXED

(Developer Services :: Mercurial: hg.mozilla.org, enhancement)

Description

[Connor Sheehan [:sheehan]]

Currently we use the version 1 stream clone bundles on hg.mozilla.org. These bundles are generated using a special command, hg debugcreatestreamclonebundle, and are unable to be examined using hg debugbundle.

A new version of the stream clone bundle (version 2) is available. These bundles are created in the same way as other bundles (using hg bundle -t <type>, where type is "none-v2;stream=v2") and can be examined using hg debugbundle. We should upgrade to the latest stream clone bundle format.

On top of this convenience change, there is also the issue of bug 1604652. Although I don't have conclusive evidence, it seems Mercurial 5.2 running on Python 3 can fail for certain repositories when cloning using v1 stream clone bundles. I say "certain repositories" as the upstream Mercurial tests seem to pass with their generated bundle, but all our "real-world" repos (m-c, m-u, v-c-t, etc) fail when stream-cloning with Py3 hg.

I tested the above switch by stream-cloning a repository using an older Mercurial (4.5, to avoid upgrading to sparserevlog), switching to a Py3 hg5.2, generating a v2 stream clone bundle with that Mercurial version, and performing a stream clone using this new bundle. It works on Py3 hg5.2, as well as Py2 hg4.5, effectively fixing the problem reported in bug 1604652.

Comment 1

[Connor Sheehan [:sheehan]]

Attachment: hgmo: teach hgweb to parse and prioritize v2 stream clone bundles (Bug 1607915) r?glob,mhentges,zeid

This commit teaches hgweb about v2 stream clone bundles and how to prioritize
them. With this patch, v2 bundles will be preferred over v1 bundles, and
v1 bundles over all other bundle types, when filtering manifests for stream
bundles (ie when cloning from AWS, GCP, etc). This means we can safely start
generating v2 bundles, and roll back to a manifest with v1 bundles in case we
hit failures with the new bundle type.

Since all hgweb machines now run Python 3, we also take this time to convert
from a cmp style function to a key function directly.

Comment 2

[Connor Sheehan [:sheehan]]

Attachment: bundles: generate v2 stream clone bundles (Bug 1607915) r?glob,mhentges,zeid

This commit updates the clonebundle generation script to create v2 stream
clone bundles instead of a v1 bundle. The bundle generation command is
updated from hg debugcreatestreamclonebundle to hg bundle with arguments
for the bundle type, bringing it in line with other bundles. The bundlespec
is updated to reflect the format of new bundles as they would be created on
hgssh.

In the event this change causes unforeseen bustages in CI or elsewhere,
we can safely roll it back and revert clonebundle manifest files to the
previous day's version.

Depends on D59236

Comment 3

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/hgcustom/version-control-tools/rev/ae8ec5436413
hgmo: teach hgweb to parse and prioritize v2 stream clone bundles r=glob,mhentges
https://hg.mozilla.org/hgcustom/version-control-tools/rev/770bb7bb2e9e
bundles: generate v2 stream clone bundles r=glob,mhentges

Comment 4

[Connor Sheehan [:sheehan]]

This is being deployed, and we'll need to wait until at least tomorrow before a new set of bundles is created.

Comment 5

[Mike Hommey [:glandium]]

it seems Mercurial 5.2 running on Python 3 can fail for certain repositories when cloning using v1 stream clone bundles.

Did you report upstream?

Comment 6

[Connor Sheehan [:sheehan]]

(In reply to Mike Hommey [:glandium] from comment #5)

it seems Mercurial 5.2 running on Python 3 can fail for certain repositories when cloning using v1 stream clone bundles.

Did you report upstream?

I don't think I ever reported it, I'll mention it in our next meeting.