1352494 - Remove bundleclone from hg.mozilla.org

Status: RESOLVED FIXED

(Developer Services :: Mercurial: hg.mozilla.org, enhancement)

Description

[Gregory Szorc [:gps]]

Mercurial 3.6 integrated the bundleclone feature as a core feature (clonebundles). Since bug 1216216, we've been supporting both the legacy bundleclone feature and the modern clonebundles.

I just looked at server logs and we're seeing maybe 20 requests per day for the bundleclone manifest. Contrast with 20,000+ per day for clonebundles.

I think it is time we remove bundleclone support from the server.

Comment 1

[Gregory Szorc [:gps]]

Attachment: hgmo: delay import mozbuildinfo module;

Without this change, an upcoming commit that removes the bundleclone
extension from hgweb breaks the hgmo extension.

After much head scratching, I tracked this down to Mercurial's demand
importer and a bug in bundleclone.

The bundleclone extension uses the legacy, non-context-manager API
for disabling the demand importer when performing a module presence
test. Instead of restoring the previous state of the demand importer,
after this operation, it always calls "demandimport.enable()" thus
ensuring it is enabled.

When the bundleclone extension is removed, the hgmo extension is
loaded with the demand importer disabled. (We stopped enabling the
demand importer in WSGI processes in 287fd5985aa2.) This caused the
mozbuildinfo module import to fail because it attempts to import
mozbuild and mozpack modules which don't exist in the hgweb
virtualenv. This caused the hgmo module load to fail and hgweb
to effectively break.

The fix for this on hgmo's side is to delay import the module
containing unknown modules until they are actually needed. A
fix for bundleclone won't be pursued because the extension is
effectively deprecated and will be removed from use shortly.

Review commit: https://reviewboard.mozilla.org/r/125596/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/125596/

Comment 2

[Gregory Szorc [:gps]]

Attachment: ansible/hg-ssh: stop generate legacy stream bundles (bug 1352494);

These are for bundleclone, which we are going to get rid of.

Review commit: https://reviewboard.mozilla.org/r/125598/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/125598/

Comment 3

[Gregory Szorc [:gps]]

Attachment: generate-hg-s3-bundles: stop supporting legacy stream bundles (bug 1352494);

We just stopped generating them in the manifest file. So let's stop
supporting their generation.

Review commit: https://reviewboard.mozilla.org/r/125600/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/125600/

Comment 4

[Gregory Szorc [:gps]]

Attachment: generate-hg-s3-bundles: stop generating bundleclone manifests (bug 1352494);

We're only getting ~20 requests a day for bundleclone manifests
(compared to ~20,000 requests a day for clonebundles manifests).
The legacy bundleclone feature has served its use. It is time to turn
it off.

This commit removes all references to bundleclone in the bundle
generation script. Tests have been updated to no longer query the
cmd=bundles endpoint, which serves the bundleclone manifest.

Review commit: https://reviewboard.mozilla.org/r/125602/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/125602/

Comment 5

[Gregory Szorc [:gps]]

Attachment: hgmo: stop supporting bundleclone for IP address filtering (bug 1352494);

We have custom code in the hgmo extension for modifying the behavior
of clonebundles and bundleclone manifest serving to filter and
re-prioritize entries depending on the source IP address.

Now that we don't care about bundleclone manifests, we can remove
support for bundleclone from this extension.

It isn't obvious from the patch, but the "stream=" syntax was only
used by bundleclone. clonebundles uses that BUNDLESPEC syntax for
similar functionality.

Review commit: https://reviewboard.mozilla.org/r/125604/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/125604/

Comment 6

[Gregory Szorc [:gps]]

Attachment: hgmo: support fetching clonebundles.manifest during pull (bug 1352494);

Currently, the bundleclone extension fetches the remote's clone bundles
manifest during pull operations and saves it locally. Since we'll be
removing the bundleclone extension, we need to implement this logic
elsewhere.

The hgmo extension is where we are putting our server-side
customizations for hg.mo. So it seems a logical place to put this
functionality.

This code is pretty much a copy of what's in the bundleclone
extension.

An explicit test has been added for good measure.

Review commit: https://reviewboard.mozilla.org/r/125606/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/125606/

Comment 7

[Gregory Szorc [:gps]]

Attachment: ansible/hg-web: stop loading bundleclone extension (bug 1352494);

We no longer generate bundles or manifests for it and all code
references to it have gone away. So let's stop loading it on the
HTTP servers.

Because the extension was syncing the clonebundles.manifest file
during pull operations, we've activated the identical feature in
the hgmo extension to take over this functionality.

Note: generate-hg-s3-bundles still calls a command provided by this
extension, so we can't remove it from hgssh just yet.

Review commit: https://reviewboard.mozilla.org/r/125608/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/125608/

Comment 8

[Gregory Szorc [:gps]]

Attachment: generate-hg-s3-bundles: use core command for producing stream clone bundles (bug 1352494);

The current behavior of generate-hg-s3-bundles is to call a command
in the bundleclone extension for producing a modern stream clone
bundle. If my recollection is correct, we implemented this functionality
in bundleclone to allow us to produce stream clone bundles before
deploying a new version of Mercurial that had built-in support for
this functionality.

Now that we have a modern version of Mercurial deployed, we don't need
to call the custom command implementation. So, this commit switches us
to the implementation in core Mercurial.

Review commit: https://reviewboard.mozilla.org/r/125610/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/125610/

Comment 9

[Gregory Szorc [:gps]]

Attachment: ansible/hg-ssh: stop loading bundleclone extension (bug 1352494);

We no longer have any references to this extension on hgssh. Let's stop
loading it.

Review commit: https://reviewboard.mozilla.org/r/125612/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/125612/

Comment 10

[Gregory Szorc [:gps]]

Attachment: bundleclone: remove extension (bug 1352494);

The bundleclone extension was a pioneer in the world of cloning from
pre-generated bundles. However, this functionality has been incorporated
into Mercurial 3.6+ and nearly all consumers of bundle-based clones now
use the built-in clonebundles feature. We recently removed support
for bundleclone on hg.mozilla.org, which should have been the only
server using this functionality. It makes little sense to continue
supporting the bundleclone extension for either server or client use.

So, this commit deletes the now-unused bundleclone extension.

After this commit, the only references to "bundleclone" in the
repository are in the documentation: the page we use to describe
cloning from bundles is named "bundleclone." There are no references
to the extension remaining.

Review commit: https://reviewboard.mozilla.org/r/125614/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/125614/

Comment 11

[:glob ✱]

Comment on attachment 8853544 [details]
hgmo: delay import mozbuildinfo module;

https://reviewboard.mozilla.org/r/125596/#review128532

Comment 12

[:glob ✱]

Comment on attachment 8853545 [details]
ansible/hg-ssh: stop generate legacy stream bundles (bug 1352494);

https://reviewboard.mozilla.org/r/125598/#review128552

Comment 13

[:glob ✱]

Comment on attachment 8853546 [details]
generate-hg-s3-bundles: stop supporting legacy stream bundles (bug 1352494);

https://reviewboard.mozilla.org/r/125600/#review128554

Comment 14

[:glob ✱]

Comment on attachment 8853547 [details]
generate-hg-s3-bundles: stop generating bundleclone manifests (bug 1352494);

https://reviewboard.mozilla.org/r/125602/#review128572

::: hgserver/tests/test-clonebundles.t
(Diff revision 1)
>    adding changesets
>    adding manifests
>    adding file changes
>    added 1 changesets with 1 changes to 1 files
>  
> -#else

can probably drop the #if hg36+ checks too

Comment 15

[Gregory Szorc [:gps]]

Comment on attachment 8853547 [details]
generate-hg-s3-bundles: stop generating bundleclone manifests (bug 1352494);

https://reviewboard.mozilla.org/r/125602/#review128572

> can probably drop the #if hg36+ checks too

Yeah, I was thinking about doing that globally in the "upgrade to 4.1" series. I didn't want to scope bloat this series any more.

Comment 16

[:glob ✱]

Comment on attachment 8853548 [details]
hgmo: stop supporting bundleclone for IP address filtering (bug 1352494);

https://reviewboard.mozilla.org/r/125604/#review128610

Comment 17

[:glob ✱]

Comment on attachment 8853549 [details]
hgmo: support fetching clonebundles.manifest during pull (bug 1352494);

https://reviewboard.mozilla.org/r/125606/#review128612

Comment 18

[:glob ✱]

Comment on attachment 8853550 [details]
ansible/hg-web: stop loading bundleclone extension (bug 1352494);

https://reviewboard.mozilla.org/r/125608/#review128614

Comment 19

[:glob ✱]

Comment on attachment 8853551 [details]
generate-hg-s3-bundles: use core command for producing stream clone bundles (bug 1352494);

https://reviewboard.mozilla.org/r/125610/#review128618

Comment 20

[:glob ✱]

Comment on attachment 8853552 [details]
ansible/hg-ssh: stop loading bundleclone extension (bug 1352494);

https://reviewboard.mozilla.org/r/125612/#review128934

Comment 21

[:glob ✱]

Comment on attachment 8853553 [details]
bundleclone: remove extension (bug 1352494);

https://reviewboard.mozilla.org/r/125614/#review128940

Comment 22

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/hgcustom/version-control-tools/rev/fc5f2907f112
ansible/hg-ssh: stop generate legacy stream bundles ; r=glob
https://hg.mozilla.org/hgcustom/version-control-tools/rev/7e5d34570bdd
generate-hg-s3-bundles: stop supporting legacy stream bundles ; r=glob
https://hg.mozilla.org/hgcustom/version-control-tools/rev/57947e522a92
generate-hg-s3-bundles: stop generating bundleclone manifests ; r=glob
https://hg.mozilla.org/hgcustom/version-control-tools/rev/ccdd561cf624
hgmo: stop supporting bundleclone for IP address filtering ; r=glob
https://hg.mozilla.org/hgcustom/version-control-tools/rev/ceb9b26fb5ee
hgmo: support fetching clonebundles.manifest during pull ; r=glob
https://hg.mozilla.org/hgcustom/version-control-tools/rev/0ffb4d71b9d0
ansible/hg-web: stop loading bundleclone extension ; r=glob
https://hg.mozilla.org/hgcustom/version-control-tools/rev/07543fc9ebbc
generate-hg-s3-bundles: use core command for producing stream clone bundles ; r=glob
https://hg.mozilla.org/hgcustom/version-control-tools/rev/c71c0cfdd31e
ansible/hg-ssh: stop loading bundleclone extension ; r=glob
https://hg.mozilla.org/hgcustom/version-control-tools/rev/c805a3fc11ca
bundleclone: remove extension ; r=glob

Comment 23

[Mozilla Autoland]

We're sorry, Autoland could not rebase your commits for you automatically. Please manually rebase your commits and try again.

hg error in cmd: hg rebase -s 34ab846f4c6a -d b895763b7fe7: rebasing 5213:34ab846f4c6a "hgmo: delay import mozbuildinfo module; r=glob"
merging hgext/hgmo/__init__.py
note: rebase of 5213:34ab846f4c6a created no changes to commit
rebasing 5214:a49dd1921980 "ansible/hg-ssh: stop generate legacy stream bundles (bug 1352494); r=glob"
note: rebase of 5214:a49dd1921980 created no changes to commit
rebasing 5215:81d8578a372c "generate-hg-s3-bundles: stop supporting legacy stream bundles (bug 1352494); r=glob"
merging hgserver/tests/test-clonebundles.t
merging scripts/generate-hg-s3-bundles
warning: conflicts while merging scripts/generate-hg-s3-bundles! (edit, then use 'hg resolve --mark')
unresolved conflicts (see hg resolve, then hg rebase --continue)