Enable sparse-revlog on Mercurial repositories
Categories
(Developer Services :: Mercurial: hg.mozilla.org, enhancement)
Tracking
(Not tracked)
People
(Reporter: glandium, Unassigned)
Details
As seen in https://glandium.org/blog/?p=3913, currently, cloning mozilla-unified (or mozilla-central, for that matter) takes an awful long time (except when doing a streaming clone).
One of the reasons is that the clonebundle is suboptimal wrt delta chains, and sparse-revlog (a feature new to mercurial 4.7) improves things.
The hg --config format.sparse-revlog=yes debugupgraderepo --run command should work to convert a repository.
If I convert a mozilla-unified clone, and create a new bundle from it, unbundling that bundle takes 10 minutes instead of the 20 minutes it takes to unbundle the last mozilla-unified zstd bundle from hg.cdn.mozilla.net.
Strictly speaking, this is better to apply on all repositories, and shouldn't affect their clonability with older versions of mercurial. Ideally, the web heads would be updated too. But the most important part is applying this to the repositories that are used to create the clonebundles.
|
||
Comment 1•5 years ago
|
||
(In reply to Mike Hommey [:glandium] from comment #0)
As seen in https://glandium.org/blog/?p=3913, currently, cloning mozilla-unified (or mozilla-central, for that matter) takes an awful long time (except when doing a streaming clone).
One of the reasons is that the clonebundle is suboptimal wrt delta chains, and sparse-revlog (a feature new to mercurial 4.7) improves things.
The
hg --config format.sparse-revlog=yes debugupgraderepo --runcommand should work to convert a repository.If I convert a mozilla-unified clone, and create a new bundle from it, unbundling that bundle takes 10 minutes instead of the 20 minutes it takes to unbundle the last mozilla-unified zstd bundle from hg.cdn.mozilla.net.
Interesting, that's a much larger improvement than I had previously seen for sparse-revlog.
Strictly speaking, this is better to apply on all repositories, and shouldn't affect their clonability with older versions of mercurial. Ideally, the web heads would be updated too. But the most important part is applying this to the repositories that are used to create the clonebundles.
As I understand it, this is not true for stream-clone bundles, since they essentially send the raw revlog data over the wire with no extra processing. From the output of hg help clonebundles:
'hg debugcreatestreamclonebundle' can be used to produce a special streaming
clonebundle. These are bundle files that are extremely efficient to produce
and consume (read: fast). However, they are larger than traditional bundle
formats and require that clients support the exact set of repository data
store formats in use by the repository that created them. Typically, a newer
server can serve data that is compatible with older clients. However,
streaming clone bundles don't have this guarantee. Server operators need
to be aware that newer versions of Mercurial may produce streaming clone
bundles incompatible with older Mercurial versions.
This is the main blocker for us, since we don't have a consistent version of hg in use in CI. We should be on 4.8 almost everywhere, however I still see older versions in Taskcluster logs from time to time. IIRC there are some versions of hg baked in to Docker images, where upgrading the Docker image is undesirable. Decision tasks come to mind as one of the main cases of this, the last time I looked into it.
With this new timing info it may be easier to justify chasing down those last few upgrades, or just doing the upgrade and letting things break to flush them out.
Anyways, this was reported a while back in bug 1513276, so lets track it over there.
Description
•