Closed Bug 687064 Opened 9 years ago Closed 8 years ago

hgtool needs option to clone entire repository even when -r is given

Categories

(Release Engineering :: General, defect, P4)

defect

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: catlee, Assigned: catlee)

References

Details

(Whiteboard: [hg])

Attachments

(1 file, 2 obsolete files)

Right now if you pass in a revision to hgtool, it clones the remote repo with 'hg clone -r <rev> <remote>'. We do this for try since we wanted to avoid cloning the entire try repository on all the slaves.

For mozilla-central, and other more stable branches, I think this turns out to be quite a bit slower than a plain clone of the repository followed by an update to the requested revision

We need to:
* make sure that 'hg clone <remote>; hg update -r <rev>' would be faster than 'hg clone -r <rev> <remote>'

* add an option to hgtool to support this mode so that try can continue to clone by revision.
Assignee: nobody → catlee
This patch adds a few options to hgtool.py:

* --mirror <url>
  adds a mirror url to use for cloning/pulling operations. multiple urls can be specified with multiple --mirror arguments

* --bundle <url>
  adds a bundle url to use for cloning.

* --clone_by_revision
  When enabled, initial clones will be done via 'hg clone -r <rev>' instead of 'hg clone ; hg update -r <rev>'. This defaults to False, which should greatly reduce the load on the hg backends. We need to use this switch for try though.

Bundles are only tried for initial clones. If downloading / unbundling a bundle is successful we'll pull in new changes from the mirrors or master repo before trying to update.

Mirrors are tried in the order specified for pulling / cloning before falling back to the master repository.

I've also added a check for bug 691467 that will clobber the checkout if .hgrc is busted in our non-share codepaths.
Attachment #570716 - Flags: review?(bhearsum)
Comment on attachment 570716 [details] [diff] [review]
handle bundles, mirrors, full clones by default and busted non-shared checkouts

Review of attachment 570716 [details] [diff] [review]:
-----------------------------------------------------------------

::: lib/python/mozilla_buildtools/test/test_util_hg.py
@@ +574,5 @@
> +            self.assertEquals(called, [True])
> +        finally:
> +            hg.unbundle = orig_unbundle
> +
> +    def testCloneWithMirror(self):

It would be good to have a test or two around cloning with multiple mirrors where the first (or all) fail.

::: lib/python/util/hg.py
@@ +475,5 @@
> +def init(dest):
> +    """Initializes an empty repo in `dest`"""
> +    run_cmd(['hg', 'init', dest])
> +
> +def unbundle(bundle_url, dest):

Nit: 'unbundle' handles local files, too. This would be better named just plain 'bundle' or similar.

@@ +479,5 @@
> +def unbundle(bundle_url, dest):
> +    """Unbundles the bundle at `bundle_url` into `dest`."""
> +    run_cmd(['hg', 'unbundle', bundle_url], cwd=dest)
> +
> +def adjust_paths(dest, **paths):

.hgrc is an .ini file, why not use ConfigParser for this?
Comment on attachment 570783 [details] [diff] [review]
handle bundles, mirrors, full clones by default and busted non-shared checkouts

Thanks for the interdiff!
Attachment #570783 - Flags: review?(bhearsum) → review+
Same as before, with this addition: 
https://github.com/catlee/tools/commit/524bf3ef0728a1edc27fa763e38846b50dbb5dd0

Fixes pulling by revision into the shared directory.

Also fixes bug 699028.
Attachment #570783 - Attachment is obsolete: true
Attachment #571647 - Flags: review?(bhearsum)
Depends on: 699407
Attachment #571647 - Flags: review?(bhearsum) → review+
Attachment #571647 - Flags: checked-in+
Status: NEW → RESOLVED
Closed: 8 years ago
Resolution: --- → FIXED
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.