687064 - hgtool needs option to clone entire repository even when -r is given

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P4)

Description

[Chris AtLee [:catlee]]

Right now if you pass in a revision to hgtool, it clones the remote repo with 'hg clone -r <rev> <remote>'. We do this for try since we wanted to avoid cloning the entire try repository on all the slaves.

For mozilla-central, and other more stable branches, I think this turns out to be quite a bit slower than a plain clone of the repository followed by an update to the requested revision

We need to:
* make sure that 'hg clone <remote>; hg update -r <rev>' would be faster than 'hg clone -r <rev> <remote>'

* add an option to hgtool to support this mode so that try can continue to clone by revision.

Comment 1

[Chris AtLee [:catlee]]

Attachment: handle bundles, mirrors, full clones by default and busted non-shared checkouts

This patch adds a few options to hgtool.py:

* --mirror <url>
  adds a mirror url to use for cloning/pulling operations. multiple urls can be specified with multiple --mirror arguments

* --bundle <url>
  adds a bundle url to use for cloning.

* --clone_by_revision
  When enabled, initial clones will be done via 'hg clone -r <rev>' instead of 'hg clone ; hg update -r <rev>'. This defaults to False, which should greatly reduce the load on the hg backends. We need to use this switch for try though.

Bundles are only tried for initial clones. If downloading / unbundling a bundle is successful we'll pull in new changes from the mirrors or master repo before trying to update.

Mirrors are tried in the order specified for pulling / cloning before falling back to the master repository.

I've also added a check for bug 691467 that will clobber the checkout if .hgrc is busted in our non-share codepaths.

Comment 2

[[e-mail] (:bhearsum)]

Comment on attachment 570716 [details] [diff] [review]
handle bundles, mirrors, full clones by default and busted non-shared checkouts

Review of attachment 570716 [details] [diff] [review]:
-----------------------------------------------------------------

::: lib/python/mozilla_buildtools/test/test_util_hg.py
@@ +574,5 @@
> +            self.assertEquals(called, [True])
> +        finally:
> +            hg.unbundle = orig_unbundle
> +
> +    def testCloneWithMirror(self):

It would be good to have a test or two around cloning with multiple mirrors where the first (or all) fail.

::: lib/python/util/hg.py
@@ +475,5 @@
> +def init(dest):
> +    """Initializes an empty repo in `dest`"""
> +    run_cmd(['hg', 'init', dest])
> +
> +def unbundle(bundle_url, dest):

Nit: 'unbundle' handles local files, too. This would be better named just plain 'bundle' or similar.

@@ +479,5 @@
> +def unbundle(bundle_url, dest):
> +    """Unbundles the bundle at `bundle_url` into `dest`."""
> +    run_cmd(['hg', 'unbundle', bundle_url], cwd=dest)
> +
> +def adjust_paths(dest, **paths):

.hgrc is an .ini file, why not use ConfigParser for this?

Comment 3

[Chris AtLee [:catlee]]

Attachment: handle bundles, mirrors, full clones by default and busted non-shared checkouts

comments addressed. interdiff is at

https://github.com/catlee/tools/compare/a66f762cc8feb89e5e9e4f7cb541f60be1195fce...12bc583dd31921fb56c836d216f8ca678e07429a

Comment 4

[[e-mail] (:bhearsum)]

Comment on attachment 570783 [details] [diff] [review]
handle bundles, mirrors, full clones by default and busted non-shared checkouts

Thanks for the interdiff!

Comment 5

[Chris AtLee [:catlee]]

Attachment: handle bundles, mirrors, full clones by default and busted non-shared checkouts

Same as before, with this addition: 
https://github.com/catlee/tools/commit/524bf3ef0728a1edc27fa763e38846b50dbb5dd0

Fixes pulling by revision into the shared directory.

Also fixes bug 699028.