627576 - replace mozilla-central 'hg clone' operations with imports from cached bundles

Status: RESOLVED WONTFIX

(Release Engineering :: General, defect, P5)

Description

[Dustin J. Mitchell [:dustin] (he/him)]

(summarized from IRC conversation)

We already generate a bundle every week.

Slaves should try to get that bundle from ftp (stashing it somewhere common, and only downloading it if it's newer) and import from it, *then* run 'hg pull' to get everything since that weekend.

Resiliency can be built in a few ways:
 - if the download fails or the import fails, fall back to a clobber and clone
or, fancier but harder:
 - if the download or import fails, fall back to the previous week's bundle

This
 * takes a great load off hg.m.o
 * will be a lot faster, especially when the bundle is already cached
 * is the "right" way to do this sort of thing with hg
 * is reasonably easy to make fault-tolerant

Comment 1

[Aki Sasaki (not active)]

Similar to bug 589885, but a different approach.

Comment 2

[Aki Sasaki (not active)]

Ah, found the other one. bug 498522, which was WONTFIXed.

Comment 3

[[e-mail] (:bhearsum)]

(In reply to comment #1)
> Similar to bug 589885, but a different approach.

I'm tempted to WONTFIX this in favour of the approach in bug 589885. That bug already some work done in it, it just needs some more attention to get finished up. Technically, we _could_ do this still, by using bundles instead of "hg clone" to create the initial shared repository. I'm not sure it's worth the work though.

Dustin, what do you think?

Comment 4

[Dustin J. Mitchell [:dustin] (he/him)]

I was under the impression that the share extension didn't buy us much in the end, since it doesn't actually share objects.  So in effect it's an internal-to-hg implementation of this bug: the share directory provides a local source from which hg can copy common objects instead of downloading them from hg.m.o.

If I've understood correctly, then this can be dup'd to 589885.  I wrote this down so that the conversation wasn't limited to #mozbuild - I have no particular attachment to the idea.

Comment 5

[[e-mail] (:bhearsum)]

(In reply to comment #4)
> I was under the impression that the share extension didn't buy us much in the
> end, since it doesn't actually share objects.  So in effect it's an
> internal-to-hg implementation of this bug: the share directory provides a local
> source from which hg can copy common objects instead of downloading them from
> hg.m.o.

That's not correct, actually. "hg share" shares the repository between multiple working copies. For example:
[] bhearsum@voot:~/Mozilla/checkouts$ hg share tools tools-share
updating working directory
228 files updated, 0 files merged, 0 files removed, 0 files unresolved
[] bhearsum@voot:~/Mozilla/checkouts$ ls -l tools/.hg
total 96
-rw-r--r-- 1 bhearsum bhearsum    57 2010-12-14 16:20 00changelog.i
-rw-r--r-- 1 bhearsum bhearsum     8 2011-01-21 10:42 branch
-rw-r--r-- 1 bhearsum bhearsum  8405 2011-01-21 10:56 branchheads.cache
-rw-r--r-- 1 bhearsum bhearsum 12060 2011-01-21 10:43 dirstate
-rw-r--r-- 1 bhearsum bhearsum   113 2010-12-14 16:21 hgrc
-rw-r--r-- 1 bhearsum bhearsum    54 2011-01-21 10:43 last-message.txt
-rw-r--r-- 1 bhearsum bhearsum    23 2010-12-14 16:20 requires
drwxr-xr-x 3 bhearsum bhearsum  4096 2011-01-21 10:43 store
drwxr-xr-x 2 bhearsum bhearsum  4096 2011-01-21 10:42 strip-backup
-rw-r--r-- 1 bhearsum bhearsum 21873 2011-01-21 10:56 tags.cache
-rw-r--r-- 1 bhearsum bhearsum     7 2011-01-21 10:43 undo.branch
-rw-r--r-- 1 bhearsum bhearsum    12 2011-01-21 10:43 undo.desc
-rw-r--r-- 1 bhearsum bhearsum 12060 2011-01-21 10:43 undo.dirstate
[] bhearsum@voot:~/Mozilla/checkouts$ ls -l tools-share/.hg
total 64
-rw-r--r-- 1 bhearsum bhearsum     8 2011-01-21 11:23 branch
-rw-r--r-- 1 bhearsum bhearsum  8405 2011-01-21 11:23 branchheads.cache
-rw-r--r-- 1 bhearsum bhearsum 12060 2011-01-21 11:23 dirstate
-rw-r--r-- 1 bhearsum bhearsum    52 2011-01-21 11:23 hgrc
-rw-r--r-- 1 bhearsum bhearsum    30 2011-01-21 11:23 requires
-rw-r--r-- 1 bhearsum bhearsum    42 2011-01-21 11:23 sharedpath
-rw-r--r-- 1 bhearsum bhearsum 21873 2011-01-21 11:23 tags.cache
[] bhearsum@voot:~/Mozilla/checkouts$ cat tools-share/.hg/sharedpath 
/home/bhearsum/Mozilla/checkouts/tools/.hg

The objects live in "store", and "tools-share" uses the objects from "tools" for updating, commiting, and any other repository operations.

Comment 6

[Mike Taylor [:bear]]

Chris,

Can you comment on this in light of the work you are doing with hg share?  Is
this a DUPE or a WONTFIX related to that?

Comment 7

[Chris AtLee [:catlee]]

I say WONTFIX.

Using shared checkouts is "close".  Using pre-built bundles could save time in the initial case where machines don't have a shared copy of the repo, but I think adds unnecessary complexity.