851398 - Avoid rm -rf; hg clone; hg up pattern and purge instead

Status: RESOLVED WORKSFORME

(Release Engineering :: General, defect)

Description

[Gregory Szorc [:gps]]

+++ This bug was initially created as a clone of Bug #851270 +++

I was poking through more of the build logs and noticed a common pattern:

  rm -rf foo
  hg clone http://hg.mozilla.org/foo foo
  hg up -r

We're doing this for build-tools, mozharness, and main source repositories. This pattern is silly for a few reasons:

1) It is more filesystem intensive than hg purge --all && hg pull -u -r
2) All the excessive cloning creates extra load on the Mercurial server.

We're shooting ourselves in both feet here. We should systematically replace all occurrences of the rm -rf; hg clone; hg up pattern with essentially the logic:

if -e $dest:
   status=`hg purge --all -a`
   if status != 0:
     # Only clobber if purge was not successful!
     rm -rf $dest

if ! -e $dest:
  hg clone -u $rev http://... $dest
else:
  hg pull
  hg up -C -r $rev

Maybe the actual logic needs to be slightly more complicated to ensure repo URLs are the same, etc. But, you get my gist.

Enacting these changes will cumulatively shave minutes off of build times and should significantly reduce load on hg.mozilla.org by eliminating thousands of clones per day.

Comment 1

[Richard Newman [:rnewman]]

Additional point: when we *do* do a clone, we should instead download and unbundle a bundle if one is available (http://ftp.mozilla.org/pub/mozilla.org/firefox/bundles/), then `hg up` to get the latest. Bundles are static files on an FTP server, and are quicker in general.

And once you've grabbed a bundle, you can use it again (until you want a new starting bundle): you never need to talk to the server for those changesets.

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to Richard Newman [:rnewman] from comment #1)
> Additional point: when we *do* do a clone, we should instead download and
> unbundle a bundle if one is available
> (http://ftp.mozilla.org/pub/mozilla.org/firefox/bundles/)...

hgtool does this already, the bundles at that location are updated by RelEng to feed it.

Comment 3

[Mike Hommey [:glandium]]

As I wrote in another bug, be aware that unbundle is actually much slower than cloning.

(In reply to Gregory Szorc [:gps] from comment #0)
>   hg clone -u $rev http://... $dest

Also note this is super slow as well (giving a revision when cloning). It's much faster to clone the repository entirely, and strip it if necessary. Or, for try, to clone *another* repository (like mozilla-central), and then pull -r $rev.

Comment 4

[Gregory Szorc [:gps]]

(In reply to Mike Hommey [:glandium] from comment #3)
> (In reply to Gregory Szorc [:gps] from comment #0)
> >   hg clone -u $rev http://... $dest
> 
> Also note this is super slow as well (giving a revision when cloning). It's
> much faster to clone the repository entirely, and strip it if necessary. Or,
> for try, to clone *another* repository (like mozilla-central), and then pull
> -r $rev.

You are confusing -r <rev> with -u <rev>. -u will clone the entire repo and update to the changeset specified. -r will only pull down ancestors of the specified changeset. I confirmed that -u doesn't bleed into -r by looking at the source code.

Comment 5

[Chris AtLee [:catlee]]

We only use pull/clone -u when fetching from try, otherwise we pull/clone the entire repo and update to the revision we want.

Is the slowness from unbundling due to network, or something else?

Comment 6

[Mike Hommey [:glandium]]

(In reply to Chris AtLee [:catlee] from comment #5)
> Is the slowness from unbundling due to network, or something else?

From bunzipping. See bug 958286 comment 6

Comment 7

[Hal Wine [:hwine] use NI!]

ipsec issues resolved without needing to implement this, no longer blocks

Comment 9

[Rail Aliiev [:rail]]

catlee, is there anything that we should do here?

Comment 10

[Rail Aliiev [:rail]]

The test machines are still the case here.

Comment 11

[Chris AtLee [:catlee]]

robustcheckout has fixed this nearly everywhere.