Avoid rm -rf; hg clone; hg up pattern and purge instead

RESOLVED WORKSFORME

Status

RESOLVED WORKSFORME
6 years ago
7 months ago

People

(Reporter: gps, Unassigned)

Tracking

(Blocks: 1 bug)

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [buildfaster:P1][capacity])

(Reporter)

Description

6 years ago
+++ This bug was initially created as a clone of Bug #851270 +++

I was poking through more of the build logs and noticed a common pattern:

  rm -rf foo
  hg clone http://hg.mozilla.org/foo foo
  hg up -r

We're doing this for build-tools, mozharness, and main source repositories. This pattern is silly for a few reasons:

1) It is more filesystem intensive than hg purge --all && hg pull -u -r
2) All the excessive cloning creates extra load on the Mercurial server.

We're shooting ourselves in both feet here. We should systematically replace all occurrences of the rm -rf; hg clone; hg up pattern with essentially the logic:

if -e $dest:
   status=`hg purge --all -a`
   if status != 0:
     # Only clobber if purge was not successful!
     rm -rf $dest

if ! -e $dest:
  hg clone -u $rev http://... $dest
else:
  hg pull
  hg up -C -r $rev

Maybe the actual logic needs to be slightly more complicated to ensure repo URLs are the same, etc. But, you get my gist.

Enacting these changes will cumulatively shave minutes off of build times and should significantly reduce load on hg.mozilla.org by eliminating thousands of clones per day.
Additional point: when we *do* do a clone, we should instead download and unbundle a bundle if one is available (http://ftp.mozilla.org/pub/mozilla.org/firefox/bundles/), then `hg up` to get the latest. Bundles are static files on an FTP server, and are quicker in general.

And once you've grabbed a bundle, you can use it again (until you want a new starting bundle): you never need to talk to the server for those changesets.
(In reply to Richard Newman [:rnewman] from comment #1)
> Additional point: when we *do* do a clone, we should instead download and
> unbundle a bundle if one is available
> (http://ftp.mozilla.org/pub/mozilla.org/firefox/bundles/)...

hgtool does this already, the bundles at that location are updated by RelEng to feed it.
Product: mozilla.org → Release Engineering
As I wrote in another bug, be aware that unbundle is actually much slower than cloning.

(In reply to Gregory Szorc [:gps] from comment #0)
>   hg clone -u $rev http://... $dest

Also note this is super slow as well (giving a revision when cloning). It's much faster to clone the repository entirely, and strip it if necessary. Or, for try, to clone *another* repository (like mozilla-central), and then pull -r $rev.
(Reporter)

Comment 4

5 years ago
(In reply to Mike Hommey [:glandium] from comment #3)
> (In reply to Gregory Szorc [:gps] from comment #0)
> >   hg clone -u $rev http://... $dest
> 
> Also note this is super slow as well (giving a revision when cloning). It's
> much faster to clone the repository entirely, and strip it if necessary. Or,
> for try, to clone *another* repository (like mozilla-central), and then pull
> -r $rev.

You are confusing -r <rev> with -u <rev>. -u will clone the entire repo and update to the changeset specified. -r will only pull down ancestors of the specified changeset. I confirmed that -u doesn't bleed into -r by looking at the source code.
We only use pull/clone -u when fetching from try, otherwise we pull/clone the entire repo and update to the revision we want.

Is the slowness from unbundling due to network, or something else?
(In reply to Chris AtLee [:catlee] from comment #5)
> Is the slowness from unbundling due to network, or something else?

From bunzipping. See bug 958286 comment 6
ipsec issues resolved without needing to implement this, no longer blocks
No longer blocks: 957502
Duplicate of this bug: 632624

Comment 9

4 years ago
catlee, is there anything that we should do here?
Flags: needinfo?(catlee)

Comment 10

4 years ago
The test machines are still the case here.
Flags: needinfo?(catlee)
robustcheckout has fixed this nearly everywhere.
Status: NEW → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → WORKSFORME
(Assignee)

Updated

7 months ago
Component: General Automation → General
Product: Release Engineering → Release Engineering
You need to log in before you can comment on or make changes to this bug.