Closed Bug 851398 Opened 10 years ago Closed 6 years ago

Avoid rm -rf; hg clone; hg up pattern and purge instead

Categories

(Release Engineering :: General, defect)

defect
Not set
normal

Tracking

(Not tracked)

RESOLVED WORKSFORME

People

(Reporter: gps, Unassigned)

References

(Blocks 1 open bug)

Details

(Whiteboard: [buildfaster:P1][capacity])

+++ This bug was initially created as a clone of Bug #851270 +++

I was poking through more of the build logs and noticed a common pattern:

  rm -rf foo
  hg clone http://hg.mozilla.org/foo foo
  hg up -r

We're doing this for build-tools, mozharness, and main source repositories. This pattern is silly for a few reasons:

1) It is more filesystem intensive than hg purge --all && hg pull -u -r
2) All the excessive cloning creates extra load on the Mercurial server.

We're shooting ourselves in both feet here. We should systematically replace all occurrences of the rm -rf; hg clone; hg up pattern with essentially the logic:

if -e $dest:
   status=`hg purge --all -a`
   if status != 0:
     # Only clobber if purge was not successful!
     rm -rf $dest

if ! -e $dest:
  hg clone -u $rev http://... $dest
else:
  hg pull
  hg up -C -r $rev

Maybe the actual logic needs to be slightly more complicated to ensure repo URLs are the same, etc. But, you get my gist.

Enacting these changes will cumulatively shave minutes off of build times and should significantly reduce load on hg.mozilla.org by eliminating thousands of clones per day.
Additional point: when we *do* do a clone, we should instead download and unbundle a bundle if one is available (http://ftp.mozilla.org/pub/mozilla.org/firefox/bundles/), then `hg up` to get the latest. Bundles are static files on an FTP server, and are quicker in general.

And once you've grabbed a bundle, you can use it again (until you want a new starting bundle): you never need to talk to the server for those changesets.
(In reply to Richard Newman [:rnewman] from comment #1)
> Additional point: when we *do* do a clone, we should instead download and
> unbundle a bundle if one is available
> (http://ftp.mozilla.org/pub/mozilla.org/firefox/bundles/)...

hgtool does this already, the bundles at that location are updated by RelEng to feed it.
Product: mozilla.org → Release Engineering
Blocks: 957502
As I wrote in another bug, be aware that unbundle is actually much slower than cloning.

(In reply to Gregory Szorc [:gps] from comment #0)
>   hg clone -u $rev http://... $dest

Also note this is super slow as well (giving a revision when cloning). It's much faster to clone the repository entirely, and strip it if necessary. Or, for try, to clone *another* repository (like mozilla-central), and then pull -r $rev.
(In reply to Mike Hommey [:glandium] from comment #3)
> (In reply to Gregory Szorc [:gps] from comment #0)
> >   hg clone -u $rev http://... $dest
> 
> Also note this is super slow as well (giving a revision when cloning). It's
> much faster to clone the repository entirely, and strip it if necessary. Or,
> for try, to clone *another* repository (like mozilla-central), and then pull
> -r $rev.

You are confusing -r <rev> with -u <rev>. -u will clone the entire repo and update to the changeset specified. -r will only pull down ancestors of the specified changeset. I confirmed that -u doesn't bleed into -r by looking at the source code.
We only use pull/clone -u when fetching from try, otherwise we pull/clone the entire repo and update to the revision we want.

Is the slowness from unbundling due to network, or something else?
(In reply to Chris AtLee [:catlee] from comment #5)
> Is the slowness from unbundling due to network, or something else?

From bunzipping. See bug 958286 comment 6
ipsec issues resolved without needing to implement this, no longer blocks
No longer blocks: 957502
Blocks: try-tracker
catlee, is there anything that we should do here?
Flags: needinfo?(catlee)
The test machines are still the case here.
Flags: needinfo?(catlee)
robustcheckout has fixed this nearly everywhere.
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → WORKSFORME
Component: General Automation → General
You need to log in before you can comment on or make changes to this bug.