+++ This bug was initially created as a clone of Bug #851270 +++ I was poking through more of the build logs and noticed a common pattern: rm -rf foo hg clone http://hg.mozilla.org/foo foo hg up -r We're doing this for build-tools, mozharness, and main source repositories. This pattern is silly for a few reasons: 1) It is more filesystem intensive than hg purge --all && hg pull -u -r 2) All the excessive cloning creates extra load on the Mercurial server. We're shooting ourselves in both feet here. We should systematically replace all occurrences of the rm -rf; hg clone; hg up pattern with essentially the logic: if -e $dest: status=`hg purge --all -a` if status != 0: # Only clobber if purge was not successful! rm -rf $dest if ! -e $dest: hg clone -u $rev http://... $dest else: hg pull hg up -C -r $rev Maybe the actual logic needs to be slightly more complicated to ensure repo URLs are the same, etc. But, you get my gist. Enacting these changes will cumulatively shave minutes off of build times and should significantly reduce load on hg.mozilla.org by eliminating thousands of clones per day.
Additional point: when we *do* do a clone, we should instead download and unbundle a bundle if one is available (http://ftp.mozilla.org/pub/mozilla.org/firefox/bundles/), then `hg up` to get the latest. Bundles are static files on an FTP server, and are quicker in general. And once you've grabbed a bundle, you can use it again (until you want a new starting bundle): you never need to talk to the server for those changesets.
(In reply to Richard Newman [:rnewman] from comment #1) > Additional point: when we *do* do a clone, we should instead download and > unbundle a bundle if one is available > (http://ftp.mozilla.org/pub/mozilla.org/firefox/bundles/)... hgtool does this already, the bundles at that location are updated by RelEng to feed it.
Product: mozilla.org → Release Engineering
As I wrote in another bug, be aware that unbundle is actually much slower than cloning. (In reply to Gregory Szorc [:gps] from comment #0) > hg clone -u $rev http://... $dest Also note this is super slow as well (giving a revision when cloning). It's much faster to clone the repository entirely, and strip it if necessary. Or, for try, to clone *another* repository (like mozilla-central), and then pull -r $rev.
(In reply to Mike Hommey [:glandium] from comment #3) > (In reply to Gregory Szorc [:gps] from comment #0) > > hg clone -u $rev http://... $dest > > Also note this is super slow as well (giving a revision when cloning). It's > much faster to clone the repository entirely, and strip it if necessary. Or, > for try, to clone *another* repository (like mozilla-central), and then pull > -r $rev. You are confusing -r <rev> with -u <rev>. -u will clone the entire repo and update to the changeset specified. -r will only pull down ancestors of the specified changeset. I confirmed that -u doesn't bleed into -r by looking at the source code.
We only use pull/clone -u when fetching from try, otherwise we pull/clone the entire repo and update to the revision we want. Is the slowness from unbundling due to network, or something else?
(In reply to Chris AtLee [:catlee] from comment #5) > Is the slowness from unbundling due to network, or something else? From bunzipping. See bug 958286 comment 6
ipsec issues resolved without needing to implement this, no longer blocks
No longer blocks: 957502
catlee, is there anything that we should do here?
The test machines are still the case here.
robustcheckout has fixed this nearly everywhere.
Status: NEW → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → WORKSFORME
Component: General Automation → General
Product: Release Engineering → Release Engineering
You need to log in before you can comment on or make changes to this bug.