Last Comment Bug 437482 - Create a bundle of mozilla-central regularly
: Create a bundle of mozilla-central regularly
Status: VERIFIED FIXED
[simple][oldbugs][buildduty]
:
Product: Release Engineering
Classification: Other
Component: Other (show other bugs)
: other
: All All
: P2 normal (vote)
: ---
Assigned To: Dustin J. Mitchell [:dustin]
:
:
Mentors:
http://benjamin.smedbergs.us/blog/200...
Depends on: 654958
Blocks: 614010 633254 644157
  Show dependency treegraph
 
Reported: 2008-06-05 12:48 PDT by Benjamin Smedberg [:bsmedberg]
Modified: 2013-08-12 21:54 PDT (History)
17 users (show)
See Also:
Crash Signature:
(edit)
QA Whiteboard:
Iteration: ---
Points: ---


Attachments
hg_templates-r1.patch (599 bytes, patch)
2010-11-12 17:40 PST, Dustin J. Mitchell [:dustin]
ted: review-
Details | Diff | Splinter Review
buildbot-configs-r1.patch (848 bytes, patch)
2010-11-12 17:41 PST, Dustin J. Mitchell [:dustin]
catlee: review+
catlee: checked‑in+
Details | Diff | Splinter Review
buildbotcustom-r1.patch (3.96 KB, patch)
2010-11-12 17:42 PST, Dustin J. Mitchell [:dustin]
catlee: feedback+
Details | Diff | Splinter Review
tools-r1.patch (881 bytes, patch)
2010-11-12 17:43 PST, Dustin J. Mitchell [:dustin]
catlee: review-
Details | Diff | Splinter Review
buildbotcustom-r2.patch (4.15 KB, patch)
2010-11-22 14:58 PST, Dustin J. Mitchell [:dustin]
catlee: review+
catlee: checked‑in+
Details | Diff | Splinter Review
tools-r2.patch (1002 bytes, patch)
2010-11-22 15:25 PST, Dustin J. Mitchell [:dustin]
catlee: review+
dustin: checked‑in+
Details | Diff | Splinter Review

Description Benjamin Smedberg [:bsmedberg] 2008-06-05 12:48:28 PDT
To get people started with mozilla-central, it would be nice to publish a bundle which people can download instead of having to start out with an HTTP clone.

See http://benjamin.smedbergs.us/blog/2008-06-05/getting-mozilla-central-with-limited-bandwidth/ for some details.

To create a bundle, do:

$ hg clone http://hg.mozilla.org/mozilla-central
$ hg -R mozilla-central bundle -a mozilla-central.bundle

I figured that since the build tinderboxes already clone mozilla-central, creating a bundle from it would be a simple extra step. Alternately, we might be able to create such a bundle nightly on hg.mozilla.org directly from a cronjob. Aravind/bhearsum thoughts about which solution is easier/better?

Ultimately these should end up on ftp.mozilla.org: I don't think we need an archive of them: just upload a new one every night or every week.
Comment 1 Daniel Brooks [:db48x] 2008-09-29 03:06:18 PDT
Not to be a wet blanket or anything, but doesn't hg clone simply download a compressed bundle? It certainly downloads a lot less data than what ends up on disk.
Comment 2 Dirkjan Ochtman (:djc) 2008-09-29 03:10:43 PDT
A bundle *is* compressed. Although in some circumstances it will use "streamclone", which is an uncompressed clone variant (also it will sometimes not use compression over SSH, I think, because SSH can provide its own compression). But I think the main advantage of pre-creating a bundle is that the producing the bundle can take up some time, so if you pre-create a bundle, that part is already down and the download time is only limited by available bandwidth.
Comment 3 Benjamin Smedberg [:bsmedberg] 2008-09-29 07:43:13 PDT
There are other advantages: if you have a sporadic connection, you can make use of download resuming over HTTP, whereas if a pull fails you have to start over from the beginning. And there's certainly less load on the server.
Comment 4 Hùng. NGUYỄN Mạnh 2008-11-01 05:46:15 PDT
When will the bundle get into the FTP server?
With my unstable connection, I couldn't pull the source via Mercurial. Thanks to Alex Hecht pointing me to Benjamin's post, I managed to do it. So I think it would be helpful for people like me.
Comment 5 Ben Hearsum (:bhearsum) 2008-11-03 09:39:41 PST
We have code that does this for releases now. We should be able to get it running a weekly or nightly basis without *too* much trouble. It'll be a few weeks at least before someone can revisit this, though.
Comment 6 Jonathan Watt [:jwatt] (back in October - email directly if necessary) 2009-01-12 04:55:16 PST
This would help me too.

Once this is done the there's no need for the bundles in the release directory IMO. We'll only need one (the latest) since the bundle will contain all previous revisions.
Comment 7 Jonathan Watt [:jwatt] (back in October - email directly if necessary) 2009-01-12 05:03:30 PST
Should separate bugs be opened for other repositories, or can all of the repositories listed at http://hg.mozilla.org/ have this done for them?

Perhaps the correct way to fix this is to fix hgweb to have a "bundle" link alongside the "zip", "gz" and "bz2" links on that page. (Upstream I mean.)
Comment 8 Dirkjan Ochtman (:djc) 2009-01-12 05:09:19 PST
Downloading bundles is pretty different from downloading archives, though (since they include full history). Not sure having the same UI for that would make sense.
Comment 9 Jonathan Watt [:jwatt] (back in October - email directly if necessary) 2009-01-12 05:54:42 PST
Yeah, the UI should show that the "zip", "gz" and "bz2" links don't contain history, whereas the "bundle" does. At least indicate that they are in some way different (different color button? red?).

I added a section to the docs linking to this bug BTW:

https://developer.mozilla.org/en/Mozilla_Source_Code_%28Mercurial%29#Bundles
Comment 10 Dirkjan Ochtman (:djc) 2009-01-12 05:59:11 PST
Well, I still think that's a bad solution. Just having a directory with periodically generated bundles sitting on a HTTP/FTP server would be better, IMO. Remember that downloading bundles is quite expensive for the server.
Comment 11 Chris Cooper [:coop] 2009-05-07 08:01:20 PDT
(In reply to comment #10)
> Well, I still think that's a bad solution. Just having a directory with
> periodically generated bundles sitting on a HTTP/FTP server would be better,
> IMO. Remember that downloading bundles is quite expensive for the server.

How expensive is expensive? I think the majority of users would continue to use standard hg methods for pulling and updating. 

Adding a "bundle" link to the existing hgweb UI keeps the source acquisition methods in one place, rather than requiring low-bandwidth users to go to ftp or elsewhere.

If it is too expensive (i.e. hgweb is going to try to bundle on-the-fly and we see a lot of concurrent requests), maybe a nighlty bundle makes more sense, but we should still have a "bundle" link in hgweb, even if it just points to latest-mozilla-central-bundle/ on ftp.
Comment 12 Dirkjan Ochtman (:djc) 2009-05-07 08:06:42 PDT
Having the link is fine with me. Generating it on the fly is a bad idea, IMO.
Comment 13 Chris Cooper [:coop] 2010-02-15 07:38:18 PST
Mass move of bugs from Release Engineering:Future -> Release Engineering. See
http://coop.deadsquid.com/2010/02/kiss-the-future-goodbye/ for more details.
Comment 14 Dustin J. Mitchell [:dustin] 2010-11-02 14:55:53 PDT
Is it possible to add a link to the hgweb page *without* generating it dynamically?

More generally, how is the header stuff on http://hg.mozilla.org generated? The "Respositories list" and "Repository layout" don't agree with one another, so I assume at least one of those is an HTML file somewhere..
Comment 15 Ted Mielczarek [:ted.mielczarek] 2010-11-02 17:06:32 PDT
Some of that is in the templates:
http://hg.mozilla.org/hgcustom/hg_templates/file/a56fa3e671a4/gitweb_mozilla
Comment 16 Dustin J. Mitchell [:dustin] 2010-11-02 22:34:08 PDT
Great - so we could potentially drop a bundle somewhere on a periodic basis, and add a static pointer to gitweb_mozilla/index.html.  That seems the ideal solution - no additional load on hg, nor even on its webserver, but the links are all still in a "logical" place.
Comment 17 Dustin J. Mitchell [:dustin] 2010-11-09 08:33:56 PST
I'll see if I can carry this forward.  I don't think we need dated bundles or anything - just a most-recent should be fine, since users need to 'hg up' once they've unbundled it anyway.  I see three parts:

a) set up a crontab to do the bundling (on stage.mozilla.org?) - nightly? weekly? depends how many calories it burns..

b) drop the bundles somewhere (probably ftp.m.o, via scp from stage) - this should be done atomically:
  scp to bundle~, ssh mv bundle~ bundle

c) update the template to point to it
Comment 18 Dustin J. Mitchell [:dustin] 2010-11-11 10:19:49 PST
bhearsum suggests that (a) should actually be done with a builder, not with a crontab.  Makes sense.  I'll see if I can cook that up.

I'm going to try to short-circuit (b) and (c) by making a bundle manually and linking to it.  Any bundle is better than no bundle at all!
Comment 19 Chris Cooper [:coop] 2010-11-12 12:58:06 PST
(In reply to comment #18)
> bhearsum suggests that (a) should actually be done with a builder, not with a
> crontab.  Makes sense.  I'll see if I can cook that up.

I'd suggest setting up another weekly builder like the code coverage or blocklist updater builders, and using a ScriptFactory to run your bundling script.
Comment 20 Dustin J. Mitchell [:dustin] 2010-11-12 17:31:44 PST
OK, the bundle is uploaded here (by running the script I'll upload shortly by hand):
  ftp://ftp.mozilla.org/pub/firefox/bundles/
Comment 21 Dustin J. Mitchell [:dustin] 2010-11-12 17:40:17 PST
Created attachment 490303 [details] [diff] [review]
hg_templates-r1.patch

It's not the prettiest HTML design, but it gets the link on the page.  The list at the top of http://hg.mozilla.org is generated from {entries%indexentry}, and from my read of http://mercurial.selenic.com/wiki/Theming there's no way to add an extra archive link only to one repo.
Comment 22 Dustin J. Mitchell [:dustin] 2010-11-12 17:41:26 PST
Created attachment 490304 [details] [diff] [review]
buildbot-configs-r1.patch

Add an 'enable_weekly_bundle' config key; set it for mozilla-central
Comment 23 Dustin J. Mitchell [:dustin] 2010-11-12 17:42:31 PST
Created attachment 490305 [details] [diff] [review]
buildbotcustom-r1.patch

Schedule weekly bundles and add a builder using ScriptFactory.
Comment 24 Dustin J. Mitchell [:dustin] 2010-11-12 17:43:45 PST
Created attachment 490306 [details] [diff] [review]
tools-r1.patch

The script to create and upload a bundle.  I ran this on a production slave to upload the bundle that's available now.
Comment 25 Dustin J. Mitchell [:dustin] 2010-11-12 17:49:47 PST
Comment on attachment 490305 [details] [diff] [review]
buildbotcustom-r1.patch

Sorry - I haven't staged this patch yet, although it passes checkconfig, so I'd like feedback that it makes sense and is the right way to do this, before I spend time in staging.
Comment 26 Chris AtLee [:catlee] 2010-11-16 18:50:45 PST
Comment on attachment 490305 [details] [diff] [review]
buildbotcustom-r1.patch

Actually, you probably need extra_args=[config['repo_path']] to handle repos like releases/mozilla-1.9.2
Comment 27 Chris AtLee [:catlee] 2010-11-16 18:52:44 PST
Comment on attachment 490306 [details] [diff] [review]
tools-r1.patch

>diff --git a/scripts/bundle/hg-bundle.sh b/scripts/bundle/hg-bundle.sh
>new file mode 100755
>--- /dev/null
>+++ b/scripts/bundle/hg-bundle.sh
>@@ -0,0 +1,26 @@
>+#! /bin/bash
>+
>+BRANCH="$1"
>+
>+if test -d clone; then
>+    rm -rf clone || exit 1
>+fi
>+
>+# checkout
>+hg clone "http://hg.mozilla.org/$BRANCH" clone || exit 1
>+
>+# bundle
>+BUNDLE=`echo $BRANCH | tr -c '\n[:alnum:]' '-'`.hg
>+( cd clone && hg bundle -a "../$BUNDLE" ) || exit 1
>+
>+# blow away the checkout to save space
>+rm -rf clone || exit 1
>+
>+# upload to a temporary name
>+scp -i ~/.ssh/ffxbld_dsa "$BUNDLE" ffxbld@stage.mozilla.org:"/pub/mozilla.org/firefox/bundles/$BUNDLE~" || exit 1

So we need to pass in the staging/production information somehow so we can upload to staging-stage.b.m.o or stage.m.o as appropriate.  Also, can we use something else other than "~" as the temporary file indication?  "~" usually has special meaning.
Comment 28 Ted Mielczarek [:ted.mielczarek] 2010-11-17 11:32:19 PST
Comment on attachment 490303 [details] [diff] [review]
hg_templates-r1.patch

I'm not wild about crufting up the templates with this. How about we just link this from the MDC docs once it's live? There's already a bundle section here:
https://developer.mozilla.org/En/Developer_Guide/Source_Code/Mercurial

Also, we probably want to link to http://ftp.mozilla.org, not FTP.
Comment 29 Dustin J. Mitchell [:dustin] 2010-11-22 10:35:28 PST
(In reply to comment #26)
> Actually, you probably need extra_args=[config['repo_path']] to handle repos
> like releases/mozilla-1.9.2

Fixed - I'll pass both the branch name (for use in naming the bundle) and the repo path (for the hg clone).

(In reply to comment #27)
> So we need to pass in the staging/production information somehow so we can
> upload to staging-stage.b.m.o or stage.m.o as appropriate.  Also, can we use
> something else other than "~" as the temporary file indication?  "~" usually
> has special meaning.

Sound good.  I learned ~ from emacs long ago, but .upload works, too.

(In reply to comment #28)
> I'm not wild about crufting up the templates with this. How about we just link
> this from the MDC docs once it's live? There's already a bundle section here:
> https://developer.mozilla.org/En/Developer_Guide/Source_Code/Mercurial
> 
> Also, we probably want to link to http://ftp.mozilla.org, not FTP.

I updated the MDN to point to the existing bundle (even if it is a bit out of date right now).  I also used the http link - thanks for spotting that!
Comment 30 Dustin J. Mitchell [:dustin] 2010-11-22 14:58:25 PST
Created attachment 492463 [details] [diff] [review]
buildbotcustom-r2.patch
Comment 31 Dustin J. Mitchell [:dustin] 2010-11-22 15:25:41 PST
Created attachment 492471 [details] [diff] [review]
tools-r2.patch

Both patches are tested in staging:
  http://staging-stage.build.mozilla.org/pub/mozilla.org/firefox/bundles/
Comment 32 Dustin J. Mitchell [:dustin] 2010-11-23 14:32:51 PST
I'll land the other two tomorrow when catlee's around.
Comment 33 Chris AtLee [:catlee] 2010-11-24 06:51:59 PST
Comment on attachment 492463 [details] [diff] [review]
buildbotcustom-r2.patch

changeset:   1132:753e5b87eb6d
Comment 34 Chris AtLee [:catlee] 2010-11-24 06:52:41 PST
Comment on attachment 490304 [details] [diff] [review]
buildbot-configs-r1.patch

changeset:   3315:632937d89dd7
Comment 35 Dustin J. Mitchell [:dustin] 2010-11-29 09:26:10 PST
the bundle was properly uploaded over the weekend.
Comment 36 Justin Wood (:Callek) 2011-02-02 20:49:23 PST
(In reply to comment #30)
> Created attachment 492463 [details] [diff] [review]
> buildbotcustom-r2.patch

Dustin, it looks like your patch here https://hg.mozilla.org/build/buildbotcustom/rev/753e5b87eb6d

Has accidentally omitted coverageBuilders from the m-c generateBranchObjects and instead added it to c-c's section. (see last hunk, rather than above)
Comment 37 Dustin J. Mitchell [:dustin] 2011-02-03 09:35:22 PST
Such are the dangers of copy-pasted code..
Comment 38 Justin Wood (:Callek) 2011-02-06 11:15:54 PST
(In reply to comment #37)
> Such are the dangers of copy-pasted code..

https://hg.mozilla.org/build/buildbotcustom/rev/c57d824be391

Will get pulled into the next production-bump here.
Comment 39 Serge Gautherie (:sgautherie) 2011-02-10 04:08:39 PST
What about doing the same for mozilla-1.9.2 and mozilla-1.9.1?
Comment 40 Ben Hearsum (:bhearsum) 2011-02-10 04:10:00 PST
(In reply to comment #39)
> What about doing the same for mozilla-1.9.2 and mozilla-1.9.1?

Not worth it, IMHO.
Comment 41 Serge Gautherie (:sgautherie) 2011-03-26 21:11:57 PDT
(In reply to comment #40)
> (In reply to comment #39)
> > What about doing the same for mozilla-1.9.2 and mozilla-1.9.1?
> 
> Not worth it, IMHO.

Eventually, bug 644157 did it...

Note You need to log in before you can comment on or make changes to this bug.