Closed Bug 437482 Opened 16 years ago Closed 14 years ago

Create a bundle of mozilla-central regularly

Categories

(Release Engineering :: General, defect, P2)

defect

Tracking

(Not tracked)

VERIFIED FIXED

People

(Reporter: benjamin, Assigned: dustin)

References

()

Details

(Whiteboard: [simple][oldbugs][buildduty])

Attachments

(3 files, 3 obsolete files)

To get people started with mozilla-central, it would be nice to publish a bundle which people can download instead of having to start out with an HTTP clone.

See http://benjamin.smedbergs.us/blog/2008-06-05/getting-mozilla-central-with-limited-bandwidth/ for some details.

To create a bundle, do:

$ hg clone http://hg.mozilla.org/mozilla-central
$ hg -R mozilla-central bundle -a mozilla-central.bundle

I figured that since the build tinderboxes already clone mozilla-central, creating a bundle from it would be a simple extra step. Alternately, we might be able to create such a bundle nightly on hg.mozilla.org directly from a cronjob. Aravind/bhearsum thoughts about which solution is easier/better?

Ultimately these should end up on ftp.mozilla.org: I don't think we need an archive of them: just upload a new one every night or every week.
Component: Release Engineering → Release Engineering: Future
Priority: -- → P3
Not to be a wet blanket or anything, but doesn't hg clone simply download a compressed bundle? It certainly downloads a lot less data than what ends up on disk.
A bundle *is* compressed. Although in some circumstances it will use "streamclone", which is an uncompressed clone variant (also it will sometimes not use compression over SSH, I think, because SSH can provide its own compression). But I think the main advantage of pre-creating a bundle is that the producing the bundle can take up some time, so if you pre-create a bundle, that part is already down and the download time is only limited by available bandwidth.
There are other advantages: if you have a sporadic connection, you can make use of download resuming over HTTP, whereas if a pull fails you have to start over from the beginning. And there's certainly less load on the server.
When will the bundle get into the FTP server?
With my unstable connection, I couldn't pull the source via Mercurial. Thanks to Alex Hecht pointing me to Benjamin's post, I managed to do it. So I think it would be helpful for people like me.
We have code that does this for releases now. We should be able to get it running a weekly or nightly basis without *too* much trouble. It'll be a few weeks at least before someone can revisit this, though.
This would help me too.

Once this is done the there's no need for the bundles in the release directory IMO. We'll only need one (the latest) since the bundle will contain all previous revisions.
Should separate bugs be opened for other repositories, or can all of the repositories listed at http://hg.mozilla.org/ have this done for them?

Perhaps the correct way to fix this is to fix hgweb to have a "bundle" link alongside the "zip", "gz" and "bz2" links on that page. (Upstream I mean.)
Downloading bundles is pretty different from downloading archives, though (since they include full history). Not sure having the same UI for that would make sense.
Yeah, the UI should show that the "zip", "gz" and "bz2" links don't contain history, whereas the "bundle" does. At least indicate that they are in some way different (different color button? red?).

I added a section to the docs linking to this bug BTW:

https://developer.mozilla.org/en/Mozilla_Source_Code_%28Mercurial%29#Bundles
Well, I still think that's a bad solution. Just having a directory with periodically generated bundles sitting on a HTTP/FTP server would be better, IMO. Remember that downloading bundles is quite expensive for the server.
(In reply to comment #10)
> Well, I still think that's a bad solution. Just having a directory with
> periodically generated bundles sitting on a HTTP/FTP server would be better,
> IMO. Remember that downloading bundles is quite expensive for the server.

How expensive is expensive? I think the majority of users would continue to use standard hg methods for pulling and updating. 

Adding a "bundle" link to the existing hgweb UI keeps the source acquisition methods in one place, rather than requiring low-bandwidth users to go to ftp or elsewhere.

If it is too expensive (i.e. hgweb is going to try to bundle on-the-fly and we see a lot of concurrent requests), maybe a nighlty bundle makes more sense, but we should still have a "bundle" link in hgweb, even if it just points to latest-mozilla-central-bundle/ on ftp.
Having the link is fine with me. Generating it on the fly is a bad idea, IMO.
Mass move of bugs from Release Engineering:Future -> Release Engineering. See
http://coop.deadsquid.com/2010/02/kiss-the-future-goodbye/ for more details.
Component: Release Engineering: Future → Release Engineering
Whiteboard: [simple]
Priority: P3 → P5
Keywords: student-project
Assignee: nobody → armenzg
Status: NEW → ASSIGNED
Is it possible to add a link to the hgweb page *without* generating it dynamically?

More generally, how is the header stuff on http://hg.mozilla.org generated? The "Respositories list" and "Repository layout" don't agree with one another, so I assume at least one of those is an HTML file somewhere..
Great - so we could potentially drop a bundle somewhere on a periodic basis, and add a static pointer to gitweb_mozilla/index.html.  That seems the ideal solution - no additional load on hg, nor even on its webserver, but the links are all still in a "logical" place.
Whiteboard: [simple] → [simple][oldbugs]
I'll see if I can carry this forward.  I don't think we need dated bundles or anything - just a most-recent should be fine, since users need to 'hg up' once they've unbundled it anyway.  I see three parts:

a) set up a crontab to do the bundling (on stage.mozilla.org?) - nightly? weekly? depends how many calories it burns..

b) drop the bundles somewhere (probably ftp.m.o, via scp from stage) - this should be done atomically:
  scp to bundle~, ssh mv bundle~ bundle

c) update the template to point to it
Assignee: armenzg → dustin
bhearsum suggests that (a) should actually be done with a builder, not with a crontab.  Makes sense.  I'll see if I can cook that up.

I'm going to try to short-circuit (b) and (c) by making a bundle manually and linking to it.  Any bundle is better than no bundle at all!
(In reply to comment #18)
> bhearsum suggests that (a) should actually be done with a builder, not with a
> crontab.  Makes sense.  I'll see if I can cook that up.

I'd suggest setting up another weekly builder like the code coverage or blocklist updater builders, and using a ScriptFactory to run your bundling script.
Keywords: student-project
OS: Mac OS X → All
Priority: P5 → P2
Hardware: x86 → All
OK, the bundle is uploaded here (by running the script I'll upload shortly by hand):
  ftp://ftp.mozilla.org/pub/firefox/bundles/
Attached patch hg_templates-r1.patch (obsolete) — Splinter Review
It's not the prettiest HTML design, but it gets the link on the page.  The list at the top of http://hg.mozilla.org is generated from {entries%indexentry}, and from my read of http://mercurial.selenic.com/wiki/Theming there's no way to add an extra archive link only to one repo.
Attachment #490303 - Flags: review?(ted.mielczarek)
Add an 'enable_weekly_bundle' config key; set it for mozilla-central
Attachment #490304 - Flags: review?(catlee)
Attached patch buildbotcustom-r1.patch (obsolete) — Splinter Review
Schedule weekly bundles and add a builder using ScriptFactory.
Attachment #490305 - Flags: review?(catlee)
Attached patch tools-r1.patch (obsolete) — Splinter Review
The script to create and upload a bundle.  I ran this on a production slave to upload the bundle that's available now.
Attachment #490306 - Flags: review?(catlee)
Comment on attachment 490305 [details] [diff] [review]
buildbotcustom-r1.patch

Sorry - I haven't staged this patch yet, although it passes checkconfig, so I'd like feedback that it makes sense and is the right way to do this, before I spend time in staging.
Attachment #490305 - Flags: review?(catlee) → feedback?(catlee)
Attachment #490305 - Flags: feedback?(catlee) → feedback+
Attachment #490304 - Flags: review?(catlee) → review+
Comment on attachment 490305 [details] [diff] [review]
buildbotcustom-r1.patch

Actually, you probably need extra_args=[config['repo_path']] to handle repos like releases/mozilla-1.9.2
Comment on attachment 490306 [details] [diff] [review]
tools-r1.patch

>diff --git a/scripts/bundle/hg-bundle.sh b/scripts/bundle/hg-bundle.sh
>new file mode 100755
>--- /dev/null
>+++ b/scripts/bundle/hg-bundle.sh
>@@ -0,0 +1,26 @@
>+#! /bin/bash
>+
>+BRANCH="$1"
>+
>+if test -d clone; then
>+    rm -rf clone || exit 1
>+fi
>+
>+# checkout
>+hg clone "http://hg.mozilla.org/$BRANCH" clone || exit 1
>+
>+# bundle
>+BUNDLE=`echo $BRANCH | tr -c '\n[:alnum:]' '-'`.hg
>+( cd clone && hg bundle -a "../$BUNDLE" ) || exit 1
>+
>+# blow away the checkout to save space
>+rm -rf clone || exit 1
>+
>+# upload to a temporary name
>+scp -i ~/.ssh/ffxbld_dsa "$BUNDLE" ffxbld@stage.mozilla.org:"/pub/mozilla.org/firefox/bundles/$BUNDLE~" || exit 1

So we need to pass in the staging/production information somehow so we can upload to staging-stage.b.m.o or stage.m.o as appropriate.  Also, can we use something else other than "~" as the temporary file indication?  "~" usually has special meaning.
Attachment #490306 - Flags: review?(catlee) → review-
Comment on attachment 490303 [details] [diff] [review]
hg_templates-r1.patch

I'm not wild about crufting up the templates with this. How about we just link this from the MDC docs once it's live? There's already a bundle section here:
https://developer.mozilla.org/En/Developer_Guide/Source_Code/Mercurial

Also, we probably want to link to http://ftp.mozilla.org, not FTP.
Attachment #490303 - Flags: review?(ted.mielczarek) → review-
(In reply to comment #26)
> Actually, you probably need extra_args=[config['repo_path']] to handle repos
> like releases/mozilla-1.9.2

Fixed - I'll pass both the branch name (for use in naming the bundle) and the repo path (for the hg clone).

(In reply to comment #27)
> So we need to pass in the staging/production information somehow so we can
> upload to staging-stage.b.m.o or stage.m.o as appropriate.  Also, can we use
> something else other than "~" as the temporary file indication?  "~" usually
> has special meaning.

Sound good.  I learned ~ from emacs long ago, but .upload works, too.

(In reply to comment #28)
> I'm not wild about crufting up the templates with this. How about we just link
> this from the MDC docs once it's live? There's already a bundle section here:
> https://developer.mozilla.org/En/Developer_Guide/Source_Code/Mercurial
> 
> Also, we probably want to link to http://ftp.mozilla.org, not FTP.

I updated the MDN to point to the existing bundle (even if it is a bit out of date right now).  I also used the http link - thanks for spotting that!
Blocks: 614010
Attachment #490303 - Attachment is obsolete: true
Attachment #490305 - Attachment is obsolete: true
Attachment #492463 - Flags: review?(catlee)
Attached patch tools-r2.patchSplinter Review
Both patches are tested in staging:
  http://staging-stage.build.mozilla.org/pub/mozilla.org/firefox/bundles/
Attachment #490306 - Attachment is obsolete: true
Attachment #492471 - Flags: review?(catlee)
Attachment #492463 - Flags: review?(catlee) → review+
Attachment #492471 - Flags: review?(catlee) → review+
Attachment #492471 - Flags: checked-in+
I'll land the other two tomorrow when catlee's around.
Flags: needs-reconfig?
Whiteboard: [simple][oldbugs] → [simple][oldbugs][buildduty]
Comment on attachment 492463 [details] [diff] [review]
buildbotcustom-r2.patch

changeset:   1132:753e5b87eb6d
Attachment #492463 - Flags: checked-in+
Comment on attachment 490304 [details] [diff] [review]
buildbot-configs-r1.patch

changeset:   3315:632937d89dd7
Attachment #490304 - Flags: checked-in+
Flags: needs-reconfig? → needs-reconfig+
Status: ASSIGNED → RESOLVED
Closed: 14 years ago
Resolution: --- → FIXED
the bundle was properly uploaded over the weekend.
Status: RESOLVED → VERIFIED
(In reply to comment #30)
> Created attachment 492463 [details] [diff] [review]
> buildbotcustom-r2.patch

Dustin, it looks like your patch here https://hg.mozilla.org/build/buildbotcustom/rev/753e5b87eb6d

Has accidentally omitted coverageBuilders from the m-c generateBranchObjects and instead added it to c-c's section. (see last hunk, rather than above)
Such are the dangers of copy-pasted code..
(In reply to comment #37)
> Such are the dangers of copy-pasted code..

https://hg.mozilla.org/build/buildbotcustom/rev/c57d824be391

Will get pulled into the next production-bump here.
What about doing the same for mozilla-1.9.2 and mozilla-1.9.1?
(In reply to comment #39)
> What about doing the same for mozilla-1.9.2 and mozilla-1.9.1?

Not worth it, IMHO.
Blocks: 633254
(In reply to comment #40)
> (In reply to comment #39)
> > What about doing the same for mozilla-1.9.2 and mozilla-1.9.1?
> 
> Not worth it, IMHO.

Eventually, bug 644157 did it...
Blocks: 644157
Depends on: 654958
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: