Closed
Bug 437482
Opened 16 years ago
Closed 14 years ago
Create a bundle of mozilla-central regularly
Categories
(Release Engineering :: General, defect, P2)
Release Engineering
General
Tracking
(Not tracked)
VERIFIED
FIXED
People
(Reporter: benjamin, Assigned: dustin)
References
()
Details
(Whiteboard: [simple][oldbugs][buildduty])
Attachments
(3 files, 3 obsolete files)
848 bytes,
patch
|
catlee
:
review+
catlee
:
checked-in+
|
Details | Diff | Splinter Review |
4.15 KB,
patch
|
catlee
:
review+
catlee
:
checked-in+
|
Details | Diff | Splinter Review |
1002 bytes,
patch
|
catlee
:
review+
dustin
:
checked-in+
|
Details | Diff | Splinter Review |
To get people started with mozilla-central, it would be nice to publish a bundle which people can download instead of having to start out with an HTTP clone. See http://benjamin.smedbergs.us/blog/2008-06-05/getting-mozilla-central-with-limited-bandwidth/ for some details. To create a bundle, do: $ hg clone http://hg.mozilla.org/mozilla-central $ hg -R mozilla-central bundle -a mozilla-central.bundle I figured that since the build tinderboxes already clone mozilla-central, creating a bundle from it would be a simple extra step. Alternately, we might be able to create such a bundle nightly on hg.mozilla.org directly from a cronjob. Aravind/bhearsum thoughts about which solution is easier/better? Ultimately these should end up on ftp.mozilla.org: I don't think we need an archive of them: just upload a new one every night or every week.
Updated•16 years ago
|
Component: Release Engineering → Release Engineering: Future
Priority: -- → P3
Comment 1•16 years ago
|
||
Not to be a wet blanket or anything, but doesn't hg clone simply download a compressed bundle? It certainly downloads a lot less data than what ends up on disk.
Comment 2•16 years ago
|
||
A bundle *is* compressed. Although in some circumstances it will use "streamclone", which is an uncompressed clone variant (also it will sometimes not use compression over SSH, I think, because SSH can provide its own compression). But I think the main advantage of pre-creating a bundle is that the producing the bundle can take up some time, so if you pre-create a bundle, that part is already down and the download time is only limited by available bandwidth.
Reporter | ||
Comment 3•16 years ago
|
||
There are other advantages: if you have a sporadic connection, you can make use of download resuming over HTTP, whereas if a pull fails you have to start over from the beginning. And there's certainly less load on the server.
When will the bundle get into the FTP server? With my unstable connection, I couldn't pull the source via Mercurial. Thanks to Alex Hecht pointing me to Benjamin's post, I managed to do it. So I think it would be helpful for people like me.
Comment 5•16 years ago
|
||
We have code that does this for releases now. We should be able to get it running a weekly or nightly basis without *too* much trouble. It'll be a few weeks at least before someone can revisit this, though.
Comment 6•15 years ago
|
||
This would help me too. Once this is done the there's no need for the bundles in the release directory IMO. We'll only need one (the latest) since the bundle will contain all previous revisions.
Comment 7•15 years ago
|
||
Should separate bugs be opened for other repositories, or can all of the repositories listed at http://hg.mozilla.org/ have this done for them? Perhaps the correct way to fix this is to fix hgweb to have a "bundle" link alongside the "zip", "gz" and "bz2" links on that page. (Upstream I mean.)
Comment 8•15 years ago
|
||
Downloading bundles is pretty different from downloading archives, though (since they include full history). Not sure having the same UI for that would make sense.
Comment 9•15 years ago
|
||
Yeah, the UI should show that the "zip", "gz" and "bz2" links don't contain history, whereas the "bundle" does. At least indicate that they are in some way different (different color button? red?). I added a section to the docs linking to this bug BTW: https://developer.mozilla.org/en/Mozilla_Source_Code_%28Mercurial%29#Bundles
Comment 10•15 years ago
|
||
Well, I still think that's a bad solution. Just having a directory with periodically generated bundles sitting on a HTTP/FTP server would be better, IMO. Remember that downloading bundles is quite expensive for the server.
Comment 11•15 years ago
|
||
(In reply to comment #10) > Well, I still think that's a bad solution. Just having a directory with > periodically generated bundles sitting on a HTTP/FTP server would be better, > IMO. Remember that downloading bundles is quite expensive for the server. How expensive is expensive? I think the majority of users would continue to use standard hg methods for pulling and updating. Adding a "bundle" link to the existing hgweb UI keeps the source acquisition methods in one place, rather than requiring low-bandwidth users to go to ftp or elsewhere. If it is too expensive (i.e. hgweb is going to try to bundle on-the-fly and we see a lot of concurrent requests), maybe a nighlty bundle makes more sense, but we should still have a "bundle" link in hgweb, even if it just points to latest-mozilla-central-bundle/ on ftp.
Comment 12•15 years ago
|
||
Having the link is fine with me. Generating it on the fly is a bad idea, IMO.
Comment 13•14 years ago
|
||
Mass move of bugs from Release Engineering:Future -> Release Engineering. See http://coop.deadsquid.com/2010/02/kiss-the-future-goodbye/ for more details.
Component: Release Engineering: Future → Release Engineering
Updated•14 years ago
|
Whiteboard: [simple]
Updated•14 years ago
|
Priority: P3 → P5
Updated•14 years ago
|
Keywords: student-project
Updated•14 years ago
|
Assignee: nobody → armenzg
Status: NEW → ASSIGNED
Assignee | ||
Comment 14•14 years ago
|
||
Is it possible to add a link to the hgweb page *without* generating it dynamically? More generally, how is the header stuff on http://hg.mozilla.org generated? The "Respositories list" and "Repository layout" don't agree with one another, so I assume at least one of those is an HTML file somewhere..
Comment 15•14 years ago
|
||
Some of that is in the templates: http://hg.mozilla.org/hgcustom/hg_templates/file/a56fa3e671a4/gitweb_mozilla
Assignee | ||
Comment 16•14 years ago
|
||
Great - so we could potentially drop a bundle somewhere on a periodic basis, and add a static pointer to gitweb_mozilla/index.html. That seems the ideal solution - no additional load on hg, nor even on its webserver, but the links are all still in a "logical" place.
Updated•14 years ago
|
Whiteboard: [simple] → [simple][oldbugs]
Assignee | ||
Comment 17•14 years ago
|
||
I'll see if I can carry this forward. I don't think we need dated bundles or anything - just a most-recent should be fine, since users need to 'hg up' once they've unbundled it anyway. I see three parts: a) set up a crontab to do the bundling (on stage.mozilla.org?) - nightly? weekly? depends how many calories it burns.. b) drop the bundles somewhere (probably ftp.m.o, via scp from stage) - this should be done atomically: scp to bundle~, ssh mv bundle~ bundle c) update the template to point to it
Assignee: armenzg → dustin
Assignee | ||
Comment 18•14 years ago
|
||
bhearsum suggests that (a) should actually be done with a builder, not with a crontab. Makes sense. I'll see if I can cook that up. I'm going to try to short-circuit (b) and (c) by making a bundle manually and linking to it. Any bundle is better than no bundle at all!
Comment 19•14 years ago
|
||
(In reply to comment #18) > bhearsum suggests that (a) should actually be done with a builder, not with a > crontab. Makes sense. I'll see if I can cook that up. I'd suggest setting up another weekly builder like the code coverage or blocklist updater builders, and using a ScriptFactory to run your bundling script.
Assignee | ||
Comment 20•14 years ago
|
||
OK, the bundle is uploaded here (by running the script I'll upload shortly by hand): ftp://ftp.mozilla.org/pub/firefox/bundles/
Assignee | ||
Comment 21•14 years ago
|
||
It's not the prettiest HTML design, but it gets the link on the page. The list at the top of http://hg.mozilla.org is generated from {entries%indexentry}, and from my read of http://mercurial.selenic.com/wiki/Theming there's no way to add an extra archive link only to one repo.
Attachment #490303 -
Flags: review?(ted.mielczarek)
Assignee | ||
Comment 22•14 years ago
|
||
Add an 'enable_weekly_bundle' config key; set it for mozilla-central
Attachment #490304 -
Flags: review?(catlee)
Assignee | ||
Comment 23•14 years ago
|
||
Schedule weekly bundles and add a builder using ScriptFactory.
Attachment #490305 -
Flags: review?(catlee)
Assignee | ||
Comment 24•14 years ago
|
||
The script to create and upload a bundle. I ran this on a production slave to upload the bundle that's available now.
Attachment #490306 -
Flags: review?(catlee)
Assignee | ||
Comment 25•14 years ago
|
||
Comment on attachment 490305 [details] [diff] [review] buildbotcustom-r1.patch Sorry - I haven't staged this patch yet, although it passes checkconfig, so I'd like feedback that it makes sense and is the right way to do this, before I spend time in staging.
Attachment #490305 -
Flags: review?(catlee) → feedback?(catlee)
Updated•14 years ago
|
Attachment #490305 -
Flags: feedback?(catlee) → feedback+
Updated•14 years ago
|
Attachment #490304 -
Flags: review?(catlee) → review+
Comment 26•14 years ago
|
||
Comment on attachment 490305 [details] [diff] [review] buildbotcustom-r1.patch Actually, you probably need extra_args=[config['repo_path']] to handle repos like releases/mozilla-1.9.2
Comment 27•14 years ago
|
||
Comment on attachment 490306 [details] [diff] [review] tools-r1.patch >diff --git a/scripts/bundle/hg-bundle.sh b/scripts/bundle/hg-bundle.sh >new file mode 100755 >--- /dev/null >+++ b/scripts/bundle/hg-bundle.sh >@@ -0,0 +1,26 @@ >+#! /bin/bash >+ >+BRANCH="$1" >+ >+if test -d clone; then >+ rm -rf clone || exit 1 >+fi >+ >+# checkout >+hg clone "http://hg.mozilla.org/$BRANCH" clone || exit 1 >+ >+# bundle >+BUNDLE=`echo $BRANCH | tr -c '\n[:alnum:]' '-'`.hg >+( cd clone && hg bundle -a "../$BUNDLE" ) || exit 1 >+ >+# blow away the checkout to save space >+rm -rf clone || exit 1 >+ >+# upload to a temporary name >+scp -i ~/.ssh/ffxbld_dsa "$BUNDLE" ffxbld@stage.mozilla.org:"/pub/mozilla.org/firefox/bundles/$BUNDLE~" || exit 1 So we need to pass in the staging/production information somehow so we can upload to staging-stage.b.m.o or stage.m.o as appropriate. Also, can we use something else other than "~" as the temporary file indication? "~" usually has special meaning.
Attachment #490306 -
Flags: review?(catlee) → review-
Comment 28•14 years ago
|
||
Comment on attachment 490303 [details] [diff] [review] hg_templates-r1.patch I'm not wild about crufting up the templates with this. How about we just link this from the MDC docs once it's live? There's already a bundle section here: https://developer.mozilla.org/En/Developer_Guide/Source_Code/Mercurial Also, we probably want to link to http://ftp.mozilla.org, not FTP.
Attachment #490303 -
Flags: review?(ted.mielczarek) → review-
Assignee | ||
Comment 29•14 years ago
|
||
(In reply to comment #26) > Actually, you probably need extra_args=[config['repo_path']] to handle repos > like releases/mozilla-1.9.2 Fixed - I'll pass both the branch name (for use in naming the bundle) and the repo path (for the hg clone). (In reply to comment #27) > So we need to pass in the staging/production information somehow so we can > upload to staging-stage.b.m.o or stage.m.o as appropriate. Also, can we use > something else other than "~" as the temporary file indication? "~" usually > has special meaning. Sound good. I learned ~ from emacs long ago, but .upload works, too. (In reply to comment #28) > I'm not wild about crufting up the templates with this. How about we just link > this from the MDC docs once it's live? There's already a bundle section here: > https://developer.mozilla.org/En/Developer_Guide/Source_Code/Mercurial > > Also, we probably want to link to http://ftp.mozilla.org, not FTP. I updated the MDN to point to the existing bundle (even if it is a bit out of date right now). I also used the http link - thanks for spotting that!
Assignee | ||
Comment 30•14 years ago
|
||
Attachment #490303 -
Attachment is obsolete: true
Attachment #490305 -
Attachment is obsolete: true
Attachment #492463 -
Flags: review?(catlee)
Assignee | ||
Comment 31•14 years ago
|
||
Both patches are tested in staging: http://staging-stage.build.mozilla.org/pub/mozilla.org/firefox/bundles/
Attachment #490306 -
Attachment is obsolete: true
Attachment #492471 -
Flags: review?(catlee)
Updated•14 years ago
|
Attachment #492463 -
Flags: review?(catlee) → review+
Updated•14 years ago
|
Attachment #492471 -
Flags: review?(catlee) → review+
Assignee | ||
Updated•14 years ago
|
Attachment #492471 -
Flags: checked-in+
Assignee | ||
Comment 32•14 years ago
|
||
I'll land the other two tomorrow when catlee's around.
Flags: needs-reconfig?
Whiteboard: [simple][oldbugs] → [simple][oldbugs][buildduty]
Comment 33•14 years ago
|
||
Comment on attachment 492463 [details] [diff] [review] buildbotcustom-r2.patch changeset: 1132:753e5b87eb6d
Attachment #492463 -
Flags: checked-in+
Comment 34•14 years ago
|
||
Comment on attachment 490304 [details] [diff] [review] buildbot-configs-r1.patch changeset: 3315:632937d89dd7
Attachment #490304 -
Flags: checked-in+
Updated•14 years ago
|
Flags: needs-reconfig? → needs-reconfig+
Assignee | ||
Updated•14 years ago
|
Status: ASSIGNED → RESOLVED
Closed: 14 years ago
Resolution: --- → FIXED
Assignee | ||
Comment 35•14 years ago
|
||
the bundle was properly uploaded over the weekend.
Status: RESOLVED → VERIFIED
Comment 36•13 years ago
|
||
(In reply to comment #30) > Created attachment 492463 [details] [diff] [review] > buildbotcustom-r2.patch Dustin, it looks like your patch here https://hg.mozilla.org/build/buildbotcustom/rev/753e5b87eb6d Has accidentally omitted coverageBuilders from the m-c generateBranchObjects and instead added it to c-c's section. (see last hunk, rather than above)
Assignee | ||
Comment 37•13 years ago
|
||
Such are the dangers of copy-pasted code..
Comment 38•13 years ago
|
||
(In reply to comment #37) > Such are the dangers of copy-pasted code.. https://hg.mozilla.org/build/buildbotcustom/rev/c57d824be391 Will get pulled into the next production-bump here.
Comment 39•13 years ago
|
||
What about doing the same for mozilla-1.9.2 and mozilla-1.9.1?
Comment 40•13 years ago
|
||
(In reply to comment #39) > What about doing the same for mozilla-1.9.2 and mozilla-1.9.1? Not worth it, IMHO.
Comment 41•13 years ago
|
||
(In reply to comment #40) > (In reply to comment #39) > > What about doing the same for mozilla-1.9.2 and mozilla-1.9.1? > > Not worth it, IMHO. Eventually, bug 644157 did it...
Blocks: 644157
Updated•11 years ago
|
Product: mozilla.org → Release Engineering
You need to log in
before you can comment on or make changes to this bug.
Description
•