Closed Bug 635132 Opened 15 years ago Closed 15 years ago

Figure out how to have less mobile releases on mirrors

Categories

(Release Engineering :: General, defect)

Product:
Release Engineering
Component:
General
Platform:
x86
All
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: nthomas, Assigned: mozilla)

Details

Attachments

(1 file)

add mozilla/releases to exclude list
999 bytes, patch
nthomas
: review+
Details | Diff | Splinter Review
(this might be a dupe) We still carry every mobile release since Maemo 1.0 on the mirrors, taking up 14G of our goal to keep the module to < 110G. Probably don't serve that much traffic for mobile, at least in comparison to desktop, so this isn't good bang for (volunteered) buck. Need some help to figure out what we can exclude, in particular for the Maemo repos. Detailed breakdown, in MB: 1690 mobile/releases/1.0 1083 mobile/releases/1.0.1 2127 mobile/releases/1.0rc1 2514 mobile/releases/1.0rc2 1662 mobile/releases/1.1 905 mobile/releases/1.1b1 1662 mobile/releases/1.1rc1 647 mobile/releases/2.0a1 353 mobile/releases/4.0b1-real 371 mobile/releases/4.0b2 403 mobile/releases/4.0b3 792 mobile/releases/4.0b4 38 mobile/releases/winmo How much do we need to put on mirrors if we're pointing people towards Ovi/Android marketplaces ? Need to take care not to break our web pages too.
Hm. In a way, our Maemo repos are served off of moff.mozilla.com, so we don't need to mirror those elsewhere. How do we limit this? A whitelist? Assuming the files still live on ftp.m.o but just don't live on mirrors after this procedure: (In reply to comment #0) > Detailed breakdown, in MB: > 1690 mobile/releases/1.0 probably remove. > 1083 mobile/releases/1.0.1 probably remove. > 2127 mobile/releases/1.0rc1 probably remove. > 2514 mobile/releases/1.0rc2 probably remove. > 1662 mobile/releases/1.1 this is our latest real release, so let's keep it. > 905 mobile/releases/1.1b1 probably remove. > 1662 mobile/releases/1.1rc1 probably remove. > 647 mobile/releases/2.0a1 probably remove. > 353 mobile/releases/4.0b1-real probably remove. > 371 mobile/releases/4.0b2 probably remove. > 403 mobile/releases/4.0b3 probably remove. > 792 mobile/releases/4.0b4 this is our latest beta until 4.0b5 is out the door. let's keep. > 38 mobile/releases/winmo probably remove. Where "probably remove" means "I think we can remove these from list of mirrored directories, if we keep the files around on ftp.m.o". > How much do we need to put on mirrors if we're pointing people towards > Ovi/Android marketplaces ? Need to take care not to break our web pages too. I say probably because I don't know everything that might be accessing these. Do we only point to mirrors via bouncer?
Also, the first directories are probably huge in comparison because I copied everything including test files and the like. We can probably clean these up if we still want to serve the deb files/installers from the old releases.
To see what the mirrors get have a look at stage:/pub/mozilla.org/zz/rsyncd-mozilla-releases.exclude. The first 98% is exclusions for old releases, and then are inclusions at the very bottom. All the files remain on ftp.m.o, modulo an http redirect for some $app/releases dirs to releases.mozilla.org (which is a set of mirrors carrying the large module). mobile doesn't currently do this, but there's ftp://ftp.m.o if required. How does moff.m.c keep up to date ? (In reply to comment #1) > > How much do we need to put on mirrors if we're pointing people towards > > Ovi/Android marketplaces ? Need to take care not to break our web pages too. > > I say probably because I don't know everything that might be accessing these. > Do we only point to mirrors via bouncer? I'm not familiar with the web pages for mobile, but taking http://www.mozilla.com/en-US/m/ as an example there's a link to bouncer for maemo, rather than anything to releases.m.o. Android goes to a market:// URL.
(In reply to comment #3) > To see what the mirrors get have a look at > stage:/pub/mozilla.org/zz/rsyncd-mozilla-releases.exclude. The first 98% is > exclusions for old releases, and then are inclusions at the very bottom. How is this file versioned? I'm afraid I already know the answer, but asking anyway. (Oh, answered in IRC; I was right =P ) > How does moff.m.c keep up to date ? I believe it's an rsync of mobile/releases/ to moff webroot. > I'm not familiar with the web pages for mobile, but taking > http://www.mozilla.com/en-US/m/ as an example there's a link to bouncer for > maemo, rather than anything to releases.m.o. Android goes to a market:// URL. I would hazard a guess that we don't actually need anything on the mirrors for mobile. I'd probably add all the "probably remove" release directories to the exclude list. Re: versioning, should we resort to something as simple as rcs? Alternately starting a repo with a bunch of flatfiles isn't an awful idea.
(In reply to comment #4) > How is this file versioned? > I'm afraid I already know the answer, but asking anyway. > (Oh, answered in IRC; I was right =P ) > Re: versioning, should we resort to something as simple as rcs? Alternately > starting a repo with a bunch of flatfiles isn't an awful idea. Yeah, it's not. We could do that pretty easily, stage has hg on it already. > > How does moff.m.c keep up to date ? > I believe it's an rsync of mobile/releases/ to moff webroot. Really depends what the details of that are. Something using mozilla-releases would be a problem. > > I'm not familiar with the web pages for mobile, but taking > > http://www.mozilla.com/en-US/m/ as an example there's a link to bouncer for > > maemo, rather than anything to releases.m.o. Android goes to a market:// URL. > > I would hazard a guess that we don't actually need anything on the mirrors for > mobile. > > I'd probably add all the "probably remove" release directories to the exclude > list. How can we be confident enough to *zot* all those probably's ?
Assignee: nobody → aki
(In reply to comment #5) > > > How does moff.m.c keep up to date ? > > I believe it's an rsync of mobile/releases/ to moff webroot. > > Really depends what the details of that are. Something using mozilla-releases > would be a problem. Justdave: do you have an answer here re: moff.m.c rsyncs?
The cron job does this: rsync -aq --timeout=900 --delete --delete-before stage.mozilla.org::mozilla-all/mobile/releases/ /data/www/moff.mozilla.com
the mozilla-all module has access to *everything* and doesn't use an exclude list, so adding the older mobile releases to the exclude list for releases won't prevent them from getting to moff.
Attachment #514658 - Flags: review?(nrthomas)
(In reply to comment #9) > Created attachment 514658 [details] [diff] [review] > add mozilla/releases to exclude list You mean mobile/releases (says so in the patch, just assuring the people who haven't looked at it ;)
Comment on attachment 514658 [details] [diff] [review] add mozilla/releases to exclude list >diff -U9 -r1.1 rsyncd-mozilla-releases.exclude >+- mobile/releases/4.0b2 >+- mobile/releases/4.0b1-real >+- mobile/releases/4.0b3 Looks good to me. I'd ask that the b1-real come before b2, and that rsyncd-mozilla-prereleases.exclude gets the same treatment.
Attachment #514658 - Flags: review?(nrthomas) → review+
Done.
Status: NEW → RESOLVED
Closed: 15 years ago
Resolution: --- → FIXED
Thanks a lot aki!
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: