Closed Bug 635132 Opened 11 years ago Closed 11 years ago

Figure out how to have less mobile releases on mirrors

Categories

(Release Engineering :: General, defect)

x86
All
defect
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: nthomas, Assigned: aki)

Details

Attachments

(1 file)

(this might be a dupe)

We still carry every mobile release since Maemo 1.0 on the mirrors, taking up 14G of our goal to keep the module to < 110G. Probably don't serve that much traffic for mobile, at least in comparison to desktop, so this isn't good bang for (volunteered) buck. Need some help to figure out what we can exclude, in particular for the Maemo repos.

Detailed breakdown, in MB:
1690	mobile/releases/1.0
1083	mobile/releases/1.0.1
2127	mobile/releases/1.0rc1
2514	mobile/releases/1.0rc2
1662	mobile/releases/1.1
 905	mobile/releases/1.1b1
1662	mobile/releases/1.1rc1
 647	mobile/releases/2.0a1
 353	mobile/releases/4.0b1-real
 371	mobile/releases/4.0b2
 403	mobile/releases/4.0b3
 792	mobile/releases/4.0b4
  38	mobile/releases/winmo

How much do we need to put on mirrors if we're pointing people towards Ovi/Android marketplaces ? Need to take care not to break our web pages too.
Hm.

In a way, our Maemo repos are served off of moff.mozilla.com, so we don't need to mirror those elsewhere.

How do we limit this? A whitelist?
Assuming the files still live on ftp.m.o but just don't live on mirrors after this procedure:

(In reply to comment #0)
> Detailed breakdown, in MB:
> 1690    mobile/releases/1.0
probably remove.

> 1083    mobile/releases/1.0.1
probably remove.

> 2127    mobile/releases/1.0rc1
probably remove.

> 2514    mobile/releases/1.0rc2
probably remove.

> 1662    mobile/releases/1.1
this is our latest real release, so let's keep it.

>  905    mobile/releases/1.1b1
probably remove.

> 1662    mobile/releases/1.1rc1
probably remove.

>  647    mobile/releases/2.0a1
probably remove.

>  353    mobile/releases/4.0b1-real
probably remove.

>  371    mobile/releases/4.0b2
probably remove.

>  403    mobile/releases/4.0b3
probably remove.

>  792    mobile/releases/4.0b4
this is our latest beta until 4.0b5 is out the door. let's keep.

>   38    mobile/releases/winmo
probably remove.

Where "probably remove" means "I think we can remove these from list of mirrored directories, if we keep the files around on ftp.m.o".

> How much do we need to put on mirrors if we're pointing people towards
> Ovi/Android marketplaces ? Need to take care not to break our web pages too.

I say probably because I don't know everything that might be accessing these.  Do we only point to mirrors via bouncer?
Also, the first directories are probably huge in comparison because I copied everything including test files and the like.  We can probably clean these up if we still want to serve the deb files/installers from the old releases.
To see what the mirrors get have a look at stage:/pub/mozilla.org/zz/rsyncd-mozilla-releases.exclude. The first 98% is exclusions for old releases, and then are inclusions at the very bottom. All the files remain on ftp.m.o, modulo an http redirect for some $app/releases dirs to releases.mozilla.org (which is a set of mirrors carrying the large module). mobile doesn't currently do this, but there's ftp://ftp.m.o if required.

How does moff.m.c keep up to date ?

(In reply to comment #1)
> > How much do we need to put on mirrors if we're pointing people towards
> > Ovi/Android marketplaces ? Need to take care not to break our web pages too.
> 
> I say probably because I don't know everything that might be accessing these. 
> Do we only point to mirrors via bouncer?

I'm not familiar with the web pages for mobile, but taking http://www.mozilla.com/en-US/m/ as an example there's a link to bouncer for maemo, rather than anything to releases.m.o. Android goes to a market:// URL.
(In reply to comment #3)
> To see what the mirrors get have a look at
> stage:/pub/mozilla.org/zz/rsyncd-mozilla-releases.exclude. The first 98% is
> exclusions for old releases, and then are inclusions at the very bottom.

How is this file versioned?
I'm afraid I already know the answer, but asking anyway.
(Oh, answered in IRC; I was right =P )

> How does moff.m.c keep up to date ?

I believe it's an rsync of mobile/releases/ to moff webroot.

> I'm not familiar with the web pages for mobile, but taking
> http://www.mozilla.com/en-US/m/ as an example there's a link to bouncer for
> maemo, rather than anything to releases.m.o. Android goes to a market:// URL.

I would hazard a guess that we don't actually need anything on the mirrors for mobile.

I'd probably add all the "probably remove" release directories to the exclude list.

Re: versioning, should we resort to something as simple as rcs?  Alternately starting a repo with a bunch of flatfiles isn't an awful idea.
(In reply to comment #4)
> How is this file versioned?
> I'm afraid I already know the answer, but asking anyway.
> (Oh, answered in IRC; I was right =P )

> Re: versioning, should we resort to something as simple as rcs?  Alternately
> starting a repo with a bunch of flatfiles isn't an awful idea.

Yeah, it's not. We could do that pretty easily, stage has hg on it already.
 
> > How does moff.m.c keep up to date ?
> I believe it's an rsync of mobile/releases/ to moff webroot.

Really depends what the details of that are. Something using mozilla-releases would be a problem.
 
> > I'm not familiar with the web pages for mobile, but taking
> > http://www.mozilla.com/en-US/m/ as an example there's a link to bouncer for
> > maemo, rather than anything to releases.m.o. Android goes to a market:// URL.
> 
> I would hazard a guess that we don't actually need anything on the mirrors for
> mobile.
> 
> I'd probably add all the "probably remove" release directories to the exclude
> list.

How can we be confident enough to *zot* all those probably's ?
Assignee: nobody → aki
(In reply to comment #5)
> > > How does moff.m.c keep up to date ?
> > I believe it's an rsync of mobile/releases/ to moff webroot.
> 
> Really depends what the details of that are. Something using mozilla-releases
> would be a problem.

Justdave: do you have an answer here re: moff.m.c rsyncs?
The cron job does this:

rsync -aq --timeout=900 --delete --delete-before stage.mozilla.org::mozilla-all/mobile/releases/ /data/www/moff.mozilla.com
the mozilla-all module has access to *everything* and doesn't use an exclude list, so adding the older mobile releases to the exclude list for releases won't prevent them from getting to moff.
(In reply to comment #9)
> Created attachment 514658 [details] [diff] [review]
> add mozilla/releases to exclude list

You mean mobile/releases (says so in the patch, just assuring the people who haven't looked at it ;)
Comment on attachment 514658 [details] [diff] [review]
add mozilla/releases to exclude list

>diff -U9 -r1.1 rsyncd-mozilla-releases.exclude
>+- mobile/releases/4.0b2
>+- mobile/releases/4.0b1-real
>+- mobile/releases/4.0b3

Looks good to me. I'd ask that the b1-real come before b2, and that rsyncd-mozilla-prereleases.exclude gets the same treatment.
Attachment #514658 - Flags: review?(nrthomas) → review+
Done.
Status: NEW → RESOLVED
Closed: 11 years ago
Resolution: --- → FIXED
Thanks a lot aki!
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.