332627 - Reduce size of the ftp.mozilla.org mirror module

Status: RESOLVED WONTFIX

(mozilla.org :: FTP: Mirrors, task, P2)

Description

[J. Paul Reed [:preed]]

I've heard a fair amount of rumbling about how big the mozilla.org FTP mirror module is, i.e. size of the bits of ftp.m.o.

I'm planning on moving the following releases to archive.mozilla.org. I will replace the directories with a README pointing people to archive.

pub/mozilla.org/firebird/:
 - everything

pub/mozilla.org/firefox/releases/
 - any alpha, beta, or RC directory (1.5.0.1rc1, 1.5b1, etc.)
 - Any release < 1.0

pub/mozilla.org/grendel/
 - grendal

pub/mozilla.org/mozilla/
 - any alpha, beta, or RC directory (mozilla1.7rc1, etc)
 - The point releases < 1.7 that aren't the latest in their series, so KEEP mozilla-1.4.4, but move to archive mozilla-1.4.*

pub/mozilla.org/phoenix
 - everything

pub/mozilla.org/webtools/archived/
 - everything

pub/mozilla.org/thunderbird/releases/
 - any alpha, beta, or RC directory (1.5.0.1rc1, 1.5b1, etc.)
 - Any release < 1.0

To reiterate, NO BITS WILL BE DESTROYED. Just moved to archive.mozilla.org, instead of in the mirrored-around-the-world FTP module.

This will make our ftp mirrors much happier.

Comment 1

[J. Paul Reed [:preed]]

Part of this bug should be making http://archive.mozilla.org/ either look like http://archive.mozilla.org/pub, or making it forward to /pub, or something.

Having archive.m.o bring up the default RHEL page telling people that we need to put up a website is confusing.

Comment 2

[Robert Kaiser]

actually, resolving the missing parts of bug 318510 (mainly within seamonkey/nightly/) would shrink the size of ftp.mozilla.org mirror mosdule a lot, I guess...

Comment 3

[Zach Lipton [:zach]]

See also Asa's bug 329863 addressing the archving of old tinderbox builds to archive.m.o. This really should be automated. Better yet, do we really need to archive tinderbox builds, or can they just be clobbered after a short time? 

Comment 4

[Vladimir Vukicevic [:vlad] [:vladv] (needinfo me, slow to respond)]

(In reply to comment #3)
> See also Asa's bug 329863 addressing the archving of old tinderbox builds to
> archive.m.o. This really should be automated. Better yet, do we really need to
> archive tinderbox builds, or can they just be clobbered after a short time? 

They're useful for people trying to find a regression range for a bug, so it would be good if they were still available.  No need to have them mirrored, though.  I would like to see a tiered expiration though -- instead of just the last N builds, maybe last 20, then 1 per month for the previous 12-24 months.  (Or maybe 1 every 2 weeks for the past year.)

Comment 5

[Zach Lipton [:zach]]

(In reply to comment #4)
> They're useful for people trying to find a regression range for a bug, so it
> would be good if they were still available.

I'm refering to the tinderbox builds (in the tinderbox-builds/) directory. The nightlies would of course still be archived to archive.m.o. Are the tinderbox builds really useful when they are older then a couple of weeks?

Comment 6

[Boris Zbarsky [:bzbarsky]]

> Are the tinderbox builds really useful when they are older then a couple of
> weeks?

Depends on how many checkins happened on the day in question, generally speaking.  Usually a 24-hour range (from nightlies) is good enough, but some days being able to narrow from that really helps...

Comment 7

[Adam Hauner]

There are probably many links on w.m.o pages pointing to files, which will be moved. Maybe you should also fill bug to update such links. Similar for Mozillazine.

Comment 8

[Robert Kaiser]

tinderbox-builds/ and tinderbox/ should actually only have the latest builds from the respective tinderbox, all others should be cleared up. Perhaps it would be a good idea to patch tinderbox itself to care about that.

We may really have a bigger list of (old) releases that are linked from www.m.o and other domains and it would need a bigger amount of work to find and correct all those links.

I'm quite sure though that completely resolving bug 318510 could derease the mirror size more than the proposed changes proposed in comment #0 - that doesn't mean those are not worth persuing, but I think the low-hanging fruit should be cared about first.

Comment 9

[Ian Tommins ('thelem')]

I assume you are planning to leave the latest alpha/beta/rc so long as there is not a newer release based on it. At the moment this would apply to the pub/mozilla.org/firefox/bonecho folder.

Comment 10

[J. Paul Reed [:preed]]

(In reply to comment #9)
> I assume you are planning to leave the latest alpha/beta/rc so long as there is
> not a newer release based on it. At the moment this would apply to the
> pub/mozilla.org/firefox/bonecho folder.

Correct.

Comment 11

[Axel Hecht [:Pike]]

I nominate the latest-l10n in nightlies for old releases, too.
http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla1.8-1.5rc3-l10n/
is an example.

And we should only push those l10n bits into the release dirs that we actually
released, instead of rsynch'ing the complete candidates dir.

Comment 12

[J. Paul Reed [:preed]]

Reassigning bugs I'm not actively working on back into the triage pool.

Comment 13

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

reassigning as requested offline.

Comment 14

[Nick Thomas [:nthomas] (UTC+12)]

Is this bug still valid ? AFAIK we don't have any external consumers of the mozilla-all module used for ftp.m.o.

Comment 15

[Dave Miller [:justdave]]

(In reply to comment #14)
> Is this bug still valid ? AFAIK we don't have any external consumers of the
> mozilla-all module used for ftp.m.o.

We're not talking about mozilla-all, we're talking about mozilla-releases.  The mozilla-releases module in and of itself is huge these days.

[root@surf ~]# cat /etc/rsyncd.conf-motd 
[...]
       Current module sizes (updated 2007-08-29)

                           mozilla-ftp: 584GB
                      mozilla-releases: 114GB
                       mozilla-current: 7.8GB

Most mirrors like it if it stays around 80 or 90 GB or so.

Comment 16

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to comment #15)

Hmm, I think the motd is slightly out of date, because the mozilla-ftp module seems to have identical configuration to mozilla-releases (and also "ftp", I'm looking on surf). It makes sense if you mentally s/mozilla-ftp/mozilla-all/. (How's the motd generated btw ?)

Please correct me if I'm wrong, but it looks like this bug dates back to when mirrors like aol, osuosl, tds etc carried nightlies as well as releases. The suggested changes in comment #0 certainly apply to ftp.m.o as it is today, while  some of the mirrors I mentioned are part of the releases.m.o round robin.

Bug 390479 has the recent changes to the mozilla-releases that brought it down from 150GB. Perhaps that's not in the right Bugzilla component for best visibility.

Comment 17

[Nick Thomas [:nthomas] (UTC+12)]

I talked with Dave about this, and he agrees that the size of this module only matters when manna (cname'd to ftp.m.o) runs out of disk space. That issue is also slated to go away with the stage migration. Resolving WONTFIX.

Comment 18

[Dave Miller [:justdave]]

Note that we are still having issues with the releases module, but that's bug 390479.