Closed Bug 390479 Opened 17 years ago Closed 17 years ago

rsync module for mirrors needs to be trimmed

Categories

(Release Engineering :: General, defect, P2)

Product:
Release Engineering
Component:
General
Type:
defect

Tracking

(Not tracked)

Status:
VERIFIED FIXED

People

(Reporter: joduinn, Assigned: nthomas)

References

Details

AOL stopped serving the nightly builds because the data go to be too big.  The releases directory is now over 140G. 

Can we trim that down? Or maybe, move some of the less needed content out of the main rsync module and move it to archives?
If this is the mozilla-releases module, then the breakdown is

70G     firefox
39G     thunderbird
11G     seamonkey
7.2G    mozilla (1.7, 1.8)
4.5G    calendar
2.8G    addons
1.6G    themes
1.2G    extensions
681M    camino
192M    xulrunner
8.0K    bouncer
8.0K    zz

[this is du output, less the excluded directories off old releases]

Of which firefox breaks down as (in megabytes)

277     deerpark/
156     1.5b1
1675    1.5b2
1755    1.5rc1
1834    1.5rc2
148     1.5rc3
1368    1.5
181     1.5.0.1rc1
2026    1.5.0.1
2408    1.5.0.2
2869    1.5.0.3
3230    1.5.0.4
2570    1.5.0.5
2181    1.5.0.6
2402    1.5.0.7
2300    1.5.0.8
2333    1.5.0.9
2356    1.5.0.10
2258    1.5.0.11
2302    1.5.0.12
426     bonecho/
1972    2.0b1
2787    2.0b2
3266    2.0rc1
2640    2.0rc2
2621    2.0rc3
2462    2.0
2825    2.0.0.1
3086    2.0.0.2
2931    2.0.0.3
3235    2.0.0.4
2963    2.0.0.5
2889    2.0.0.6
633     granparadiso/

4720    partners/

And thunderbird 

154     1.5b1
1828    1.5b2
1928    1.5rc1
1985    1.5rc2
1946    1.5
1903    1.5.0.2
2458    1.5.0.4
3025    1.5.0.5
2374    1.5.0.7
2375    1.5.0.8
2426    1.5.0.9
2463    1.5.0.10
2366    1.5.0.12
2660    2.0.0.0rc1
177     2.0a1
170     2.0b1
2081    2.0b2
2580    2.0.0.0
2726    2.0.0.4
2719    2.0.0.5

Suggestion - All Firefox 1.5 except 1.5.0.12, and deerpark - 34327MB 
Hrm... I thought I sent this already... but I guess not:

I said I think we can actually archive all of Firefox 1.x, (since it's not supported anymore) and put all of the bonecho/2.0 alphas, betas, and rcs into archive.

If we want to go for more, we could also get archive some of the older Thunderbird bits.
If I'm reading the rsync config on stage properly, the quick win is to add the old versions to 
  /etc/rsyncd-mozilla-releases.exclude

I'm assuming rsync removes files from the target mirrors, or that those admins call it with arguments to do so. This would fix anyone pulling the mozilla-ftp, mozilla-releases, or (legacy) ftp module, eg releases.m.o and bouncer mirrors.

We might have to be a bit careful with 1.5.0.12, since it's still at http://www.mozilla.com/en-US/firefox/all-older.html. Perhaps it's time to revisit that. Also Bouncer should probably have some products disabled.

Plus I saw some IRC chat about moving old releases over to archive.m.o. Any ideas about how we document that change, so that testers/archaeologists can still find old releases ? With http://ftp.m.o/pub/*/releases bouncing to http://releases.m.o they're not accesssible today (unless you use ftp).
Priority: -- → P3
Yeah, adding old releases to the exclude file is the quick win.  That will remove them from all the mirrors as well.  The only way to access those directories would then be via ftp.mozilla.org rather than releases.mozilla.org.
(In reply to comment #3)

> We might have to be a bit careful with 1.5.0.12, since it's still at
> http://www.mozilla.com/en-US/firefox/all-older.html. Perhaps it's time to
> revisit that.

Yeah, we should fix that, or remember to repoint the link as part of this...

> Also Bouncer should probably have some products disabled.

Bug 390200.

> Plus I saw some IRC chat about moving old releases over to archive.m.o. Any
> ideas about how we document that change, so that testers/archaeologists can
> still find old releases ? With http://ftp.m.o/pub/*/releases bouncing to
> http://releases.m.o they're not accesssible today (unless you use ftp).

We can (and should) announce this move widely, and then move it to archive.m.o.

I believe archive is the correct place, based upon our support statements/policy for older releases.


When we're done with the stage migration, ftp.m.o and archive.m.o are going to be physically the same site anyway.  So adding them to the exclude file for the releases module needs to be done regardless.
(In reply to comment #6)
> When we're done with the stage migration, ftp.m.o and archive.m.o are going to
> be physically the same site anyway.  So adding them to the exclude file for the
> releases module needs to be done regardless.

Good to know... I had forgotten all the gotchas of the stage migration.
We should also look at what obsolete products to trim from bouncer to make sure its all consistent. For details, see tracking bug#390200
Depends on: 384031
Assignee: build → nrthomas
Priority: P3 → P2
reassigning as requested offline.
Summary: rsync module needs to be trimmed → rsync module for mirrors needs to be trimmed
Added these to rsyncd-mozilla-releases.exclude, which affects the mozilla-ftp, mozilla-releases, and ftp modules.

firefox/releases/deerpark
firefox/releases/1.5b1
firefox/releases/1.5b2
firefox/releases/1.5rc1
firefox/releases/1.5rc2
firefox/releases/1.5rc3 
firefox/releases/1.5
firefox/releases/1.5.0.1rc1
firefox/releases/1.5.0.1
firefox/releases/1.5.0.2
firefox/releases/1.5.0.3
firefox/releases/1.5.0.4
firefox/releases/1.5.0.5
firefox/releases/1.5.0.6
firefox/releases/1.5.0.7
firefox/releases/1.5.0.8
firefox/releases/1.5.0.9          
firefox/releases/1.5.0.10
firefox/releases/1.5.0.11

Module size dropped from 150GB to 116 GB. Matching bouncer changes in bug 390200.
OS: Mac OS X → All
Hardware: PC → All
(In reply to comment #5)
> (In reply to comment #3)
> 
> > We might have to be a bit careful with 1.5.0.12, since it's still at
> > http://www.mozilla.com/en-US/firefox/all-older.html. Perhaps it's time to
> > revisit that.
> 
> Yeah, we should fix that, or remember to repoint the link as part of this...
> 
Bug#394847 filed to track updating www.mozilla.com.
Depends on: 396440
Added these to the exclude list:

firefox/releases/1.5.0.12
firefox/releases/latest-1.5
firefox/releases/bonecho 
firefox/releases/2.0b1
firefox/releases/2.0b2
firefox/releases/2.0b3
firefox/releases/2.0rc1
firefox/releases/2.0rc2 
firefox/releases/2.0rc3 

Module size dropped from 114 GB to 98 GB. Bug 396440 for any release note and other website changes.
Co-ordinated with KaiRo and removed 

seamonkey/releases/1.0a
seamonkey/releases/1.0b
seamonkey/releases/1.0
seamonkey/releases/1.0.1
seamonkey/releases/1.0.2
seamonkey/releases/1.0.3
seamonkey/releases/1.0.4
seamonkey/releases/1.0.5
seamonkey/releases/1.0.6
seamonkey/releases/1.0.7
seamonkey/releases/1.0.8
seamonkey/releases/1.0.9

as that series is no longer supported. That saves a bit over 7 GB.
removed those or added them to the exclude list?
(In reply to comment #14)
> removed those or added them to the exclude list?

I meant "added to exclude list". Sorry about that. 


Added these to the exclude list

thunderbird/releases/1.5b1
thunderbird/releases/1.5b2
thunderbird/releases/1.5b1
thunderbird/releases/1.5rc1
thunderbird/releases/1.5rc2
thunderbird/releases/1.5
thunderbird/releases/1.5.0.2
thunderbird/releases/1.5.0.4
thunderbird/releases/1.5.0.5
thunderbird/releases/1.5.0.7
thunderbird/releases/1.5.0.8
thunderbird/releases/1.5.0.9
thunderbird/releases/2.0a1
thunderbird/releases/2.0b1
thunderbird/releases/2.0b2
thunderbird/releases/2.0.0.0rc1

That drops the Thunderbird contribution from 45G to 18G, and the overall size to 85G. Any objections to closing this out ?
Status: NEW → RESOLVED
Closed: 17 years ago
Resolution: --- → FIXED
Status: RESOLVED → VERIFIED
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.