Closed Bug 390479 Opened 18 years ago Closed 18 years ago

rsync module for mirrors needs to be trimmed

Categories

(Release Engineering :: General, defect, P2)

Product:
Release Engineering
Component:
General
Type:
defect

Tracking

(Not tracked)

Status:
VERIFIED FIXED

People

(Reporter: joduinn, Assigned: nthomas)

References

Details

AOL stopped serving the nightly builds because the data go to be too big. The releases directory is now over 140G. Can we trim that down? Or maybe, move some of the less needed content out of the main rsync module and move it to archives?
If this is the mozilla-releases module, then the breakdown is 70G firefox 39G thunderbird 11G seamonkey 7.2G mozilla (1.7, 1.8) 4.5G calendar 2.8G addons 1.6G themes 1.2G extensions 681M camino 192M xulrunner 8.0K bouncer 8.0K zz [this is du output, less the excluded directories off old releases] Of which firefox breaks down as (in megabytes) 277 deerpark/ 156 1.5b1 1675 1.5b2 1755 1.5rc1 1834 1.5rc2 148 1.5rc3 1368 1.5 181 1.5.0.1rc1 2026 1.5.0.1 2408 1.5.0.2 2869 1.5.0.3 3230 1.5.0.4 2570 1.5.0.5 2181 1.5.0.6 2402 1.5.0.7 2300 1.5.0.8 2333 1.5.0.9 2356 1.5.0.10 2258 1.5.0.11 2302 1.5.0.12 426 bonecho/ 1972 2.0b1 2787 2.0b2 3266 2.0rc1 2640 2.0rc2 2621 2.0rc3 2462 2.0 2825 2.0.0.1 3086 2.0.0.2 2931 2.0.0.3 3235 2.0.0.4 2963 2.0.0.5 2889 2.0.0.6 633 granparadiso/ 4720 partners/ And thunderbird 154 1.5b1 1828 1.5b2 1928 1.5rc1 1985 1.5rc2 1946 1.5 1903 1.5.0.2 2458 1.5.0.4 3025 1.5.0.5 2374 1.5.0.7 2375 1.5.0.8 2426 1.5.0.9 2463 1.5.0.10 2366 1.5.0.12 2660 2.0.0.0rc1 177 2.0a1 170 2.0b1 2081 2.0b2 2580 2.0.0.0 2726 2.0.0.4 2719 2.0.0.5 Suggestion - All Firefox 1.5 except 1.5.0.12, and deerpark - 34327MB
Hrm... I thought I sent this already... but I guess not: I said I think we can actually archive all of Firefox 1.x, (since it's not supported anymore) and put all of the bonecho/2.0 alphas, betas, and rcs into archive. If we want to go for more, we could also get archive some of the older Thunderbird bits.
If I'm reading the rsync config on stage properly, the quick win is to add the old versions to /etc/rsyncd-mozilla-releases.exclude I'm assuming rsync removes files from the target mirrors, or that those admins call it with arguments to do so. This would fix anyone pulling the mozilla-ftp, mozilla-releases, or (legacy) ftp module, eg releases.m.o and bouncer mirrors. We might have to be a bit careful with 1.5.0.12, since it's still at http://www.mozilla.com/en-US/firefox/all-older.html. Perhaps it's time to revisit that. Also Bouncer should probably have some products disabled. Plus I saw some IRC chat about moving old releases over to archive.m.o. Any ideas about how we document that change, so that testers/archaeologists can still find old releases ? With http://ftp.m.o/pub/*/releases bouncing to http://releases.m.o they're not accesssible today (unless you use ftp).
Priority: -- → P3
Yeah, adding old releases to the exclude file is the quick win. That will remove them from all the mirrors as well. The only way to access those directories would then be via ftp.mozilla.org rather than releases.mozilla.org.
(In reply to comment #3) > We might have to be a bit careful with 1.5.0.12, since it's still at > http://www.mozilla.com/en-US/firefox/all-older.html. Perhaps it's time to > revisit that. Yeah, we should fix that, or remember to repoint the link as part of this... > Also Bouncer should probably have some products disabled. Bug 390200. > Plus I saw some IRC chat about moving old releases over to archive.m.o. Any > ideas about how we document that change, so that testers/archaeologists can > still find old releases ? With http://ftp.m.o/pub/*/releases bouncing to > http://releases.m.o they're not accesssible today (unless you use ftp). We can (and should) announce this move widely, and then move it to archive.m.o. I believe archive is the correct place, based upon our support statements/policy for older releases.
When we're done with the stage migration, ftp.m.o and archive.m.o are going to be physically the same site anyway. So adding them to the exclude file for the releases module needs to be done regardless.
(In reply to comment #6) > When we're done with the stage migration, ftp.m.o and archive.m.o are going to > be physically the same site anyway. So adding them to the exclude file for the > releases module needs to be done regardless. Good to know... I had forgotten all the gotchas of the stage migration.
We should also look at what obsolete products to trim from bouncer to make sure its all consistent. For details, see tracking bug#390200
Depends on: 384031
Assignee: build → nrthomas
Priority: P3 → P2
reassigning as requested offline.
Summary: rsync module needs to be trimmed → rsync module for mirrors needs to be trimmed
Added these to rsyncd-mozilla-releases.exclude, which affects the mozilla-ftp, mozilla-releases, and ftp modules. firefox/releases/deerpark firefox/releases/1.5b1 firefox/releases/1.5b2 firefox/releases/1.5rc1 firefox/releases/1.5rc2 firefox/releases/1.5rc3 firefox/releases/1.5 firefox/releases/1.5.0.1rc1 firefox/releases/1.5.0.1 firefox/releases/1.5.0.2 firefox/releases/1.5.0.3 firefox/releases/1.5.0.4 firefox/releases/1.5.0.5 firefox/releases/1.5.0.6 firefox/releases/1.5.0.7 firefox/releases/1.5.0.8 firefox/releases/1.5.0.9 firefox/releases/1.5.0.10 firefox/releases/1.5.0.11 Module size dropped from 150GB to 116 GB. Matching bouncer changes in bug 390200.
OS: Mac OS X → All
Hardware: PC → All
(In reply to comment #5) > (In reply to comment #3) > > > We might have to be a bit careful with 1.5.0.12, since it's still at > > http://www.mozilla.com/en-US/firefox/all-older.html. Perhaps it's time to > > revisit that. > > Yeah, we should fix that, or remember to repoint the link as part of this... > Bug#394847 filed to track updating www.mozilla.com.
Depends on: 396440
Added these to the exclude list: firefox/releases/1.5.0.12 firefox/releases/latest-1.5 firefox/releases/bonecho firefox/releases/2.0b1 firefox/releases/2.0b2 firefox/releases/2.0b3 firefox/releases/2.0rc1 firefox/releases/2.0rc2 firefox/releases/2.0rc3 Module size dropped from 114 GB to 98 GB. Bug 396440 for any release note and other website changes.
Co-ordinated with KaiRo and removed seamonkey/releases/1.0a seamonkey/releases/1.0b seamonkey/releases/1.0 seamonkey/releases/1.0.1 seamonkey/releases/1.0.2 seamonkey/releases/1.0.3 seamonkey/releases/1.0.4 seamonkey/releases/1.0.5 seamonkey/releases/1.0.6 seamonkey/releases/1.0.7 seamonkey/releases/1.0.8 seamonkey/releases/1.0.9 as that series is no longer supported. That saves a bit over 7 GB.
removed those or added them to the exclude list?
(In reply to comment #14) > removed those or added them to the exclude list? I meant "added to exclude list". Sorry about that.
Added these to the exclude list thunderbird/releases/1.5b1 thunderbird/releases/1.5b2 thunderbird/releases/1.5b1 thunderbird/releases/1.5rc1 thunderbird/releases/1.5rc2 thunderbird/releases/1.5 thunderbird/releases/1.5.0.2 thunderbird/releases/1.5.0.4 thunderbird/releases/1.5.0.5 thunderbird/releases/1.5.0.7 thunderbird/releases/1.5.0.8 thunderbird/releases/1.5.0.9 thunderbird/releases/2.0a1 thunderbird/releases/2.0b1 thunderbird/releases/2.0b2 thunderbird/releases/2.0.0.0rc1 That drops the Thunderbird contribution from 45G to 18G, and the overall size to 85G. Any objections to closing this out ?
Status: NEW → RESOLVED
Closed: 18 years ago
Resolution: --- → FIXED
Status: RESOLVED → VERIFIED
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.