Closed Bug 418661 Opened 17 years ago Closed 17 years ago

Trim mozilla-releases rsync module

Categories

(Release Engineering :: General, defect, P2)

Product:
Release Engineering
Component:
General
Type:
defect

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: nthomas, Assigned: nthomas)

Details

It's that time again - we need to trim the 120GB size of the rsync module that our major mirrors carry. (last time was bug 390479) The current breakdown is (in MB): 50274 firefox/ 24462 mozilla/ 22898 thunderbird/ 10714 calendar 7508 seamonkey 3580 addons 1174 camino 291 xulrunner 1 zz 1 bouncer All the old releases in mozilla/ needn't be carried by the mirrors any more, since demand is likely to be very low. I've already excluded them from the rsync module, and bug 418646 is to remove that dir from the module. The other obvious target is the firefox directory (MB again): 2463 2.0 2827 2.0.0.1 3088 2.0.0.2 2933 2.0.0.3 3237 2.0.0.4 2965 2.0.0.5 3112 2.0.0.6 2874 2.0.0.7 2936 2.0.0.8 2925 2.0.0.9 3013 2.0.0.10 3025 2.0.0.11 3035 2.0.0.12 1354 3.0b1 2034 3.0b2 2625 3.0b3 792 granparadiso/ 5494 partners/ We'll need to keep 2.0.0.6 for the major update from 1.5.0.12 (similarly a portion of partners/), but we can probably stop carrying everything else between 2.0 and 2.0.0.9. For thunderbird: 2465 1.5.0.10 2368 1.5.0.12 2277 1.5.0.13 2262 1.5.0.14 2582 2.0.0.0 2728 2.0.0.4 2721 2.0.0.5 2662 2.0.0.6 2828 2.0.0.9 Can probably drop all of 1.5.0.x except 1.5.0.14, and 2.0.0.0 thru 2.0.0.6. That would trim off about 67GB (including the mozilla/ stuff), and bring the size down to about 52GB. Comments ?
Status: NEW → ASSIGNED
Priority: -- → P2
Added Tb 1.5.0.10, .12, and .13, since that's obvious and we're shipping 2.0.0.12 today.
I've been thinking about this some more, and I think we only need the following for end-users: * Firefox: 1.5.0.12, 2.0.0.6, <latest_200x-1>, <latest_200x>, <latest_30y-1>, <latest_30y> * Thunderbird: 1.5.0.14, <latest_200x-1>, <latest_200x> * AMO (/addons) * other apps manage themselves if they stay small To explain FF, * <latest_200x> is the current release and gets the most traffic, <latest_200x-1> is the previous release which seems to get 10k's of daily requests after a new release (declines over time) * similarly <latest_30y> and <latest_30y -1> * 1.5.0.12 and 2.0.0.6 provide the current major update path for anyone stuck with 1.5.0.11 or older It's similar for TB, except 1.5.0.14 is the last 1.5.0.x release and we don't have a 2.0.0.x landing zone for a major update at the moment. All the other FF versions get a few hundred requests a day, for reasons that aren't obvious to me. Anyone ever looked into why ? (I used stats on the dashboard for installer downloads). We currently cover this with mirrors which don't delete old files properly, which isn't so good. Perhaps we could include a machine in bouncer which has full copy of <apps>/releases ? Maybe ftp.m.o with a really small weight, or a new host with a moderate one ? I think we could handle Fx1.5.0.12 and 2.0.0.6 this way too.
(In reply to comment #2) > We currently cover this with mirrors which don't delete old files properly, > which isn't so good. Perhaps we could include a machine in bouncer which has > full copy of <apps>/releases ? Maybe ftp.m.o with a really small weight, or a > new host with a moderate one ? I think we could handle Fx1.5.0.12 and 2.0.0.6 > this way too. The first sentence is referring to 1.5.0.12, which is excluded from the mozilla-release module at the moment. We could get the mirrors to carry it again, or add a host to bouncer. It would apply to all the removed versions if we cull it back as outlined at the top of comment #2.
Just added these to rsyncd-mozilla-releases.exclude on surf & dm-stage02: firefox/releases/2.0 firefox/releases/2.0.0.1 firefox/releases/2.0.0.2 firefox/releases/2.0.0.3 firefox/releases/2.0.0.4 firefox/releases/2.0.0.5 thunderbird/releases/2.0.0.0 thunderbird/releases/2.0.0.4 thunderbird/releases/2.0.0.5 calendar/sunbird/releases/0.2 calendar/sunbird/releases/0.3a1 calendar/sunbird/releases/0.3a2 calendar/sunbird/releases/0.3 calendar/sunbird/releases/0.3.1rc1 calendar/sunbird/releases/0.3.1rc2 calendar/sunbird/releases/0.3.1 calendar/sunbird/releases/0.5rc1 calendar/sunbird/releases/0.5rc2 calendar/sunbird/releases/0.7rc1 calendar/sunbird/releases/0.7rc2 calendar/sunbird/releases/0.7rc3 calendar/lightning/releases/0.1 calendar/lightning/releases/0.3 calendar/lightning/releases/0.3.1rc1 calendar/lightning/releases/0.3.1rc2 calendar/lightning/releases/0.3.1 calendar/lightning/releases/0.5rc1 calendar/lightning/releases/0.5rc2 calendar/lightning/releases/0.7rc1 calendar/lightning/releases/0.7rc2 calendar/lightning/releases/0.7rc3 Between Fx3.0b5 and Sunbird/Lightning 0.8rc2 we nearly blew our free space.
i plan to do sunbird/lightning 0.8 today (~3gb). are there any concerns regarding diskspace and how to check myself?
Put back the accidentally removed firefox/releases/2.0rc3 and added firefox/releases/2.0.0.7 firefox/releases/2.0.0.8 firefox/releases/2.0.0.9 firefox/releases/2.0.0.10 firefox/releases/2.0.0.11
Bug 428080 for cleaning up firefox/releases/partners
Haven't heard of any fallout from the changes above and we're down to 64GB for the module. Lets keep continue to monitor this and clean up periodically.
Status: ASSIGNED → RESOLVED
Closed: 17 years ago
Resolution: --- → FIXED
Bug 426720 Comment #2 pointed me to this bug. Where do I find the old releases now? Is there a dedicated archive server?
You can use ftp://ftp.mozilla.org/pub/mozilla.org/calendar/sunbird/releases/ The key is using ftp, as http is redirected to http://releases.m.o when the url ends in release.
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.