Closed Bug 390479 Opened 13 years ago Closed 13 years ago
rsync module for mirrors needs to be trimmed
AOL stopped serving the nightly builds because the data go to be too big. The releases directory is now over 140G. Can we trim that down? Or maybe, move some of the less needed content out of the main rsync module and move it to archives?
If this is the mozilla-releases module, then the breakdown is 70G firefox 39G thunderbird 11G seamonkey 7.2G mozilla (1.7, 1.8) 4.5G calendar 2.8G addons 1.6G themes 1.2G extensions 681M camino 192M xulrunner 8.0K bouncer 8.0K zz [this is du output, less the excluded directories off old releases] Of which firefox breaks down as (in megabytes) 277 deerpark/ 156 1.5b1 1675 1.5b2 1755 1.5rc1 1834 1.5rc2 148 1.5rc3 1368 1.5 181 220.127.116.11rc1 2026 18.104.22.168 2408 22.214.171.124 2869 126.96.36.199 3230 188.8.131.52 2570 184.108.40.206 2181 220.127.116.11 2402 18.104.22.168 2300 22.214.171.124 2333 126.96.36.199 2356 188.8.131.52 2258 184.108.40.206 2302 220.127.116.11 426 bonecho/ 1972 2.0b1 2787 2.0b2 3266 2.0rc1 2640 2.0rc2 2621 2.0rc3 2462 2.0 2825 18.104.22.168 3086 22.214.171.124 2931 126.96.36.199 3235 188.8.131.52 2963 184.108.40.206 2889 220.127.116.11 633 granparadiso/ 4720 partners/ And thunderbird 154 1.5b1 1828 1.5b2 1928 1.5rc1 1985 1.5rc2 1946 1.5 1903 18.104.22.168 2458 22.214.171.124 3025 126.96.36.199 2374 188.8.131.52 2375 184.108.40.206 2426 220.127.116.11 2463 18.104.22.168 2366 22.214.171.124 2660 126.96.36.199rc1 177 2.0a1 170 2.0b1 2081 2.0b2 2580 188.8.131.52 2726 184.108.40.206 2719 220.127.116.11 Suggestion - All Firefox 1.5 except 18.104.22.168, and deerpark - 34327MB
Hrm... I thought I sent this already... but I guess not: I said I think we can actually archive all of Firefox 1.x, (since it's not supported anymore) and put all of the bonecho/2.0 alphas, betas, and rcs into archive. If we want to go for more, we could also get archive some of the older Thunderbird bits.
If I'm reading the rsync config on stage properly, the quick win is to add the old versions to /etc/rsyncd-mozilla-releases.exclude I'm assuming rsync removes files from the target mirrors, or that those admins call it with arguments to do so. This would fix anyone pulling the mozilla-ftp, mozilla-releases, or (legacy) ftp module, eg releases.m.o and bouncer mirrors. We might have to be a bit careful with 22.214.171.124, since it's still at http://www.mozilla.com/en-US/firefox/all-older.html. Perhaps it's time to revisit that. Also Bouncer should probably have some products disabled. Plus I saw some IRC chat about moving old releases over to archive.m.o. Any ideas about how we document that change, so that testers/archaeologists can still find old releases ? With http://ftp.m.o/pub/*/releases bouncing to http://releases.m.o they're not accesssible today (unless you use ftp).
13 years ago
Priority: -- → P3
Yeah, adding old releases to the exclude file is the quick win. That will remove them from all the mirrors as well. The only way to access those directories would then be via ftp.mozilla.org rather than releases.mozilla.org.
(In reply to comment #3) > We might have to be a bit careful with 126.96.36.199, since it's still at > http://www.mozilla.com/en-US/firefox/all-older.html. Perhaps it's time to > revisit that. Yeah, we should fix that, or remember to repoint the link as part of this... > Also Bouncer should probably have some products disabled. Bug 390200. > Plus I saw some IRC chat about moving old releases over to archive.m.o. Any > ideas about how we document that change, so that testers/archaeologists can > still find old releases ? With http://ftp.m.o/pub/*/releases bouncing to > http://releases.m.o they're not accesssible today (unless you use ftp). We can (and should) announce this move widely, and then move it to archive.m.o. I believe archive is the correct place, based upon our support statements/policy for older releases.
When we're done with the stage migration, ftp.m.o and archive.m.o are going to be physically the same site anyway. So adding them to the exclude file for the releases module needs to be done regardless.
(In reply to comment #6) > When we're done with the stage migration, ftp.m.o and archive.m.o are going to > be physically the same site anyway. So adding them to the exclude file for the > releases module needs to be done regardless. Good to know... I had forgotten all the gotchas of the stage migration.
We should also look at what obsolete products to trim from bouncer to make sure its all consistent. For details, see tracking bug#390200
13 years ago
Assignee: build → nrthomas
Priority: P3 → P2
reassigning as requested offline.
Summary: rsync module needs to be trimmed → rsync module for mirrors needs to be trimmed
Added these to rsyncd-mozilla-releases.exclude, which affects the mozilla-ftp, mozilla-releases, and ftp modules. firefox/releases/deerpark firefox/releases/1.5b1 firefox/releases/1.5b2 firefox/releases/1.5rc1 firefox/releases/1.5rc2 firefox/releases/1.5rc3 firefox/releases/1.5 firefox/releases/188.8.131.52rc1 firefox/releases/184.108.40.206 firefox/releases/220.127.116.11 firefox/releases/18.104.22.168 firefox/releases/22.214.171.124 firefox/releases/126.96.36.199 firefox/releases/188.8.131.52 firefox/releases/184.108.40.206 firefox/releases/220.127.116.11 firefox/releases/18.104.22.168 firefox/releases/22.214.171.124 firefox/releases/126.96.36.199 Module size dropped from 150GB to 116 GB. Matching bouncer changes in bug 390200.
OS: Mac OS X → All
Hardware: PC → All
(In reply to comment #5) > (In reply to comment #3) > > > We might have to be a bit careful with 188.8.131.52, since it's still at > > http://www.mozilla.com/en-US/firefox/all-older.html. Perhaps it's time to > > revisit that. > > Yeah, we should fix that, or remember to repoint the link as part of this... > Bug#394847 filed to track updating www.mozilla.com.
Added these to the exclude list: firefox/releases/184.108.40.206 firefox/releases/latest-1.5 firefox/releases/bonecho firefox/releases/2.0b1 firefox/releases/2.0b2 firefox/releases/2.0b3 firefox/releases/2.0rc1 firefox/releases/2.0rc2 firefox/releases/2.0rc3 Module size dropped from 114 GB to 98 GB. Bug 396440 for any release note and other website changes.
Co-ordinated with KaiRo and removed seamonkey/releases/1.0a seamonkey/releases/1.0b seamonkey/releases/1.0 seamonkey/releases/1.0.1 seamonkey/releases/1.0.2 seamonkey/releases/1.0.3 seamonkey/releases/1.0.4 seamonkey/releases/1.0.5 seamonkey/releases/1.0.6 seamonkey/releases/1.0.7 seamonkey/releases/1.0.8 seamonkey/releases/1.0.9 as that series is no longer supported. That saves a bit over 7 GB.
removed those or added them to the exclude list?
(In reply to comment #14) > removed those or added them to the exclude list? I meant "added to exclude list". Sorry about that.
Added these to the exclude list thunderbird/releases/1.5b1 thunderbird/releases/1.5b2 thunderbird/releases/1.5b1 thunderbird/releases/1.5rc1 thunderbird/releases/1.5rc2 thunderbird/releases/1.5 thunderbird/releases/220.127.116.11 thunderbird/releases/18.104.22.168 thunderbird/releases/22.214.171.124 thunderbird/releases/126.96.36.199 thunderbird/releases/188.8.131.52 thunderbird/releases/184.108.40.206 thunderbird/releases/2.0a1 thunderbird/releases/2.0b1 thunderbird/releases/2.0b2 thunderbird/releases/220.127.116.11rc1 That drops the Thunderbird contribution from 45G to 18G, and the overall size to 85G. Any objections to closing this out ?
Status: NEW → RESOLVED
Closed: 13 years ago
Resolution: --- → FIXED
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.