We've progressively disabled the volunteer mirror network by * swapping all downloads to our meta-CDN (download.cdn.mozilla.net, many folks) * disabled all non-CDN mirrors in download.m.o (bouncer, jakem) * deprecated the internal mirrors (pv-mirror01/02), and moved rsync hosting in house (digi) * swapped AMO downloads to akamai (bug 806615, oremj) At this point I'm not aware of anything still using the mozilla-releases module, the contents of which you can see at http://releases.mozilla.org/pub/mozilla.org/ However we're still carrying these burdens * volunteer mirrors are still rsyncing new releases (which annoys IT by causing nagios alerts that need no action, bug 765623) * RelEng need to periodically trim the module so the disk usage stays sane for the above (eg bug 791205) * RelEng need to periodically trim the bouncer products with the Check Now bitset, so that we don't check too many things every 5 mins in sentry This bug is to address the first two of those, by * emptying the rsync module * pointing releases.m.o at the CDN in DNS * removing a bunch of nagios alerts Alex, do you have any objections to this ? It's effectively the last nail in the coffin for falling back to the volunteer mirrors if we have CDN issues. At this point we haven't seen any failure modes that say that'd be a bad idea.
(In reply to Nick Thomas [:nthomas] from comment #0) > Alex, do you have any objections to this ? It's effectively the last nail in > the coffin for falling back to the volunteer mirrors if we have CDN issues. > At this point we haven't seen any failure modes that say that'd be a bad > idea. Given the CDN uptime that we can expect, and bhearsum's assertion in yesterday's channel meeting that external mirrors already aren't being used as a failover, I don't have a problem with hammering that nail in.
Given recent discussion on the mirror's list, what does IT have to say about this proposal ? We could leave mozilla-current alive (and up to date!) if there are external parties that still want to sync.
I suspect there may be some interest from mirror admins to running local mirrors even if Mozilla doesn't use them... however, I'm not sure how widespread this would be, and if we do want to support it I'm not sure what it should look like. Perhaps, for instance, it could be greatly simplified, and just direct folks to the FTP cluster rsync, and let them set up their own excludes. Since it's not for our usage, one could argue we don't need to provide predetermined exclude lists.
Ok. Can we protect ourselves from people accidentally syncing 'the world' (TM) ?
Not easily, as far as I'm aware. We can have the server-side only allow certain things to be served up, but we can't really prevent someone from requesting everything we make available... at least not automatically (we can obviously block IPs on demand as usual). Handling of accidental or intentional abuse would need to be part of a larger discussion/decision around what (if anything) should still be available via rsync. I'm not sure I should have a voice in that. My 2c: I don't *mind* making rsync access available for those who want it, but I don't want it to be a burden on maintenance (which includes any manual abuse handling, extra server setups, running our own local mirror, etc). Without knowing what kind of demand there would be for this, I couldn't make any value judgements as to how much time or effort its worth to streamline this vs just removing it entirely. I believe the current scheme is far to complicated for what we need nowadays, so we should definitely do *something*.
Found during triage. nthomas, nmaul: Given our CDN and now S3 experiences, I think we're ok... so question is: whats left to do here?
(In reply to Nick Thomas [:nthomas] from comment #0) > However we're still carrying these burdens > * volunteer mirrors are still rsyncing new releases (which annoys IT by > causing nagios alerts that need no action, bug 765623) > * RelEng need to periodically trim the module so the disk usage stays sane > for the above (eg bug 791205) > * RelEng need to periodically trim the bouncer products with the Check Now > bitset, so that we don't check too many things every 5 mins in sentry > > This bug is to address the first two of those, by > * emptying the rsync module > * pointing releases.m.o at the CDN in DNS > * removing a bunch of nagios alerts We haven't addressed this yet, except that the point about volunteer mirrors is no longer valid. There isn't much traffic out of releases-rsync.mozilla.org: https://ganglia-scl3.mozilla.org/ganglia/graph_all_periods.php?c=productdelivery-rsync&m=load_one&r=hour&s=by%20name&hc=4&mc=2&st=1386143808&g=network_report&z=large&c=productdelivery-rsync It looks like the long-term plots chop the tops of any peaks in traffics, eg compare last week with last month, but even still there isn't much traffic even at release time.
We're hitting nagios alerts trying to check the mounts on the rsync hosts, because they're so busy serving rsync (see bug 1013624). And it got zero response to http://ftangftang.wordpress.com/2014/05/12/rethinking-rsync-at-mozilla/ (which was on planet).
We're going to go ahead with disabling the existing modules in the absence of any feedback on how they are still useful. We can look at tearing down the infrastructure later.
Created attachment 8480316 [details] [diff] [review] [svn] Remove mozilla-releases and mozilla-current modules
Comment on attachment 8480316 [details] [diff] [review] [svn] Remove mozilla-releases and mozilla-current modules Sending files/rsync/rsyncd.conf Transmitting file data . Committed revision 92750.
Updated the motd generation script to not try to find the size of the deprecated modules, leave a pointer to this bug. Sending files/bin/motd-gen.sh Transmitting file data . Committed revision 92751.
As for why the rsync mozilla-releases module was not being used much, there is a very simple reason for the lack. THE @#$% NAME DID NOT RESOLVE VIA THE DNS FOR THE LONGEST TIME!!!! It would go to a CNAME entry which went to another CNAME entry which would not resolve to an address! I sent notes to email@example.com complaining about it with -=*<NO>*=- results. I finally changed the script I used for downloading new releases to check the return code of rsync for a simple listing of releases-rsync.mozilla.org. If that didn't work, I used a mirror. The comment in my script was, and I quote, # Does releases-rsync work today? It finally started working regularly, after months of NOT working, shortly before you decided it wasn't being used enough. TALK ABOUT A SELF-FULFILLING PROPHECY! Also, when FTP'ing to ftp.mozilla.org, it says not to use the address for high volume transfers and to use release.mozilla.org. This would be fine if FTP worked for release.mozilla.org WHICH IT DOESN'T as you well know!
I had been maintaining a local mirror for internal use by my organization (save some of our Internet bandwidth, as well as that of whoever we would otherwise download Mozilla releases from) - which I just realized no one was using because it broke a few months ago as part of this. Is my current understanding correct that there is no longer an rsync server available for such use? If there is still a server for acceptable use, what is the appropriate address? If rsync is no longer an option, is there a recommended alternative - or are mirrors no longer supported here?
By the way - https://wiki.mozilla.org/MirrorNetwork ranks up pretty high in the Google search results, and it is in desperate need for some updates. Many of the pages that are linked to no longer exist.
At this point I think we should confirm that rsync is gone for good. I appreciate the input in comments #13 and onwards but there isn't sufficient interest in rsync to continue providing it. We are also planning to deprecate ftp.mozilla.org this year, which removes the natural place to host an rsync server. For people/orgs who still wish to mirror releases I suggest using http://download.cdn.mozilla.net/pub/firefox/releases/ which is a high bandwidth host. It doesn't support the ftp protocol but alternatives like 'wget -m' can get you pretty close.
Per the product delivery meetings, the rsync hosts have been decom'ed (bug 1162938). Unless there's some more housekeeping I missed, this is done.
Agreed. RIP rsync.