807543 - Empty mozilla-releases rsync module

Status: RESOLVED FIXED

(Release Engineering :: Release Automation: Other, defect, P3)

Description

[Nick Thomas [:nthomas] (UTC+12)]

We've progressively disabled the volunteer mirror network by
* swapping all downloads to our meta-CDN (download.cdn.mozilla.net, many folks)
* disabled all non-CDN mirrors in download.m.o (bouncer, jakem)
* deprecated the internal mirrors (pv-mirror01/02), and moved rsync hosting in house (digi)
* swapped AMO downloads to akamai (bug 806615, oremj)

At this point I'm not aware of anything still using the mozilla-releases module, the contents of which you can see at
   http://releases.mozilla.org/pub/mozilla.org/

However we're still carrying these burdens
* volunteer mirrors are still rsyncing new releases (which annoys IT by causing nagios alerts that need no action, bug 765623)
* RelEng need to periodically trim the module so the disk usage stays sane for the above (eg bug 791205)
* RelEng need to periodically trim the bouncer products with the Check Now bitset, so that we don't check too many things every 5 mins in sentry

This bug is to address the first two of those, by
* emptying the rsync module
* pointing releases.m.o at the CDN in DNS
* removing a bunch of nagios alerts

Alex, do you have any objections to this ? It's effectively the last nail in the coffin for falling back to the volunteer mirrors if we have CDN issues. At this point we haven't seen any failure modes that say that'd be a bad idea.

Comment 1

[Alex Keybl [:akeybl]]

(In reply to Nick Thomas [:nthomas] from comment #0)
> Alex, do you have any objections to this ? It's effectively the last nail in
> the coffin for falling back to the volunteer mirrors if we have CDN issues.
> At this point we haven't seen any failure modes that say that'd be a bad
> idea.

Given the CDN uptime that we can expect, and bhearsum's assertion in yesterday's channel meeting that external mirrors already aren't being used as a failover, I don't have a problem with hammering that nail in.

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

Given recent discussion on the mirror's list, what does IT have to say about this proposal ? We could leave mozilla-current alive (and up to date!) if there are external parties that still want to sync.

Comment 3

[Jake Maul [:jakem]]

I suspect there may be some interest from mirror admins to running local mirrors even if Mozilla doesn't use them... however, I'm not sure how widespread this would be, and if we do want to support it I'm not sure what it should look like.

Perhaps, for instance, it could be greatly simplified, and just direct folks to the FTP cluster rsync, and let them set up their own excludes. Since it's not for our usage, one could argue we don't need to provide predetermined exclude lists.

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

Ok. Can we protect ourselves from people accidentally syncing 'the world' (TM) ?

Comment 5

[Jake Maul [:jakem]]

Not easily, as far as I'm aware. We can have the server-side only allow certain things to be served up, but we can't really prevent someone from requesting everything we make available... at least not automatically (we can obviously block IPs on demand as usual). Handling of accidental or intentional abuse would need to be part of a larger discussion/decision around what (if anything) should still be available via rsync. I'm not sure I should have a voice in that.

My 2c: I don't *mind* making rsync access available for those who want it, but I don't want it to be a burden on maintenance (which includes any manual abuse handling, extra server setups, running our own local mirror, etc). Without knowing what kind of demand there would be for this, I couldn't make any value judgements as to how much time or effort its worth to streamline this vs just removing it entirely. I believe the current scheme is far to complicated for what we need nowadays, so we should definitely do *something*.

Comment 6

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Found during triage. 

nthomas, nmaul: Given our CDN and now S3 experiences, I think we're ok... so question is:  whats left to do here?

Comment 7

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to Nick Thomas [:nthomas] from comment #0)
> However we're still carrying these burdens
> * volunteer mirrors are still rsyncing new releases (which annoys IT by
> causing nagios alerts that need no action, bug 765623)
> * RelEng need to periodically trim the module so the disk usage stays sane
> for the above (eg bug 791205)
> * RelEng need to periodically trim the bouncer products with the Check Now
> bitset, so that we don't check too many things every 5 mins in sentry
> 
> This bug is to address the first two of those, by
> * emptying the rsync module
> * pointing releases.m.o at the CDN in DNS
> * removing a bunch of nagios alerts

We haven't addressed this yet, except that the point about volunteer mirrors is no longer valid. There isn't much traffic out of releases-rsync.mozilla.org:
https://ganglia-scl3.mozilla.org/ganglia/graph_all_periods.php?c=productdelivery-rsync&m=load_one&r=hour&s=by%20name&hc=4&mc=2&st=1386143808&g=network_report&z=large&c=productdelivery-rsync

It looks like the long-term plots chop the tops of any peaks in traffics, eg compare last week with last month, but even still there isn't much traffic even at release time.

Comment 8

[Nick Thomas [:nthomas] (UTC+12)]

We're hitting nagios alerts trying to check the mounts on the rsync hosts, because they're so busy serving rsync (see bug 1013624). And it got zero response to http://ftangftang.wordpress.com/2014/05/12/rethinking-rsync-at-mozilla/ (which was on planet).

Comment 9

[Nick Thomas [:nthomas] (UTC+12)]

We're going to go ahead with disabling the existing modules in the absence of any feedback on how they are still useful. We can look at tearing down the infrastructure later.

Comment 10

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [svn] Remove mozilla-releases and mozilla-current modules

Comment 11

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8480316 [details] [diff] [review]
[svn] Remove mozilla-releases and mozilla-current modules

Sending        files/rsync/rsyncd.conf
Transmitting file data .
Committed revision 92750.

Comment 12

[Nick Thomas [:nthomas] (UTC+12)]

Updated the motd generation script to not try to find the size of the deprecated modules, leave a pointer to this bug.

Sending        files/bin/motd-gen.sh
Transmitting file data .
Committed revision 92751.

Comment 13

[Carolina Calling]

As for why the rsync mozilla-releases module was not being used much, there is a very simple reason for the lack.  THE @#$% NAME DID NOT RESOLVE VIA THE DNS FOR THE LONGEST TIME!!!!  It would go to a CNAME entry which went to another CNAME entry which would not resolve to an address!  I sent notes to [e-mail] complaining about it with -=*<NO>*=- results.

I finally changed the script I used for downloading new releases to check the return code of rsync for a simple listing of releases-rsync.mozilla.org.  If that didn't work, I used a mirror.  The comment in my script was, and I quote, # Does releases-rsync work today?  It finally started working regularly, after months of NOT working, shortly before you decided it wasn't being used enough.  TALK ABOUT A SELF-FULFILLING PROPHECY!

Also, when FTP'ing to ftp.mozilla.org, it says not to use the address for high volume transfers and to use release.mozilla.org.  This would be fine if FTP worked for release.mozilla.org WHICH IT DOESN'T as you well know!

Comment 14

[Mark A. Ziesemer]

I had been maintaining a local mirror for internal use by my organization (save some of our Internet bandwidth, as well as that of whoever we would otherwise download Mozilla releases from) - which I just realized no one was using because it broke a few months ago as part of this.

Is my current understanding correct that there is no longer an rsync server available for such use?  If there is still a server for acceptable use, what is the appropriate address?

If rsync is no longer an option, is there a recommended alternative - or are mirrors no longer supported here?

Comment 15

[Mark A. Ziesemer]

By the way - https://wiki.mozilla.org/MirrorNetwork ranks up pretty high in the Google search results, and it is in desperate need for some updates.  Many of the pages that are linked to no longer exist.

Comment 16

[Mark A. Ziesemer]

Bump?

Comment 17

[Mark A. Ziesemer]

Bump?

Comment 18

[Nick Thomas [:nthomas] (UTC+12)]

At this point I think we should confirm that rsync is gone for good. I appreciate the input in comments #13 and onwards but there isn't sufficient interest in rsync to continue providing it. We are also planning to deprecate ftp.mozilla.org this year, which removes the natural place to host an rsync server.

For people/orgs who still wish to mirror releases I suggest using
   http://download.cdn.mozilla.net/pub/firefox/releases/
which is a high bandwidth host. It doesn't support the ftp protocol but alternatives like 'wget -m' can get you pretty close.

Comment 19

[Greg Cox [:gcox]]

Per the product delivery meetings, the rsync hosts have been decom'ed (bug 1162938).
Unless there's some more housekeeping I missed, this is done.

Comment 20

[Nick Thomas [:nthomas] (UTC+12)]

Agreed. RIP rsync.