632283 - Break mirrorservice.org into several mirrors

Status: RESOLVED INCOMPLETE

(mozilla.org :: FTP: Mirrors, task)

Description

[Nick Thomas [:nthomas] (UTC+12)]

mirrorservice.org is a round-robin DNS with 10 machines in it, which are proxies back to a smaller number of backend filestores (bug 540326). This doesn't play very well with bouncer's sentry when the filestores take different time to sync (bug 540326, other occasions when we've disabled mirrorservice until they all catch up), or when individual machines have problems (bug 632273), because sentry tests a random IP every 5 minutes.

I'm proposing we split the single mirror definition in bouncer to have a separate mirror for each machine in the DNS pool. That would make redirecting from download.m.o to mirrorservice more reliable for users, and make it plain at https://nagios.mozilla.org/sentry/ when there were problems. Whether we hardwire IPs or request mozilla{1..9}.mirrorservice.org records is an open question.

Comment 1

[Dave Miller [:justdave]]

This would require cooperation from mirrorservice to set up, of course.  I see the admin address is CCed here.  You guys have any preference for how to do this?

Comment 2

[archive-admin]

hmmm the front-end backend mapping is random so that doesnt really work for us or you (and we will be moving to an lvs fronted system shortly), what about a quick iterate through all the ip's to check /zz/marker.txt is within a set of bounds?

Comment 3

[Dave Miller [:justdave]]

(In reply to comment #2)
> what about a quick iterate through all the ip's to check /zz/marker.txt is
> within a set of bounds?

That's pretty much what we're asking for, except they'd each need to be in bouncer separately.

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

I've disabled www.mirrorservice.org until the sync of 3.6.20 and 6.0 has completed on all nodes, after seeing some failures in our automated update checks. Can you let us know when we can re-enable ?

Comment 5

[Shyam Mani [:fox2mike]]

Timeout.

Comment 6

[Tim Bishop]

I didn't know about this bug; I've only been running mirrorservice.org for the past year. We've completely rebuilt the system now and it runs behind a single IP. What's the status of this? Is the mirror still disabled?

Anyhow, is it still relevant now you've moved to a CDN for distribution?

Tim.