773439 - No add-ons on mozilla.isc.org mirror (affects releases.mozilla.org, AMO downloads)

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Wladimir Palant]

A user reported that his Adblock Plus download resulted in a 404 error. I suspected a mirror error and went through the releases.mozilla.org mirrors manually - turned out that 204.152.184.113 (mozilla.isc.org) indeed doesn't have the addons directory (http://mozilla.isc.org/pub/mozilla.org/ only has Firefox and Thunderbird releases, nothing else).

Comment 1

[Jake Maul [:jakem]]

I don't really know how this sort of thing works... kicking to AMO Operations, in the hope that they'll have a better grasp on it. Also CC'ing some non-AMO-Ops folks that might know what's up here.

Comment 2

[Jason Thomas [:jason]]

Looking at another release mirror includes the addons directory [1].

Do we have a contact over at ISC that we can reach out to regarding this issue?

[1]http://mozilla.cs.utah.edu/pub/mozilla.org/

Comment 3

[Jason Thomas [:jason]]

We should pull this mirror out of geodns until it is fixed. @justdave can you assist with this?

Comment 4

[Dave Miller [:justdave]]

wifi sucks at the conference.  it needs to be disabled on geodns with ./geodns.pl --disable --id X  where the X is the ID you find for it from ./geodns.pl --list | grep release
*before* you do the above, the nagios check script check_releaseslag.pl will need to have it added to the ignore list so it won't immediately re-enable it.

Comment 5

[Dave Miller [:justdave]]

I managed to get it added to the ignore list in the nagios check script, and it's disabled now.  Still need to get ahold of isc to find out why it was removed.

Comment 6

[Jason Thomas [:jason]]

@justdave it looks like mozilla.isc.org has been re-enabled. Could you please take a look?

>dig releases.mozilla.org +short
releases.geo.mozilla.com.
128.61.111.9
64.50.236.214
131.188.12.212
216.165.129.141
149.20.36.135
202.177.202.154
204.246.0.136
155.98.64.83
156.56.247.196
204.152.184.113 <-

Comment 7

[Jeremy Orem [:oremj]]

mozilla.isc.org is breaking a good portion of addons downloads. We need to make sure it remains out of rotation until the mirror is fixed.

Comment 8

[Dumitru Gherman [:dumitru]]

Dave pointed to the issue, and we fixed it.
He is emailing isc as we speak.

Comment 9

[Jeremy Orem [:oremj]]

We also need to pull http://131.188.12.212. Looks like it hasn't updated since July 18th.

Comment 10

[Dumitru Gherman [:dumitru]]

Disabled.

Comment 11

[Dave Miller [:justdave]]

Received this from ISC on Wednesday:

(apologies for the delayed response) 

Thanks - the reason we pulled the addons subdir was due to a emergency disk space issue on the server.  We are working on solving the issue, but it requires getting more HW so it will take some time...

-Peter

Comment 12

[Dave Miller [:justdave]]

I haven't heard anything else from them, and the addons directory still isn't present.  We'll just leave them out of the pool until they tell us otherwise.

Comment 13

[Dave Miller [:justdave]]

I guess this should be fixed... we fixed the reported problem by removing them from the pool.