451769 - services.addons.mozilla.org is down

Status: VERIFIED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Laura Thomson :laura]

This host is not responding to http requests, which means the Addons manager in Fx3 is not returning any data to users.

Comment 1

[matthew zeier [:mrz]]

Netscaler issue, failing over resolved it.  

need to get monitors on samo

Comment 2

[matthew zeier [:mrz]]

Citrix case 31840567.

Comment 3

[Fred Wenzel [:wenzel]]

(In reply to comment #1)
> Netscaler issue, failing over resolved it.  

Correct: The Fx3 add-on manager shows SAMO content again.

Comment 4

[matthew zeier [:mrz]]

Config typo on the second Netscaler pair (copy-paste error).  No idea why it took so long to manifest but it was a duplicate IP error.  Failing over forces the new primary to send out gratuitous ARPs which is why it "fixed".

Double checked config and SAMO's in Nagios - Reed please confirm?

Comment 5

[Reed Loden [:reed]]

Monitoring not set up completely yet. I have to edit SAMO's vhost to allow nagios to access /services/monitor.php like it does for FAMO and AMO.

Comment 6

[matthew zeier [:mrz]]

Reed - can this get done by the end of Monday?  What's blocking it if anything?

Comment 7

[Reed Loden [:reed]]

(In reply to comment #6)
> Reed - can this get done by the end of Monday?  What's blocking it if anything?

I'd like to know if WebDev wants anything else under /services/ accessible to people besides monitor.php, but that can wait, I guess. I committed the change for just /services/monitor.php for now, but it needs to be pushed out.

Comment 8

[Reed Loden [:reed]]

Ok, now monitoring SAMO in multiple ways.

https://nagios.mozilla.org/nagios/cgi-bin/status.cgi?host=dyna-services-amo.nslb.sj

Comment 9

[Stephen Donner [:stephend] Not actively reading bugmail]

Verified FIXED; been back up for a while, and is now monitored.