Closed Bug 1064791 Opened 10 years ago Closed 10 years ago

addons.mozilla.org knocked offline.

Categories

(Cloud Services :: Operations: Marketplace, task)

Product:
Cloud Services
Component:
Operations: Marketplace
Platform:
x86
macOS
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: rwatson, Unassigned)

Details

As per the time stamps:
[10:55:05]  nagios-phx1	 Tue 02:55:04 PDT [1330] addons.dynect.mozilla.net is DOWN :PING CRITICAL - Packet loss = 100%
[10:55:08]  nagios-phx1	 Tue 02:55:07 PDT [1331] marketplace.firefox.com is DOWN :PING CRITICAL - Packet loss = 100%
[10:56:53]  nagios-phx1	 Tue 02:56:53 PDT [1332] geo.marketplace.firefox.com is DOWN :PING CRITICAL - Packet loss = 100%
[10:57:28]  nagios-phx1	 Tue 02:57:27 PDT [1333] services.addons.mozilla.org is DOWN :PING CRITICAL - Packet loss = 100%
[10:57:30]  nagios-phx1	 Tue 02:57:29 PDT [1334] static.addons.mozilla.net is DOWN :PING CRITICAL - Packet loss = 100%

Confirmed via browser/curl

I rang oremj and he came online. It appears to have recovered a few minutes later.
[11:06:06]  nagios-phx1	 Tue 03:06:06 PDT [1336] addons.mozilla.org is UP :PING OK - Packet loss = 0%, RTA = 0.89 ms
It appears both netscaler units went offline.

mktns1 is still offline, it has a remote PDU (tripp-lite-03894.console.phx1.mozilla.com), but none of the typical oob credentials work, so I've been unable to access it.

Will look in to the root cause further in the morning.
Filed Support ticket with Citrix #63967137.
Status: NEW → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.