707630 - PHX1 rolling network outage

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: NetOps, task)

Description

[Ashish Vijayaram [:ashish]]

Network issues in PHX beginning at ~02:33 PDT for about 10 minutes. The issue affected addons.m.o, bugzilla.m.o, Services (sync/weave), Socorro and Input - none of which were reachable or would show a Service Unavailable message.

Comment 1

[Dave Miller [:justdave]]

and IRC (concrete), according to 3crowd.  3crowd shows concrete offline from 10:33 UTC to 10:39 UTC.

Comment 2

[Dave Miller [:justdave]]

and again from 10:46 to 10:49 UTC.

Comment 3

[Dave Miller [:justdave]]

Sorry, I misread the timestamps, the second outage 3crowd reported on the irc server was 10:43 to 10:49 UTC.

Comment 4

[Ashish Vijayaram [:ashish]]

Another very short one between 08:23 and 08:24 Pacific. Nagios caught it but being so short, the external monitors didn't.

Comment 5

[Ravi Pina [:ravi]]

All clear was given at 0400 UTC after a OS upgrade was performed on equipment.

Full details will be provided once remaining data is collected.

Comment 7

[Ravi Pina [:ravi]]

Applied configuration changes per vendor recommendation and rebooted firewalls.  Sustained a 8m outage, but things are stable with all AMO online.

CPU and memory utilization nominal.  We will continue to monitor things especially around 1200 GMT.

Comment 8

[Dumitru Gherman [:dumitru]]

Not sure if this is the appropriate bug to add the comment or not, but:
Starting 9:20 AM until 9:25 AM , PHX was experiencing latencies. Netops were alerted and on it promptly.

Comment 9

[Ravi Pina [:ravi]]

I think we are all going to selectively forget the month of December 2011...