Closed Bug 1286460 Opened 8 years ago Closed 8 years ago

tpe1 network issues

Categories

(Infrastructure & Operations :: MOC: Problems, task)

Product:
Infrastructure & Operations
Component:
MOC: Problems
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED WORKSFORME

People

(Reporter: mdevney, Unassigned)

Details

02:10 <@nagios-scl3> Tue 23:10:10 PDT [5638] admin1a.private.tpe1.mozilla.com:Time Sync is CRITICAL: CHECK_NRPE: Socket timeout after 15 seconds.  (http://m.mozilla.org/Time+Sync)


This is the first in a series of alerts for just about everything in tpe1 going down.

No planned maintenance in Whistlepig.

CAN ping fw1.tpe1.mozilla.net
CAN ping fw1.tier2.tpe1.mozilla.net

Checked Observium: 
Both checks for fw1.ops.tpe1.mozilla.net are red - last changed 3h8m ago.  

This doesn't appear to be a telco thing, as I can ping both fw1 connections.  Paging netops.
Came back up.
Final up alert:
02:31 <@nagios-scl3> (IRC) Tue 23:31:46 PDT [5760] endpoint1.kenya.av.tpe1.mozilla.com:Check vidyo port 80 is OK: TCP OK - 0.152 second response time on 10.247.48.46 port

Total downtime: 21 minutes.  
NOT following up with netops.
Status: NEW → RESOLVED
Closed: 8 years ago
Resolution: --- → WORKSFORME
Component: MOC: Incidents → MOC: Problems
You need to log in before you can comment on or make changes to this bug.