759248 - SCL2 and MTV1 network unreachable from Boris, mradm01; external health checks failing as well

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: NetOps, task)

Description

[:Atoll]

tcptraceroute -p 22 from mradm01 and boris to adm1.scl2.svc.m.c (22/tcp), mon1.scl2.svc.m.c, and so forth.

we also just got notified that scl2 has been removed from BrowserID service automatically due to health check failures

Comment 1

[Ravi Pina [:ravi]]

I'm not sure if there is anything actionable for this event anymore.  I cleared xlate within minutes of SRE pinging on #netops.

Please close or let me know what else we can do.

Comment 2

[:Atoll]

slate clear confirmed to fix site issue, resolving. listed hosts known to be affected by xlate issues at https://intranet.mozilla.org/Services/Ops/FWSM and linked from Oncall page.

Comment 3

[Ashish Vijayaram [:ashish]]

This occurred again at 00:49 and recovered 5 mins later (going by nagios).

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

Only partially resolved I think. eg mv-moz2-linux-ix-slave04.build.mtv1 can reach
 buildbot-master32.srv.releng.scl3.mozilla.com:9001 (10.26.48.19)
but not 
 stage.mozilla.org 22 or 80
 hg.mozilla.org 22 or 80

Both of the latter are 63.245.215 IPs.

Comment 5

[Aaron]

Cleared xlates and things began to heal.  We need to get off this firewall.

Comment 6

[Ashish Vijayaram [:ashish]]

Looks like this is happening again. I see HOST DOWNs MTV1 from dm-nagios01 and was briefly unable to ping SCL2 host db1.iddb.scl2.svc.mozilla.com. The nagios bots in mtv1 and scl2 were knocked off too:

04:16:51 -!- nagios-mtv1 [[e-mail]] has quit [Ping timeout]
04:16:51 -!- nagios-svc-scl2 [[e-mail]] has quit [Ping timeout]

Comment 7

[Ashish Vijayaram [:ashish]]

(In reply to Ashish Vijayaram [:ashish] from comment #6)
> Looks like this is happening again. I see HOST DOWNs MTV1 from dm-nagios01
> and was briefly unable to ping SCL2 host db1.iddb.scl2.svc.mozilla.com. The
> nagios bots in mtv1 and scl2 were knocked off too:
> 
> 04:16:51 -!- nagios-mtv1 [[e-mail]] has quit
> [Ping timeout]
> 04:16:51 -!- nagios-svc-scl2 [[e-mail]] has
> quit [Ping timeout]

:casey says this wasn't the FWSM. Opened Bug 762857.