1125678 - mozilla.okta.com is no longer responding to pings

Status: RESOLVED FIXED

(Infrastructure & Operations :: MOC: Problems, task)

Description

[Ashlee Nguyen [:ashlee]]

Its possible that one of the IP's is down. We should probably change that check to look up DNS on the fly rather than leting nagios/puppet set the address.

The next time puppet runs on the nagios server, it'll pick a new address for that host and the check will recover.

Comment 1

[Justin Dow [:jabba]]

I just tried pinging the other IPs that are listed in DNA for mozilla.okta.com and none of them are responding, but they are listening on port 80 and 443. I think Okta just started blocking ICMP today, so we'll have to remove this nagios check (currently acked the alert), and replace with a TCP or HTTP check instead.

Comment 2

[Ludovic Hirlimann [:Usul]]

(In reply to Justin Dow [:jabba] from comment #1)
> I just tried pinging the other IPs that are listed in DNA for
> mozilla.okta.com and none of them are responding, but they are listening on
> port 80 and 443. I think Okta just started blocking ICMP today, so we'll
> have to remove this nagios check (currently acked the alert), and replace
> with a TCP or HTTP check instead.

We should do a http check with transactions see bug 1082144

Comment 3

[Salvador Espinoza [:sal]]

21:05 < nagios-scl3> Tue 21:05:52 PST [5954] mozilla.okta.com (54.235.64.96) is                      DOWN :PING CRITICAL - Packet loss = 100%

Comment 4

[Ryan Watson [:w0ts0n]]

Since it's an external service and it's just a ping check, wouldn't it make more sense to have this in pingdom, via the URL, vs IP in nagios? I think so.

Comment 5

[Peter Radcliffe [:pir]]

It might, if it responded to pings... which it now does not.

pir@shiny> ping mozilla.okta.com
PING mozilla.okta.com (54.235.64.96): 56 data bytes
Request timeout for icmp_seq 0
Request timeout for icmp_seq 1
Request timeout for icmp_seq 2
Request timeout for icmp_seq 3
^C
--- mozilla.okta.com ping statistics ---
5 packets transmitted, 0 packets received, 100.0% packet loss


It needs to be checked on port 80/443, not by ping. I'd say checking from nagios makes sense since it has to be reachable from the location of the ldap server it talks to so it can function.

Comment 6

[Ryan Watson [:w0ts0n]]

Attachment: okta.png

Pingdom has an option to check via https (attached) and works.

I personally think it makes sense in Pingdom with the rest of our external services. We use nagios to check the internal server, okta1.private.scl3.mozilla.com

Comment 7

[Ryan Watson [:w0ts0n]]

Well. I've added the pingdom check and removed the nagios ones (just the ping checks, the okta1.private still exists) these can be put back in if there is an issue. Would like +R from ashish.

Comment 8

[Ashish Vijayaram [:ashish]]

WFM